Initialization, BatchNorm, and LayerNorm: Beyond textbook definitions
Author(s): Adam Elimadi Originally published on Towards AI. The Holy Trilogy There are a ton of blog posts out there breaking down both initialization and normalization. However, I feel like most authors fail to get into the apprentice’s shoe especially those that …