Batch Normalization (BN) is a technique in deep learning that normalizes the inputs of a neural network layer by adjusting and scaling them. It was introduced to address issues related to internal covariate shift and has become a standard component in many modern neural network architectures. Here’s an in-depth look at Batch Normalization:
1. Internal Covariate Shift:
Definition:
- Internal covariate shift refers to the change in the distribution of the input to a neural network layer during training.
Issue:
- The shift in input distributions can slow down the training process as each layer has to continuously adapt to the changing inputs.
2. Batch Normalization Concept:
Normalization:
- Batch Normalization normalizes the inputs of a layer by subtracting the mean and dividing by the standard deviation.
Scale and Shift:
- The normalized values are then scaled and shifted using learnable parameters to allow the network to adapt during training.
3. Batch Normalization Procedure:
For a Mini-Batch:
- Given a mini-batch of activations, calculate the mean and standard deviation for each feature.
- Normalize the features using the mean and…
