Batch Normalization

Batch Norm at train time

$\mu = \frac{1}{m}\sum_iz^{(i)}$

$\sigma^2 = \frac{1}{m}\sum_i(z^{(i)}-\mu)^2$

$z_{norm}^{(i)} = \frac{z^{(i)} - \mu}{\sqrt{\sigma^2 + \epsilon}}$

$\bar{z}^{(i)} = \gamma z_{norm}^{(i)} + \beta$

Batch Norm at test time

$\mu$, $\sigma$: estimate using exponentially weighted average (accross mini-batch in training procedure)

$z_{norm} = \frac{z^{(i)} - \mu}{\sqrt{\sigma^2 + \epsilon}}$

$\bar{z} = \gamma z_{norm} + \beta$