↓ Skip to main content

leaky ReLU

·85 words·1 min
Dave the human
Author
Dave the human
Homo sapiens in the loop

LeakyReLU is a variation of the ReLU activation function:

$$ f(x) = \max(\alpha x, x) $$

Human evolution illustration

It leaves positive inputs unvaried while multiplying the negative ones by a small positive constant $\alpha$. Unlike ReLU, which outputs zero for any negative input, Leaky ReLU keeps a small slope on the negative side, allowing gradients to flow through neurons with negative inputs during backpropagation, and mitigating the dying ReLU problem.

In standard Leaky ReLU, $\alpha$ is a fixed hyperparameter, while in [PReLU] it is learned during training.


← weights biases →

Comments