leaky ReLU

Mon, 28 Sep 2026 00:00:00 +0000

LeakyReLU is a variation of the ReLU activation function:

$$ f(x) = \max(\alpha x, x) $$

It leaves positive inputs unvaried while multiplying the negative ones by a small positive constant $\alpha$. Unlike ReLU, which outputs zero for any negative input, Leaky ReLU keeps a small slope on the negative side, allowing gradients to flow through neurons with negative inputs during backpropagation, and mitigating the dying ReLU problem.

In standard Leaky ReLU, $\alpha$ is a fixed hyperparameter, while in [PReLU] it is learned during training.

ReLU

Tue, 22 Sep 2026 00:00:00 +0000

ReLU (Rectified Linear Unit) is an activation function with formula:

$$ f(x) = \max(0, x) $$

It is the default activation function in the hidden layers of modern neural networkss (FNN, CNN, etc.) and it just checks if a number is larger than zero, allowing for quick computation and fast convergence.

Pros:

it tackles the vanishing gradient problem, keeping the gradient flow strong during backpropagation - this happens because its derivative (gradient) for any positive input is always exactly 1
it allows sparse activation, meaning that only relevant neurons fire, while negative inputs are zeroed out

Cons:

Activation on davethehuman.com

leaky ReLU

ReLU