An Activation Function is a mathematical operation applied to the linear output of a neuron to introduce non-linearity into a neural network.
Without activation functions, a multi-layer neural network collapses into a single linear transformation (similar to Linear Regression) regardless of its depth:
$$ W_2(W_1 x + b_1) + b_2 = (W_2 W_1)x + (W_2 b_1 + b_2) = W'x + b' $$Non-linear activations allow networks to learn complex, non-linear decision boundaries.
Common examples of activation functions used in neural networks are:
With ReLU and its variations used for hidden layers nodes, and sigmoid, tanh and softmax used for output layers nodes.