Biases $b$ are constant values added to the sum of inputs and weights before passing through an activation function. Mathematically, their function is the same as the intercept for Linear Regression, and in this case they shift the activation curve to help the model fit complex patterns even when input values are zero.
Together with weights, they are the core parameters for neural networks.
The difference between bias and intercept and the shift they cause is where the shift happens:
- In linear regression, $b$ is the predicted output when all inputs are zero.
$\hat{y} = \sum_{i=1}^{n} w_i x_i + b$ - when weights are zero $\hat{y} = b$ - In a neural network, $b$ is the value before activation when all inputs are zero. The output is therefore $\phi(b)$ because the bias undergoes a non-linear transformation through the activation function.
$a = \phi\left(\sum_{i=1}^{n} w_i x_i + b\right)$ - when weights are zero $a = \phi\left(b\right)$ - In logistic regression, $b$ is the intercept of the log-odds; the probability at zero input is $\sigma(b)$, as the bias goes through the sigmoid function.