In Linear Regression, the intercept is the expected value of the dependent variable $Y$ when all independent variables are equal to zero. Graphically, it is represented by the intersection point between the regression line and the Y axis on a chart; its value shift the regression line up and down without affecting its tilt or slope.
It can be interpreted as a baseline value, since it acts as the starting point or baseline prediction for the linear regression model before any independent variables effect is added.
It is also called the constant or the bias term, and its mathematical function is indeed the same as the one of the bias parameter in neural networks: it shifts a weighted sum.
The subtlety is where the shift happens:
- In linear regression, $b$ is the predicted output when all inputs are zero $\hat{y} = \sum_{i=1}^{n} w_i x_i + b$ - when weights are zero $\hat{y} = b$
- In a neural network, $b$ is the value before activation when all inputs are zero. The output is therefore $\phi(b)$ because the bias undergoes a non-linear transformation through the activation function. $a = \phi\left(\sum_{i=1}^{n} w_i x_i + b\right)$ - when weights are zero $a = \phi\left(b\right)$
- In logistic regression, $b$ is the intercept of the log-odds; the probability at zero input is $\sigma(b)$, as the bias goes through the sigmoid function.