The bias-variance trade-off describes the balance between learning the undelying pattern and being too sensitive to the particular training sample:
- A model with high bias is too simple, makes too restrictive assumptions, and consequently makes systematic errors; for example, a straight line cannot fully capture a curved relationship. This is associated with underfitting.
- A model with high variance (ML) is too complex and sensitive to the examples it sees. It may follow random fluctuations in the training data and perform poorly on new examples. This is associated with overfitting.
For squared-error regression:
$$ error = \text{Bias}^2 + \text{Variance} + \text{Irreducible noise} $$With $\text{Irreducible noise} =$ target variation that cannot be predicted from the available inputs.
Minimizing error is not enough, we need to find a model complexity that gives the lowest generalization error on unseen data. Increasing model flexibility often reduces bias but increases variance, and the other way around.
High Bias examples#
- Training error high + Validation error high (underfitting)
- Linear model fitted to a highly non-linear relationship
High Variance examples#
- Training error low + Validation error high (overfitting)
- Deep decision tree with very low training error
Ways to reduce bias#
- Increasing model flexibility/complexity (at the expense of a higher variance)
- Adding features
- Reducing regularization
Ways to reduce variance#
- More training data
- Regularization (L1, L2, early stopping, dropout)
- Simplifying the model
- Feature selection (so reducing the number of features)
- Ensembling