Modelcomplexity on davethehuman.com

early stopping

Thu, 08 Oct 2026 00:00:00 +0000

Early stopping is a regularization technique that stops training when performance on a held-out validation set stops improving, helping prevent overfitting.

Training loss may keep decreasing while validation loss begins to rise: the model is increasingly fitting details that do not generalize.

A typical procedure monitors validation loss, saves the best model checkpoint, and stops after a chosen number of evaluations without meaningful improvement. This waiting period is called patience. The best checkpoint is then restored.

Another way to apply early stopping is to monitor the algorithm convergence: if further iterations barely change the parameters, we can stop when the largest absolute update of w and b is below tolerance. This is implemented in Linear Regression from scratch and Logistic Regression from scratch.

bias

Tue, 29 Sep 2026 00:00:00 +0000

In statistics and machine learning, Bias refers to the error coming from model oversimplification and leading to underfitting.
A model with high bias is not able to capture the true complexity of the data, leading to poor performance on both training set and test set.
The bias is represented by the difference between the expected or true value of the target variable and the predicted value of the model.

bias-variance trade-off

Tue, 29 Sep 2026 00:00:00 +0000

The bias-variance trade-off describes the balance between learning the undelying pattern and being too sensitive to the particular training sample:

A model with high bias is too simple, makes too restrictive assumptions, and consequently makes systematic errors; for example, a straight line cannot fully capture a curved relationship. This is associated with underfitting.
A model with high variance (ML) is too complex and sensitive to the examples it sees. It may follow random fluctuations in the training data and perform poorly on new examples. This is associated with overfitting.

variance (ML)

Tue, 29 Sep 2026 00:00:00 +0000

In machine learning, Variance refers to the error coming from the model being too complex and leading to overfitting.
A model with high variance has a prediction error that changes a lot depending on the training input because it fits the training set very closely; its performance drop dramatically on the test set.

Together with bias, this is a fundamental concept to understand the bias-variance trade-off.