Early stopping is a regularization technique that stops training when performance on a held-out validation set stops improving, helping prevent overfitting.
Training loss may keep decreasing while validation loss begins to rise: the model is increasingly fitting details that do not generalize.
A typical procedure monitors validation loss, saves the best model checkpoint, and stops after a chosen number of evaluations without meaningful improvement. This waiting period is called patience. The best checkpoint is then restored.
Another way to apply early stopping is to monitor the algorithm convergence: if further iterations barely change the parameters, we can stop when the largest absolute update of w and b is below tolerance. This is implemented in Linear Regression from scratch and Logistic Regression from scratch.
So in a hypothetical fit method, we can add early stopping as:
#early-stopping
largest_update = max(np.max(np.abs(self.learning_rate * dw )), abs(self.learning_rate * db))
if largest_update<=tolerance:
break