MLfromScratch on davethehuman.com

KNN from scratch

Mon, 12 Oct 2026 00:00:00 +0000

Algorithm and implementation
#

KNN is a so-called lazy learner because it does not have a training step and it is immediately ready to predict, using euclidean distance to find the k points that are closest to the point we are predicting onto, and outputting a prediction based on these k points (either using the most common y class among them (classification) or averaging those closest points’ value of y (regression) ).

K-means from scratch

Sat, 10 Oct 2026 00:00:00 +0000

Algorithm and implementation
#

K-means algorithm assigns labels to data points by iteratively updating centroids and clusters’ labels and using euclidean distance to calculate their proximity until convergence (or until a maximum number of iterations is reached).

Class structure
#

main attributes:
- the number of clusters K
- n_iterations
- clusters, which is the list of sample indices for each cluster initialized in the __init__
- centroid which is the center (or mean vector) for each cluster
predict method It is worth to note that there’s no fit method and we can do the predict straight away.

import numpy as np

class KMeans:
	def __init__(self, K=5, n_iterations=100):
		self.K = K
		self.n_iterations = n_iterations
		
		#list of sample indices for each cluster
		self.clusters = [[] for _ in range(self.K)]
		
		#mean vector for each cluster
		self.centroids = []
		
	def predict(self, X):
		pass

Inference
#

The predict method consists of the following steps:

early stopping

Thu, 08 Oct 2026 00:00:00 +0000

Early stopping is a regularization technique that stops training when performance on a held-out validation set stops improving, helping prevent overfitting.

Training loss may keep decreasing while validation loss begins to rise: the model is increasingly fitting details that do not generalize.

A typical procedure monitors validation loss, saves the best model checkpoint, and stops after a chosen number of evaluations without meaningful improvement. This waiting period is called patience. The best checkpoint is then restored.

Another way to apply early stopping is to monitor the algorithm convergence: if further iterations barely change the parameters, we can stop when the largest absolute update of w and b is below tolerance. This is implemented in Linear Regression from scratch and Logistic Regression from scratch.

Logistic Regression from scratch

Thu, 08 Oct 2026 00:00:00 +0000

Algorithm and implementation
#

Logistic Regression formula is the same as the Linear Regression’s one passed to a sigmoid function to obtain a probability score from logits and can be replicated on numpy with:
sigmoid(X @ w + b) (or sigmoid(np.dot(X,w)).
The implementation of gradient descent as an optimizer is the same of Linear Regression from scratch, what changes is the loss and the error since we deal with probabilities:

standardization

Thu, 08 Oct 2026 00:00:00 +0000

In several algorithms, gradient descent uses one learning rate for every weight; this means that if one feature ranges from 0 to 1 and another from 0 to 100000 their gradients can have very different size, with training that may zig-zag, diverge or require a very small learning rate.

One solution is to add standardization which means transforming each column on the training dataset so that its mean is 0 and standard deviation is 1: X_scaled = (X - mean) / scale.
The training mean and scale need to be saved and used for validation, test and future predictions too.

Linear Regression from scratch

Tue, 06 Oct 2026 00:00:00 +0000

Algorithm and implementation
#

Linear Regression formula can be replicated on numpy with X @ w + b (or np.dot(X,w)).

To implement gradient descent as an optimizer, we need to calculate the following:

errors = predictions - y
The loss is RSS , evaluated with MSE.
loss = np.mean(errors - y)**2 In this case, we will actually be using the half-MSE to have similarities with Logistic Regression from scratch. $$L=\frac{1}{2N}\sum_i(\hat y_i-y_i)^2.$$Half-MSE has the same minimizer as MSE and its 1/2 cancels the derivative’s 2 giving the same gradient template as Logistic Regression: $$dw=X^T errors/N, \qquad db=mean(errors).$$ Using “pure” MSE the derivative would have been: $$dw=2X^T errors/N, \qquad db=mean(errors).$$
weights w and biases are initialized as np.zeros and np.mean(y) respectively, since the weights act as a sort of feature importance and the bias works as an intercept . After choosing a learning rate, at further steps w and b will be updated as:
- w-=dw * learning_rate
- b-=db * learning_rate

Class structure
#

main attributes:
- learning_rate
- n_iterations
- weights w_ and bias b_ initialized as None
fit method
predict method

import numpy as np

class LinearRegression:
	def __init__(self, learning_rate=0.05, n_iterations=1000):
		self.learning_rate = learning_rate
		self.n_iterations = n_iterations
		self.w_ = None
		self.b_ = None
		
	def fit(self, X, y):
		pass
		
	def predict(self, X):
		pass

Training
#

The fit method can be implemented following these steps:

MLfromScratch on davethehuman.com

KNN from scratch

Algorithm and implementation #

K-means from scratch

Algorithm and implementation #

Class structure #

Inference #

early stopping

Logistic Regression from scratch

Algorithm and implementation #

standardization

Linear Regression from scratch

Algorithm and implementation #

Class structure #

Training #

Algorithm and implementation
#

Algorithm and implementation
#

Class structure
#

Inference
#

Algorithm and implementation
#

Algorithm and implementation
#

Class structure
#

Training
#