ClassicML on davethehuman.com

gini index

Tue, 13 Oct 2026 00:00:00 +0000

Gini Index or Gini Impurity is a metric used to measure the frequency with which a randomly chosen element from a dataset would be incorrectly labeled if it were randomly labeled according to the distribution of labels in the subset.

In Machine Learning, it is primarily used as a splitting criterion by the algorithm to build decision trees.

Formula
#

The Formula For a dataset with $C$ total classes, where $p_i$ is the probability of an item belonging to class $i$:

KNN from scratch

Mon, 12 Oct 2026 00:00:00 +0000

Algorithm and implementation
#

KNN is a so-called lazy learner because it does not have a training step and it is immediately ready to predict, using euclidean distance to find the k points that are closest to the point we are predicting onto, and outputting a prediction based on these k points (either using the most common y class among them (classification) or averaging those closest points’ value of y (regression) ).

entropy

Sat, 10 Oct 2026 00:00:00 +0000

Entropy is a statistics concept used to measure disorder in a sample.
It is applied to calculate the information gain of a decision tree’s split through node impurity, and can be described with the formula:

$$ E = -\sum_{i=1}^{n} P(X) \log_2 P(X) $$

In a decision tree’s node, $P(X)$ is the number of times a given class has occurred in the node divided by the number of total examples $n$ in the node.
It is basically the probability of encountering that class in the node’s sample.

K-means

Sat, 10 Oct 2026 00:00:00 +0000

K-means is an unsupervised learning algorithm that assigns labels to the data points (clustering) with an iterative optimization that repeats until convergence.

Optimization
#

initialize clusters centers (also called centroids) randomly
update cluster labels by assigning points to the nearest cluster center
update cluster centers by setting the center to the mean of the corresponding cluster

Loss function
#

Euclidean distance is the metric used to find the nearest centroids.

K-means from scratch

Sat, 10 Oct 2026 00:00:00 +0000

Algorithm and implementation
#

K-means algorithm assigns labels to data points by iteratively updating centroids and clusters’ labels and using euclidean distance to calculate their proximity until convergence (or until a maximum number of iterations is reached).

Class structure
#

main attributes:
- the number of clusters K
- n_iterations
- clusters, which is the list of sample indices for each cluster initialized in the __init__
- centroid which is the center (or mean vector) for each cluster
predict method It is worth to note that there’s no fit method and we can do the predict straight away.

import numpy as np

class KMeans:
	def __init__(self, K=5, n_iterations=100):
		self.K = K
		self.n_iterations = n_iterations
		
		#list of sample indices for each cluster
		self.clusters = [[] for _ in range(self.K)]
		
		#mean vector for each cluster
		self.centroids = []
		
	def predict(self, X):
		pass

Inference
#

The predict method consists of the following steps:

KNN

Sat, 10 Oct 2026 00:00:00 +0000

K nearest neighbours is a supervised learning algorithm that, given a data point to predict onto, uses euclidean distance to calculate the distance between that point and the training data to make a prediction based on the closest $k$ points ($k$ is then an hyperparameter defined by the user a-priori).
In case of a classification task, it will assign a label with majority voting; in case of a regression task it will compute the average of those closest points’ value.

early stopping

Thu, 08 Oct 2026 00:00:00 +0000

Early stopping is a regularization technique that stops training when performance on a held-out validation set stops improving, helping prevent overfitting.

Training loss may keep decreasing while validation loss begins to rise: the model is increasingly fitting details that do not generalize.

A typical procedure monitors validation loss, saves the best model checkpoint, and stops after a chosen number of evaluations without meaningful improvement. This waiting period is called patience. The best checkpoint is then restored.

Another way to apply early stopping is to monitor the algorithm convergence: if further iterations barely change the parameters, we can stop when the largest absolute update of w and b is below tolerance. This is implemented in Linear Regression from scratch and Logistic Regression from scratch.

Logistic Regression from scratch

Thu, 08 Oct 2026 00:00:00 +0000

Algorithm and implementation
#

Logistic Regression formula is the same as the Linear Regression’s one passed to a sigmoid function to obtain a probability score from logits and can be replicated on numpy with:
sigmoid(X @ w + b) (or sigmoid(np.dot(X,w)).
The implementation of gradient descent as an optimizer is the same of Linear Regression from scratch, what changes is the loss and the error since we deal with probabilities:

standardization

Thu, 08 Oct 2026 00:00:00 +0000

In several algorithms, gradient descent uses one learning rate for every weight; this means that if one feature ranges from 0 to 1 and another from 0 to 100000 their gradients can have very different size, with training that may zig-zag, diverge or require a very small learning rate.

One solution is to add standardization which means transforming each column on the training dataset so that its mean is 0 and standard deviation is 1: X_scaled = (X - mean) / scale.
The training mean and scale need to be saved and used for validation, test and future predictions too.

Linear Regression from scratch

Tue, 06 Oct 2026 00:00:00 +0000

Algorithm and implementation
#

Linear Regression formula can be replicated on numpy with X @ w + b (or np.dot(X,w)).

To implement gradient descent as an optimizer, we need to calculate the following:

errors = predictions - y
The loss is RSS , evaluated with MSE.
loss = np.mean(errors - y)**2 In this case, we will actually be using the half-MSE to have similarities with Logistic Regression from scratch. $$L=\frac{1}{2N}\sum_i(\hat y_i-y_i)^2.$$Half-MSE has the same minimizer as MSE and its 1/2 cancels the derivative’s 2 giving the same gradient template as Logistic Regression: $$dw=X^T errors/N, \qquad db=mean(errors).$$ Using “pure” MSE the derivative would have been: $$dw=2X^T errors/N, \qquad db=mean(errors).$$
weights w and biases are initialized as np.zeros and np.mean(y) respectively, since the weights act as a sort of feature importance and the bias works as an intercept . After choosing a learning rate, at further steps w and b will be updated as:
- w-=dw * learning_rate
- b-=db * learning_rate

Class structure
#

main attributes:
- learning_rate
- n_iterations
- weights w_ and bias b_ initialized as None
fit method
predict method

import numpy as np

class LinearRegression:
	def __init__(self, learning_rate=0.05, n_iterations=1000):
		self.learning_rate = learning_rate
		self.n_iterations = n_iterations
		self.w_ = None
		self.b_ = None
		
	def fit(self, X, y):
		pass
		
	def predict(self, X):
		pass

Training
#

The fit method can be implemented following these steps:

Logistic Regression

Sat, 26 Sep 2026 00:00:00 +0000

Formula
#

Taking as a reference the Linear Regression, in Logistic Regression the linear output is converted into a probability between 0 and 1 using the sigmoid function:

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

With $z = \beta_0 + X\beta$ as the linear score, the model’s predicted probability is $\hat{p} = \sigma(z) = \sigma(\beta_0 + X\beta) = \frac{1}{1 + e^{-(\beta_0 + X\beta)}}$.

Random Forest

Sat, 26 Sep 2026 00:00:00 +0000

The Random Forest is an ensemble of decision trees that reduces overfitting by using:

bagging to reduce variance
random selection of features from the full set of features at each split to prevent the important features from always being present at the top of each tree

It is important to note that we can’t overfit by putting in too many trees as they just reduce variance; at some point, variance reduction will just stop and we need to take into account computation time.

Linear Regression

Thu, 24 Sep 2026 00:00:00 +0000

Formula
#

When using Linear Regression, the goal is to estimate a function $f(x)$ such that each feature has a linear relationship to the target variable $y$.
This is a supervised learning model written as:

$$ y = \beta_0 + X\beta $$

with:

$\beta_0$ = intercept (or bias)
$X$ = matrix of predictor variables
$\beta$ = vector of parameters that determine the weights for each variable. For a single predictor, the sign of $\beta$ gives the direction of the relationship:
- $\beta > 0$ is positive
- $\beta < 0$ is negative
- $\beta = 0$ means no linear effect.

intercept

Sun, 20 Sep 2026 00:00:00 +0000

In Linear Regression, the intercept is the expected value of the dependent variable $Y$ when all independent variables are equal to zero. Graphically, it is represented by the intersection point between the regression line and the Y axis on a chart; its value shift the regression line up and down without affecting its tilt or slope.

It can be interpreted as a baseline value, since it acts as the starting point or baseline prediction for the linear regression model before any independent variables effect is added.

ClassicML on davethehuman.com

gini index

Formula #

KNN from scratch

Algorithm and implementation #

entropy

K-means

Optimization #

Loss function #

K-means from scratch

Algorithm and implementation #

Class structure #

Inference #

KNN

early stopping

Logistic Regression from scratch

Algorithm and implementation #

standardization

Linear Regression from scratch

Algorithm and implementation #

Class structure #

Training #

Logistic Regression

Formula #

Random Forest

Linear Regression

Formula #

intercept

Formula
#

Algorithm and implementation
#

Optimization
#

Loss function
#

Algorithm and implementation
#

Class structure
#

Inference
#

Algorithm and implementation
#

Algorithm and implementation
#

Class structure
#

Training
#

Formula
#

Formula
#