Supervisedlearning on davethehuman.com

KNN from scratch

Mon, 12 Oct 2026 00:00:00 +0000

Algorithm and implementation
#

KNN is a so-called lazy learner because it does not have a training step and it is immediately ready to predict, using euclidean distance to find the k points that are closest to the point we are predicting onto, and outputting a prediction based on these k points (either using the most common y class among them (classification) or averaging those closest points’ value of y (regression) ).

KNN

Sat, 10 Oct 2026 00:00:00 +0000

K nearest neighbours is a supervised learning algorithm that, given a data point to predict onto, uses euclidean distance to calculate the distance between that point and the training data to make a prediction based on the closest $k$ points ($k$ is then an hyperparameter defined by the user a-priori).
In case of a classification task, it will assign a label with majority voting; in case of a regression task it will compute the average of those closest points’ value.

Logistic Regression from scratch

Thu, 08 Oct 2026 00:00:00 +0000

Algorithm and implementation
#

Logistic Regression formula is the same as the Linear Regression’s one passed to a sigmoid function to obtain a probability score from logits and can be replicated on numpy with:
sigmoid(X @ w + b) (or sigmoid(np.dot(X,w)).
The implementation of gradient descent as an optimizer is the same of Linear Regression from scratch, what changes is the loss and the error since we deal with probabilities:

Linear Regression from scratch

Tue, 06 Oct 2026 00:00:00 +0000

Algorithm and implementation
#

Linear Regression formula can be replicated on numpy with X @ w + b (or np.dot(X,w)).

To implement gradient descent as an optimizer, we need to calculate the following:

errors = predictions - y
The loss is RSS , evaluated with MSE.
loss = np.mean(errors - y)**2 In this case, we will actually be using the half-MSE to have similarities with Logistic Regression from scratch. $$L=\frac{1}{2N}\sum_i(\hat y_i-y_i)^2.$$Half-MSE has the same minimizer as MSE and its 1/2 cancels the derivative’s 2 giving the same gradient template as Logistic Regression: $$dw=X^T errors/N, \qquad db=mean(errors).$$ Using “pure” MSE the derivative would have been: $$dw=2X^T errors/N, \qquad db=mean(errors).$$
weights w and biases are initialized as np.zeros and np.mean(y) respectively, since the weights act as a sort of feature importance and the bias works as an intercept . After choosing a learning rate, at further steps w and b will be updated as:
- w-=dw * learning_rate
- b-=db * learning_rate

Class structure
#

main attributes:
- learning_rate
- n_iterations
- weights w_ and bias b_ initialized as None
fit method
predict method

import numpy as np

class LinearRegression:
	def __init__(self, learning_rate=0.05, n_iterations=1000):
		self.learning_rate = learning_rate
		self.n_iterations = n_iterations
		self.w_ = None
		self.b_ = None
		
	def fit(self, X, y):
		pass
		
	def predict(self, X):
		pass