Linearregression on davethehuman.com

Linear Regression from scratch

Tue, 06 Oct 2026 00:00:00 +0000

Algorithm and implementation
#

Linear Regression formula can be replicated on numpy with X @ w + b (or np.dot(X,w)).

To implement gradient descent as an optimizer, we need to calculate the following:

errors = predictions - y
The loss is RSS , evaluated with MSE.
loss = np.mean(errors - y)**2 In this case, we will actually be using the half-MSE to have similarities with Logistic Regression from scratch. $$L=\frac{1}{2N}\sum_i(\hat y_i-y_i)^2.$$Half-MSE has the same minimizer as MSE and its 1/2 cancels the derivative’s 2 giving the same gradient template as Logistic Regression: $$dw=X^T errors/N, \qquad db=mean(errors).$$ Using “pure” MSE the derivative would have been: $$dw=2X^T errors/N, \qquad db=mean(errors).$$
weights w and biases are initialized as np.zeros and np.mean(y) respectively, since the weights act as a sort of feature importance and the bias works as an intercept . After choosing a learning rate, at further steps w and b will be updated as:
- w-=dw * learning_rate
- b-=db * learning_rate

Class structure
#

main attributes:
- learning_rate
- n_iterations
- weights w_ and bias b_ initialized as None
fit method
predict method

import numpy as np

class LinearRegression:
	def __init__(self, learning_rate=0.05, n_iterations=1000):
		self.learning_rate = learning_rate
		self.n_iterations = n_iterations
		self.w_ = None
		self.b_ = None
		
	def fit(self, X, y):
		pass
		
	def predict(self, X):
		pass

Training
#

The fit method can be implemented following these steps:

RSS

Tue, 29 Sep 2026 00:00:00 +0000

The Residual Sum of Squares (RSS) is the loss function used in Linear Regression to find the weights.

$$ \operatorname{RSS} = \sum_{i=1}^{n} (y_i - \hat{y}_i)^2, \qquad \hat{y}_i = \beta_0 + x_i^{\mathsf{T}}\beta $$

It is also the core error metric used to build and optimize regression decision trees by selecting the best split that results in the lowest total RSS for the resulting child nodes:

$$RSS = \sum_{i \in R_1} (y_i - \hat{y}_{R_1})^2 + \sum_{i \in R_2} (y_i - \hat{y}_{R_2})^2$$

residual

Sat, 26 Sep 2026 00:00:00 +0000

The residual is the distance between $\tilde{y}$ (what the model predicted) and the actual value $y$.

It is particularly important for the Linear Regression model, that minimizes residual sum of squares (RSS) to estimate weights.

Linear Regression

Thu, 24 Sep 2026 00:00:00 +0000

Formula
#

When using Linear Regression, the goal is to estimate a function $f(x)$ such that each feature has a linear relationship to the target variable $y$.
This is a supervised learning model written as:

$$ y = \beta_0 + X\beta $$

with:

$\beta_0$ = intercept (or bias)
$X$ = matrix of predictor variables
$\beta$ = vector of parameters that determine the weights for each variable. For a single predictor, the sign of $\beta$ gives the direction of the relationship:
- $\beta > 0$ is positive
- $\beta < 0$ is negative
- $\beta = 0$ means no linear effect.

intercept

Sun, 20 Sep 2026 00:00:00 +0000

In Linear Regression, the intercept is the expected value of the dependent variable $Y$ when all independent variables are equal to zero. Graphically, it is represented by the intersection point between the regression line and the Y axis on a chart; its value shift the regression line up and down without affecting its tilt or slope.

It can be interpreted as a baseline value, since it acts as the starting point or baseline prediction for the linear regression model before any independent variables effect is added.

Linearregression on davethehuman.com

Linear Regression from scratch

Algorithm and implementation #

Class structure #

Training #

RSS