Deeplearning on davethehuman.com

biases

Mon, 28 Sep 2026 00:00:00 +0000

Biases $b$ are constant values added to the sum of inputs and weights before passing through an activation function. Mathematically, their function is the same as the intercept for Linear Regression, and in this case they shift the activation curve to help the model fit complex patterns even when input values are zero.

Together with weights, they are the core parameters for neural networks.

The difference between bias and intercept and the shift they cause is where the shift happens:

leaky ReLU

Mon, 28 Sep 2026 00:00:00 +0000

LeakyReLU is a variation of the ReLU activation function:

$$ f(x) = \max(\alpha x, x) $$

It leaves positive inputs unvaried while multiplying the negative ones by a small positive constant $\alpha$. Unlike ReLU, which outputs zero for any negative input, Leaky ReLU keeps a small slope on the negative side, allowing gradients to flow through neurons with negative inputs during backpropagation, and mitigating the dying ReLU problem.

In standard Leaky ReLU, $\alpha$ is a fixed hyperparameter, while in [PReLU] it is learned during training.

weights

Mon, 28 Sep 2026 00:00:00 +0000

In neural networks, Weights $W$ are numerical coefficients assigned to connections between neurons. They determine how much influence one neuron has on the next; higher weights mean stronger connections and greater importance given to that signal.

Together with biases, they are the core parameters for neural networks.

activation function

Tue, 22 Sep 2026 00:00:00 +0000

An Activation Function is a mathematical operation applied to the linear output of a neuron to introduce non-linearity into a neural network.

Without activation functions, a multi-layer neural network collapses into a single linear transformation (similar to Linear Regression) regardless of its depth:

$$ W_2(W_1 x + b_1) + b_2 = (W_2 W_1)x + (W_2 b_1 + b_2) = W'x + b' $$

Non-linear activations allow networks to learn complex, non-linear decision boundaries.

Common examples of activation functions used in neural networks are:

ReLU

Tue, 22 Sep 2026 00:00:00 +0000

ReLU (Rectified Linear Unit) is an activation function with formula:

$$ f(x) = \max(0, x) $$

It is the default activation function in the hidden layers of modern neural networkss (FNN, CNN, etc.) and it just checks if a number is larger than zero, allowing for quick computation and fast convergence.

Pros:

it tackles the vanishing gradient problem, keeping the gradient flow strong during backpropagation - this happens because its derivative (gradient) for any positive input is always exactly 1
it allows sparse activation, meaning that only relevant neurons fire, while negative inputs are zeroed out

Cons:

neural network

Mon, 21 Sep 2026 00:00:00 +0000

A neural network is a computational model inspired by the biological brain structure, consisting of interconnected layers of neurons that map input features to target outputs.

A standard feedforward neural network (FNN) is organized into three main types of layers:

Input Layer: Receives the raw input vector $x$.
Hidden Layer(s): Extracts latent representations using weighted sums and activation function.
Output Layer: Produces the final prediction $\hat{y}$.

The core mechanism of the learning process of a neural network operates in a two-step feedback loop:

neuron

Mon, 21 Sep 2026 00:00:00 +0000

In deep learning, a neuron is the foundational building block of a neural network. It computes a scalar output from a vector of inputs.

Formula
#

Given an input vector $\mathbf{x} = [x_1, x_2, \dots, x_n]$, a weight vector $\mathbf{w} = [w_1, w_2, \dots, w_n]$, and a scalar bias $b$:

$$ z = \mathbf{w}^T \mathbf{x} + b = \sum_{i=1}^{n} w_i x_i + b $$

The final output $a$ is obtained by applying an activation function $g(\cdot)$:

batch

Sun, 20 Sep 2026 00:00:00 +0000

The batch is the sample used to perform a single forward pass and backward pass before the model updates its weights. The model training is strongly influenced by the batch size.

batch size

Sun, 20 Sep 2026 00:00:00 +0000

Batch size refers to the number of training examples processed in a batch, which corresponds to a single forward pass and backward pass before the model updates its weights.

For instance, if a dataset contains 10,000 images and the batch size is set to 64, the model splits the dataset into subsets of 64 images, calculates the average error (loss) across those 64 samples, updates its weights, and then moves to the next batch.

intercept

Sun, 20 Sep 2026 00:00:00 +0000

In Linear Regression, the intercept is the expected value of the dependent variable $Y$ when all independent variables are equal to zero. Graphically, it is represented by the intersection point between the regression line and the Y axis on a chart; its value shift the regression line up and down without affecting its tilt or slope.

It can be interpreted as a baseline value, since it acts as the starting point or baseline prediction for the linear regression model before any independent variables effect is added.

Attention Is All You Need