PyTorch on davethehuman.com

softmax

Tue, 13 Oct 2026 00:00:00 +0000

The softmax function (also known as normalized exponential) is a generalization and extension of the sigmoid to move from binary classification to multi-class classification.

It takes a vector of $K$ real numbers (logits) and normalizes it into a probability distribution consisting of $K$ probabilities proportional to the exponentials of the input numbers.

For an input vector $\mathbf{z} = (z_1, z_2, \dots, z_K) \in \mathbb{R}^K$, the Softmax function $\sigma(\mathbf{z})_i$ for a specific component $i$ is defined as:

variance (statistics)

Tue, 13 Oct 2026 00:00:00 +0000

In statistics, Variance measures data dispersion around a mean.
It is a descriptive statistical measure that quantifies how far individual data points deviate from the mean of a dataset.

Depending on the fact that the dataset represent a full population or a sample, the formula is:

$\sigma^2 = \frac{\sum_{i=1}^{N} (x_i - \mu)^2}{N}$ for the entire population
$s^2 = \frac{\sum_{i=1}^{n} (x_i - \bar{x})^2}{n - 1}$ for a sample

In numpy, it can be implemented as:

euclidean distance

Tue, 29 Sep 2026 00:00:00 +0000

The Euclidean distance is the measure of the true straight-line distance between two points in a Euclidean space. It is the geometric distance you would measure with a ruler, representing the shortest path between those points.

The Formula
#

$$ d(p, q) = \sqrt{\sum_{i=1}^{n} (q_i - p_i)^2} $$

In a 2D Space, for two points $P_1(x_1, y_1)$ and $P_2(x_2, y_2)$:

Dataloader

Mon, 21 Sep 2026 00:00:00 +0000

The DataLoader is a PyTorch data primitive that allows to manage how a Dataset is loaded onto a model. In LLMs tasks, it is usually included in a funcion with the tokenizer so that the input data is already processed to be ingested by the model, e.g.:

def create_dataloader_v1(txt, 
						 batch_size=4, 
						 max_length=256, 
 stride=128, #number of positions the input shifts across batches
 shuffle=True,
 drop_last=True, 
 num_workers=0):
 
 tokenizer = tiktoken.get_encoding("gpt2")
 
 dataset = GPTDatasetV1(txt, tokenizer, max_length, stride)
 
 dataloader = DataLoader(
 dataset,
 batch_size=batch_size,
 shuffle=shuffle,
 drop_last=drop_last, #if True, drops the last batch if shorter than batch_size - to prevent loss spikes during training
 num_workers=num_workers #n. of CPU processes to use 
 )

 return dataloader

Dataset