LLM on davethehuman.com

leaky ReLU

Mon, 28 Sep 2026 00:00:00 +0000

LeakyReLU is a variation of the ReLU activation function:

$$ f(x) = \max(\alpha x, x) $$

It leaves positive inputs unvaried while multiplying the negative ones by a small positive constant $\alpha$. Unlike ReLU, which outputs zero for any negative input, Leaky ReLU keeps a small slope on the negative side, allowing gradients to flow through neurons with negative inputs during backpropagation, and mitigating the dying ReLU problem.

In standard Leaky ReLU, $\alpha$ is a fixed hyperparameter, while in [PReLU] it is learned during training.

activation function

Tue, 22 Sep 2026 00:00:00 +0000

An Activation Function is a mathematical operation applied to the linear output of a neuron to introduce non-linearity into a neural network.

Without activation functions, a multi-layer neural network collapses into a single linear transformation (similar to Linear Regression) regardless of its depth:

$$ W_2(W_1 x + b_1) + b_2 = (W_2 W_1)x + (W_2 b_1 + b_2) = W'x + b' $$

Non-linear activations allow networks to learn complex, non-linear decision boundaries.

Common examples of activation functions used in neural networks are:

ReLU

Tue, 22 Sep 2026 00:00:00 +0000

ReLU (Rectified Linear Unit) is an activation function with formula:

$$ f(x) = \max(0, x) $$

It is the default activation function in the hidden layers of modern neural networkss (FNN, CNN, etc.) and it just checks if a number is larger than zero, allowing for quick computation and fast convergence.

Pros:

it tackles the vanishing gradient problem, keeping the gradient flow strong during backpropagation - this happens because its derivative (gradient) for any positive input is always exactly 1
it allows sparse activation, meaning that only relevant neurons fire, while negative inputs are zeroed out

Cons:

Dataloader

Mon, 21 Sep 2026 00:00:00 +0000

The DataLoader is a PyTorch data primitive that allows to manage how a Dataset is loaded onto a model. In LLMs tasks, it is usually included in a funcion with the tokenizer so that the input data is already processed to be ingested by the model, e.g.:

def create_dataloader_v1(txt, 
						 batch_size=4, 
						 max_length=256, 
 stride=128, #number of positions the input shifts across batches
 shuffle=True,
 drop_last=True, 
 num_workers=0):
 
 tokenizer = tiktoken.get_encoding("gpt2")
 
 dataset = GPTDatasetV1(txt, tokenizer, max_length, stride)
 
 dataloader = DataLoader(
 dataset,
 batch_size=batch_size,
 shuffle=shuffle,
 drop_last=drop_last, #if True, drops the last batch if shorter than batch_size - to prevent loss spikes during training
 num_workers=num_workers #n. of CPU processes to use 
 )

 return dataloader

Dataset

Mon, 21 Sep 2026 00:00:00 +0000

The Dataset is a PyTorch data class that allows to manage how the individual rows are fetched and then loaded into the model by the DataLoader. In LLMs tasks, it is usually included in a funcion with the tokenizer so that the input data is already processed to be ingested by the model, e.g.:

class GPTDatasetV1(Dataset):
 
 def __init__(self, txt, tokenizer, max_length, stride):
 self.input_ids = []
 self.target_ids = []

 token_ids = tokenizer.encode(txt) #this tokenizes the entire text

 for i in range(0, len(token_ids) - max_length, stride): #sliding window
 input_chunk = token_ids[i:i + max_length]
 target_chunk = token_ids[i+1:i+max_length+1]

 self.input_ids.append(torch.tensor(input_chunk))
 self.target_ids.append(torch.tensor(target_chunk))

 def __len__(self):
 return len(self.input_ids)

 def __getitem__(self, idx):
 return self.input_ids[idx], self.target_ids[idx]

Among the common args we find the batch size, the stride and max_length.

max_length

Mon, 21 Sep 2026 00:00:00 +0000

Talking about LLMs and the Datasets used for their training, max_length usually indicates the length of the overlapping sequences that get loaded by the DataLoader in a sliding window fashion.

neural network

Mon, 21 Sep 2026 00:00:00 +0000

A neural network is a computational model inspired by the biological brain structure, consisting of interconnected layers of neurons that map input features to target outputs.

A standard feedforward neural network (FNN) is organized into three main types of layers:

Input Layer: Receives the raw input vector $x$.
Hidden Layer(s): Extracts latent representations using weighted sums and activation function.
Output Layer: Produces the final prediction $\hat{y}$.

The core mechanism of the learning process of a neural network operates in a two-step feedback loop:

neuron

Mon, 21 Sep 2026 00:00:00 +0000

In deep learning, a neuron is the foundational building block of a neural network. It computes a scalar output from a vector of inputs.

Formula
#

Given an input vector $\mathbf{x} = [x_1, x_2, \dots, x_n]$, a weight vector $\mathbf{w} = [w_1, w_2, \dots, w_n]$, and a scalar bias $b$:

$$ z = \mathbf{w}^T \mathbf{x} + b = \sum_{i=1}^{n} w_i x_i + b $$

The final output $a$ is obtained by applying an activation function $g(\cdot)$:

stride

Mon, 21 Sep 2026 00:00:00 +0000

The stride dictates the number of positions the inputs shift across batches, creating a sliding window approach. It is used for example in the DataLoader; in the following example we see that a stride = 1 shifts the second batch by one position compared to the first one, with the target portion as an anchor point for the shift.

dataloader = create_dataloader_v1(raw_text, batch_size=1, 
 max_length=4, #usually much larger (e.g. 256)
 stride=1, 
 shuffle=False)

data_iter = iter(dataloader)
inputs, targets = next(data_iter)
print("Inputs:\n", inputs)
print("\nTargets:\n", targets)

```
Inputs:
 tensor([[ 40, 367, 2885, 1464]])

Targets:
 tensor([[ 367, 2885, 1464, 1807]])
```

#with a stride=1 the second batch will be shifted by 1 position 
inputs, targets = next(data_iter)
print("Inputs:\n", inputs)
print("\nTargets:\n", targets)

```
Inputs:
 tensor([[ 367, 2885, 1464, 1807]])

Targets:
 tensor([[2885, 1464, 1807, 3619]])
```

The anchor point concept is mote visible in an example where we use batch_size = 2 , and let’s use a larger stride = 2 too:

padding

Sat, 19 Sep 2026 00:00:00 +0000

When training LLMs with batch size sizes larger than one, the batch might contain texts of varying lengths. To ensure all texts have the same length, the shorter texts are extended or “padded” up to the length of the longest text in the batch.

embedding

Fri, 18 Sep 2026 00:00:00 +0000

An embedding is a mapping from discrete objects, such as words, images, or entire documents, to points in a continuous vector space. Its primary purpose is to convert non-numeric data into a format that neural networks can process.

One of the earlier and most popular examples of word embedding is the Word2Vec approach.

Such embedding models can be used as pretrained models to generate embeddings for Machine Learning models, but LLMs produce their own embeddings as part of the input layer and they get updated during training. The advantage of optimizing the embeddings as part of the training is that they get optimized to the specific task and data at hand.

Word2Vec

Fri, 18 Sep 2026 00:00:00 +0000

Word2Vec is one of the earlier and most popular examples of word embedding approach. It was created training a neural network architecture that predicts the context of a word given the target word or vice versa.

The main idea behind this technique is that words that appear in similar contexts tend to have similar meanings, and consequently, when projected into two-dimensional word embeddings for visualization, simple terms are displayed as clustered together.

self-supervised learning

Mon, 17 Aug 2026 00:00:00 +0000

LLMs use self-supervised learning, where the model generates its own labels from the input data (as opposed to the conventional supervised learning paradigm).

BPE

Sat, 30 May 2026 00:00:00 +0000

BPE stands for Byte Pair Encoding, a method to split into tokens that can represent both common and rare words using a mix of full words and sub-words units. Spaces are also included in tokens, and this helps LLM detect word boundaries.

Using a BPE tokenizer, the correct encoding and decoding of unknown out-of-vocabulary words is possible without an explicit <|unk|> token.

In short, BPE builds its vocabulary by iteratively merging frequent characters into subwords and frequent subwords into words. For example, it starts with adding single characters to the vocabulary (a, b, etc.), then at the following stage it merges the most frequent characters combinations into subwords (e.g. d and e may be merged into de), dependent from a frequency cutoff.

decoding

Sat, 30 May 2026 00:00:00 +0000

When a tokenizer performs the process of decoding (through its decode method), a list of IDs representing tokens is converted to natural language.

input_token_ids_list = [19152, 20238, 11, 358, 646, 944, 653, 429]
text = tokenizer.decode(input_token_ids_list)
print(text)

Sorry Dave, I can't do that

The other way around from natural language to IDs is called encoding.

encoding

Sat, 30 May 2026 00:00:00 +0000

When a tokenizer performs the process of encoding (through its encode method), a natural language text is broken into tokens that are then converted to IDs.

# Even notes can have code
prompt = "Sorry Dave, I can't do that"
input_token_ids_list = tokenizer.encode(prompt)
print(input_token_ids_list)

[19152, 20238, 11, 358, 646, 944, 653, 429]

The way back from IDs to natural language is called decoding.

vocabulary

Sat, 30 May 2026 00:00:00 +0000

The number of tokens that can be handled by a tokenizer.

A larger vocabulary in LLMs:

increases the model size because the embedding and output layers must store more token representations
increases the per-token compute cost of producing next-token probabilities
allows more words to be represented as single tokens rather than being split into subword components; this can reduce the sequence length since less tokens are required to represent a sentence

So the tradeoff is between a larger vocabulary with somewhat higher per-token cost and a smaller vocabulary that often produces longer token sequences.

RLHF

Fri, 15 May 2026 00:00:00 +0000

RLHF (Reinforcement Learning From Human Feedback) involves human evaluations or rankings of model outputs as reward signals; this means that humans are involved in the process to guide the model toward human-preferred behaviors.

This is in contrast with RL in the context of reasoning, where the models rely on automated or environment-based reward signals (more objective but potentially less aligned with human preference).

distillation

Thu, 14 May 2026 00:00:00 +0000

Distillation (also called knowledge distillation) consists of transferring complex reasoning patterns learned by larger models into smaller ones. In deep learning, distillation happens when a smaller “student” model learns from outputs and logits of a larger “teacher” model; when talking about Large Language Models, distillation typically means performing supervised fine-tuning using high-quality labeled instruction datasets generated by a more capable LLM.

inference-compute scaling