Nlp on davethehuman.com

F1-score

Tue, 29 Sep 2026 00:00:00 +0000

F1-score (also called harmonic mean) is a metric that takes into account both precision and recall:

$$ \mathrm{F1} = \frac{2*Precision*Recall}{Precision + Recall} $$

It is widely used in NLP and it balances precision and recall, handling class imbalance and highlighting poor performance on minority classes.

Dataloader

Mon, 21 Sep 2026 00:00:00 +0000

The DataLoader is a PyTorch data primitive that allows to manage how a Dataset is loaded onto a model. In LLMs tasks, it is usually included in a funcion with the tokenizer so that the input data is already processed to be ingested by the model, e.g.:

def create_dataloader_v1(txt, 
						 batch_size=4, 
						 max_length=256, 
 stride=128, #number of positions the input shifts across batches
 shuffle=True,
 drop_last=True, 
 num_workers=0):
 
 tokenizer = tiktoken.get_encoding("gpt2")
 
 dataset = GPTDatasetV1(txt, tokenizer, max_length, stride)
 
 dataloader = DataLoader(
 dataset,
 batch_size=batch_size,
 shuffle=shuffle,
 drop_last=drop_last, #if True, drops the last batch if shorter than batch_size - to prevent loss spikes during training
 num_workers=num_workers #n. of CPU processes to use 
 )

 return dataloader

Dataset

Mon, 21 Sep 2026 00:00:00 +0000

The Dataset is a PyTorch data class that allows to manage how the individual rows are fetched and then loaded into the model by the DataLoader. In LLMs tasks, it is usually included in a funcion with the tokenizer so that the input data is already processed to be ingested by the model, e.g.:

class GPTDatasetV1(Dataset):
 
 def __init__(self, txt, tokenizer, max_length, stride):
 self.input_ids = []
 self.target_ids = []

 token_ids = tokenizer.encode(txt) #this tokenizes the entire text

 for i in range(0, len(token_ids) - max_length, stride): #sliding window
 input_chunk = token_ids[i:i + max_length]
 target_chunk = token_ids[i+1:i+max_length+1]

 self.input_ids.append(torch.tensor(input_chunk))
 self.target_ids.append(torch.tensor(target_chunk))

 def __len__(self):
 return len(self.input_ids)

 def __getitem__(self, idx):
 return self.input_ids[idx], self.target_ids[idx]

Among the common args we find the batch size, the stride and max_length.

max_length

Mon, 21 Sep 2026 00:00:00 +0000

Talking about LLMs and the Datasets used for their training, max_length usually indicates the length of the overlapping sequences that get loaded by the DataLoader in a sliding window fashion.

stride

Mon, 21 Sep 2026 00:00:00 +0000

The stride dictates the number of positions the inputs shift across batches, creating a sliding window approach. It is used for example in the DataLoader; in the following example we see that a stride = 1 shifts the second batch by one position compared to the first one, with the target portion as an anchor point for the shift.

dataloader = create_dataloader_v1(raw_text, batch_size=1, 
 max_length=4, #usually much larger (e.g. 256)
 stride=1, 
 shuffle=False)

data_iter = iter(dataloader)
inputs, targets = next(data_iter)
print("Inputs:\n", inputs)
print("\nTargets:\n", targets)

```
Inputs:
 tensor([[ 40, 367, 2885, 1464]])

Targets:
 tensor([[ 367, 2885, 1464, 1807]])
```

#with a stride=1 the second batch will be shifted by 1 position 
inputs, targets = next(data_iter)
print("Inputs:\n", inputs)
print("\nTargets:\n", targets)

```
Inputs:
 tensor([[ 367, 2885, 1464, 1807]])

Targets:
 tensor([[2885, 1464, 1807, 3619]])
```

The anchor point concept is mote visible in an example where we use batch_size = 2 , and let’s use a larger stride = 2 too:

padding

Sat, 19 Sep 2026 00:00:00 +0000

When training LLMs with batch size sizes larger than one, the batch might contain texts of varying lengths. To ensure all texts have the same length, the shorter texts are extended or “padded” up to the length of the longest text in the batch.

embedding

Fri, 18 Sep 2026 00:00:00 +0000

An embedding is a mapping from discrete objects, such as words, images, or entire documents, to points in a continuous vector space. Its primary purpose is to convert non-numeric data into a format that neural networks can process.

One of the earlier and most popular examples of word embedding is the Word2Vec approach.

Such embedding models can be used as pretrained models to generate embeddings for Machine Learning models, but LLMs produce their own embeddings as part of the input layer and they get updated during training. The advantage of optimizing the embeddings as part of the training is that they get optimized to the specific task and data at hand.

Word2Vec

Fri, 18 Sep 2026 00:00:00 +0000

Word2Vec is one of the earlier and most popular examples of word embedding approach. It was created training a neural network architecture that predicts the context of a word given the target word or vice versa.

The main idea behind this technique is that words that appear in similar contexts tend to have similar meanings, and consequently, when projected into two-dimensional word embeddings for visualization, simple terms are displayed as clustered together.

BPE

Sat, 30 May 2026 00:00:00 +0000

BPE stands for Byte Pair Encoding, a method to split into tokens that can represent both common and rare words using a mix of full words and sub-words units. Spaces are also included in tokens, and this helps LLM detect word boundaries.

Using a BPE tokenizer, the correct encoding and decoding of unknown out-of-vocabulary words is possible without an explicit <|unk|> token.

In short, BPE builds its vocabulary by iteratively merging frequent characters into subwords and frequent subwords into words. For example, it starts with adding single characters to the vocabulary (a, b, etc.), then at the following stage it merges the most frequent characters combinations into subwords (e.g. d and e may be merged into de), dependent from a frequency cutoff.

decoding

Sat, 30 May 2026 00:00:00 +0000

When a tokenizer performs the process of decoding (through its decode method), a list of IDs representing tokens is converted to natural language.

input_token_ids_list = [19152, 20238, 11, 358, 646, 944, 653, 429]
text = tokenizer.decode(input_token_ids_list)
print(text)

Sorry Dave, I can't do that

The other way around from natural language to IDs is called encoding.

encoding

Sat, 30 May 2026 00:00:00 +0000

When a tokenizer performs the process of encoding (through its encode method), a natural language text is broken into tokens that are then converted to IDs.

# Even notes can have code
prompt = "Sorry Dave, I can't do that"
input_token_ids_list = tokenizer.encode(prompt)
print(input_token_ids_list)

[19152, 20238, 11, 358, 646, 944, 653, 429]

The way back from IDs to natural language is called decoding.

vocabulary

Sat, 30 May 2026 00:00:00 +0000

The number of tokens that can be handled by a tokenizer.

A larger vocabulary in LLMs:

increases the model size because the embedding and output layers must store more token representations
increases the per-token compute cost of producing next-token probabilities
allows more words to be represented as single tokens rather than being split into subword components; this can reduce the sequence length since less tokens are required to represent a sentence

So the tradeoff is between a larger vocabulary with somewhat higher per-token cost and a smaller vocabulary that often produces longer token sequences.

token

Wed, 13 May 2026 00:00:00 +0000

A token is not necessary an entire word: it can be defined as a small unit of text that gets processed by a language model.

It can be a full word, part of a word, or even punctuation depending on how the text is split and mapped to IDs by the tokenizer. E.g. the sentence Sorry Dave, I can't do that can be broken into tokens like this:

--> Sorry
--> Dave
--> ,
--> I
--> can
--> 't
--> do
--> that

The ID mapping is necessary to make the model ingest the tokens.

tokenizer

Wed, 13 May 2026 00:00:00 +0000

A tokenizer is a critical component of the LLM text processing and generation pipeline even though it is not directly part of it. It splits the text into tokens that get converted into numerical IDs to be ingested by the language model (encoding) and it decodes back the LLM’s output to human-readable text (decoding).

Tokenization is usually done on a Dataset within a DataLoader.

Here’s an example code for a basic tokenizer:

class SimpleTokenizerV2:
 def __init__(self, vocab):
 self.str_to_int = vocab #stores the vocabulary
 self.int_to_str = {i:s for s,i in vocab.items()} #creates inverse vocabulary

 def encode(self, text): 
	 #processes the input text into token ids
 preprocessed = re.split(r'([,.:;?_!"()\']|--|\s)',text)
 preprocessed = [item.strip() for item in preprocessed if item.strip()]

 #replaces unknown words by <|unk|> tokens
 preprocessed = [item if item in self.str_to_int else "<|unk|>" for item in preprocessed]
 
 ids = [self.str_to_int[s] for s in preprocessed]
 return ids

 def decode(self, ids): #converts token ids back into text
 text = " ".join([self.int_to_str[i] for i in ids])
 text = re.sub(r'\s+([,.?!"()\'])', r'\1', text) #removes spaces before the specified punctuation
 return text

LLM

Sat, 02 May 2026 00:00:00 +0000

An LLM (Large Language Model) is a type of artificial intelligence trained on humongous datasets of text to understand, generate, and manipulate human language. They are built using transformer architectures and function by predicting the next most likely token (word or part of a word) in a sequence.

The core technical concepts:

scale: “Large” refers to both the training data (petabytes of text) and the parameters count (billions)
LLM training pipeline
context window: the maximum amount of text the model can “hold in mind” at one time during a single conversation

The main capabilities: