BPE

Sat, 30 May 2026 00:00:00 +0000

BPE stands for Byte Pair Encoding, a method to split into tokens that can represent both common and rare words using a mix of full words and sub-words units. Spaces are also included in tokens, and this helps LLM detect word boundaries.

Using a BPE tokenizer, the correct encoding and decoding of unknown out-of-vocabulary words is possible without an explicit <|unk|> token.

In short, BPE builds its vocabulary by iteratively merging frequent characters into subwords and frequent subwords into words. For example, it starts with adding single characters to the vocabulary (a, b, etc.), then at the following stage it merges the most frequent characters combinations into subwords (e.g. d and e may be merged into de), dependent from a frequency cutoff.

Tokenization on davethehuman.com

BPE