BPE

BPE stands for Byte Pair Encoding, a method to split into tokens that can represent both common and rare words using a mix of full words and sub-words units. Spaces are also included in tokens, and this helps LLM detect word boundaries.