A tokenizer is a critical component of the LLM text processing and generation pipeline even though it is not directly part of it. It splits the text into tokens that get converted into numerical IDs to be ingested by the language model (encoding) and it decodes back the LLM’s output to human-readable text (decoding).
Tokenization is usually done on a Dataset within a DataLoader.
Here’s an example code for a basic tokenizer:
class SimpleTokenizerV2:
def __init__(self, vocab):
self.str_to_int = vocab #stores the vocabulary
self.int_to_str = {i:s for s,i in vocab.items()} #creates inverse vocabulary
def encode(self, text):
#processes the input text into token ids
preprocessed = re.split(r'([,.:;?_!"()\']|--|\s)',text)
preprocessed = [item.strip() for item in preprocessed if item.strip()]
#replaces unknown words by <|unk|> tokens
preprocessed = [item if item in self.str_to_int else "<|unk|>" for item in preprocessed]
ids = [self.str_to_int[s] for s in preprocessed]
return ids
def decode(self, ids): #converts token ids back into text
text = " ".join([self.int_to_str[i] for i in ids])
text = re.sub(r'\s+([,.?!"()\'])', r'\1', text) #removes spaces before the specified punctuation
return text