token

A token is not necessary an entire word: it can be defined as a small unit of text that gets processed by a language model.

It can be a full word, part of a word, or even punctuation depending on how the text is split and mapped to IDs by the tokenizer. E.g. the sentence Sorry Dave, I can't do that can be broken into tokens like this:

19152 --> Sorry
20238 -->  Dave
11 --> ,
358 -->  I
646 -->  can
944 --> 't
653 -->  do
429 -->  that

The ID mapping is necessary to make the model ingest the tokens.

In some case, even the end of a document is marked with a token like <|endoftext|>: when working with multiple independent text sources, this type of token signals the start or end of a segment to the LLM, that can process and “understand” better the input - this is what was used in GPT.

Other “special” tokens (not used in GPT) are:

• [BOS] Beginning Of Sequence - this token marks the start of a text.
• [EOS] End Of Sequence - This is positioned at the end of a text and is particularly useful when concatenating multiple unrelated texts, similar to <|endoftext|>.
• [PAD] Corresponding to the padding operations.
• <|unk|> Corresponding to an Unknown word

There are various way to split a sentence into tokens, the one used in GPT is BPE.