Skip to main content

token

·116 words·1 min
Dave the human
Author
Dave the human
Homo sapiens in the loop

A token is not necessary an entire word: it can be defined as a small unit of text that gets processed by a language model.

It can be a full word, part of a word, or even punctuation depending on how the text is split and mapped to IDs by the tokenizer. E.g. the sentence Sorry Dave, I can't do that can be broken into tokens like this:

19152 --> Sorry
20238 -->  Dave
11 --> ,
358 -->  I
646 -->  can
944 --> 't
653 -->  do
429 -->  that

The ID mapping is necessary to make the model ingest the tokens.

There are various way to split a sentence into tokens, like BPE.


Comments