When training LLMs with batch size sizes larger than one, the batch might contain texts of varying lengths. To ensure all texts have the same length, the shorter texts are extended or “padded” up to the length of the longest text in the batch.
padding
·44 words·1 min
Author
Dave the human
Homo sapiens in the loop