The DataLoader is a PyTorch data primitive that allows to manage how a Dataset is loaded onto a model. In LLMs tasks, it is usually included in a funcion with the tokenizer so that the input data is already processed to be ingested by the model, e.g.:
def create_dataloader_v1(txt,
batch_size=4,
max_length=256,
stride=128, #number of positions the input shifts across batches
shuffle=True,
drop_last=True,
num_workers=0):
tokenizer = tiktoken.get_encoding("gpt2")
dataset = GPTDatasetV1(txt, tokenizer, max_length, stride)
dataloader = DataLoader(
dataset,
batch_size=batch_size,
shuffle=shuffle,
drop_last=drop_last, #if True, drops the last batch if shorter than batch_size - to prevent loss spikes during training
num_workers=num_workers #n. of CPU processes to use
)
return dataloader