The Dataset is a PyTorch data class that allows to manage how the individual rows are fetched and then loaded into the model by the DataLoader. In LLMs tasks, it is usually included in a funcion with the tokenizer so that the input data is already processed to be ingested by the model, e.g.:
class GPTDatasetV1(Dataset):
def __init__(self, txt, tokenizer, max_length, stride):
self.input_ids = []
self.target_ids = []
token_ids = tokenizer.encode(txt) #this tokenizes the entire text
for i in range(0, len(token_ids) - max_length, stride): #sliding window
input_chunk = token_ids[i:i + max_length]
target_chunk = token_ids[i+1:i+max_length+1]
self.input_ids.append(torch.tensor(input_chunk))
self.target_ids.append(torch.tensor(target_chunk))
def __len__(self):
return len(self.input_ids)
def __getitem__(self, idx):
return self.input_ids[idx], self.target_ids[idx]Among the common args we find the batch size, the stride and max_length.