relative positional embedding

Instead of focusing on the absolute position of a token (as it happens in absolute positional embedding, relative positional embedding focuses on the relative position between tokens. The advantage is that the model can generalize better to sequences of varying lengths, even if it hasn’t seen such lengths during training.