Transformer on davethehuman.com

Attention Is All You Need

Mon, 17 Aug 2026 00:00:00 +0000

The paper that introduced the self-attention mechanism.

BERT

Mon, 17 Aug 2026 00:00:00 +0000

BERT (Bidirectional Encoder Representations from Transformer) is a variant of the transformer architecture built upon the original encoder submodule. It specialises in masked word prediction and it is very strong in classification tasks, like sentiment prediction and document categorization.

few-shot

Mon, 17 Aug 2026 00:00:00 +0000

Few-shot refers to the ability of a model to learn from a minimal number of examples provided by a user as input.

GPT

Mon, 17 Aug 2026 00:00:00 +0000

GPT (Generative Pretrained Trasnsformer) is a variant of the transformer architecture built upon the original decoder submodule and is designed for generative tasks. It was introduced by the paper “Improving Language Understanding by Generative Pre-Training” and specialises in generating texts and its application includes machine translation, text summarization, fiction writing, writing computer code, and more.

This type of models were primarily designed and trained for text completion tasks but they showed very versatile capabilities, both with zero-shot and few-shot learning tasks.

Improving Language Understanding by Generative Pre-Training

Mon, 17 Aug 2026 00:00:00 +0000

The paper that introduced GPT.

self-attention mechanism

Mon, 17 Aug 2026 00:00:00 +0000

The self-attention mechanism allows the model to weigh the importance of different words or tokens in a sequence relative to each other, allowing to capture contextual relationships and long-range dependencies within the input data.

This weighting mechanism enhance the ability of the model to generate coherent and relevant output.

transformer

Mon, 17 Aug 2026 00:00:00 +0000

The transformer architecture is a deep neural network architecture introduced in the 2017 paper Attention Is All You Need.

Originally, the transformer was developed for machine tranlsation, specifically to translate English texts to German and French. Its architecture consists of two submodules:

the encoder (that performs the encoding and produces an embedding representation)
the decoder (that performs the decoding)

Both of them consist of many layers connected by the self-attention mechanism. Famous variants of the transformer architecture are BERT and GPT.

zero-shot

Mon, 17 Aug 2026 00:00:00 +0000

Zero-shot refers to the ability of a model to generalise to completely unseen tasks without any prior specific example.