The transformer architecture is a deep neural network architecture introduced in the 2017 paper Attention Is All You Need.
Originally, the transformer was developed for machine tranlsation, specifically to translate English texts to German and French. Its architecture consists of two submodules:
- the encoder (that performs the encoding and produces an embedding representation)
- the decoder (that performs the decoding)
Both of them consist of many layers connected by the self-attention mechanism. Famous variants of the transformer architecture are BERT and GPT.