The Transformer Architecture From a Top View

Author(s): Dimitris Effrosynidis

Originally published on Towards AI.

Exploring the encoder-decoder magic in NLP behind LLMs
Image created by the author.

The state-of-the-art Natural Language Processing (NLP) models used to be Recurrent Neural Networks (RNN) among others.

And then came Transformers.

Transformer architecture significantly improved natural language task performance compared to earlier RNNs.

Developed by Vaswani et al. in their 2017 paper “Attention is All You Need,” Transformers revolutionized NLP by leveraging self-attention mechanisms, allowing the model to learn the relevance and context of all words in a sentence.

Unlike RNNs that process data sequentially, Transformers analyze all parts of the sentence simultaneously. This parallel processing capability allows Transformers to learn the context and relevance of each word about every other word in a sentence or document, overcoming limitations related to long-term dependency and computational efficiency found in RNNs.

But let’s explore the architecture step by step.

Image created by the author.There are two components in a Transformer Architecture: the Encoder and the Decoder.These components work in conjunction with each other and they share several similarities.Encoder: Converts an input sequence of tokens into a rich, continuous representation that captures the context of each token within the sequence. Its output is a sequence of embedding vectors, often called the hidden state or context.Decoder: Uses the encoder’s hidden state to iteratively generate an output sequence… Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI