The neural-network design behind modern LLMs, from the 2017 paper 'Attention Is All You Need.' It reads a whole sequence in parallel using self-attention.
Last verified · 2026-09-22 · by Moe Ameen
The Transformer is the neural-network architecture that nearly every modern large language model is built on. It was introduced in the 2017 paper "Attention Is All You Need" by eight researchers at Google, and its central idea is in the title: instead of processing text word by word in order, a Transformer looks at the whole sequence at once and uses a mechanism called attention to weigh how much every token should be influenced by every other token. The "T" in GPT and BERT stands for Transformer, and it is the reason those models exist in their current form.
What made it a break from the past is that it dispensed with recurrence and convolutions — the two techniques that dominated sequence modeling before it. Earlier models like RNNs and LSTMs read a sentence left to right, one step at a time, carrying a running memory forward; that ordering made them slow to train and prone to losing track of long-range connections. A Transformer has no such loop. Every position is processed in parallel, and self-attention lets any word directly consult any other word in the input regardless of distance, so "it" at the end of a paragraph can look straight back at the noun it refers to near the start. Because there is no left-to-right loop to encode word order, the model adds positional encodings — signals that tell it where each token sits in the sequence.
The mechanical heart is scaled dot-product attention over three learned projections of each token: a query, a key, and a value. Loosely, each token's query is compared against every token's key to produce relevance weights, and those weights pull a blend of the values — a data-driven way of deciding what to pay attention to. The model runs several of these in parallel (multi-head attention) so different heads can track different kinds of relationships — grammar, reference, topic — at once. Around the attention layers sit feed-forward networks, residual connections, and layer normalization, stacked dozens of times deep. The original design was an encoder-decoder built for translation, but the two halves split into their own lineages: encoder-only models like BERT (built to understand text) and decoder-only models like the GPT and Claude families (built to generate it).
The quiet reason the Transformer took over is not just quality but parallelism. Removing the sequential bottleneck meant the architecture could be trained efficiently on modern GPUs across enormous datasets, and it kept improving as models, data, and compute grew rather than plateauing. That scalability is what turned a 2017 translation paper into the foundation of the entire generative-AI era — the same architecture, scaled up, powers today's chatbots, coding assistants, and image and video models.
The Transformer was published on June 12, 2017, in the arXiv paper "Attention Is All You Need" (Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin), most of them at Google, and presented at the NeurIPS conference later that year. The immediate goal was machine translation, where the model set new state-of-the-art BLEU scores on English-to-German and English-to-French while training in a fraction of the time earlier architectures needed — the paper's headline results were 28.4 BLEU on English-to-German and 41.8 on English-to-French. Attention itself was not new; it had been bolted onto recurrent translation models since around 2014. The paper's radical move was to throw the recurrence away and keep only the attention.
From there the lineage is direct. In 2018 Google released BERT, an encoder-only Transformer that learned deep bidirectional representations of language and reset benchmarks across natural-language understanding. In parallel OpenAI built the GPT series — decoder-only Transformers trained to predict the next token — scaling from GPT-1 in 2018 through GPT-2 and GPT-3 and into the models that put generative AI in front of the public. The pattern that emerged, sometimes called the scaling era, was that the same architecture kept getting better as parameters, data, and compute increased, which is why the industry poured resources into scaling Transformers rather than replacing them. Nearly a decade on, the Transformer remains the dominant design; challengers like state-space models and, more recently, [diffusion language models](/glossary/diffusion-language-model) exist and win specific tradeoffs, but as of 2026 the strongest general models are still Transformers or close hybrids of one.
The Transformer is one of those rare ideas that is worth understanding at exactly one level of depth if you make content rather than build models: know that it reads everything at once and decides, through attention, what matters to what. That single property — any token can consult any other token directly — is why LLMs got good at holding a thought across a long prompt, and it is why context quality matters so much (see [context engineering](/glossary/context-engineering)). You are not tuning the architecture; you are feeding it, and it rewards a clean, well-ordered input because attention is only as useful as the tokens you give it to attend to.
The practical mindset I would keep is that the architecture is a commodity and the operation around it is not. Every serious model — the ones drafting your captions, blogs, and scripts — is a Transformer, so the drafting layer is not where anyone wins anymore; the win is in what you do with the draft. That is deliberately the layer Kompozy owns and the reason it stays model-agnostic: it runs the same content operation on top of whichever Transformer is best today (Claude and OpenAI for copy right now) and could adopt a stronger one tomorrow without the workflow changing, because a [Persona Brief](/glossary/persona-brief) governs the voice and the publishing engine handles the eight social platforms plus blog and email. Understand the Transformer well enough to respect what it can and cannot do — it predicts, it does not know — and then spend your effort on the part it does not solve: turning a good prediction into finished, on-brand, scheduled content.
The Transformer is a neural-network architecture, introduced in the 2017 paper "Attention Is All You Need," that processes an entire sequence of text in parallel and uses a mechanism called attention to weigh how much each token should be influenced by every other token. It replaced the older recurrent and convolutional approaches to sequence modeling and is the foundation of modern large language models — the "T" in GPT and BERT stands for Transformer.
It was introduced on June 12, 2017, in the paper "Attention Is All You Need" by eight researchers — Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin — most of them at Google, and presented at the NeurIPS conference in 2017. It was built for machine translation and set new state-of-the-art results while training far faster than the recurrent models it replaced.
RNNs and LSTMs read a sequence one step at a time, carrying a running memory forward, which makes them slow to train and prone to losing long-range connections. A Transformer removes that loop: it processes every position in parallel and uses self-attention so any token can directly connect to any other token regardless of distance. Because there is no left-to-right loop, it adds positional encodings to represent word order.
Self-attention is the core operation of a Transformer. Each token is projected into a query, a key, and a value; every token's query is compared against every token's key to produce relevance weights, and those weights blend the values into a new representation. In effect it is a data-driven way of deciding which other words each word should pay attention to. Multi-head attention runs several of these in parallel so different heads can track grammar, reference, and topic at once.
Yes — it remains the dominant architecture behind large language models nearly a decade after its introduction. Newer designs like state-space models and diffusion language models exist and win specific tradeoffs such as speed or long-context efficiency, but as of 2026 the strongest general-purpose models are still Transformers or close hybrids of one. The architecture has kept improving with scale rather than being replaced.
The original 2017 Transformer was an encoder-decoder built for translation. Those halves later split into their own families. Encoder-only models like BERT read text bidirectionally to understand it, which suits classification and search. Decoder-only models like the GPT and Claude families generate text one token at a time, which suits chat and content generation. Encoder-decoder designs are still used for tasks like translation and summarization that map one sequence to another.