Blog/What is a transformer?
Definition

What is a transformer?

By Samuel Seidel · Published September 9, 2026

A transformer is the neural network architecture behind nearly every current large language model. It's a stack of identical layers. Each layer has a self-attention step, where every token weighs how relevant every other token is, and a feed-forward step that transforms each token's representation independently. The design, introduced in a 2017 Google paper, broke from earlier sequence models by processing a whole sequence in parallel instead of one token at a time, while still letting distant tokens influence each other directly.

What it replaced

Before transformers, the dominant architectures for sequence data were recurrent networks, which processed a sequence one token at a time, carrying forward a hidden state that summarized everything seen so far. That worked, at two costs. Training couldn't parallelize across the sequence, since each step depended on the previous one's output. And information from early tokens had to survive compression through many intermediate steps, which made long-range dependencies hard to learn.

Self-attention avoids both: every token gets a direct path to every other token, and all of those relationships are scored at once instead of one after another.

What self-attention actually does

For each token, the model computes a query vector and compares it against key vectors from every token in the sequence to get a relevance score, then combines value vectors weighted by those scores. Flash attention is an optimized way of computing exactly this, and the key and value tensors it produces are what the KV cache stores so they aren't recomputed at each decode step. Stacking many attention and feed-forward layers (dozens in most current models) lets the network build increasingly abstract representations of how tokens relate to each other across the sequence.

Where the compute and memory cost actually goes

A transformer's parameters are the trained weights inside its attention and feed-forward layers; they are what a model's weight file stores and what a parameter count counts. Running the model means reading those weights for every generated token while holding the growing KV cache alongside them, so memory bandwidth and capacity, more than raw compute, are the usual bottleneck for token-by-token generation. When a model outgrows one device's memory, tensor parallelism splits individual layer computations across several GPUs without changing the architecture.

Why this matters for choosing hardware

Nearly every open-weight model on the GPUwerk models page is a transformer, so its memory-bandwidth-bound decode pattern, not FLOPs alone, sets serving speed in practice. A DGX Spark's 128 GB of unified memory holds the full weight stack and the KV cache in one pool, which lets it run larger transformer models than a GPU with a smaller, separate VRAM pool. The benchmarks page has measured throughput and latency for several transformer models on that hardware.

Related pages

First top-up: pay $20, get $30 in credit

Run open-weight transformer models on dedicated hardware.

One dedicated Spark with 128 GB of unified memory, from $0.79/hour.

Create account