Understanding LLMs: How GPT-Style Models Actually Work

Understanding LLMs: How GPT-Style Models Actually Work

Start from one concrete fact about how these systems are built, one that most explanations skim past: a GPT-style model takes text apart into subword units, turns them into vectors, and runs them through a stack of attention and feed-forward transformations, all to choose one next token at a time. The prediction part is real, but everything interesting lives in how the prediction happens: how text becomes numbers, what actually occurs inside a forward pass, and why sampling parameters change personality. This article follows a single token through the entire machine, because once you can trace that path, model behavior stops being mysterious and starts being engineering.

The core idea to hold onto: a GPT-style model is an autoregressive transformer trained to predict one token given all previous tokens. Nothing in the architecture knows what truth is, what your intent is, or what happens after generation ends. Every capability, from writing sonnets to debugging code, emerges from next-token prediction at enormous scale, then gets shaped by post-training into something that follows instructions.

From Text to Numbers#

Models cannot read characters. The first layer of the system is a tokenizer, which chops text into subword units using byte-pair encoding. BPE starts with single bytes and iteratively merges the most frequent adjacent pairs until a fixed vocabulary exists, typically somewhere between 50,000 and 200,000 entries depending on the model.

You can watch it happen with tiktoken:

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o-mini")
text = "Tokenization determines what the model can even see."

ids = enc.encode(text)
print(len(ids))                                  # 10
print([enc.decode([i]) for i in ids])
# ['Token', 'ization', ' determines', ' what', ' the', ' model', ' can', ' even', ' see', '.']

Notice what happened there. "Tokenization" split into two pieces, common words stayed whole, and punctuation became its own token. This matters more than beginners expect. Billing counts tokens, context limits count tokens, and tasks involving character-level manipulation, like counting letters or doing digit-by-digit arithmetic, are hard partly because numbers arrive as opaque chunks rather than place-value digits.

That loop back edge is the autoregression: generation means running the same forward pass repeatedly, appending each sampled token to the input, until a stop condition fires.

One Forward Pass, Step by Step#

Take the token IDs and look up their embedding vectors from a learned table. Each ID maps to a vector of several thousand floats, the numeric representation the network actually manipulates. Position information gets mixed in here too; older architectures used learned absolute position vectors, while most current open models use rotary position embeddings (RoPE), which rotate attention queries and keys proportionally to distance so relative order falls out of the math.

Then the vectors flow through a stack of identical transformer blocks. The depth varies by model, but the shape is constant, and it helps to picture the residual stream: a highway of vectors where every block reads the current state, computes a small update, and adds its contribution back.

Each block contains exactly two sublayers:

1. Attention, which lets every token gather information from earlier tokens.
2. An MLP (feed-forward network), which transforms each token's vector independently, expanding it roughly fourfold through a gated activation such as SwiGLU before projecting back down.

Both sublayers are wrapped in normalization (RMSNorm, applied before the sublayer in modern designs) and residual connections. If attention is how tokens communicate, the MLP is where most of the model's learned knowledge lives; scaling studies and interpretability work both suggest factual associations concentrate in those feed-forward weights, while attention handles routing information between positions.

Early blocks tend toward local patterns like syntax and phrase structure; deeper blocks carry more abstract relationships. Nobody placed that hierarchy deliberately; gradient descent discovered it because it compressed training data efficiently.

Attention Without the Mysticism#

Self-attention answers one question for every token: which earlier tokens should I pull information from, and how strongly? Each token's vector is projected three ways, into a query, a key, and a value. Compare my query against your key with a dot product, scale it, softmax across positions so the weights sum to one, and output the weighted average of values.

A tiny worked example makes it concrete. Three tokens with two-dimensional queries and keys:

import numpy as np

def softmax(x):
    e = np.exp(x - x.max())
    return e / e.sum()

q_new = np.array([1.0, 0.2])          # query from the newest token
keys = np.array([
    [0.9, 0.1],                        # key for token A
    [0.2, 0.8],                        # key for token B
    [0.1, 0.3],                        # key for token C
])
scores = keys @ q_new / np.sqrt(2)     # scaled dot products
weights = softmax(scores)
print(weights)                         # e.g. [0.52, 0.31, 0.17]
output = weights @ values              # weighted blend of value vectors

Token A dominated because its key aligned with the new query. Multiply by values instead of raw vectors and you get a context-aware representation for the newest position. A causal mask forces attention weights onto future positions to zero, which is what makes this a language model rather than a bidirectional encoder.

Multi-head attention runs several of these operations in parallel with different projections, letting different heads specialize: some track syntactic dependencies, others long-range references, others positional patterns. Interpretability research has found heads that copy names, heads that track induction (repeating earlier sequences), heads that suppress irrelevant context. The behaviors were not programmed; they were the cheapest solutions gradient descent found.

Where attention really shows its value is reference resolution. In "The robot lifted the box because it was heavy," the word "it" needs to bind to "box", not "robot". Attention weights do exactly this binding:

How Sampling Turns Scores into Words#

The final block produces logits: one score per vocabulary entry. Softmax converts them to probabilities, and then a sampler picks. This stage is where "the model" becomes configurable behavior.

Temperature divides logits before softmax. Low temperature sharpens the distribution toward the top candidate; high temperature flattens it and lets unlikely words through:

import numpy as np

logits = np.array([4.0, 2.5, 1.0, 0.2])

def sample_probs(logits, temperature=1.0):
    scaled = logits / temperature
    e = np.exp(scaled - scaled.max())
    return e / e.sum()

print(sample_probs(logits, 0.3).round(3))
# [0.994 0.006 0.    0.   ]   nearly deterministic
print(sample_probs(logits, 1.5).round(3))
# [0.664 0.158 0.038 0.008]   exploratory

The same logits produce entirely different behavior under the two settings, which is why temperature is the first parameter to check whenever outputs feel "too random" or "too repetitive":

Top-p sampling (nucleus) keeps only the smallest set of candidates whose probabilities sum past p, renormalizes, and samples within that set, which bounds worst-case weirdness better than raw temperature alone. In practice: temperature near 0.2-0.4 for extraction and classification, 0.7-1.0 for creative generation, and never treat temperature as a creativity dial for tasks where correctness is binary.

What Training Actually Optimizes#

Pretraining is brutally simple: predict the next token on trillions of web pages, books, and code repositories, scored by cross-entropy loss, optimized by gradient descent. Compression pressure forces the network to internalize grammar, facts, reasoning patterns, and programming semantics, because predicting text well requires modeling the processes that produced it.

But a raw pretrained model only continues text. It does not know it should answer you. Post-training fixes that in stages. Supervised fine-tuning trains on curated instruction-response pairs, teaching the chat format itself. Then preference optimization aligns behavior: either RLHF (train a reward model on human preference comparisons, optimize the policy against it) or direct preference optimization (DPO), which tunes weights directly on preference pairs without a separate reward model. After post-training, the assistant persona, refusals, and instruction-following are baked into weights, which is why the same base model behind two chat products can behave completely differently.

Where the Mental Model Breaks#

Understanding the mechanism explains the failure modes instead of just cataloging them. Hallucination is sampling continuing plausibly when evidence runs out; nothing in the objective rewards saying "I don't know" unless post-training taught it. Knowledge freezes at the training cutoff; weights cannot learn your product docs afterward. Context windows are finite and attention quality sags for material buried mid-context, the effect documented in the "lost in the middle" research from Liu and colleagues in 2023. There is no memory beyond the window: every request re-reads the entire conversation. And tokenization quietly sabotages tasks that assume character-level access.

These are not bugs awaiting patches. They are properties of the machine, and robust systems are designed around them with retrieval, structured outputs, and evaluation harnesses rather than wishes.

Strip away the mystique and a GPT-style model is an enormously sophisticated stochastic function from token sequences to probability distributions. It has no goals between requests, no state across them, and no self-correcting relationship with truth. That framing sounds deflationary until you notice it is empowering: stochastic functions can be constrained with schemas, grounded with retrieved evidence, evaluated against golden sets, and routed based on cost. Engineers who internalize the mechanism stop asking the model to be reliable and start building systems that make reliability measurable. The payoff of the mental model is practical: the failure modes above stop looking like betrayals and start looking like measurable properties you can design around.