7 AI areas everyone should understand in 2026

Ask a frontier model how many times the letter r appears in "strawberry" and it may still get it wrong. The same model will write a working distributed lock or explain a proof in measure theory. That gap is not a bug anyone forgot to fix. It falls directly out of the first mechanism in this article, and once you see it, a dozen other behaviours stop being mysterious.

The seven areas below are the ones that carry the most explanatory weight. Learn them and most AI news becomes readable: you can tell which claims are load-bearing, which are marketing, and which are physically impossible given the arithmetic. Each section gives the mechanism at the level of the actual equations, then hands you a figure that computes those equations live so you can push on the assumptions yourself.

What you will be able to explain afterwards

• Why a model that reasons about topology cannot reliably count letters, and what that costs per API call.
• Why doubling a model's context window more than doubles what it costs to serve, and where the quadratic term actually bites.
• How a compute budget in FLOPs turns into a specific parameter count and token count, and why the most cited number in that calculation was quietly wrong.
• What the KL penalty in RLHF does mechanically, and why removing it produces a model that games its own reward.

What is helpful to know first

Token

The unit a language model actually reads and writes. Usually a common word fragment rather than a character or a word.

Parameter

One learned number inside the model. A 70B model has 70 billion of them, and they stay fixed once training ends.

Embedding

A list of numbers standing in for a token or a piece of text, arranged so that similar meanings land near each other.

Logit

The raw score a model assigns each possible next token, before those scores are turned into probabilities.

Softmax

The function that turns a list of scores into probabilities that sum to one, exaggerating the gaps between them as it goes.

FLOP

One floating-point operation. Training compute is measured in total FLOPs, and frontier runs are now in the range of 10²⁵ to 10²⁶.

Pretraining

The first and most expensive training stage, where the model learns to predict the next token over a very large corpus.

Post-training

Everything done after pretraining to turn a raw text predictor into something that follows instructions and refuses harmful requests.

The model never sees your letters

A language model never receives your text. It receives a list of integers. The component that produces those integers is the tokenizer, and it is the only part of the stack that is not learned by gradient descent. It is fit once, before training, and then frozen for the life of the model.Production tokenizers work on bytes rather than characters, so every possible input has some representation and there is no unknown-token case. The cost is that one non-ASCII character can consume several tokens.

Nearly every production model uses byte pair encoding, introduced as a compression algorithm by Gage in 1994 and adapted for machine translation by Sennrich, Haddow and Birch in 2016.1 The training procedure is short enough to state completely:

1. Split the corpus into words, keeping the leading space attached so that the and the stay distinct.
2. Represent every word as a sequence of single characters.
3. Count every adjacent pair of symbols across the whole corpus, weighted by how often each word occurs.
4. Merge the single most frequent pair everywhere it appears, and record that merge with its rank.
5. Repeat from step 3 until you have as many merges as your vocabulary budget allows.

To encode new text you replay those merges in the order they were learned, applying the lowest-rank applicable merge at each step until none apply. GPT-4's tokenizer runs about 100,000 merges, and the tokenizer introduced with GPT-4o roughly 200,000. The figure below runs the identical algorithm with a budget of a few hundred merges on a small corpus embedded in this page, so you can watch the merge count change what the model sees.

This is the whole explanation for the strawberry problem. By the time the model sees the word, it is two or three opaque integers. The letters are gone. Nothing in the architecture can recover them except by having memorised, during pretraining, statements about how those particular tokens are spelled. Counting letters is not a reasoning failure. It is a perception failure, and it is baked in before the first layer runs.

The same mechanism explains a set of behaviours that otherwise look unrelated. Arithmetic is unreliable in part because number tokenization is irregular, so 1234 and 1235 can decompose completely differently. Non-English text costs more, sometimes several times more per sentence, because the merges were fit on a corpus that was mostly English, so other scripts fall back to short tokens. Rare proper nouns fragment into pieces that carry no useful meaning individually.

For a tokenizer you can inspect against the real GPT vocabularies, Tiktokenizer shows the exact token boundaries for the production encodings side by side.

Every position compares itself to every other

Once text is a list of integers, each integer indexes a row of the embedding matrix, producing a vector of numbers. The stack of these vectors, one per position, is the residual stream. Every layer reads from it and writes back into it, and after the final layer the vector at the last position is multiplied by an output matrix to produce a logit for every token in the vocabulary.

Attention is the only operation in the whole architecture that moves information between positions. Everything else, the MLP blocks and the normalisation, acts on each position independently. From Vaswani et al. in 2017, the operation is:2

Read left to right, at each position the model produces three vectors from the residual stream. The query $q$ encodes what this position is looking for. The key $k$ encodes what each position offers. The value $v$ encodes what gets copied if that position is selected. The dot product $q\cdot k_i$ scores how well position $i$ answers the query, softmax turns those scores into weights that sum to one, and the output is the weighted average of the values.

Two details in that formula do real work. The division by $\sqrt{d_k}$ exists because the dot product of two random vectors of dimension $d_k$ has variance proportional to $d_k$. Without the correction, scores grow with dimension, softmax saturates onto a single position, and the gradient through it goes to nearly zero. The causal mask, which sets scores to negative infinity for any position later than the current one, is what makes the model a next-token predictor rather than a document autoencoder. It is also what makes the KV cache possible, since a past position's key and value can never change.

The lower panel is where the economics live. Attention compares every position against every earlier position, so the number of scored pairs for a sequence of length $n$ is $n$($n$+1)/2, which grows as $n$². Separately, serving requires holding the key and value vectors for every past token in memory:

That term is linear in sequence length, and the leading 2 is simply keys plus values. It is also the reason grouped-query attention exists: by letting several query heads share one key-value head, Ainslie et al. cut $h_{kv}$ by a large factor and shrink the cache proportionally, at a small quality cost.3 When a provider announces a longer context window, this is the line item they had to pay for.

Two visual references are worth the detour here. Poloclub's Transformer Explainer runs a real GPT-2 in the browser and lets you watch attention weights change as you edit the prompt, and Brendan Bycroft's LLM visualisation walks a single token through every matrix multiply in the network at full scale.

How a compute budget turns into a model

Before a frontier run starts, someone has to decide how big the model will be and how many tokens it will see. Those two choices are not independent, because they are both paid for out of one compute budget. The standard approximation, from Kaplan et al. in 2020, is that a transformer costs about $C\approx 6ND$ FLOPs to train, where $N$ is parameters and $D$ is training tokens.4 The 6 is two floating-point operations per parameter per token on the forward pass and about twice that on the backward pass.This counts only the multiplies against parameters. It ignores attention's own quadratic term, which is negligible while the context is short relative to model width and stops being negligible at long context.

Hoffmann et al. then fit an explicit form for the loss you get from any pair of choices:5

Each term has a plain reading. $E$ is the irreducible loss, the entropy of the text itself, which no model of any size can beat. $A/N^{\alpha }$ is the penalty for being too small to represent what is in the data. $B/D^{\beta }$ is the penalty for not having seen enough data to pin the parameters down. Minimising this subject to $C=6ND$ gives the compute-optimal split, and the paper's headline was that $N$ and $D$ should scale in equal proportion. Chinchilla itself was 70B parameters on 1.4T tokens, a ratio of 20 tokens per parameter, and it beat the 280B-parameter Gopher trained on the same compute budget.

Here is the part that rarely survives into secondhand summaries. In 2024 Besiroglu, Erdil and colleagues re-extracted the underlying data and found the published constants do not reproduce the paper's own recommendation.6 The figure below lets you check that yourself. Switch between the two fits at Chinchilla's actual compute budget and watch the optimum move.

The disagreement matters beyond bookkeeping. Under Hoffmann's published constants the optimal token-to-parameter ratio drifts upward with scale, from roughly 34 at 10²¹ FLOPs to over 90 at 10²⁶. Under the refit it stays near 20 across the entire range, which is what "scale them equally" actually means. The refit is the one consistent with the paper's own conclusion and with the model DeepMind shipped.

Every frontier lab now trains well past the compute-optimal token count anyway, and the reason is in the objective. Chinchilla optimises training cost alone. If a model will serve billions of tokens after training, a smaller model trained longer is cheaper over its life even though it cost more to build, an argument formalised by Sardana et al. in 2024.7 This is why an 8B model may see 15T tokens, nearly 2,000 per parameter, roughly a hundred times the Chinchilla point. That is a deliberate and correctly reasoned departure, not a contradiction.

From text predictor to assistant

A model fresh out of pretraining does one thing: continue text plausibly. Given a question it is as likely to produce more questions as an answer, because lists of questions are common in the corpus. Post-training is what closes that gap, and it uses a tiny fraction of the compute pretraining did.8

Supervised fine-tuning comes first. Train on curated prompt and response pairs with the ordinary next-token objective, computing loss only on the response tokens. This is enough to fix the format problem. It cannot fix quality, because it can only imitate demonstrations, and it has no way to express that one answer is better than another.

Preference learning supplies that missing signal. Collect pairs where an annotator marked one response as better, then model the probability of that judgement with Bradley-Terry:

Fit a reward model $r$ to maximise the likelihood of the observed comparisons, and you have a scorer that generalises past the responses humans actually ranked. Now optimise the policy against it. The objective is not raw reward, because raw reward gets gamed. It is reward minus a penalty for drifting from the model you started with:

That constrained problem has an exact solution, and it is worth staring at because it explains the entire tuning dynamic:

The aligned model is the original model reweighted by an exponential in reward.No trainer computes this directly, since the partition function $Z(x)$ sums over every possible response. PPO approximates it by sampling. The closed form is still worth knowing, because it says what PPO is approximating and therefore what moving $\beta$ actually does. Nothing is created. Probability mass is moved from responses the reward model dislikes to ones it likes, and $\beta$ sets how far that mass is allowed to travel. The figure below evaluates this formula directly over a set of candidate responses.

Rafailov et al. noticed in 2023 that you can rearrange that closed form to express reward in terms of the policy, substitute it back into the Bradley-Terry likelihood, and eliminate the reward model entirely.9 What remains is direct preference optimization, a plain classification loss on preference pairs:

No reward model to train, no rollouts to generate, no PPO machinery. The same $\beta$ appears, doing the same job, because it is the same objective algebraically rearranged.

Verifiable rewards are the change that reshaped post-training most recently. Where a task has a checkable answer, a unit test that passes, a proof that type-checks, a final numeric result that matches, you can delete the learned reward model and score the outcome directly. Lambert et al. formalised this as RLVR in Tulu 3,10 and DeepSeek used the same idea at scale with group relative policy optimization, which drops PPO's value network and computes each sample's advantage against the mean reward of a group of samples for the same prompt.11 The reward is no longer a model that can be fooled. It is a program that either passes or does not, which is exactly why this technique works so well for mathematics and code and does not straightforwardly extend to essay quality.

Thinking for longer instead of training bigger

Until 2024 nearly all of the compute in a model's life was spent before release. A prompt got one forward pass per output token and that was the budget. The reasoning models changed the shape of that curve by making the amount of computation spent per question a variable you can turn up.

The simplest version is sampling more than once. If a model solves a problem with probability $p$ on any single attempt, and attempts are independent, then the chance at least one of $k$ attempts is correct is:

Brown et al. measured exactly this in 2024 and found coverage keeps climbing over four orders of magnitude of $k$.12 On SWE-bench Lite, DeepSeek-V2-Coder went from 15.9 percent with one sample to 56 percent with 250, beating the single-attempt state of the art at the time. On GSM8K and MATH, coverage with Llama-3 passed 95 percent at 10,000 samples.

The catch is in the word coverage. It counts problems where a correct answer appears somewhere in the pile. Turning that into an answer you can ship requires picking the right one, and that is a separate and much harder problem. Both halves are below.

Two things close that gap. A verifier, when the domain permits one, collapses selection to a search problem. Compilers, test suites and proof checkers are verifiers, which is precisely why coding and mathematics were the first places reasoning models became dramatically better. Where no verifier exists, the alternative is training the model to do its own search internally, which is what long chain-of-thought reasoning is: the model spends tokens proposing, checking and revising before committing to an answer.

Snell et al. quantified the trade in 2024.13 On problems where a small base model already has a non-trivial success rate, allocating test-time compute optimally lets it outperform a model 14 times larger under a matched FLOP budget, and their adaptive allocation strategy beat a plain best-of-N baseline by more than four times in efficiency. Compute spent at inference substitutes for compute spent at training, within limits.

What a model can reach at run time

A model's weights are frozen at training time. Everything it knows about your codebase, your documents or today's date has to arrive through the context window. Deciding what goes in there is now the central engineering problem in applied AI, and it is the same problem whether you call it retrieval or agent design.

The retrieval pipeline has five steps and each one loses something:

1. Chunk. Split documents into passages. Too small and a passage loses the context that made it meaningful. Too large and its embedding averages several topics into a vector that matches none of them well.
2. Embed. Map each chunk to a vector with a trained encoder, so that semantic similarity becomes geometric proximity.
3. Index. Store the vectors in a structure that supports approximate nearest-neighbour search, since exact search over a hundred million vectors per query is not affordable.
4. Retrieve. Embed the query the same way, pull the top $k$ chunks by cosine similarity.
5. Rerank and assemble. Optionally rescore the candidates with a slower cross-encoder that reads query and chunk together, then place the survivors in the prompt.

The figure below runs steps 4 and 5 over a small corpus. It scores with term-frequency vectors rather than a neural encoder, which is the honest limitation to state, and the ranking mechanics are identical: vectorise, take cosines, sort.

Longer context windows look like they should make retrieval unnecessary. They do not, for two measured reasons. Liu et al. found in 2023 that accuracy follows a U-shape in the position of the relevant passage: information at the very start or very end of the context is used reliably, and the same information in the middle is often missed, a pattern that persists across models and survives instruction tuning.14 Chroma's 2025 evaluation of 18 models extended this, finding that degradation arrives in sudden cliffs instead of a gradual slope, that semantic similarity between the target and its surrounding distractors predicts failure better than length alone, and, counterintuitively, that coherently structured filler degrades retrieval more than shuffled filler does.15

An agent is what you get when the model is allowed to decide what enters its own context. The loop is short:

1. The model receives the goal, the conversation so far, and a schema for each tool it may call.
2. It emits either a final answer or a structured tool call.
3. The harness executes the call, which is ordinary code, and appends the result to the context.
4. The loop repeats until the model answers or a limit is hit.

The model does not execute anything. It emits a request, and a program the developer wrote decides whether to honour it. Anthropic's Model Context Protocol, open-sourced in November 2024, standardises the interface between the two so that any compliant model can use any compliant tool, turning an M times N integration problem into M plus N.

The mathematics of the loop is where intuition usually fails. If each step succeeds independently with probability $p$, an $n$-step task succeeds with probability $p$^$n$.

Reading the weights is still mostly unsolved

Everything so far describes what the architecture computes. None of it says what any particular parameter means. That question is the domain of mechanistic interpretability, and the honest summary in 2026 is that real progress has been made and the problem is not solved.

The natural hope is that individual neurons correspond to individual concepts. Some do. Most do not, and the reason is a counting argument. A model has some number of dimensions in its residual stream, on the order of thousands. The number of distinguishable things a model needs to represent, every entity, idiom, syntactic role and domain convention in its training corpus, is far larger. There is no way to give each one its own direction.

Elhage et al. showed in 2022 what a network does instead.16 When features are sparse, meaning any given input activates only a few of them, a model can store more features than it has dimensions by assigning them directions that are not quite orthogonal and relying on the nonlinearity to clean up the resulting interference. They call this superposition, and it is the reason a single neuron so often fires for an unrelated-looking set of inputs. The figure below trains their toy model in your browser.

The technique that turned this insight into a usable tool is the sparse autoencoder. Train a wide layer to reconstruct a model's activations under a sparsity penalty, and the dictionary it learns tends to contain directions that are individually interpretable, even though the underlying neurons were not. Anthropic scaled this to a production model in 2024, extracting up to 34 million features from the middle layer of Claude 3 Sonnet, and reported features that fire for concepts as abstract as sarcasm, code errors, deception and sycophancy.17 The features are also causal handles: clamping one changes the model's behaviour in the direction its interpretation predicts.

What remains unsolved is coverage and composition. A dictionary of 34 million features is large and still incomplete. Knowing which features exist does not yet tell you which circuits connect them into the computation that produced a specific answer, and reconstructing those circuits is currently slow and largely manual. Distill's Zoom In and Anthropic's Toy Models of Superposition are the two clearest entry points if you want the visual version of this argument.

The bottom line

Seven mechanisms carry most of the explanatory load. Tokenization determines what the model can perceive and what you pay per call. Attention determines what it can relate and why long context is expensive. Scaling laws determine how a compute budget becomes a specific model. Post-training determines what it will actually do with a request, through an objective whose closed form makes the alignment trade-off explicit. Inference-time compute determines how much a hard question can be improved after training is over. Context and tool design determine what a deployed system can reach, with per-step reliability setting a hard ceiling on task length. Interpretability determines how much of any of this we can inspect, and it is the one where the honest answer is still partial.

The open problems are specific. Nobody has a selection method that keeps pace with coverage in domains without verifiers. Nobody has a scaling law for post-training that works the way Chinchilla works for pretraining. Long-context degradation is measured but not explained, and the mechanism behind those cliffs is unknown. Feature dictionaries are incomplete and circuit-level explanation does not yet scale. Those four gaps are where most of the interesting work of the next two years will be.