# MicroGPT, Simply Explained

#96 Apr 6 LLMs

https://www.aibutsimple.com/p/microgpt-simply-explained

Hello from the AI, but simple team! If you enjoy our content and custom visuals, consider sharing this newsletter with others or [upgrading](http://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=microgpt-simply-explained) so we can keep doing what we do.

[Andrej Karpathy](https://karpathy.ai/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=microgpt-simply-explained) is a well-renowned computer scientist and leader in today’s Artificial Intelligence public forum. Some of his best work stems from the craft that the community deems "minimalist engineering.”

On February 12th, he posted on his GitHub a blog called [MicroGPT](https://karpathy.github.io/2026/02/12/microgpt/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=microgpt-simply-explained), where he showcases his latest creation: a 200-line project containing the very backbone of frontier model chatbots like ChatGPT.

Today, we’ll visually break down the project, piece by piece, unraveling the engineering marvel. We’ll look at the complete architecture, from the dataset to the last layer output, and understand deeply how each component contributes to the full mechanism.

MicroGPT is not meant to replace the frontier models, by no means. In fact, the model simply outputs predictions for how to complete names, which its original database is full of.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7ec0e698-77da-4c96-a8cd-28ca56dc8aba/MicroGPT__AI__But_Simple_Visuals_March_26__6_.png)

However, it represents something bolder. Even with how sophisticated modern LLMs become, the core of what makes them work can be simple.

In Karpathy’s breakdown, these principles are elegantly on full display.

### The Scope

The premise of Andrej Karpathy’s work is simple: build a viable text autocomplete engine. This reflects early-stage LLMs and chatbots, which were just really good at autocompleting sentences.

In this case, MicroGPT is trained on a dataset of 32,000 names, in contrast to other LLMs that may have been trained on millions of documents and webpages.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/18f89032-1003-4384-af07-02f3fc99af86/MicroGPT__AI__But_Simple_Visuals_March_26__1_.png)

Karpathy’s goal for MicroGPT is for it to recognize patterns from the 32,000 names and eventually use the model to generate his own names, which ideally are unique from the original set.

---

### [Level Up Your Career With AI, But Simple PRO](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=microgpt-simply-explained)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/281697a8-c915-435c-a68b-43374b671698/aibutsimple_academy.png)

Supporting us by purchasing a PRO membership will get you:

- Exclusive Hands-on Tutorials With Code
- Real-World Interview Questions and Cases (From Real Companies)
- Full Math Explanations and Beautiful Visualizations
- Ad-Free Reading for an Uninterrupted Learning Experience

[Break Into AI With PRO](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=microgpt-simply-explained)

---

### MicroGPT Architecture

MicroGPT depends on three main components:

1. A Way to Classify Parameters
2. Robust Backpropagation
3. Self-Attention

The way that parameters are dealt with is through the creation of a Value class. The Value Class is an object class initialized as a way to show components of the neural network behind the LLM architecture. Here’s a quick preview.

```
class Value:
    # Python optimization for memory usage
    __slots__ = ('data', 'grad', '_children', '_local_grads')

    def __init__(self, data, children=(), local_grads=()):
        self.data = data
        self.grad = 0            
        self._children = children
        self._local_grads = local_grads
    ...
    def backward(self): # topological sort to order
        topo = []
        visited = set()
        def build_topo(v):
            if v not in visited:
                visited.add(v)
                for child in v._children:
                    build_topo(child)
                topo.append(v)
        build_topo(self)
    ...
```

Value is built with a key property of backpropagation in mind: the ability to trace partial derivatives backward.

- In a graph-like fashion, each Value object holds a representative scalar, children value objects, and a local gradient with respect to its children.
- Each value object represents a node in a computation graph, a directed graph showing how values are computed from inputs. Computation graphs are used in PyTorch’s [autograd](https://docs.pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=microgpt-simply-explained), among other deep learning frameworks.

The children value objects help that specific node “remember” the nodes that contributed to its own representative scalar.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/371f5154-b806-461a-9222-c43b0b1fabde/MicroGPT__AI__But_Simple_Visuals_March_26__4_.png)

Besides the initialization, the Value class also holds numerous methods for operations that are necessary to carry out the process of predictive generation.

Notably, the backward() method is contained in the Value class. The backward function is what performs the process of accumulating partial derivatives, which are associated with derivatives of the Loss function.

These derivatives are very useful when combined to form a gradient since it gives the direction of steepest ascent, telling the model how to tweak its own parameters.

A simple example of this backpropagation can be shown here, given that the scalar representatives of a and b are 2 and 3, respectively.

```
a = Value(2.0)
b = Value(3.0)
L = a * b + a   # L = 8.0
L.backward()
print(a.grad)   # 4.0 (dL/da = b + 1 = 3 + 1, via both paths)
print(b.grad)   # 2.0 (dL/db = a = 2)
```

A more complicated example can be drawn below, with more math:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6f020a26-fbe0-4a3d-9eac-da5fe63c7570/MicroGPT__AI__But_Simple_Visuals_March_26__7_.png)

This is the same process that PyTorch’s .backward method does. For simplicity, MicroGPT operates on scalars, while PyTorch has been designed to be ultra-efficient and computes on tensors.

- If you’re building any industry-grade application, it’s probably best to stick with PyTorch.

Returning to the fundamentals of transformer models, MicroGPT implements [self-attention](https://aibutsimple.beehiiv.com/p/transformers-and-the-attention-mechanism?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=microgpt-simply-explained) utilizing this Value class.

Weight matrices for key, value, and query matrices are all randomly initialized and used to multiply with embedding vectors, carried out through the training and inference cycles.

After being randomly initialized from the standard distributions, the Value objects are treated as nodes that hold the weights of each neuron for each layer. The code is shown below.

```
def gpt(token_id, pos_id, keys, values):
    ...
    for li in range(n_layer):
        # 1) Multi-head Attention block
        x_residual = x
        x = rmsnorm(x)
        q = linear(x, state_dict[f'layer{li}.attn_wq'])
        k = linear(x, state_dict[f'layer{li}.attn_wk'])
        v = linear(x, state_dict[f'layer{li}.attn_wv'])
        keys[li].append(k)
        values[li].append(v)
        x_attn = []
        for h in range(n_head):
            hs = h * head_dim
            q_h = q[hs:hs+head_dim]
            k_h = [ki[hs:hs+head_dim] for ki in keys[li]]
            v_h = [vi[hs:hs+head_dim] for vi in values[li]]
            attn_logits = [sum(q_h[j] * k_h[t][j] for j in range(head_dim)) / head_dim**0.5 for t in range(len(k_h))]
            attn_weights = softmax(attn_logits)
            head_out = [sum(attn_weights[t] * v_h[t][j] for t in range(len(v_h))) for j in range(head_dim)]
            x_attn.extend(head_out)
        x = linear(x_attn, state_dict[f'layer{li}.attn_wo'])
        x = [a + b for a, b in zip(x, x_residual)]
     ...
```

But before any attention can happen, the model needs a way to represent tokens as numbers it can actually work with.

The way Karpathy collects these tokens is by scraping all the unique tokens from the dataset and using that as his vocabulary, along with an additional delimiter token, which tells the model where outputs begin and end.

In the end, the model itself is made up of 4,192 parameters, as opposed to GPT-2’s 1.5 billion parameters, which, again, are all just weight values inside of the Value Class.

- Parameters are organized into named matrices: embedding tables, attention weights, MLP weights, and a final output projection.

Two of these matrices are special: the token embedding matrix and the position embedding matrix. The token embedding table contains the 16-dimensional vector, and the position embedding table contains the position of that vector in the sequence that is being fed in.

```
tok_emb = state_dict['wte'][token_id] # token emb
pos_emb = state_dict['wpe'][pos_id] # position emb
x = [t + p for t, p in zip(tok_emb, pos_emb)] # joint
```

When a token enters the model, both vectors are looked up and added together, giving the model a single representation that captures both what the token is and where it sits.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b4dd9ab8-c962-4e4b-a1a2-b77ec12debad/MicroGPT__AI__But_Simple_Visuals_March_26__8_.png)

For the full 200 lines of code that is used, it would be best to take a look at the source article of MicroGPT. If you’d also like to see other optimizations that could be made to reflect actual modern LLM output, check out our code tutorial, linked in the [PRO Version](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=microgpt-simply-explained).

### The Forward Pass

The embedding flows through the gpt() function, which processes one token at a time. The sequence is as follows:

First, the combined embedding from both tables is normalized using RMSNorm (Root-mean-squared normalization), which essentially just rescales the vector so it doesn’t grow too large as it goes through the layers.

```
x = rmsnorm(x)
```

Next, the embedding is processed by the attention block. The current token is projected into a query (Q), a key (K), and a value (V) vector using learned weight matrices, which are randomly initialized.

```
q = linear(x, state_dict[f'layer{li}.attn_wq'])
k = linear(x, state_dict[f'layer{li}.attn_wk'])
v = linear(x, state_dict[f'layer{li}.attn_wv'])
keys[li].append(k)
values[li].append(v)
```

The key and value are stored in the [KV cache](https://aibutsimple.beehiiv.com/p/llm-kv-caching-simply-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=microgpt-simply-explained) so that future tokens can reference them. The query is compared against all cached keys via dot products to produce attention scores, which are passed through softmax to become weights.

These weights determine how much the current token should "attend" each past token. The weighted sum of cached values becomes the attention output. This is the only point in the model where tokens communicate with each other.

After attention, the output passes through an MLP block, a two-layer feed-forward network that projects up to 4x the embedding dimension, applies ReLU, and projects back down. Unlike attention, this step doesn’t pass information between tokens, transforming vectors separately.

```
x_residual = x
x = rmsnorm(x)
x = linear(x, state_dict[f'layer{li}.mlp_fc1'])
x = [xi.relu() for xi in x]
x = linear(x, state_dict[f'layer{li}.mlp_fc2'])
x = [a + b for a, b in zip(x, x_residual)]
```

Both the attention block and the MLP block use residual connections, meaning their input is added back to their output. This allows gradients to be stable throughout backpropagation.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/fbc8a09c-f22a-4d28-8a96-b9ddf2484129/MicroGPT__AI__But_Simple_Visuals_March_26__5_.png)

The final hidden state is projected through one last weight matrix to produce 27 numbers: one logit for each possible token in the vocabulary.

```
logits = linear(x, state_dict['lm_head'])
return logits
```

The higher the logit, the more likely the model is to spit it out next. These logits are what get passed to softmax during training (to compute loss) and during generation (to sample the next character).

### Training and Inference

During training, each step picks a name, puts the delimiter token on both sides, and feeds it through gpt(). At each character, the model's logits are compared to what actually comes next using cross-entropy loss.

- This is no different than the standard cycle of training for LLMs!

Over 1,000 steps, Karpathy shows that the loss drops from around 3.3 (random guessing) to ~2.37. This is calculated by calling loss.backward.

```
num_steps = 1000 # number of training steps
for step in range(num_steps):
    ...
    loss = (1 / n) * sum(losses) # average loss
    loss.backward()
    ...
```

Once trained, generation is just our same forward pass, just for the opposite purpose. Start with a delimiter token, take a random character from the model's predicted probabilities, feed it back in, and repeat until the delimiter appears again.

The result is names like "kamon," "karai," and "vialan" that don't exist in the original 32,000 but sound like they could.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/8d66e77f-6ff3-4a15-8a68-e912c32399ba/MicroGPT__AI__But_Simple_Visuals_March_26__9_.png)

### The Bottom Line

Remember, these features all fit into the same 200 line file, which you can run from your very own computer with just a couple commands.

While the notion of an LLM’s “scale” is thrown around, it does depict a very serious limitation. Modern LLMs will only ever be as robust as the training data, an obvious takeaway.

Perhaps less obvious, however, is how simple the core mechanisms are, and if we can find possible alternatives to the underlying mechanism, maybe we can see significant differences in the output.

---

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=microgpt-simply-explained):

- Sushant Waidande
- Sai Krishna Pashapu
- Bob Boncella
- Demet Yazici
- Yassine Ennaji

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Want to reach 50000+ ML engineers? Let’s work together:

[WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=microgpt-simply-explained)

Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

If you like, you can also donate to help our team push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=microgpt-simply-explained)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team