# GPT-OSS, Simply Explained (The Real Architecture of LLMs)

#73 Oct 20, 2025 LLMs

https://www.aibutsimple.com/p/gpt-oss-simply-explained-the-real-architecture-of-llms

In August 2025, [OpenAI](https://openai.com/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpt-oss-simply-explained-the-real-architecture-of-llms) made a surprising announcement: they were going to release open-source and fully open-weight models.

They ended up releasing two new state-of-the-art models under the [gpt-oss](https://openai.com/index/introducing-gpt-oss/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpt-oss-simply-explained-the-real-architecture-of-llms) family, with a larger variant (gpt-oss-120b) and a smaller one (gpt-oss-20b), all available under an Apache 2.0 license.

For a company that had spent years building its reputation on proprietary, API-only models like GPT-4, releasing rivalling open-source models seemed far away.

- This marks OpenAI's first large, fully open-weight release since [GPT-2](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpt-oss-simply-explained-the-real-architecture-of-llms), which may be a signal for what’s to come in terms of collaboration within the AI ecosystem.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/832e21b6-ab14-4157-95b9-3fce83df2a92/Copy_of_The_red_dog_barks.__5_.png)

---

### What 100K+ Engineers Read to Stay Ahead

[![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f2b94c8c-8e22-488b-99eb-49460702595a/The__1_Tech_newsletter_for_AI___ML_Engineers_-_V4.jpg)](https://codenewsletter.ai/forms/14166360-de71-46c4-8722-878d417fab5c)

Your GitHub stars won't save you if you're behind on tech trends.

That's why over 100K engineers read [The Code](https://codenewsletter.ai/forms/14166360-de71-46c4-8722-878d417fab5c) to spot what's coming next.

- Get curated tech news, tools, and insights twice a week
- Learn about emerging trends you can leverage at work in just 10 mins
- Become the engineer who always knows what's next

[Join 100k+ engineers](https://codenewsletter.ai/forms/14166360-de71-46c4-8722-878d417fab5c)

---

### What Makes GPT-OSS Different

The GPT-OSS series consists of two models:

- gpt-oss-120b with 117 billion total parameters and 5.1 billion active parameters
- gpt-oss-20b with 21 billion total parameters and 3.6 billion active parameters

Both models implement a mixture-of-experts (MoE) architecture, which allows them to activate only a fraction of their total parameters during inference.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c8a1497a-3bc7-46f0-9626-d00d2f1158fe/MoE_for_SLM__1_.png)

A benefit of utilizing this architecture is that it reduces computational expenses while maintaining a high level of competency. [You can read about MoE in a previous issue here](https://www.aibutsimple.com/p/mixture-of-experts-moe-models?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpt-oss-simply-explained-the-real-architecture-of-llms).

The gpt-oss-120b model remarkably achieves on-par performance when compared with OpenAI's o4-mini on core reasoning benchmarks while running efficiently on a single 80GB GPU.

Similarly, the gpt-oss-20b model provides about the same performance as the o3-mini on common benchmarks and can run on devices with just 16GB of memory.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/ff5e6df1-b1fd-449c-a7ab-80e4a0031778/GPT_OSS_120_B_vs_20_B.png)

The reason this is simultaneously so impressive and exciting is because developers can run and test these models from their own home GPU (for the larger model) or even a good laptop (for the smaller model). These models promise accessibility like never before.

The [Apache 2.0 license](https://www.apache.org/licenses/LICENSE-2.0?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpt-oss-simply-explained-the-real-architecture-of-llms) is particularly significant. Unlike restrictive licenses that limit commercial use, Apache 2.0 allows organizations to modify and commercialize these models without the parent provider limiting the scope of the application.

This freedom is quite essential for research teams and companies building their own tools, as they can finely adapt the models to their specific needs without legal/financial complications.

### Technical Architecture and Implementation

Gpt-oss uses many of the same architectural elements as many other LLMs: it’s a transformer base, it still uses blocks of multi-headed attention with linear layers, and it still uses token embeddings.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/e9ec5361-752b-47e6-9a48-9e31d8289cca/GPT_OSS_120_B_vs_20_B__1080_x_650_px_.png)

Current available LLMs generally use similar base architectures and then apply small changes to increase performance. Gpt-oss is no exception—it is structurally similar to the other GPT models like GPT-2.

To understand the performance tweaks to gpt-oss, let’s first review GPT-2’s base architecture. GPT-2 uses the decoder portion of the transformer in the [original 2017 paper](https://arxiv.org/abs/1706.03762?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpt-oss-simply-explained-the-real-architecture-of-llms).

- Want to learn more about transformers? [Check out our issue detailing transformers here](https://www.aibutsimple.com/p/transformers-and-multi-head-attention-mathematically-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpt-oss-simply-explained-the-real-architecture-of-llms).

It is made up of a sequence of transformer decoder blocks, with each block containing a multi-head attention segment and a feed-forward network. Dropout and [layer normalization](https://www.aibutsimple.com/p/normalization-in-deep-learning?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpt-oss-simply-explained-the-real-architecture-of-llms) were applied to reduce overfitting. A diagram of GPT-2’s architecture is found below.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/3ec2f053-9f21-4ea2-9c61-70b0e6d47a4a/image.png)

Now that we have the GPT-2 base architecture in mind, let’s dive into gpt-oss’s changes.

The first notable change is the use of MoE blocks instead of fully connected feed-forward blocks, as mentioned earlier.

Each MoE block is made up of a fixed number of experts, with 128 for gpt-oss120b and 32 for gpt-oss-20b, as well as a router (or gating network) that determines which experts to use.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f06c363d-4408-4734-b3be-05aa10490a5d/GPT_OSS_Moe__1_.png)

The top-4 experts are chosen by the route for each token, and the weighted output is computed for only the selected experts. The MoE blocks use the gated SwiGLU activation function, but more on that later.

Another change is the removal of dropout. Dropout is a machine learning technique to prevent overfitting by randomly setting certain nodes of a layer to 0. Recent LLMs have all phased out dropout, and it follows suit that gpt-oss does the same.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f741f140-b02c-4754-ab10-5097d3cc12f0/image.png)

The attention mechanism is also a key component that has been heavily modified. Regular attention (dense attention) is now used together with sliding window attention. The main difference of sliding window attention is that its attention range is a restricted window.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a5cb3c23-1a26-4350-ae5e-5ca712073f1e/image.png)

Sliding window attention figure, pulled directly from the Mistral 7B paper.

Sliding window attention is similar to other sliding window techniques in computer science and can be visualized as a window that slides over all input tokens, computing attention only for the tokens within the window at each stage.

This saves compute and reduces memory usage by not computing all the non-masked tokens at once. For gpt-oss, the window size is 128 tokens, and the model applies it every second layer.

Another change to attention is the use of grouped query attention (GQA). GQA risen in popularity in recent years as a more efficient alternative to multi-head attention (MHA)—it saves parameters and compute.

In MHA, each head has its own set of keys and values provided by unique weight matrices for each head. GQA groups multiple heads together, so each head in the group shares the same keys and values.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f8af645e-8c99-418d-a1f7-dbdc0f936f35/image.png)

The grouping decreases the total number of computations and parameters, leading to lower memory usage and higher efficiency without majorly impacting performance.

In gpt-oss, each transformer block has 64 query heads of dimension 64 and uses GQA with 8 key-value heads.

The attention heads also have a learned bias in the denominator of the softmax, similar to off-by-one attention and attention sinks, which enables the attention mechanism to pay no attention to any tokens.

An interesting change they implemented in GPT-OSS is the use of the [gated linear unit (GLU)](https://arxiv.org/pdf/2002.05202?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpt-oss-simply-explained-the-real-architecture-of-llms) to replace the standard feed-forward block (a small multi-layer perceptron).

The gated linear unit uses 3 fully connected layers instead of 2 and has some gating logic that includes an additional weight matrix.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/acd59429-ca72-411e-8ebc-ed4f5c6fa74c/SwiGLU_Gated_linear_unit_gpt-oss.png)

Using the gated linear unit, specifically the SwiGLU version that contains the SiLU activation function, results in higher performance, as proven in the [PaLM model](https://research.google/blog/pathways-language-model-palm-scaling-to-540-billion-parameters-for-breakthrough-performance/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpt-oss-simply-explained-the-real-architecture-of-llms).

Typically, the standard linear layers will include more neurons to “expand” the embedding dimensions, but the GLU variants do not, resulting in a similar amount of parameters.

Gpt-oss also swaps out the normalization technique by applying [root mean square normalization (RMSnorm)](https://arxiv.org/abs/1910.07467?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpt-oss-simply-explained-the-real-architecture-of-llms) instead of layer normalization on the data before each attention and MoE block.

The final key change is the use of [rotary position embeddings](https://arxiv.org/abs/2104.09864?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpt-oss-simply-explained-the-real-architecture-of-llms)[(RoPE)](https://arxiv.org/abs/2104.09864?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpt-oss-simply-explained-the-real-architecture-of-llms)[](https://arxiv.org/abs/2309.00071?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpt-oss-simply-explained-the-real-architecture-of-llms)for better-performing and more efficient encoding of positional information. The context window of dense layers is extended to 131,072 tokens using [YaRN](https://arxiv.org/abs/2309.00071?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpt-oss-simply-explained-the-real-architecture-of-llms).

### GPT-OSS Training Process

Gpt-oss models were trained using OpenAI's harmony chat format, which is a structured form that is as follows:

1. System messages - Instructions and configuration for the model
2. User messages - Input from users
3. Assistant responses - Model outputs including chain-of-thought reasoning
4. Tool calls - Integration with functions like web browsing and Python execution
5. Reasoning effort levels - Configurable settings (low, medium, high) for how much the model "thinks"

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/10a443b6-a001-4826-9cb3-bcc8f237bc74/Copy_of_The_red_dog_barks.__8_.png)

These models also use MXFP4 quantization, which stores MoE weights in 4-bit precision. This quantization technique is key to the models' efficiency: it reduces memory requirements without significant degradation in performance.

However, implementation comes with trade-offs. Open-source models running on your infrastructure can vary widely in performance. A well-configured GPU setup might match or exceed API speed, while CPU-only deployment could take over ten seconds for the same task.

The gpt-oss models are trained on NVIDIA H100 GPUs using the PyTorch framework with Triton kernels. The training run for gpt-oss-120b required 2.1 million H100-hours to complete, with gpt-oss-20b needing about 10x fewer.

Both models leverage FlashAttention to reduce memory consumption and accelerate training.

### The Target Audience

GPT-OSS models have been designed for agentic workflows with strong instruction following, tool use capabilities like web search and code execution, and Chain of Thought (CoT) reasoning.

These were tested through numerous benchmarks and are contemporary to market needs. OpenAI seems to have positioned these models for three primary audiences/users:

1. Developers and Researchers: Teams building custom AI applications who need full control over their model can directly fine-tune GPT-OSS for domain-specific tasks. This is incredibly utilitarian, as it grants users the ability to change model weights and parameters, unlike API-based use.
2. Enterprises with Data Sovereignty Requirements: Organizations in regulated industries like healthcare, finance, and government can utilize GPT-OSS internally, eliminating security and privacy concerns. This way, confidential data is contained within the computer(s) that the model is stored on and not stored in an external database. Some early adopters include AI Sweden, Orange, and Snowflake.
3. Consumer Applications: The 20-billion parameter model's ability to run locally creates possibilities for offline AI assistants for scenarios where cloud connectivity is unreliable or undesirable. For instance, in places where WiFi is unavailable or spotty, like airplanes and boats.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/edfc9934-a44f-43e5-bcfa-857cad2fdbab/Copy_of_The_red_dog_barks.__6_.png)

### What GPT-OSS Enables Versus Flagship API Models

With GPT-OSS, developers gain access to the model's complete chain-of-thought reasoning process, enabling debugging and verification impossible with black-box APIs.

The models support different “effort” levels, allowing organizations to optimize between speed and thoroughness for different tasks. This way, the models can be fine-tuned to match the sophistication of the tasks they will be used for.

Fine-tuning capabilities present another major advantage. API providers like OpenAI usually offer limited fine-tuning options for their proprietary API models, but GPT-OSS allows customization down to each individual parameter.

Organizations can even use model distillation to produce smaller, more efficient versions that can tackle very particular problems.

From a cost perspective, open-source LLMs are most expensive in the initial stage, compared to API access.

However, this changes if the models are to expect high-volume customer traffic, such as millions of user interactions monthly: the cost factor flips and actually favors open-source implementation.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/9e000a99-3b32-4868-ba75-510fd58b45a6/Copy_of_The_red_dog_barks.__7_.png)

API providers charge per token, which can quickly get expensive on large scales. With self-hosted GPT-OSS, after the initial infrastructure investment, marginal costs drop to essentially the electricity required to power the device the model lives in. It’s like the classic linear systems question.

---

### [Level Up Your Career With AI, But Simple PRO](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpt-oss-simply-explained-the-real-architecture-of-llms)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/281697a8-c915-435c-a68b-43374b671698/aibutsimple_academy.png)

Supporting us by purchasing a PRO membership will get you:

- Exclusive Hands-on Tutorials With Code
- Full Math Explanations and Beautiful Visualizations
- Ad-Free Reading for an Uninterrupted Learning Experience

[Break into AI or sharpen your edge by joining today!](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpt-oss-simply-explained-the-real-architecture-of-llms)

---

### Does GPT-OSS Threaten API Providers?

The release of GPT-OSS raises a question that is quite timely (Google’s monopoly case): if these open models are actively matching the flagship providers’ performance, why would anyone pay for API access?

We’ve discussed cost already, but the reality is more nuanced. While open-source models provide greater control, privacy, and cost advantages, the latest proprietary models still beat them performance-wise (GPT-5, Claude Sonnet 4.5, etc.).

For startups and small projects, it’s clear that API services remain attractive; they offer near-instant access without the hassle of “infra management,” predictable costs for low-volume use, and guaranteed service and support.

Essentially, as long as the parent provider’s servers are up, the startup/organization has nothing to worry about.

In particular, OpenAI's API delivers responses in one to three seconds for most queries, providing smooth user experiences in chat interfaces and AI assistants.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/9d9cecf0-f02e-4f13-8ef3-2b8d4c6cb028/image__1_-modified.png)

OpenAI API featured models

However, the competitive landscape is evolving in front of our very eyes. As open models continue improving, base LLM capabilities will be more accessible than ever. Releasing open models can also be viewed as a strategic move.

When big companies such as Meta and OpenAI release these open-source models, it attracts a “developer mindshare” and accelerates ecosystem development, positioning them at the center of the AI economy even when giving away core technology.

Open models still lag behind frontier closed models in other ways too. GPT-OSS-120B's English-focused training limits multilingual capabilities when compared to alternatives, and its 5.1 billion active parameters may underperform on specialized tasks. Proprietary models like GPT-4.1 and Claude 3.7 Sonnet continue to lead on the most complex reasoning tasks and maintain advantages in multimodal capabilities.

### Safety, Limitations, and the Future

Safety is foundational to OpenAI's approach to open models, which present a different risk profile than proprietary models since determined attackers could fine-tune them to bypass safety guardrails.

To address these concerns, OpenAI ran capability evaluations and fine-tuned gpt-oss-120b against adversarial queries, concluding it did not reach high capability in biological, chemical, or cyber risk categories, a comforting result.

Recently, OpenAI has actually launched a $500,000 Red Teaming Challenge to crowdsource AI safety research, demonstrating a commitment to the developer environment in identifying and addressing vulnerabilities as a team. This approach is a meaningful milestone in open model safety standards.

What all of this means for the future is that AI development and research will likely accelerate. Accessibility to running ultra-powerful models locally is the first step in enabling researchers and the general public to work on AI infrastructure.

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpt-oss-simply-explained-the-real-architecture-of-llms):

- Sushant Waidande
- Sai Krishna Pashapu
- Bob Boncella

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Want to reach 50000+ ML engineers? Let’s work together:

[WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpt-oss-simply-explained-the-real-architecture-of-llms)

Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

If you like, you can also donate to our team to push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpt-oss-simply-explained-the-real-architecture-of-llms)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team