# Small Language Models (The Future?)

#68 Sep 15, 2025 LLMs

https://www.aibutsimple.com/p/small-language-models-the-future

In recent times, the popularity of transformer-based LLMs and LLM applications such as AI agents has skyrocketed. Compute is in high demand, while models soar in parameter count—reaching hundreds of billions and trillions of parameters in the largest LLMs.

If you wanted to run your own language model, this would be impossible on consumer-grade hardware. Even small-to-medium sized companies don’t have the budget to train a model of this size.

Luckily, researchers have been moving towards quantization and other techniques to reduce the compute and VRAM needed to store, train, and run models. One of the recent methods in research is to reduce the physical size (in parameters) of the models themselves.

This is where small language models (SLMs) come in. Small language models are neural language models that are much smaller in size (typically billions of parameters or fewer) than today’s massive LLMs (which often have hundreds of billions).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b4ccb94f-7e5a-42ed-a851-15a85f53acae/SLMs__1080_x_800_px_.png)

By design, SLMs can run on consumer-grade devices like smartphones, embedded systems, or PCs, offering fast inference and a much lower cost.

Researchers often consider models under about 10 billion parameters to be SLMs, since such models can fit on common hardware with low latency.

SLMs and LLMs share the same basic architecture (typically Transformers) but differ in scale and usage. LLMs excel at general tasks because their huge size lets them memorize broad knowledge—this is exactly why they need tens or hundreds of billions of parameters.

---

### [Level Up Your Career With AI, But Simple PRO](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/281697a8-c915-435c-a68b-43374b671698/aibutsimple_academy.png)

Supporting us by purchasing a PRO membership will get you:

- Exclusive Hands-on Tutorials With Code
- Full Math Explanations and Beautiful Visualizations
- Ad-Free Reading for an Uninterrupted Learning Experience

[Break into AI or sharpen your edge by joining today!](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future)

---

### SLMs in Recent Research

In June of 2025, Nvidia research released a paper detailing the potential of SLMs, titled “[Small Language Models are the Future of Agentic AI](https://arxiv.org/pdf/2506.02153?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future).”

One of the key takeaways from the paper is that since agents are typically tailored towards solving very specific tasks, a full hundred billion parameter LLM is not required to be proficient at the task.

They show that SLMs are in fact enough for specific agentic applications with examples in specific industries.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/52b73692-e3dc-4ed7-b02f-5433c1ef272e/SLMs_enough_for_small_applications__1080_x_550_px_.png)

For instance, take the [Microsoft Phi series](https://azure.microsoft.com/en-us/products/phi?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future). Phi-2, with 2.7 billion parameters, achieves commonsense reasoning scores and code generation scores similar to 30 billion parameter models while running over 10 times faster.

Phi-3 small (7 billion parameters) achieves language understanding, reasoning, and code generation scores on par with 70 billion parameter models at the time of release.

The NVIDIA [Nemotron-H family](https://research.nvidia.com/labs/adlr/nemotronh/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future) (2, 4.8, 9 billion parameters) is also a good example. These models are actually hybrid transformer-[mamba](https://www.aibutsimple.com/p/mamba-and-state-space-models-simply-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future) models, and they achieve instruction following and code-generation accuracy similar to 30 billion parameter LLMs at a fraction of compute FLOPs.

NVIDIA believes that SLMs will become the standard of agentic AI. They make a statement with three main components:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/ebf889c2-f904-44cd-829a-dcb17ed97bab/image.png)

V1 describes what we’ve discussed earlier, how SLMs are enough for domain-specific tasks—which are typically what agents are tasked to handle.

V2 describes how SLMs are more suitable for agentic systems since they are more aligned with the specific task and are smaller, so they don’t use as many computational resources.

As we save on computational resources, the SLM is more economical, as described in V3.

Another note on SLMs vs. LLMs is accessibility. The scale means LLMs require massive compute and are typically offered only via cloud APIs. SLMs are much more accessible and can even be hosted locally.

As one survey notes, “[LLMs […] face limitations due to large parameter sizes and computational demands, often requiring cloud API use which raises privacy concerns, limits real-time applications, and increases costs](https://arxiv.org/html/2411.03350v1?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future).”

### How Did SLMs Get So Small?

SLMs use many state-of-the-art optimization techniques and fine-tuning to decrease the model size and improve efficiency. Some of these techniques allow SLMs to be decently powerful and useful at small sizes.

One of the most important techniques to reduce model size is quantization. Quantization lowers the precision of parameters (e.g., from [FP32](https://en.wikipedia.org/wiki/Single-precision_floating-point_format?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future) to [INT4](https://developer.nvidia.com/blog/int4-for-ai-inference/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future)) to reduce memory usage and to speed up inference. Quantization trades some model accuracy for model size reduction.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a226144c-20c3-4438-b59c-f66b19edae2f/Quantization_with_Tensors__2_.png)

For instance, optimized INT4 inference can roughly double the rate of output over time compared to INT8 on a modern GPU. A quantized version of Llama-7B from FP16 to INT4 can see a 3x reduction from 138GB to 40.7GB.

Many tools like [BitsAndBytes](https://pypi.org/project/bitsandbytes/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future) and [GPTQ](https://huggingface.co/docs/transformers/en/quantization/gptq?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future) support 4-bit or 8-bit weight quantization. Quantization is generally feasible: INT4 quantization has been shown to match or exceed INT8 performance in some cases.

- Want to learn more about quantization? We dive deep on number precision and format in another issue, found [here](https://www.aibutsimple.com/p/quantization-for-llms-simply-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future).

Another technique making headlines recently is Low-Rank Adaptation (LoRA). This technique is used for fine-tuning language models and diffusion models by adding trainable low-rank matrices to each layer’s weights. LoRA can be applied to SLMs to efficiently tune them.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/4a583b52-2dd5-43e4-96bf-a581af81b05c/LoRA__1_.png)

Only small low-rank matrices A and B are learned during fine-tuning, keeping most original pre-trained weights frozen. Transfer learning can be achieved while using a fraction of the trainable parameters.

Another technique used to decrease the size of the language model is pruning. Pruning (removing) redundant weights or less useful attention heads can shrink model size.

Magnitude pruning and structured pruning are examples of pruning techniques that reduce computation at the cost of some loss in accuracy.

Self-attention is also a target for optimization. Researchers use techniques to make self-attention more efficient. For instance, the popular library [FlashAttention](https://github.com/Dao-AILab/flash-attention?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future) is a fused GPU kernel that computes attention in a memory-efficient and parallel way.

It speeds up attention computations and reduces the memory overhead. SLMs like TinyLLaMA and Falcon-RW use FlashAttention to boost the output rate for faster inference. We’ll go over more common SLMs later in the issue.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2f28941b-6fcb-490c-90d9-1fed8b6bb50c/Copy_of_LoRA__1100_x_500_px_.png)

Another technique used to reduce the size of models (effective size) is to use a Mixture-of-Experts (MoE) architecture. MoE is a sparse technique, meaning that the full model is not used at any point in inference.

MoE models do this by taking each input and then choosing an “expert” subnetwork to route it through and produce an output. This way, only that one subnetwork is activated. SLMs can use MoE to increase effective model capacity without scaling compute too much.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7914b8cc-7514-49ba-9edf-1e10bf8e99c4/MoE_for_SLM.png)

For example, Google’s [Switch Transformers](https://arxiv.org/abs/2101.03961?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future) and [DeepSeek r1](https://huggingface.co/deepseek-ai/DeepSeek-R1?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future) use MoE layers, where only a few experts are active per token.

Other, more niche techniques are also used to reduce the size of models. Examples include [Rotary Positional Embeddings](https://arxiv.org/abs/2104.09864?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future) or ROPE; [Grouped-Query Attention](https://arxiv.org/pdf/2305.13245?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future) or GQA; and sharding to save memory.

- Models like Phi-2 and TinyLLaMA use RoPE as their positional scheme, while other models use GQA to reduce [KV cache](https://www.aibutsimple.com/p/llm-kv-caching-simply-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future) memory by having multiple query heads share each KV head.

### Popular SLM Architectures

SLMs are gaining significant traction, with a growing number of specialized architectures used to address specific tasks and applications.

[DistilBERT (encoder-only)](https://huggingface.co/docs/transformers/en/model_doc/distilbert?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future) is a compact [BERT](https://arxiv.org/abs/1810.04805?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future) variant and uses 6 transformer layers instead of BERT-base’s 12 layers.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c4a668f4-0e97-44a3-a4bf-533678ba093e/image.png)

DistilBERT side-by-side with regular BERT.

It removes token-type embeddings and pooling and was [distilled from BERT to result in only ~66M parameters](https://arxiv.org/pdf/1910.01108?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future). This is about 40% smaller than BERT-base, while retaining ~97% of BERT’s benchmark performance (GLUE).

[TinyLLaMA](https://arxiv.org/abs/2401.02385?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future) is an open-source 1.1B-parameter autoregressive model built with the LLaMA-2 architecture and tokenizer. It has 22 decoder layers, 16 attention heads, and a relatively small context window or length of 2048 tokens.

It uses RoPE, GQA, and FlashAttention, while being trained on a mixture of code and text—outperforming similarly sized open models despite its small size.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/3f1b379a-d673-481d-bc30-5d7e1b90510c/image.png)

Training Process For TinyLlama.

[Phi-2](https://huggingface.co/microsoft/phi-2?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future) is a 2.7B-parameter model from [Microsoft Research](https://www.microsoft.com/en-us/research/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future). It uses 32 transformer decoder layers and 32 heads of attention. It has the same context window as TinyLLaMA, sitting at 2048 tokens, which is relatively small compared to recent models.

Phi-2 is the smallest model in the Phi family, with the most recent Phi-4 having 14B parameters. Phi-2 uses GELU activation and RoPE with no scaling. Phi-2 impressively achieves strong reasoning performance, matching or exceeding larger open models on many benchmarks.

Moving on to larger SLMs, [Mistral-7B](https://mistral.ai/news/announcing-mistral-7b?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future), by Mistral AI, is a 7B-parameter model that uses 32 transformer layers and 32 attention heads. Mistral-7B achieves strong performance for its scale by using previously mentioned techniques such as GQA but also a special optimization for self-attention.

Mistral-7B is specifically trained with an attention technique called sliding-window attention to support an 8192-token context window. This technique is similar to the well-known [sliding window optimization](https://www.geeksforgeeks.org/dsa/window-sliding-technique/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future) in many computer science contexts.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/5700ca51-4d17-4da3-9ea3-f1a8e23d34d9/image.png)

A Visual Demonstration of Sliding Window Attention.

By processing attention in chunks and caching previous calculations, it handles longer input sequences than most other SLMs.

[Falcon-RW-7B](https://huggingface.co/tiiuae/falcon-7b?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future) is a lesser-known 7B-parameter decoder model from a research team in Abu Dhabi, using 36 transformer layers with 64 attention heads. It uses [ALiBi](https://arxiv.org/abs/2108.12409?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future) positional biases and FlashAttention to accelerate computation.

### Moving Towards SLMs—SLMs vs. LLMs

Training an LLM is an extremely resource-intensive process, requiring thousands (and even tens of thousands in some cases) of high-end GPUs for months at a time.

It is also very costly—the estimated cost for training a large model like GPT-5 is in the hundreds of millions.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/91cb463e-3c44-410a-9105-578518850a92/LLMs_to_SLMs.png)

SLMs are significantly cheaper and faster to train. They can be trained on small cloud clusters or even on consumer hardware, making them accessible to a wider range of businesses and people. SLMs are useful in applications where speed, efficiency, or data privacy is required.

SLMs are small enough to run locally, making them inherently more efficient and customizable. They are also powerful enough (as NVIDIA has deemed) for domain-specific knowledge and many agentic applications.

In one line, LLMs are for broad topics and general-purpose intelligence, while SLMs are for efficiency and specialization.

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future):

- Sushant Waidande
- Sai Krishna Pashapu

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Want to reach 50000+ ML engineers? Let’s work together:

[WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future)

Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

If you like, you can also donate to our team to push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=small-language-models-the-future)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team