# Reasoning LLMs, Simply Explained #55 Jun 16, 2025 LLMs https://www.aibutsimple.com/p/reasoning-llms-simply-explained Reasoning large language models (LLMs) are transformer-based LLMs that appear to “think” by breaking a more complex question down into smaller steps, resulting in intermediate steps before an output. They are a very hot topic in research recently, and you can find them just about anywhere—DeepSeek-R1, OpenAI o1, and Google Gemini 2.0 Flash Thinking are examples of LLMs with “reasoning“ capabilities. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6ad343b1-7ad7-4b7b-ac6b-bae2b8d0d859/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_CV_illust__1_.png) But what do we mean by reasoning? We define “reasoning” as the process of answering questions that require complex, multi-step generation with intermediate steps. These steps are often called reasoning steps or thought processes. For instance, factual question answering, such as “Which ocean is the largest in the world?” does not involve reasoning. However, a question like “If a car speeds by at 40 mph and travels 160 miles, how much time did it take?” requires some simple reasoning. The model would need to recognize the relationship between speed, distance, and time before arriving at the answer—it can do this much better with reasoning steps. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/e5104558-6b94-414e-9937-9755a9943e53/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_CV_illust__1_.png) But when should we use a reasoning model? We use them for complex tasks such as solving puzzles, solving advanced math problems, and working out challenging coding problems. - Reasoning models are typically not necessary for simpler tasks such as summarization, translation, or knowledge-based tasks. In fact, using reasoning models for everything can be inefficient and expensive. Reasoning models are indicative of a fundamental change in approach to training LLMs. Instead of scaling train-time compute, reasoning models allocate more resources to scaling test-time compute. So what does “test-time compute” or “train-time compute” actually mean? --- ### Streamline IT management with 'The World at Work 2024: Deel IT Discover how you can transform your IT operations, enhancing operational efficiency & compliance across borders. [Our guide](https://www.deel.com/resources/deel-guide-to-the-world-of-work-in-2024-with-deel-it/?utm_medium=sponsored-newsletter&utm_source=beehiiv&utm_term=XIAEVJHTGJ&utm_campaign=ww_engage_download_beehiiv_sponnewsletter_it-theworldatwork-feb25_it_all&utm_content=engage_it_sponnewsletter_theworldatwork-sponnews180-it_en&_bhiiv=opp_bdad29d7-2824-4bb8-b9c6-a9d52665c497_c39077bb) has essential strategies for managing a global workforce. [Download free guide](https://www.deel.com/resources/deel-guide-to-the-world-of-work-in-2024-with-deel-it/?utm_medium=sponsored-newsletter&utm_source=beehiiv&utm_term=XIAEVJHTGJ&utm_campaign=ww_engage_download_beehiiv_sponnewsletter_it-theworldatwork-feb25_it_all&utm_content=engage_it_sponnewsletter_theworldatwork-sponnews180-it_en&_bhiiv=opp_bdad29d7-2824-4bb8-b9c6-a9d52665c497_c39077bb) --- ### Train-Time Compute Up until late 2024, researchers relied on three main ways to scale the performance of LLMs during pre-training. They increased the number of: - Parameters (model size) - Tokens (dataset size) - Floating point operations per second, or FLOPs (measure of compute) These three factors affect what we call “train-time compute,” which is the total amount of computational power and resources consumed during training. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/cd297645-0da1-4764-b19c-16f05e39a723/Copy_of_Copy_of_Copy_of_saduiagduih908hdiso__1_.png) Train-time compute may include both the compute needed during pre-training as well as during fine-tuning. In general, the larger your train-time compute budget, the better the performance of the model. #### Neural Scaling Laws Neural scaling laws describe how a model’s performance scales with compute, dataset size, and model size. They are power laws, meaning an increase in one variable (like dataset size) results in a proportional change in another variable (performance). - If you want to learn more about neural scaling laws, we did a deep dive in a previous issue—found [here](https://www.aibutsimple.com/p/neural-scaling-laws-simply-explained-16c1?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reasoning-llms-simply-explained). When plotted on logarithmic scales for both axes (what we call a log-log plot), the graph of a power law becomes a straight line. This is the most common way to visualize power laws. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/26d6814d-421b-490e-a4d1-8289aca3cd5a/image.png) Model loss (y) is plotted against resources (x) [OpenAI’s scaling laws](https://arxiv.org/pdf/2001.08361?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reasoning-llms-simply-explained) and the [Chinchilla scaling laws](https://arxiv.org/pdf/2203.15556?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reasoning-llms-simply-explained) are the most well-known in the space. Essentially, the laws simply reiterate the fact that the performance of a model will increase with more compute, tokens, and parameters. All three factors must be scaled properly for optimal performance. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/57035814-6edf-43e5-88f5-78a9188485b8/image.png) Chinchilla scaling laws As research progressed, performance gains started displaying diminishing returns even though compute, dataset size, and model parameters have steadily grown. This is inevitable with power laws—there are diminishing returns as you scale up. Researchers started to look for other ways to improve model performance as it was too expensive to increase train-time compute. This sparked interest in alternative methods, with the most notable being test-time compute. --- ### Used by Execs at Google and OpenAI [![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6c0da898-5b8a-420c-af25-2beb71256358/4.png)](https://www.theaireport.ai/welcome-page) Join 400,000+ professionals who rely on [The AI Report](https://www.theaireport.ai/welcome-page) to work smarter with AI. Delivered daily, it breaks down tools, prompts, and real use cases—so you can implement AI without wasting time. If they’re reading it, why aren’t you? [Join 400K+ pros](https://www.theaireport.ai/welcome-page) --- ### Test-Time Compute Test-time compute is an alternative way of increasing model performance without increasing pre-training resources. Test-time compute allows models to reason longer during inference. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/d9c20404-9ca3-4faf-80e5-12856c5ed3e8/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_CV_illust__2_.png) With non-reasoning models, they would normally only output the answer and skip any reasoning steps. Reasoning models, however, would instead use more tokens to generate their answer through a reasoning process. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/801caa7d-db25-480e-b0ff-ead10c42ae36/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_CV_illust__720_x_550_px_.png) With reasoning, the LLM has to spend more resources, like compute, to create and verify a better answer. Here, the model generates intermediate tokens beforehand that contain additional information and relationships to produce a more accurate output. This is an example of inference-time scaling—we increased the computational resources used (in tokens) during inference to improve the model’s reasoning capabilities, improving output quality. Instead of focusing purely on train-time compute (pre-training and fine-tuning), reasoning models allocate more resources to inference, creating a more balanced model. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/25ca3eff-a4b1-4682-88fb-c4d00b020b8f/Inference__1_.png) #### Test-Time Compute Scaling Laws Compared to train-time compute, scaling laws for test-time compute are relatively new and not yet developed. However, there are some discoveries on scaling that may be promising (on top of the LLM benchmarks that consistently crown reasoning models). OpenAI released a [blog post](https://openai.com/index/learning-to-reason-with-llms/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reasoning-llms-simply-explained) hypothesizing that scaling test-time compute might also follow the same trend as scaling train-time compute. They produced this graph that compared train-time compute and test-time compute: ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/79ff627e-6414-4c4c-b671-a46b488db0ac/image.png) If a regression line is drawn, test-time compute seems to be steeper—it might scale faster than train-time compute With test-time compute scaling like train-time compute, researchers are increasingly shifting towards implementing better reasoning in the newest LLMs using more test-time compute. ### Types of Test-Time Compute Test-time compute can be scaled through multiple methods. Different types of methods have different ways of obtaining a better answer, whether through Chain-of-Thought (CoT), verifiers, majority vote, or many others. The methods can be very roughly separated into two main categories: verifiers and modifying proposal distribution. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/d1fb0511-3d8b-45a2-8514-f8b8b7085d43/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_CV_illust__720_x_550_px___770_x_600_px_.png) RM—Reward Model The verifier method checks the validity of generated answers (through sampling) and picks the best one. Modifying the proposal distribution trains the model to have better reasoning steps, resulting in a better answer. Verifier methods focus on the output, while the modifying proposal distribution method focuses on the input. #### Verifiers The first major way of scaling test-time compute is to use a verifier. Most verifier algorithms will have 2 main steps. First, many different samples of reasoning steps and outputs are generated. Second, a verifier/reward model gives a numerical score to each output, and the highest-scoring output is chosen. The verifier is usually another fine-tuned LLM that judges the output or the reasoning using some reward model. These reward models assign weights, giving the best outputs or thought processes higher values and worse ones lower values. A major advantage of using verifiers is that you don’t need to retrain or fine-tune the main LLM that you initially trained to answer questions. #### Majority Voting We can also scale test-time compute without a reward model. Instead of using a reward model or verifier, we might want to go with the most simple method—to perform a majority vote. We let the model output many different answers, and the answer that is seen most often (majority) will be the final answer. Majority voting can be considered in the first category of scaling methods as it picks the best option from sampled outputs. #### Modifying Proposal Distribution The second method of scaling test-time compute is called modifying the proposal distribution. Instead of sampling the best answers or reasoning steps using verifiers (which focuses on the output), the model is trained to generate better reasoning steps, focusing on the input. This method gets its name from the probability distribution of standard transformers. After passing through a softmax function, data is transformed into a probability distribution, each with weights for different tokens. We then sample from this distribution to produce an output. A common strategy might be to choose the highest-scoring token: ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a8e538a7-9c6f-4251-8be9-d8f1930e3408/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_saduiagduih908hdiso__1_.png) In the image above, one of the tokens is colored red—this is a token that is likely to lead to a reasoning process. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/707ae7a8-f98f-4b27-bb7b-d872fd123776/Copy_of_Copy_of_Copy_of_Copy_of_saduiagduih908hdiso__1_.png) Taking the highest-scoring token is not a bad way to generate an output, but choosing a token that results in a full reasoning process typically leads to an improved answer. Thus, when we modify the proposal distribution (the token probability distribution), at a high level, we are basically making it so that the model re-ranks the distribution such that tokens that are likely to lead to reasoning are chosen more often. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/0bc6a6b6-94eb-457b-8cb7-c35ea27b1f23/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_saduiagduih908hdiso__2_.png) ### DeepSeek R1 A major breakthrough in reasoning model development was the release of DeepSeek-R1. It took the LLM community by storm as its benchmark surpassed state-of-the-art models with its reasoning abilities. In their paper, DeepSeek mentioned that R1’s reasoning system was inspired by an experimental model called DeepSeek-R1 Zero. To develop R1 Zero, starting with DeepSeek-V3-Base, they used reinforcement learning (RL) to allow the model to reason. The reinforcement learning algorithm used for this process is called [Group Relative Policy Optimization (GRPO](https://arxiv.org/pdf/2402.03300?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reasoning-llms-simply-explained))—a form of RL introduced by DeepSeek in 2024. DeepSeek used a very clever method that paired really well with the RL. They start with a system prompt that details how a user should ask a question and how the model should solve it using reasoning. The prompt specifically mentions that the reasoning process should go between tags and answers should go between tags. They do not specify what reasoning should look like to the model; it’s all learned. During reinforcement learning, all choices that led to a correct answer were made more likely, and choices that led to the incorrect answer were made less likely. Additionally, two specific rule-based rewards were created. The first reward was the accuracy reward, which rewards the answer by testing it out. The second was format rewards, which rewards the model if the and tags were used properly. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/1774eb84-a259-4662-961b-1ff50f397e5b/Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_Copy_of_CV_illust__720_x_550_px___720_x_600_px___1_.png) As mentioned previously, no examples were given of what the thinking process should look like. However, by providing indirect rewards (using increases the reward) related to Chain-of-Thought behavior, the model learned by itself that the longer and more complex the reasoning process, the more likely the answer was correct. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/d968def1-561b-40a0-bae0-f32c2278f125/image.png) The image above shows how R1-Zero slowly learned to increase the length of its responses to produce a more accurate output. The model shifted from train-time compute to test-time compute. As the model generated longer sequences, it relied heavily on test-time compute. Using this training pipeline, the model learned its own optimal Chain-of-Thought-like behavior, incorporating advanced reasoning such as self-reflection and self-verification. However, R1-Zero still had a significant flaw—its outputs were hard to read and tended to mix languages. So, the researchers at DeepSeek explored an alternative, the now well-known DeepSeek R1. --- Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reasoning-llms-simply-explained): Sushant Waidande --- If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend! Want to reach 50000+ ML engineers? Let’s work together: [WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reasoning-llms-simply-explained) Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com). If you like, you can also donate to our team to push out better newsletters every week! [Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=reasoning-llms-simply-explained) --- That’s it for this week’s issue of AI, but simple. See you next week! —AI, but simple team