# Context Self-Management, Generation for Understanding, Disaggregated Quantization #122 Oct 5 Inference https://www.aibutsimple.com/p/claude-opus-5-5-agent-swarms-writing-proofs-cheap-sparse-models-1 Welcome back to AI, but simple, your AI research beacon with visual explanations. AI research is revisiting design choices kept constant prior, across context management, computer vision, and quantization. A few developments have emerged: - Meta and MIT are training models to modify their own context - In language models, understanding helps generation, but generation may also aid understanding - Reading and writing tokens may benefit from different quantization formats We'll unpack what’s happening behind these developments (visually) and track key movements to get a sense of where the field is heading. | The week, ranked | | [Gemini 4 is out: The competition has woken up](https://www.reddit.com/r/ClaudeAI/comments/1wufk1u/gemini_4_is_out_the_competition_has_woken_up/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=context-self-management-generation-for-understanding-disaggregated-quantization) | ▲ 1.2K RDDT | | [Anthropic's Claude Sonnet 5.5 Released](https://www.reddit.com/r/singularity/comments/1wslxcu/claude_sonnet_55_released/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=context-self-management-generation-for-understanding-disaggregated-quantization) | ▲ 853 RDDT | | [How to Perform In-Context Learning for Robots](https://huggingface.co/papers/2609.36012?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=context-self-management-generation-for-understanding-disaggregated-quantization) | ▲ 383 HF | | [On-policy Self-Distillation for Multimodal Self-improvement](https://huggingface.co/papers/2609.38721?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=context-self-management-generation-for-understanding-disaggregated-quantization) | ▲ 274 HF | | [An Open-Source Version of OpenAI's Dots](https://github.com/feder-cr/dots?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=context-self-management-generation-for-understanding-disaggregated-quantization) | ▲ 2.6K GH | | [All entries](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=context-self-management-generation-for-understanding-disaggregated-quantization)Ranked by [AIBUTSIMPLE PRO](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=context-self-management-generation-for-understanding-disaggregated-quantization) | | THIS WEEK'S TOPICS | | [![Map of AI research areas with this week's three topics](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/3569aea6-1c49-42a2-96cf-b98683ea17c4/image.png)](https://www.aibutsimple.com/topics?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=context-self-management-generation-for-understanding-disaggregated-quantization) | | 1 | [Context engineering](https://www.aibutsimple.com/topics/context-engineering?family=technique&utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=context-self-management-generation-for-understanding-disaggregated-quantization), Context Language Models | | 2 | [Unified models](https://www.aibutsimple.com/search?q=unified+multimodal+models&utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=context-self-management-generation-for-understanding-disaggregated-quantization), OmniTaskonomy | | 3 | [Model quantization](https://www.aibutsimple.com/topics/model-quantization?family=technique&utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=context-self-management-generation-for-understanding-disaggregated-quantization), Disaggregated Quantization | | MAPPED BY [AIBUTSIMPLE PRO](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=context-self-management-generation-for-understanding-disaggregated-quantization) | --- #### Should language models rewrite their own context? During inference, a language model repeatedly generates output tokens to add to its context. By design, the model can’t remove from the context. To manage the context, we delegate that task to a harness, the code infrastructure around the model that builds prompts and manages tool use. Common harnesses such as Claude Code, Codex, and Cursor replace the history with a summary (known as compaction) once it reaches a set length. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a9ed3770-3232-4e73-864b-d8330185440c/Context_Language_Models__Generation_for_Understanding__Disaggregrated_Quantization__1_.png) Evidently, compaction isn’t the most elegant or robust solution. A fixed rule doesn’t keep one important fact intact or specifically drop one stale search result. So what if the model managed its own context? Context Language Models (CLMs) [[arxiv]](https://arxiv.org/abs/2609.37725?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=context-self-management-generation-for-understanding-disaggregated-quantization) from Meta Superintelligence Labs and MIT propose exactly that. The key idea is that the model is in a better position to judge what it needs than a fixed rule, so the authors let the model decide. A standard language model takes its current context and performs a direct sum, while a context language model produces the next context itself. The difference is shown below. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a2fe6e36-13f3-4ec9-9096-b68f3e2cc1e2/image.png) Here ct is the context at turn t and fθ is the model. Since the model creates the entire next context, it can delete, shorten, or rewrite anything that came before, as well as add to the end. So how did they implement this? - They used a harness to store the model’s context as a file - They then instructed the model to edit that file with bash or python No training was needed, only some instructions (e.g., a system prompt), but they discuss RL tuning and ICL, which we’ll get into below. The model may choose to not modify the file, in which the harness appends the generated output tokens into the context as usual. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f4cb9870-3577-44c1-9355-1e615df43799/Context_Language_Models__Generation_for_Understanding__Disaggregrated_Quantization__2_.png) What about the cost? Well, editing the context mid-task does require additional compute. In regular context, to avoid reprocessing the entire prompt every turn, we use prefix caching (which is just a KV cache). - Each token's keys and values are stored and reused on the next turn. - A token's keys and values depend on earlier tokens, so a cached entry is only valid when all tokens before a given token are unchanged. When the model edits something early in its context, we have to recompute everything after that point. So to answer the cost question, the paper argues that the cost to recompute keys and values for context edits is offset by a smaller, managed context. To decrease the cost further, the paper introduces Suffix Cache Reuse. To understand how it works, let’s go through an example with 3 cache blocks. Let’s say the original context is split into 3 sections: A, B, and C. If we want to replace the middle section B with a new cache state B’, then A will be the prefix cache, and C will be the suffix cache. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/cd667388-46a6-45d6-b75c-8c4ca5dcf024/Context_Language_Models__Generation_for_Understanding__Disaggregrated_Quantization__3_.png) Regular prefix caching reuses the cached KV values for section A, while pre-filling (pre-processing) B and C. SCR chooses to reuse both A and C, only recomputing the new B’, attending to A’s cache states. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/ec4eaf79-b136-4c67-8c0b-55e2401144d7/Context_Language_Models__Generation_for_Understanding__Disaggregrated_Quantization__5_.png) This means instead of computing C’ (which attends to A and B’), C’s old states are kept, still encoding information from B, which is no longer in the context. The paper mentions that this can be beneficial in some cases, since it retains richer information from the past. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7418afd2-1417-4e42-a3c7-c93843d8824b/Context_Language_Models__Generation_for_Understanding__Disaggregrated_Quantization__7_.png) But it’s also helpful to think about when this wouldn’t be desired, like if a model deletes an unwanted instruction, traces of it could still remain in the cache. The authors test CLMs on deep research and on long-horizon repository optimization benchmarks and show decent efficiency savings and performance increases. You can see them for yourself below: ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a8b726f1-f606-4de7-a3bc-0ba9af6f9a0a/image.png) ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/49525937-a759-4bd6-b0a7-d99658552029/image.png) Training a CLM can be used to store context selection strategies in its parameters. The paper RL-tunes a pre-trained LLM using stepwise GRPO [[paper]](https://arxiv.org/abs/2503.12937?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=context-self-management-generation-for-understanding-disaggregated-quantization) and a technique they refer to as a success-gated efficiency advantage. It basically adds an extra advantage term that rewards a trajectory for FLOP savings relative to the average of the correct attempts in its group. They found that their RL-tuned CLM matches the performance of typical harness compaction while roughly halving the compute per question solved. They also mention that you can steer CLMs simply by providing in-context instruction or skill documents. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/14d59382-0d06-42f5-9e92-41e534cdab12/image.png) Above, we can see some examples of steering phrases given to the model and how it changes the context management policy. We can observe similar ideas in other recent papers around context: - Context-Folding [[arxiv]](https://arxiv.org/abs/2510.11967?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=context-self-management-generation-for-understanding-disaggregated-quantization) (ByteDance, CMU, Stanford) gives the model a branch action for subtasks and folds them into a short summary when completed. - MEMENTO [[arxiv]](https://arxiv.org/abs/2604.09852?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=context-self-management-generation-for-understanding-disaggregated-quantization) (Microsoft) trains a model to summarize each reasoning block, caching only the summary. [Read The Context Language Model Paper](https://arxiv.org/pdf/2609.37725?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=context-self-management-generation-for-understanding-disaggregated-quantization) | PRESENTED BY | ![ai, but simple pro](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/59bb099e-a38a-445e-8f0d-b6cd300aa85d/_UPDATED__AI__but_simple_Pro_banner__1_.png) | ### Simplify Your R&D Workflow With A Single Dashboard ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/86d2e5d3-ca74-4e1e-aa58-945c43dd040f/068-ascii-muted-pro-billboard-560x290-7VA4Q50Z.png) With [PRO](https://newsletter.aibutsimple.com/upgrade), unlock access to our AI research layer. It synthesizes emerging techniques and hot topics across thousands of papers into a single screen. - Get benchmarks, relation graphs, authorship, industry adoption, and open questions for every technique. - Perform smart searches across our vast database of techniques, papers, and code. - Get personalized daily overviews and a feed just for you. Pick from 10 preset roles or customize your dashboard. [Get AI Research Intelligence](https://www.newsletter.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=context-self-management-generation-for-understanding-disaggregated-quantization) #### Learning to draw might help a model see Unified (omni) models natively process, reason, and output across multimodal inputs within a shared representational space. Research has used these types of models to both understand (e.g., QA on visual features) and generate images. So far, the field has found: - Understanding helps generation: aligning a diffusion generation model's features with a frozen representation alignment vision encoder makes it train much faster and increases generation quality. - Whether generation helps understanding is still unclear (driven by papers like MetaMorph [[arxiv]](https://arxiv.org/abs/2412.14164?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=context-self-management-generation-for-understanding-disaggregated-quantization)) ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/3b04fdb2-6e62-45f1-b80e-4b42320e0684/Context_Language_Models__Generation_for_Understanding__Disaggregrated_Quantization__8_.png) OmniTaskonomy [[arxiv]](https://arxiv.org/abs/2609.38079?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=context-self-management-generation-for-understanding-disaggregated-quantization) (UC Berkeley, CMU, Elorian) tests generation for understanding more carefully on a 7B model. The authors fine-tune BAGEL-7B-MoT, a mixture-of-transformers model (separate weights per modality) with two expert networks: one for understanding and one for generation. Both experts have their own weights (projections, norm layers, and MLPs), but they share an attention mechanism (joint attention) in every layer. The global attention allows experts to attend over the same token sequence and is how a generation loss can reach the understanding side. When the model trains to generate an image, the gradient flows back through attention into the understanding expert, its connector, and the vision encoder. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c892a785-f716-41f6-9f7e-20ba611dc45a/Context_Language_Models__Generation_for_Understanding__Disaggregrated_Quantization__10_.png) To determine whether generation can help speed up understanding training (or increase overall understanding), they initially tested it on a jigsaw puzzle problem. In Jigsaw, the model sees a 2×2 image with its patches shuffled. The generation expert generates a correctly arranged puzzle image, while the understanding expert outputs the correct piece order. They found that understanding only improves when the model first trains on generation to update the shared weights, then trains on understanding. Training on both from the start (mixed), or freezing the shared weights during the generation stage, shows reduced performance. The gain also grows with the amount of generation data. The authors then scale this up to 19 generation tasks, such as depth maps, segmentation, and more, against 25 understanding capabilities from 7 VQA benchmarks. This is their “omni-taxonomy.” ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/71e987e2-df3c-442c-b605-d30566c398af/image.png) The authors also measure gradient alignment, meaning how closely the generation and understanding losses push the shared weights in the same direction, with stronger alignment associated with larger transfer gains. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/505efa74-b083-4ec4-975a-8cbd4d07e95e/image.png) Other recent papers with different angles: - Vision Banana [[arxiv]](https://arxiv.org/abs/2604.20329?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=context-self-management-generation-for-understanding-disaggregated-quantization) (Google) instruction-tunes Nano Banana Pro (image generation model) to output depth maps and segmentations, beating specialist models like SAM3. [Read The OmniTaskonomy Paper](https://arxiv.org/abs/2609.38079?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=context-self-management-generation-for-understanding-disaggregated-quantization) #### Using different quantizations for prefill and decode Serving a language model occurs in two phases. - The first is prefill, where a model preprocesses the whole prompt at once and builds the KV cache. - The second is decode, where the model generates the output tokens while reading from the cache. Both phases are limited by two resource bottlenecks of the GPU. To illustrate, let’s say you have a linear layer with an HxH weight matrix applied to T tokens. The paper discusses decode, with batch size 1 (so serving one request autoregressively), which gives T = 1, so each weight is loaded from memory for a multiply-add operation. Here, memory bandwidth is the bottleneck, since it’s how quickly numbers can be moved from HBM memory to compute units, with an H100 doing around 3.35TB/s. In prefill, T can be thousands of tokens from the input, so each weight is reused many times, and compute speed (or arithmetic throughput) is the bottleneck. This speed is how fast the compute units (tensor cores) perform floating-point operations, and an H100 does 990 TFLOPS in BF16. If we divide the arithmetic throughput by the bandwidth for our linear example, we obtain the arithmetic intensity, which is the ratio of FLOPs computed per byte loaded. The intensity for the example in BF16 (2-byte weights) is approximately T FLOPs per byte (T << H), which is simply the number of tokens. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6d7c197f-2346-4bd8-b226-5c2402456a2b/image.png) Compared to the H100 GPU’s hardware ratio (FLOPs/bytes) of 300, this means that for T well below 300 is memory-bound. The compute units finish their work instantly and are idle waiting for the next weights. T well above 300 is compute-bound. The weights arrive faster than the compute units use them. We can observe how the two phases benefit from different kinds of quantization. - Decode gets faster with lower-bit quantization in any storage format; we will expand them as they load and keep activations in BF16. - Prefill only gets quicker with cheaper arithmetic, such as NVFP4 on Blackwell GPUs (4-bit values with one scale per block of 16). ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/5b9d6065-d583-4d2f-8a6a-57e40cb3f02a/Context_Language_Models__Generation_for_Understanding__Disaggregrated_Quantization.png) The issue is that current models use one quantization format for both phases. Disaggregated Quantization [[arxiv]](https://arxiv.org/abs/2609.26333?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=context-self-management-generation-for-understanding-disaggregated-quantization) (NVIDIA) specializes arithmetic formats and weights for both prefill and decode in three steps: 1. Format disaggregation. Weights used for prefill are NVFP4 with 4-bit activations, and quantization for decode is weight-only with BF16 activations. 2. Full disaggregation. Prefill uses NVFP4 weights, and decode gets separate 2/3-bit (LUT2/LUT3) weights. Both weights meet in the KV cache, which maintains the original shapes (no change to attention). 3. Offloaded prefill (ODP). Prefill weights are only needed when reading the prompt, so ODP streams them from SSD block by block, reusing idle decode weights. Due to its fixed cost, it benefits only long prompts. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/17ac1ae2-08fb-44e0-80b6-32226a5bfeea/Context_Language_Models__Generation_for_Understanding__Disaggregrated_Quantization__9_.png) To train both weight sets together, the authors introduce QADD, which reuses the fine-tuning mask that differentiates prompt and answer tokens. Typically, that mask only tells the loss to score the answer, while QADD also uses it to route each token. Prompt tokens run through the prefill weights, and answer tokens run through the decode weights. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a65f194c-148a-408d-8a03-3843cf169145/image.png) As the distillation loss only scores answer tokens, the prefill weights learn through the keys and values from the answer. In effect, the prefill network learns to write a KV cache that the decoder can read. A notable point is that the training works with frozen decoders, so the authors train NVFP4 "prefillers" for existing quantizations of models. In terms of results, using the prefillers shows some accuracy improvements, which you can see below. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2f18525c-c482-40a3-89ca-ed03b1c2706c/image.png) There exist similar papers to this idea, and another paper earlier this year demonstrates it: - Mix-Quant [[arxiv]](https://arxiv.org/abs/2605.20315?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=context-self-management-generation-for-understanding-disaggregated-quantization) runs prefill with low-precision compute and keeps decode in BF16. [Read The Disaggregated Quantization Paper](https://arxiv.org/abs/2609.26333?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=context-self-management-generation-for-understanding-disaggregated-quantization) #### Wrapping up Meta Superintelligence is letting models manage their own context. Training on generation for understanding through joint attention updates provides a useful initialization for understanding training. Learning to draw images might help learning to understand them. NVIDIA shows how using different quantization formats for prefill and decode can provide increased efficiency and performance gains. That's it for this week's free version of AI, but simple. Get deeper analysis, ad-free reading, and full access to the full archive with PRO. See you next week! --- Here's a special thanks to our biggest supporters: Sushant Waidande, Sai Krishna Pashapu, Bob Boncella, Demet Yazici, Imran Khan Want to reach 10000+ AI engineers, researchers, and technical founders? Advertising Inquiries: [WORK WITH US 📈](https://tally.so/r/A7Dgdk?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=context-self-management-generation-for-understanding-disaggregated-quantization) Feedback or inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com)