# RL Training Inside Harnesses, Stolen Reasoning Traces, Judging Agents

#118 Sep 7 Agents

https://www.aibutsimple.com/p/rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents

This week, AI research has landed on training agents to maximize harness scores, research on stealing reasoning traces from models, and a focus on benchmarking the reliability of LLM-as-a-judge.

We’ll get into the developments of these fields, get into how their underlying mechanisms work, cover some benchmark results, and key research movements in the topics to keep you updated in no time.

THE WEEK, RANKED

| [Compile by Training: Turning Natural-Language Specifications into Local Neural Functions](https://huggingface.co/papers/2609.04199?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents) | ▲ 315HF |
| [Fable 5.1 vs GPT 6 Astra, 3D Blender, mind blowing difference!](https://www.reddit.com/r/OpenAI/comments/1w7ppcj/fable_51_vs_gpt_6_astra_3d_blender_mind_blowing/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents) | ▲ 2.2KRD |
| [Building Strong Image Generators with Fully Open Training (LLADA-Image)](https://huggingface.co/papers/2609.03796?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents) | ▲ 226HF |
| [DeepSeek’s new experimental multimodal model with 305B parameters](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents) | ▲ 638HF |
| [Microsoft Research’s new framework for training personalized AI “student simulators"](https://huggingface.co/papers/2609.01591?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents) | ▲ 480HF |

[ALL ENTRIES](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents)RANKED BY [AIBUTSIMPLE PRO](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents)

QUICK LINKS

Papers

[LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering](https://huggingface.co/papers/2608.28281?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents)[6 min]

Treats the agent development loop itself as the unit of evaluation, measuring whether a model can monitor progress, assign work, and decide the next action.

[Sliding-window beats linear attention](https://huggingface.co/papers/2608.28444?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents)[7 min]

A direct challenge to the linear-attention line of work on long-context, appearing in paper and in discussion.

Code

[deepseek-ai/DeepSeek-V4-Flash-Vision-Exp](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents)[6 min]

The week's highest-scoring new model release, DeepSeek's new fast multimodal model.

[alphaXiv/openresearch-cli](https://github.com/alphaXiv/openresearch-cli?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents)[3 min]

Parallel research agents using any LLM written in Rust, from the alphaXiv team.

Jobs

Agent-related roles and infrastructure were the categories to add meaningful headcount.

[Agent roles are the only growing technical category](https://jobs.ashbyhq.com/sierra/1d5cf6f0-feba-46a6-98bc-70a1627a76d0?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents)[10 min]

Agent-tagged demand rose as agentic companies like Sierra alone are adding dozens of new agent roles.

[Data center buildout is xAI's hiring story, not modeling](https://job-boards.greenhouse.io/xai/jobs/5226701007?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents)[10 min]

xAI's net additions this week are compute-capacity related, such as lead electrical reliability, facilities electrical, and more.

Pulse

[Terminal Bench 4.0 just dropped, GLM-5.3 is at the same level as Fable 5, accounting for margin of error](https://www.reddit.com/r/LocalLLaMA/comments/1w1fpxi/terminal_bench_40_just_dropped_glm53_is_at_the/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents)[5 min]

An open-weight model reaching frontier parity on a terminal-agent benchmark is the strong claim in the week's discussion.

[Qwen 3.8 27B at 50 tok/s with 100k Context on a 16GB GPU](https://www.reddit.com/r/LocalLLaMA/comments/1w1lq7u/qwen_38_27b_at_50_toks_with_100k_context_on_a/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents)[7 min]

The week's most-discussed thread, and evidence that long-context local serving on consumer hardware is possible.

| Presented by | ![ai, but simple pro](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/59bb099e-a38a-445e-8f0d-b6cd300aa85d/_UPDATED__AI__but_simple_Pro_banner__1_.png) |

### Discover The Techniques That Matter Before Everyone Else

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/9800b14c-2d79-4aa8-a8b3-71548f2af40e/image.png)

With PRO, unlock access to our AI research layer. Discover emerging techniques and hot topics across thousands of recent papers.

- Perform smart searches across papers, code, jobs, topics, and even interview questions.
- Get personalized daily overviews and a feed just for you. Pick from 10 preset roles or customize your dashboard.

Get full access to this article, as well as:

- Signals. Adopt, test, or skip recent techniques and models. We test efficiency savings and benchmark impact on our own infrastructure.
- Code with every issue (with practice examples + solutions)
- Ad-free reading

[Get AI Research Intelligence](https://www.newsletter.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents)

#### RL Training Inside Harnesses

Agentic systems are made up of two main components.

- The agent, which is the model or LLM that generates output tokens
- The harness, which is the software around it that calls tools, handles memory and state, handles permissions, and more. A popular example is [Claude Code](https://claude.com/product/claude-code?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents).

The LLMs used by agents typically undergo post-training, where they learn to follow instructions (instruction tuning) and are tuned with reinforcement learning (RL) to optimize multi-step output trajectories for performance.

GRPO is a common strategy (and its improved variants), where a model’s score for an attempt at a task is the difference between its reward and the average reward of a group of attempts at the same task (with normalization).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c052e4da-92dc-44fe-872a-ed21805dbbc0/RL_Training_Inside_Harnesses__Stolen_Reasoning_Traces__Judging_Agents__1_.png)

In RL-tuning, we use what we call a trainer, a software abstraction that holds the model weights, runs attempts at tasks, scores them, and performs parameter updates.

Until recently, the trainer also had to run the agent itself.

Older versions of frameworks like [verl](https://github.com/verl-project/verl?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents) and [slime](https://github.com/THUDM/slime?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents) make you rebuild a simplified version of the agent loop inside the trainer.

The reimplementation might not compact context, inject tool schemas in order, append memory files, spawn subagents, or retry malformed tool calls, among other benefits of a proper harness.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/786cb199-13a4-42a3-935c-dfd2603a0f60/RL_Training_Inside_Harnesses__Stolen_Reasoning_Traces__Judging_Agents__10_.png)

Every one of those changes the sequence of tokens the model conditions on.

So, the policy is trained for states that don’t occur in deployment, and the agent is shipped into a harness that produces different states. The state distribution is different.

Newer methods like [Agent Lightning v1.0](https://arxiv.org/abs/2608.17528?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents) (Microsoft) keep the harness in charge.

The harness now runs the loop, calling the model API through a proxy that records prompts, replies, and the exact tokens the model sampled.

This changes what the trainer receives. It used to generate one long sequence together per attempt at a task (e.g., GitHub issue) and train on it.

Under a harness, the harness builds each prompt on its own, and the trainer only receives the list of calls it made:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c43e7d80-47ed-4412-9c31-d5fc276489a2/image.png)

Here, pi is the prompt sent on the i-th call and ai​ is the model's reply.

Everything that happened between calls (e.g., compaction, dropping old turns, injecting memory files) is invisible to the trainer.

So, for any given attempt, we may not be able to string everything together but instead be grouped into rows of calls (done by the harness).

In the paper, on average, one attempt has 2.41 training rows.

This slightly changes how the GRPO averaging will work. Since only one reward is given for a single attempt (e.g., right or wrong), we have to decide whether to average the reward over attempts or rows.

- For example, let’s say 2 attempts get rewards 1 and 0, and the successful attempt (1) splits into 3 rows.
- Averaging over attempts gives a baseline of (1+0)/2 = 1/2
- Averaging over rows gives (1+1+1+0)/4 = 3/4, which is higher and favors a larger amount of splits.

The paper chooses to average per attempt, since however many pieces an attempt splits doesn’t say much about how well the agent did.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/ecf6a425-bc92-4545-93c1-217e94493eb1/RL_Training_Inside_Harnesses__Stolen_Reasoning_Traces__Judging_Agents__4_.png)

With their approach, RL through the harness improved Qwen3.5-9B from 41.8% to 56.4% on SWE-bench Verified using 6,000 examples.

The method above trains the model and leaves the harness alone. Other directions search over the harness itself:

- [StateM](https://arxiv.org/abs/2608.15089?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents) improves only the harness with the model frozen
- [DarwinX](https://arxiv.org/abs/2608.07545?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents) evolves a population of harnesses
- [JIT-Agent](https://arxiv.org/abs/2608.25593?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents) trains a model to write harnesses, and [HarnessDev](https://arxiv.org/abs/2609.01437?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents) pushes back on how well a model can do this by itself.

If you want to read deeper into harness optimization and harness self-improvement, Lilian Weng has a good [blog post](https://lilianweng.github.io/posts/2026-07-04-harness/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents) mapping out the ways researchers are doing it today.

---

#### Reasoning Traces Can be Stolen (with Cheaper Models)

Reasoning models typically generate intermediate thinking tokens during the reasoning process before the final output.

Companies (e.g., Anthropic, OpenAI, and Google) hide those tokens because competitors could train on them and because they can contain tokens that are not very user-facing or friendly.

The model still needs its own earlier thinking back on the next turn, and storing it server-side costs money.

So, they encrypt the thinking, send the encrypted block to your application, and require your application to send it back with the next request.

A [blog post](https://blog.cryptographyengineering.com/2026/05/29/fooling-around-with-encrypted-reasoning-blobs/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents) in May 2026 looked into these reasoning blocks and concluded that each provider is most likely using one encryption key for every block.

That meant blocks from one conversation were accepted in other conversations, as well as across accounts.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/8982634b-f59c-4013-a1c2-7c22c65dad2d/image.png)

Encryption procedure for GPT 5.5 from the blog, which is “partially guessed”

[Panfilov et al.](https://arxiv.org/abs/2608.09867?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents) then showed a block also works in a different model from the same family.

Frontier models are trained hard to refuse to reveal their thinking, and the cheap models mostly are not.

So, how can we extract reasoning traces? First, we give a larger target (e.g., Opus 4.8) model a prompt to obtain an answer and an encrypted thinking block.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/9179c300-ddfb-4b34-88d4-57975f3d56e3/RL_Training_Inside_Harnesses__Stolen_Reasoning_Traces__Judging_Agents__5_.png)

We send that block to a cheaper model (e.g., Haiku 4.5) and ask it to write out the thinking word for word.

The server decrypts the block and places it in the cheaper model’s context, and since it has little training against emitting it, it outputs the whole block.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/72b1908c-4dde-4ac1-840d-2f103c3ba87f/RL_Training_Inside_Harnesses__Stolen_Reasoning_Traces__Judging_Agents__6_.png)

We are now able to access the larger model’s private reasoning without attacking it directly. A [project site](https://stolen-thoughts.com/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents) called “stolen thoughts” and its [corresponding paper](https://stolen-thoughts.com/paper.pdf?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents) show what the recovered thinking looks like.

Well, why does this matter?

One problem that arises is the publishing of agent logs, since the encrypted blocks will travel with them.

The authors decoded blocks from almost 7,000 public agent logs from GitHub and Hugging Face and found around 700 privacy artifacts containing API keys, passwords, and personal data.

This could be caused by something like asking an agent to clean secrets out of a repository, listing them in the agent’s thinking.

Another consideration is how a frontier model's reasoning can be copied at scale. It’s decently cheap as well, around $720 for 10K blocks at Haiku prices.

Blocks can also carry instruction. If we obtain a reasoning trace about uploading files to your server and pass that block into someone else's conversation, their model will treat the instruction as its own thought.

The fix (by agent providers) now is to bind each block to a user and conversation by placing the user ID inside the encrypted envelope and chaining blocks to previous blocks.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/9cb09cba-7064-401d-bd74-aaee5dbf6e7f/RL_Training_Inside_Harnesses__Stolen_Reasoning_Traces__Judging_Agents__7_.png)

For more on this topic, the paper [Hidden Thoughts Are Not Secret](https://arxiv.org/abs/2606.00642?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents) obtains similar results with prompting alone, and [How to Steal Reasoning Without Reasoning Traces](https://arxiv.org/abs/2603.07267?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents) rebuilds reasoning traces from output answers and summaries.

---

#### LLM-as-a-Judge is Unreliable in a Structured Way

LLMs are commonly used as judges, since it is unreasonable to read thousands of agent runs by hand.

This is known as LLM-as-a-judge, and it works reasonably well when we grade simple text output.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7e3072a6-1763-40c1-b014-07fa8fe0676c/RL_Training_Inside_Harnesses__Stolen_Reasoning_Traces__Judging_Agents__11_.png)

Agent runs are different, containing tool calls and turns, with the state of the task changing as it continues.

A recent paper, [Catching One in Five](https://arxiv.org/abs/2606.10315?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents) (Zhang et al.), measures this on a food-ordering agent.

Humans read transcripts and recorded problems, then the authors checked how many of those the agent's judge had reported.

The results show that LLM judges aren’t as reliable as we once thought.

In one batch, the judge caught 2 out of 9, while in another it flagged nothing across 100 rounds, while reviewers found 23 defects.

The issue here is that while the judge catches visible within a single turn (e.g., made-up statistic), it miscategorizes issues that exist across multiple turns.

- For instance, a cart holding items a user didn’t order or an agent that won't escalate the conversation to a human.

The judge does see these problems, and it catches some and writes them down, but since the scoring rubric had no category for those issues, 113/114 of the notes ended up under "brand voice."

All rounds ended up passing since the gate they used didn’t read the notes.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a4e5683b-919b-4c30-9737-e24707eb1185/RL_Training_Inside_Harnesses__Stolen_Reasoning_Traces__Judging_Agents__9_.png)

Could we have written guidelines for those? Yes, however, you can only write guidelines for a failure you already know about. So, an LLM judge is cheap, always-on, and good at catching things you’ve already defined.

It can’t always reliably tell you whether an agent is truly effective at a task (especially with all passes but hidden issues), so a human will probably need to be involved there.

A larger judge does not fix this. [AgentJudgeBench](https://arxiv.org/abs/2608.26623?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents) ran six judges of different sizes over 3,800 tool-calling tasks.

On the hardest tasks, with no correct answer supplied, every judge lands between 77% and 82% agreement.

Other directions in this field include [MobileJudgeBench](https://arxiv.org/abs/2608.11434?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents), which finds a plain judge given screenshots beats purpose-built pipelines, and [Inside the Unfair Judge](https://arxiv.org/abs/2607.11871?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=rl-training-inside-harnesses-stolen-reasoning-traces-judging-agents), which analyzes judge bias through model activations.

---

#### Conclusion

Agents are now being RL-tuned inside proper harnesses, where prior, they were trained on a simplified loop (and deployed in a different one).

Researchers are also beginning to search over the harness.

Averaging across attempts or examples is a design choice tackled by many papers.

Researchers found a way to extract reasoning traces from large AI model providers’ frontier models using cheap models.

Reasoning blocks were assumed to be private, yet it was found to use the same key across users and models.

It has now been fixed, but models still need earlier thinking context, so there’s some space for attacks (and protection) there.

As a reminder, strip encrypted reasoning blocks out of any log before you publish it.

LLM judges were assumed to be about reliable on agent runs (as on text). However, they miss problems that span multiple turns, and a bigger judge does not help, rather the performance depends on the task and its rules.

That's it for this week's free version of AI, but simple. Get the code and signals for deeper analysis with PRO below.