# Persistent AI memory, AI watermarking, on-policy distillation

#116 Aug 24 Training

https://www.aibutsimple.com/p/persistent-ai-memory-ai-watermarking-on-policy-distillation

This week, AI research has landed on persistent memory for AI agents, a focus on Claude’s new watermarking system (and LLM watermarking as a whole), and taking on-policy distillation as a post-training default.

We’ll get into the intricacies of how these systems work, cover some benchmark results, and the key research movements in every topic to keep you updated in no time.

THE WEEK, RANKED

| [Open source DeepSeek agent harness](https://github.com/deepseek-ai/deepseek-harness?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation) | ▲ +11KGH |
| [AI agent penetration testing tool](https://github.com/usestrix/strix?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation) | ▲ +1KGH |
| [Minimax H3 has absurd adherence](https://www.reddit.com/r/StableDiffusion/comments/1vq0ry7/minimax_h3_wasnt_released_as_an_image_model_but/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation) | ▲ 594RDDT |
| [Qwen3.8 27B GGUF quantized](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation) | ▲ 2.7KHF |
| [EnvHarness: awakening static worlds for agent learning](https://huggingface.co/papers/2608.19880?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation) | ▲ 255HF |

[ALL ENTRIES](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation)RANKED BY [BASIS, OUR TRACKER](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation)

| Presented by | ![ai, but simple pro](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/59bb099e-a38a-445e-8f0d-b6cd300aa85d/_UPDATED__AI__but_simple_Pro_banner__1_.png) |

### You Build. We Verify, Then Send The Most Impactful Techniques.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/acd2356a-bb4c-4761-bff6-c70321592212/image.png)

With PRO, unlock access to Signals. We tell you whether to adopt, test, or skip recent techniques and models.

Get full access to this article, in addition to:

- Basis, our AI research layer: personalized daily overviews, top models + code, community topics, interview questions + job trends
- Code with every issue (with practice examples + solutions)
- Ad-free reading

We use a ranking system based on 50+ factors to ensure you get the most relevant entries for your interests and career.

[Get AI Research Intelligence](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation)

#### Persistent Agent Memory

Long-term agent memory has become a popular research theme in the last month.

Papers like [Metis](https://arxiv.org/abs/2607.26760?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation), [Memory Decoder at Scale](https://arxiv.org/abs/2607.27919?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation), and [Alaya-EVOKE](https://arxiv.org/abs/2608.13546?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation) are all converging on the idea that memory should no longer be an add-on.

But first, what is memory? Modern agents use memory systems to deliberately store relevant context to the conversation.

The stored information will be useful later, allowing the agent to understand your long-term goals and preferences and feel “smarter.”

- Agents benefit from memory since there is often too much context to store in the context window, requiring linear growth for every token.

The basic approach is to use an external memory storage (e.g., vector database) paired with retrieval.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/972f8513-27a3-4da1-874a-dab209679fce/Persistent_Agent_Memory__Watermarking__On-policy_distillation.png)

This works decently well, but the model itself never changes and doesn’t adapt (forget/remember) to a user’s changing preferences and context.

The key direction this month has been placing memory inside the models themselves.

[Metis](https://arxiv.org/abs/2607.26760?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation) is a popular paper that took this approach, calling itself a “memory foundation model.” Instead of storing history in a vector data store, Metis keeps a memory state inside the network.

There are 2 memory components in a Metis transformer block that live in a memory attention:

1. Local Memory Block, holding the memory state
2. Hyper Memory Block, which learns the procedures for storing and retrieving from it.

In a RAG pipeline, storage, retrieval, and reranking are rules, but here, the model learns them.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/814df419-5f74-4744-89ba-fbc346fc71a2/Persistent_Agent_Memory__Watermarking__On-policy_distillation__1_.png)

The update mechanism is practical since writing to memory is gradient-free (costing only a forward pass), and only the memory state changes during inference. The rest of the LLM stays frozen.

The session compresses into a fixed-size state that later queries read through memory attention, so the token count term stops growing.

[Memory Decoder at Scale](https://arxiv.org/abs/2607.27919?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation) takes the same position from a different direction.

It treats long-term memory as its own pretrained module and scales it independently, reaching 6.9B parameters trained on 300B tokens.

Having a 410M [Pythia](https://github.com/eleutherai/pythia?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation) backbone, it scores higher than Pythia-12B with 39% fewer total parameters.

The claim here is that memory capacity and reasoning capacity are separate things that should also scale separately.

The rest of the field keeps memory outside the model but develops the structure.

- [ABot-AgentOS](https://arxiv.org/abs/2607.10350?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation) stores robot experiences in a graph and retrieves evidence subgraphs so a failure can be traced to the stage that caused it.
- [Agent Memory Distillation](https://arxiv.org/abs/2608.07169?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation) is training-free, copying a large teacher’s successful trajectories into 3 tiers: workflow strategy, subtask examples, and per-tool calling.
- [Spatial Memory Agent](https://arxiv.org/abs/2608.12743?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation) and [LightMem-Ego](https://arxiv.org/abs/2607.11487?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation) build memory for spatial reasoning and everyday procedures, respectively.

---

#### AI Watermarking and Claude’s Controversy

As of August 2nd, the EU requires mandatory LLM watermarking to determine if an output was AI generated, and Claude is incorporating this into every new model.

Take the sentence: "The weather today was …"

A potential next word could be sunny or cloudy, determined by the output distribution produced by the LLM.

A watermark changes the probability distribution by injecting a statistical bias. This is done using secret keys combined with a window of prior tokens.

Anyone holding the key can verify whether the sequences match what the key would’ve produced, while the text still looks random to a reader.

Watermarking was [introduced in 2022](https://scottaaronson.blog/?p=6823&utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation), and [SynthID-Text](https://www.nature.com/articles/s41586-024-08025-4?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation), the algorithm Claude’s watermarking system is based on, made it deployable in 2024.

SynthID-Text works by using tournament sampling.

In tournament sampling, we pick a secret key with m components that corresponds to m tournament layers. This determines the number of bits per token.

For every token, the secret key and a sliding window of the last few tokens are hashed into a seed.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/ab96cba7-df74-4b3d-8b9f-20795357b462/Persistent_Agent_Memory__Watermarking__On-policy_distillation__2_.png)

The seed (or components of the seed) assigns every token in the vocabulary a set of m random bits (0 or 1).

The model then samples 2m candidate tokens from its distribution, passing them into a knockout bracket.

Layers pair tokens together and compare their bits. The higher bit wins, and ties are broken at random. The token that wins the last layer is outputted.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7d6efdcf-8d8a-431f-8cc9-e9a2a3f17aca/Persistent_Agent_Memory__Watermarking__On-policy_distillation__3_.png)

To detect whether text is watermarked, we average the g-values across n tokens and all m layers:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/403a9a1e-990d-442a-917e-7652fa030cd8/image.png)

If the text was never watermarked, the g-values behave like coin flips and the mean sits near 0.5.

Watermarked text will hold mostly winning tokens (with high g-values) by default, so the mean will sit above it.

We can then use z-score thresholds and other methods to determine the probability that the text was watermarked.

Two properties of LLM watermarking:

- Watermarking costs no extra tokens, since it only re-sorts choices the model was already making.
- Anthropic [confirmed](https://www.anthropic.com/news/claude-text-watermark?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation) that Claude's watermark is a version of SynthID-Text, stating that light editing probably will not remove the watermark completely, but a full rewrite will.

Watermarking is also spreading past text into 3D Gaussian splatting ([NGS-Marker](https://arxiv.org/abs/2608.17447?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation)) and C2PA embedding vectors ([HaloMark](https://arxiv.org/abs/2608.08645?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation)), which matters because regulation covers the whole of AI-generated content.

---

#### On-policy distillation is the new default

Regular knowledge distillation trains a small student to copy a larger teacher's output distribution over a fixed dataset.

At inference, the student generates its own text, sampling trajectories that the dataset may never have covered. Here, small errors are able to compound.

On-policy distillation, introduced by [Agarwal et al.](https://arxiv.org/abs/2306.13649?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation) (2024), fixes this by supervising the student model on the distribution it will actually use.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/fa14ea70-d03a-4cf5-82d3-98906a9dbfa5/GKD_and_RL__5_.png)

The student generates the trajectory, then the teacher scores the tokens the student produces.

OPD rapidly gained popularity in post-training, being an alternative to [reinforcement learning with verifiable rewards (RLVR)](https://arxiv.org/abs/2506.14245?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation).

- The downside of RLVR is that the reward signal is sparse: an entire multi-thousand-token solution obtains one scalar reward at the end.
- On-policy distillation returns a full probability distribution at every position instead, so the same rollouts yield much more supervision.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/cc51dfb0-fad9-4d48-bdfa-7c68c5e2868b/RLVR_Overview.png)

The issue here is cost, since a separate larger teacher has to run online for the training process.

Recent research is shifting to [on-policy self-distillation (OPSD)](https://arxiv.org/pdf/2601.18734?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation), where you run the same model twice, once as a student and once as a teacher. The authors report 8 to 12 times the token efficiency of GRPO.

The teacher obtains a ground-truth reasoning trace or answer, and the model is trained on the gap between the student and teacher distributions, typically some divergence.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/236c12e9-5263-41d8-ae74-49a80612bc4d/image.png)

The teacher pT here is conditioned on the solution trace y*.

A basic intuition can be built by imagining this scenario:

- We initialize a parameterized model, with x as the query and y* as the correct answer
- The model acts as a teacher for the first forward pass, feeding x and y*, then obtaining a teacher output distribution. This is detached from the computation graph so gradients don’t flow back to the teacher side.
- The model acts as a student for a second forward pass, feeding in only x, obtaining a student distribution.
- A divergence (e.g. per-token KL) is computed between the output distributions, then the loss gradient is backpropagated, updating the model’s parameters.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/e52e28aa-6439-4641-a880-172bcec9040a/GKD_and_RL__On_policy_distillation.png)

The parameters will modify both the student’s and teacher’s distribution on the next pass, but we only want the gradient to be backpropagated on the student’s side since the teacher is the target.

Given this pattern, recent research differs in where the solution trace comes from.

- [OPSD](https://arxiv.org/pdf/2601.18734?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation): A ground-truth solution (verifier needed)
- [SEED](https://arxiv.org/abs/2607.14777?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation): self-generated completed trajectories
- [U-OPSD](https://arxiv.org/abs/2608.06296?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation): its own majority vote
- [S2VOPD](https://arxiv.org/abs/2608.14144?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation): nothing at all, by weakening the student

---

#### Conclusion

1. Memory is being pulled out of the RAG pipeline into weights or structured state.
2. Watermarking is now deployed for Claude’s new models, based on the Synth-ID system. It hasn’t been fully tested for accuracy.
3. On-policy distillation has moved towards using the same model for both student and teacher, just with different conditioning.

It’s interesting to see where these fields will move from here. That's it for this week's issue of AI, but simple. See you next week!

---

Here's a special thanks to our biggest supporters: Sushant Waidande, Sai Krishna Pashapu, Bob Boncella, Demet Yazici, Imran Khan

Want to reach 10000+ AI researchers? Advertising Inquiries:

[WORK WITH US 📈](https://tally.so/r/A7Dgdk?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=persistent-ai-memory-ai-watermarking-on-policy-distillation)

Feedback or inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).