# Random Attention, Flow Matching for VLAs, Tencent's Speech Foundation Model

#119 Sep 14 LLMs

https://www.aibutsimple.com/p/random-attention-flow-matching-for-vlas-tencent-s-speech-foundation-model

This week, AI research landed on random attention, flow matching as the method for action generation, and an open-source model that can generate and edit speech from instructions.

We'll explain these developments and how they work visually, then take a look at some results and key movements to keep you updated in no time.

THE WEEK, RANKED

| [DeepSeek's new smallest model with native visual understanding](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=random-attention-flow-matching-for-vlas-tencent-s-speech-foundation-model) | ▲ 1.8KHF |
| [A tiny open-source SoTA 2B parameter model that beats 4B models](https://huggingface.co/openbmb/MiniCPM5-2B?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=random-attention-flow-matching-for-vlas-tencent-s-speech-foundation-model) | ▲ 1KHF |
| [HuggingFace's adds "no need to hack us" on security.txt after OpenAI incident](https://www.reddit.com/r/singularity/comments/1wclhgv/huggingface_security_txt_after_the_openai_incident/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=random-attention-flow-matching-for-vlas-tencent-s-speech-foundation-model) | ▲ 2.8KRDDT |
| [Git-Native persistent project memory for Agents using the Open Knowledge Format](https://github.com/okf-memory/okf-agent-memory?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=random-attention-flow-matching-for-vlas-tencent-s-speech-foundation-model) | ▲ 596GH |
| [A step towards recursive self-improvement with Agentic Post-Training](https://huggingface.co/papers/2609.08183?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=random-attention-flow-matching-for-vlas-tencent-s-speech-foundation-model) | ▲ 391HF |

[ALL ENTRIES](https://www.aibutsimple.com/pro?utm_source=rankedweek)RANKED BY [AIBUTSIMPLE PRO](https://www.aibutsimple.com/pro?utm_source=rankedweek)

| Supported by | ![brightdata](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7abbd1bd-5279-4147-95db-11cc7e8e326e/image.png) |

### Turn the Web Into A Data Pipeline. Generate Your API, in Minutes.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/ba3b1a85-85b1-4126-bec8-7654947bb280/image__332___1_.png)

If your team is still patching selectors and rotating proxies, you're maintaining infrastructure your competitors stopped owning.

[Scraper Studio](https://brightdata.com/products/web-scraper/studio?utm_source=brand&utm_campaign=brnd-mkt_newsletter_AIsimple&promo=entail) from Bright Data turns any public website into a hosted data API from a single prompt:

- Self-healing scrapers that detect site changes and patch themselves
- Proxies, unblocking, and parsing handled - not your problem
- Build from Claude Code, Cursor, or Codex via CLI - or visually in the control panel
- Scale production data pipelines without owning the stack
- 5,000 free credits every month

No sales call. No demo. Just spin up a scraper and watch it run.

[Start Free](https://brightdata.com/products/web-scraper/studio?utm_source=brand&utm_campaign=brnd-mkt_newsletter_AIsimple&promo=entail)

#### Random Attention or Selective Attention?

Language models store keys and values computed for input tokens in a KV cache.

They do this so they can reuse these cached keys and values instead of recomputing them for every output token.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7ecd5e26-20c3-4834-9d06-6878b6057000/Random_Attention__VLAs__Speech_Generation__3_.png)

The issue is that the cache grows with every token, and reasoning models generate a lot of them:

- A 200-token math question can turn into a chain of thought of over 10,000 tokens.
- On Qwen3-32B, one 32k-token answer fills more than 8 GB of KV cache.

At that point, memory bounds how many requests a GPU can serve at once.

KV cache eviction caps the cache at K entries per attention head.

When the entries reach the limit, we score every cached token, then evict everything outside the top K.

Most KV cache research has focused largely on that score. Papers such as [SnapKV](https://arxiv.org/abs/2404.14469?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=random-attention-flow-matching-for-vlas-tencent-s-speech-foundation-model) check which tokens recent queries attended to, and [TriAttention](https://arxiv.org/abs/2604.04921?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=random-attention-flow-matching-for-vlas-tencent-s-speech-foundation-model) obtains importance from a token’s relative distance (RoPE) in the past.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/5242d5e4-7f91-4cbf-9a5f-f19165604cc3/Random_Attention__VLAs__Speech_Generation__7_.png)

Salesforce AI and UIUC take a surprisingly different approach in their new paper, [Random Attention](https://arxiv.org/abs/2609.03430?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=random-attention-flow-matching-for-vlas-tencent-s-speech-foundation-model).

Their core approach is to protect the prompt tokens and select everything else randomly.

The paper models their approach quite simply with the below cases:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7e2a9545-8412-4405-98e0-8c37c86dfc57/image.png)

Here, prompt tokens score infinity to prevent eviction, while generated tokens get a random number (from 0 to 1) every eviction round.

The numbers are drawn separately for each KV head, and the newest 64 tokens skip this scoring.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b35f4890-b848-4a8a-9765-f4c69734afcc/Random_Attention__VLAs__Speech_Generation__8_.png)

The random approach accounts for token age automatically.

Old tokens have to win a new draw every round (about 94% at K=1,024), so on average, older tokens are evicted first.

For the results:

- Random attention was compared against 4 eviction methods (SnapKV, R-KV, VaSE, TriAttention)
- Four reasoning models were tested (Qwen 3 family, Phi 4) on five tasks (MATH500, GPQA-D, AIME, etc.), for a total of 60 comparisons
- Random attention performed better on 31/60 comparisons and was only significantly worse in 1.

With no scoring mechanism, it serves about 32 to 43% more tokens per second than TriAttention.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c99cf122-4c3e-4005-a67f-c985c8ee87f2/image__326___1_.png)

So why does random work?

The prompt really is the most important element. Without keeping the full prompt tokens, random attention performance on Qwen3 drops almost 50% on MATH500.

The authors also applied prompt keeping to the other KV eviction methods, showing notable performance increases in practically all of them.

And where does random fail?

It fails on small lines of details, such as facts stated once (and never repeated).

For instance, given a passcode, after 57 eviction rounds, Random Attention was never able to recover it. In contrast, methods like [R-KV](https://arxiv.org/pdf/2505.24133?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=random-attention-flow-matching-for-vlas-tencent-s-speech-foundation-model), whose score tracks attention over the whole history, found it 84% of the time.

The paper also doesn't test methods that learn token ranks:

- [TRIM-KV](https://arxiv.org/abs/2512.03324?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=random-attention-flow-matching-for-vlas-tencent-s-speech-foundation-model) (Yale and JPMorganChase) trains a small gate that gives each token a retention score βi​ between 0 and 1, which decays as the token ages. It was able to beat a full cache’s score on certain benchmarks.
- Its follow-up, [DBTrimKV](https://arxiv.org/abs/2605.09649?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=random-attention-flow-matching-for-vlas-tencent-s-speech-foundation-model), shares the gates' last layer across every layer and head, so cached tokens compete for a global budget.
- Apple's [KV Policy](https://arxiv.org/abs/2602.10238?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=random-attention-flow-matching-for-vlas-tencent-s-speech-foundation-model) trains a tiny RL agent for each KV head that ranks tokens using only their keys, values, and positions.

---

#### VLAs are moving from action tokens to flow matching

A vision-language-action model (VLA) is a vision-language model (VLM) fine-tuned to control a robot.

Typically, they take camera images and instructions, then output actions for the robot to take.

They predict what’s known as action chunks, and they correspond to tensors of some amount of future actions (given by the chunk size) per action dimension.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/bdae7d9d-9cd0-4e02-9729-907e82d88669/Random_Attention__VLAs__Speech_Generation__1_.png)

Early VLAs represented each action dimension as a sequence of discrete bins, using the bins as tokens.

[RT-2](https://arxiv.org/abs/2307.15818?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=random-attention-flow-matching-for-vlas-tencent-s-speech-foundation-model) (2023) and [OpenVLA](https://arxiv.org/abs/2406.09246?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=random-attention-flow-matching-for-vlas-tencent-s-speech-foundation-model) (2024) used 256 bins, predicting one bin token per action dimension at a time.

So the models would predict one bin token (out of 256) for the first action dimension, then another bin token for the next action dimension, and so on.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/1d340ddc-920f-48cd-9ed4-f2ca1cc989d1/Random_Attention__VLAs__Speech_Generation__2_.png)

For example, a 7-dimensional arm action takes 7 tokens, so a 50-step chunk would take 350 bin tokens.

Later on, Physical Intelligence's [π](https://arxiv.org/abs/2410.24164?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=random-attention-flow-matching-for-vlas-tencent-s-speech-foundation-model)[0](https://arxiv.org/abs/2410.24164?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=random-attention-flow-matching-for-vlas-tencent-s-speech-foundation-model) (2024) introduces an action expert to generate the whole chunk with flow matching, while the VLM reads the scene.

This week, Qwen released their open-source autonomous driving VLA, [Qwen-Drive 1.0,](https://qwen.ai/blog?id=qwen-drive-1.0&utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=random-attention-flow-matching-for-vlas-tencent-s-speech-foundation-model) which also uses a planning expert with flow matching for action generation.

But first, what is flow matching?

Let's say you have a dataset of actions, and you sample a real action chunk x1 from a demonstration.

You also sample random noise x0, typically from a Gaussian, with the same dimension as the chunk.

During training, flow matching randomly chooses a time between 0 and 1 and linearly interpolates between the noise and the real action:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/32205d53-f612-46ca-848b-cea108af7cbc/image.png)

At t=0, we have pure noise x0, and at t=1, we have the action chunk x1.

This constructs a straight-line trajectory between the data distribution and the source distribution (noise), and the velocity along this line is constant:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/156db6f0-8419-4525-9d68-9822c6cc6036/image.png)

For VLAs, the flow matching model learns this velocity (vθ) from the current point xt, the time, and conditioning information c from the VLM.

Here’s the loss function:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b05a1ef4-909c-4bc8-b987-6268cccc8ae1/image.png)

This is what we call conditional flow matching, getting its name from conditioning on pairs (x0, x1) to learn the velocity field. It’s the common form of flow matching that is computationally tractable.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a9ceb4aa-590f-4874-be11-0250ec31ecd2/diffusion_paths_vs_flow_matching_paths__2_.png)

During inference, we start from pure noise and repeatedly follow the learned velocity field.

By integrating the field from t=0 to t=1 and incorporating conditioning information, the noise is transformed into an action chunk.

Now what about driving?

Qwen is using this VLM and action expert (with flow matching) approach for self-driving cars.

Typical driving VLAs tune the VLM on driving questions and text. But text answers don't identify where things are in 3D, and heavy fine-tuning can wear away the VLM’s general skills.

[Qwen-Drive-1.0](https://arxiv.org/abs/2609.00111?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=random-attention-flow-matching-for-vlas-tencent-s-speech-foundation-model) unifies 3D perception and VQA by keeping its VLM (Qwen3.5-4B) frozen and training a 3D object head on top of it.

- Its action (planning) expert is a 1.1B-parameter transformer that reads the VLM's cached keys and values.
- It draws the next 5 seconds of driving as 50 waypoints, using 10 flow matching steps.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a9717ec3-0548-4e2e-a2b4-cc7fee3b8d63/image.png)

Architecture from the paper

Qwen-Drive, like many other driving models, is tuned with RL to optimize for safe and effective driving.

But RL needs probabilities for each step, and in flow matching, once the random noise is sampled, the rest of the trajectory steps are deterministic (following the learned velocity).

This makes it difficult to treat the action expert as a conventional stochastic policy that assigns probabilities to the actions it takes.

To make RL possible, we add controlled noise during the final 3 flow-matching steps.

The noise changes the trajectory in smooth patterns, allowing the model to generate slightly different (still coherent) driving paths and giving RL a policy distribution to optimize.

But is flow matching why these action policies perform so well across robotics, self-driving, and more?

Recent work has started to question how much of this advantage comes from flow matching itself, with simpler iterative regression policies sometimes achieving similar performance.

- [Much Ado About Noising](https://arxiv.org/abs/2512.01809?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=random-attention-flow-matching-for-vlas-tencent-s-speech-foundation-model)’s (ICLR 2026) simple two-step regression policy essentially matched flow-based policies
- [StarVLA](https://arxiv.org/abs/2604.05014?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=random-attention-flow-matching-for-vlas-tencent-s-speech-foundation-model)'s regression head scored 96.6%, and its flow head 95.7%

---

#### Speech generation and editing are merging into one model

Speech AI typically uses separate models for various voice tasks.

For example, one model is used for text-to-speech, while another is used for removing background noise. Capabilities like voice cloning need an additional speech transcription model.

This week, Tencent’s Hunyuan lab released [AuK](https://arxiv.org/abs/2609.08936?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=random-attention-flow-matching-for-vlas-tencent-s-speech-foundation-model), an open-source foundation model for both speech generation and editing.

The model can perform 5 task families using natural-language instructions:

1. Speech generation
2. Audio enhancement and separation (e.g., speech separation)
3. Paralinguistic editing (e.g., emotion editing)
4. Content editing (e.g., inserting words)
5. Acoustic editing (e.g., pitch or speed editing)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/304d5d70-b92d-407b-b248-f0fc670da285/image.png)

AuK’s 5 task families (quite a lot)

It can even clone a voice from the reference audio and target text alone with no transcription model.

Here’s how it works. AuK is broken into 3 components: a multimodal LLM (MLLM) semantic encoder, a VAE for audio, and a hybrid DiT.

First, we pass an NL instruction and input audio as input to a frozen MLLM (in this case, [Qwen2.5-Omni](https://github.com/qwenlm/qwen2.5-omni?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=random-attention-flow-matching-for-vlas-tencent-s-speech-foundation-model)).

AuK blends all of the MLLM’s layers with learned weights so it can capture high-level semantic intent and low-level acoustics.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/9be11114-11c9-423c-8dc7-cff657c348ff/image.png)

Architecture diagram from the paper

In parallel, a 24 kHz frozen VAE compresses audio into a 50 Hz latent space (64-D latents), capturing things like fine-grained speaker timbre, room acoustics, and more.

The MLLM and VAE outputs are then passed into a 1.5B parameter rectified-flow Transformer that predicts the target audio latent in two stages.

The first 10 blocks are processed by a multimodal DiT, with the semantic stream (instructions) and the acoustic stream (reference audio latents plus noisy target latents) running in parallel.

The streams interact through joint attention, letting the model cross-reference against the source audio (with residual pathways).

The following 20 blocks are single-stream DiT, merging streams into a single sequence.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/59dd62eb-21a8-4057-bcc1-5e2085795aba/Random_Attention__VLAs__Speech_Generation__10_.png)

This allows self-attention to refine the features jointly and predict the velocity of the target audio latents, using the same flow matching loss as other VLAs.

At the end, the VAE decoder projects the denoised latent back into a 24 kHz output waveform.

AuK performs well specifically on speech editing. It scores 91.83% on SpeechEditBench, while earlier models scored a good amount lower, like [Ming-UniAudio](https://arxiv.org/abs/2511.05516?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=random-attention-flow-matching-for-vlas-tencent-s-speech-foundation-model) at 76.46%.

For voice cloning, AuK's average error rate is 2.65% against 3.07% for Qwen3-TTS on Seed-TTS-Eval.

AuK still has clear gaps:

- Emotion edits reach only 9.94% joint success
- Free-form prompts require a separate “prompt enhancer” LLM to rewrite them into formats AuK saw in training

---

#### Conclusion

Efficiency in the KV cache used to focus more on top-k scoring in eviction, and while research is still focused in this direction, Random Attention shows that keeping the prompt matters the most for performance.

Many VLAs since π0 generate actions with flow matching instead of tokens, but research is also looking into small regression policies that show promising results (matching flow).

Tencent’s AuK combined speech generation and editing in one model and open-sourced it.

That's it for this week's free version of AI, but simple. Get the code and signals for deeper analysis with PRO below. See you next week!

---

Here's a special thanks to our biggest supporters: Sushant Waidande, Sai Krishna Pashapu, Bob Boncella, Demet Yazici, Imran Khan

Want to reach 10000+ AI engineers, researchers, and technical founders? Advertising Inquiries:

[WORK WITH US 📈](https://tally.so/r/A7Dgdk?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=random-attention-flow-matching-for-vlas-tencent-s-speech-foundation-model)

Feedback or inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).