# NeurIPS 2025 Paper Awards (The Best Papers of 2025)

#90 Feb 23 Papers

https://www.aibutsimple.com/p/neurips-2025-paper-awards-the-best-papers-of-2025

The rapid progress of AI in recent years has raised many questions and challenges. In 2025, researchers responded with breakthroughs that pushed the boundaries of deep learning, machine learning, and other related fields.

As deep learning advances into 2026, these developments continue to shape its trajectory. In this issue, we’ll take a look at a selection of highly impactful NeurIPS 2025 papers, highlighting the insights and innovations that are defining the “cutting-edge” of AI research.

### A Better Real-Time Object Detection Algorithm (YOLOv10)

Paper: [YOLOv10: Real-Time End-to-End Object Detection](https://arxiv.org/abs/2405.14458?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neurips-2025-paper-awards-the-best-papers-of-2025)

Wang, Chen, Liu et al., Tsinghua University - NeurIPS 2024

The YOLO family has long dominated real-time object detection, among other image tasks, but every version up to now has shared a small issue: they all rely on Non-Maximum Suppression (NMS) after inference to filter out duplicate detections.

NMS adds latency and introduces sensitivity to hyperparameters that are hard to tune in production. YOLOv10 sets out to fix this and does so convincingly.

The centerpiece of the paper is a training strategy called consistent dual assignments.

The idea is that during training, the model runs two object detection heads in parallel.

One head uses the standard one-to-many label assignment, which provides rich supervision and is responsible for most of YOLO's historical accuracy.

The other head uses one-to-one assignment, which naturally produces a single prediction per object and therefore needs no NMS.

At inference time, you simply discard the first head and use the second. But how could this work? The authors have a clever approach.

The two heads are actually aligned using a shared matching metric, minimizing the "supervision gap" between the branches. As a result, the one-to-one head learns from the strength of the one-to-many head without drifting.

The math behind this matching is grounded in a [Wasserstein-distance](https://en.wikipedia.org/wiki/Wasserstein_metric?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neurips-2025-paper-awards-the-best-papers-of-2025) analysis of the classification targets.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/e265f244-33ef-453a-913c-ebfa03bcf417/NMS-free_YOLO_v10__1_.png)

Above, the PAN is the Path Aggregation Network, used from YOLOv4 to v10.

On the architecture side, the authors don't just make one or two tweaks but do a systematic audit of the entire YOLO pipeline.

They show through error analysis that the classification head in YOLOv8 is over-engineered relative to the regression head and replace it with a much lighter depthwise separable design.

This design is known as the Depthwise Separable Convolution (DSConv) head. They decouple downsampling into a separate channel and spatial operations to cut FLOPs without losing information.

In addition, they introduce a rank-guided block design, which works by measuring the intrinsic rank of feature maps at each stage, identifying which stages are redundant, and replacing their blocks with a compact inverted structure only where it's safe to do so.

For accuracy, they add large-kernel convolutions in deep stages (only for smaller models, where the receptive field is otherwise too narrow) and a partial self-attention module that applies attention to only half the channels at the lowest resolution, keeping the compute cost manageable.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/ba8e4646-d23c-45c8-b5df-8ab6f5712de1/Compact_Inverted_Block_and_Partial_Self_Attention_NeurIPS_2025.png)

The benchmark numbers are strong. YOLOv10-S is 1.8x faster than RT-DETR-R18 at similar accuracy. YOLOv10-B matches YOLOv9-C with 46% less latency and 25% fewer parameters.

They performed an ablation study (removing certain elements then seeing performance) that explains the remaining performance gap between NMS-free and NMS-based training.

- This gap is most noticeable in the smallest models, which have difficulty distinguishing features compared to larger models.

And the main limitation that is discussed in the paper is that the authors didn't test with large-scale pretraining (e.g., on Objects365), so it's unclear how much of the gains would compound in that setting.

Overall, this is a tightly engineered research that advances the state of the art on multiple fronts simultaneously.

---

### [Level Up Your Career With AI, But Simple PRO](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neurips-2025-paper-awards-the-best-papers-of-2025)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/281697a8-c915-435c-a68b-43374b671698/aibutsimple_academy.png)

Supporting us by purchasing a PRO membership will get you:

- Exclusive Hands-on Tutorials With Code
- Full Math Explanations and Beautiful Visualizations
- Ad-Free Reading for an Uninterrupted Learning Experience

[Break into AI or sharpen your edge by joining today!](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neurips-2025-paper-awards-the-best-papers-of-2025)

---

## Very Deep Networks For Self-Supervised RL

Paper: [1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities](https://arxiv.org/abs/2503.14858?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neurips-2025-paper-awards-the-best-papers-of-2025)

Wang, Javali, Bortkiewicz et al., Princeton / Warsaw - NeurIPS 2025 - Best Paper

For years, the standard wisdom in reinforcement learning (RL) has been that networks should be shallow. Typically, RL networks are two to five layers, which is the norm across virtually every major codebase.

There’s a pretty good reason why it is this way, since RL provides very sparse feedback signals, so training deep networks with so few gradient bits per parameter should be hopeless.

This paper challenges that assumption directly, and the results are striking enough to require a rethink.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/3662625a-815e-4043-af26-76536c4e654a/Copy_of_Input_State.png)

The setup is deliberately constrained, using unsupervised goal-conditioned RL, meaning no reward shaping, demonstrations, or hand-crafted paths.

The agent must explore from scratch and figure out how to reach goals purely from self-supervised learning.

The algorithm of choice is contrastive RL (CRL), which frames the goal-reaching problem as a classification task through an InfoNCE objective.

The architectural recipe for enabling depth is not new itself. The authors use residual connections, layer normalization, and Swish activations, which are all well-known.

However, they are the first to combine them systematically in this RL setting and scale depth to 1024 layers.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/0479b285-278d-4d13-8840-c36d783d9bb6/Copy_of_Contrastive_RL__CRL__Framework.png)

The main results are performance improvements of 2x to 50x over the standard 4-layer baseline, with the largest gains on humanoid-based tasks.

But the more interesting finding is not the magnitude of the improvement but the shape of it. Performance doesn't climb smoothly with depth. Instead, it stays flat and then jumps sharply at specific critical depths.

At depth 16 on humanoid locomotion, the agent goes from falling over to walking upright. At depth 256 on a maze task, it learns to vault over walls. These can be considered as genuinely emergent behaviors and not incremental refinements.

The paper then does what good empirical work should: it tries to understand why this is happening.

- A "collector-learner" experiment cleanly separates exploration from expressivity by having one network collect data while two others (one deep, one shallow) train on the same replay buffer.

The conclusion is that depth improves both, but in a coupled way. Better models explore better, but better exploration is needed to fully use the capacity of better models.

- If you actually want to get hands-on with this paper, check out our [PRO edition](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neurips-2025-paper-awards-the-best-papers-of-2025) for a code implementation for boosted understanding.

The authors also show that deeper networks learn richer representations of the environment (visualzed via Q-value maps in maze environments) and allocate more representational capacity to goal-relevant states.

Although exciting, there is one important caveat: the depth scaling is specific to contrastive RL.

Other standard RL algorithms like SAC or TD3 show zero or negative returns from added depth, which suggests the classification-like structure of the InfoNCE loss is a prerequisite, not just a convenient choice.

The method also doesn't yet transfer to offline RL, which limits its immediate applicability. And the computational cost scales roughly linearly with depth, so the 1024-layer runs are decently expensive.

Still, this paper is a shift in how we should think about network depth in RL.

## Why Diffusion Models Don't Memorize

Paper: [Why Diffusion Models Don't Memorize: The Role of Implicit Dynamical Regularization in Training](https://arxiv.org/abs/2505.17638?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neurips-2025-paper-awards-the-best-papers-of-2025)

Bonnaire, Urfin, Biroli & Mézard, École Normale Supérieure / Bocconi - NeurIPS 2025 - Best Paper Awards

[Diffusion models](https://www.aibutsimple.com/p/diffusion-models-simply-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neurips-2025-paper-awards-the-best-papers-of-2025) can generate photorealistic images from datasets of a few thousand samples without reproducing any of them.

This is deeply puzzling, since a model with millions of parameters trained on thousands of images seems quite overparameterized, and yet it doesn't simply memorize its training dataset.

Existing explanations such as architectural inductive biases (the diffusion process and noise) and finite learning rates are real but partial.

This paper identifies the dominant mechanism, which is an implicit dynamical regularization built into the training process itself.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/515e8de7-8579-41ab-9b0f-463eade23b4e/Diffusion_Training_Process_NeurIPS_2025.png)

The core finding is elegant. During training, two timescales emerge. The first, gen, is when the model starts producing high-quality, novel samples.

The second, mem, is when memorization begins and generated images start resembling specific training examples.

The critical observation is that gen is roughly constant regardless of dataset size, while mem grows linearly with the number of training samples n.

This means that as you increase your dataset, you get a wider window of training times during which the model generalizes perfectly. Early stopping anywhere in this window gives you good generation without memorization.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/eaadea31-aabe-41aa-88a6-8464befe8cf0/Two-Timescale_Training_NeurIPS_2025.png)

The authors establish this scaling law in two ways. First, through controlled experiments: U-Net models trained on downscaled CelebA with training set sizes ranging from 128 to 32,768.

They track both generation quality (FID) and a memorization fraction metric across training, and the linear scaling of mem holds cleanly across all configurations, such as different model widths, batch sizes, and even optimizers (SGD and Adam).

They also rule out the naive explanation that memorization is caused by seeing the same samples repeatedly. Full-batch training (where every sample is seen every step) still shows mem is still proportional to n.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/0ace2e69-3f3c-4ee9-b8b0-d48756d19a5b/Nearest_neighbor_search_in_Training_Set_NeurIPS_2025.png)

Second, and more ambitiously, the authors build a simplified neural network model using random feature neural networks (networks where only the final layer is trained, with random hidden layers) that still captures the training behavior.

This model is simple enough that it is analytically tractable (solvable with math) without simulation. By computing the [eigenvalue spectrum](https://en.wikipedia.org/wiki/Eigenvalues_and_eigenvectors?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neurips-2025-paper-awards-the-best-papers-of-2025) of the correlation matrix of gradient flow, they show that the spectrum splits into two well-separated bulks.

The fast bulk drives generalization (learning the smooth, population-level score function), and the slow bulk drives memorization (fitting the high-frequency, dataset-specific structure).

The gap between these bulks is exactly what produces the two-timescale behavior, and it widens as n increases in the same way the experiments show.

The practical takeaway is clear: for data-scarce applications (medical imaging, scientific data), controlling training time is a simple and effective anti-memorization strategy.

The main open questions are how the absolute values of gen and mem depend on conditioning (e.g., in text-to-image models) and whether the theory extends cleanly to more expressive architectures like diffusion transformers.

But as a piece of analysis, this paper is exceptionally clean, and the central insight is likely to be influential well beyond its immediate setting.

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neurips-2025-paper-awards-the-best-papers-of-2025):

- Sushant Waidande
- Sai Krishna Pashapu
- Bob Boncella
- Demet Yazici

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Want to reach 50000+ ML engineers? Let’s work together:

[WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neurips-2025-paper-awards-the-best-papers-of-2025)

Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

If you like, you can also donate to help our team push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=neurips-2025-paper-awards-the-best-papers-of-2025)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team