# Conformal Prediction: Reliable Uncertainty for Machine Learning

#94 Mar 23 Theory

https://www.aibutsimple.com/p/conformal-prediction-reliable-uncertainty-for-machine-learning

Machine learning systems are now increasingly deployed in high-stakes domains such as healthcare, autonomous vehicles, finance, and security.

In these environments, incorrect predictions can have serious consequences. Yet most machine learning models provide only a single prediction such as a class label or numerical value without conveying how confident the model actually is.

This limitation has motivated the development of uncertainty quantification techniques, which aim to measure how reliable a model’s predictions are.

In this issue, we will look at one of the most powerful and widely applicable uncertainty techniques called conformal prediction, a statistical framework that produces prediction sets with guaranteed coverage.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/623ba83c-9824-43b3-afc2-6a97c49a8c77/Conformal_Prediction__4_.png)

Unlike the previous methods, which rely on the distribution of the output data, conformal prediction works as a “distribution-free” method.

The paper “[A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://arxiv.org/abs/2107.07511?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=conformal-prediction-reliable-uncertainty-for-machine-learning)” (Angelopoulos and Bates, 2021) provides a practical and accessible introduction to this idea.

It explains how conformal prediction can be applied to basically any machine learning model while providing rigorous statistical guarantees.

### The Problem with Standard Predictions

Traditional machine learning models output a single prediction.

For example:

- A classifier outputs a label: “cat”
- A regression model outputs a number: house price = $450,000

However, these predictions do not express uncertainty or how confident the model is in its predictions.

Consider a medical diagnosis model. A prediction of “disease present” might be correct 95% of the time or only 55% of the time. Without a measure of uncertainty, it is difficult to determine how much trust should be placed in the model.

Standard approaches attempt to address this using:

- Bayesian models
- Bootstrapping
- [Ensemble methods](https://aibutsimple.beehiiv.com/p/ensemble-models-simply-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=conformal-prediction-reliable-uncertainty-for-machine-learning)

While useful, these methods often rely on assumptions about the data distribution or the model itself. When those assumptions fail, the uncertainty estimates may become unreliable.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/3f4ce857-c4a8-4101-a416-19f1219d71b3/Conformal_Prediction_Confidence.png)

Conformal prediction takes a different approach: it provides guarantees without making strong assumptions about the underlying distribution.

### The Core Idea of Conformal Prediction

The central concept behind conformal prediction is simple but powerful. Instead of predicting a single answer, the algorithm outputs a set of possible answers.

For example:

- Classification prediction: Prediction set = {cat, fox}
- Regression prediction: Prediction interval = [420000, 470000]

These sets are constructed so that the true label lies inside the set with high probability, such as 90% or 95%.

The remarkable property of conformal prediction is that this guarantee holds regardless of the underlying data distribution, as long as the data are exchangeable (roughly meaning the samples are drawn from the same process).

This makes conformal prediction a distribution-free method.

### The Calibration Idea

The method works by splitting data into two parts:

1. Training set
2. Calibration set

The training set is used to train any machine learning model normally. This could be:

- [A neural network](https://aibutsimple.beehiiv.com/p/neural-networks-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=conformal-prediction-reliable-uncertainty-for-machine-learning)
- A random forest
- [A gradient boosting model](https://aibutsimple.beehiiv.com/p/boosting-models-in-machine-learning-simply-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=conformal-prediction-reliable-uncertainty-for-machine-learning)
- [A linear model](https://aibutsimple.beehiiv.com/p/linear-regression-mathematically-explained?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=conformal-prediction-reliable-uncertainty-for-machine-learning)

Conformal prediction treats the model as a black box. After training, the calibration set is used to measure how large prediction errors tend to be.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/240807d5-0762-4bf9-8670-2ac0432bdb76/Conformal_Prediction_Calibration_Split_Flow__1_.png)

For each calibration example:

1. The model makes a prediction
2. The error between prediction and true label is computed
3. These errors are collected into a distribution

The algorithm then determines a threshold based on these errors. This threshold determines how large the prediction set should be in order to achieve the desired confidence level.

---

### [Level Up Your Career With AI, But Simple PRO](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=conformal-prediction-reliable-uncertainty-for-machine-learning)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/281697a8-c915-435c-a68b-43374b671698/aibutsimple_academy.png)

Supporting us by purchasing a PRO membership will get you:

- Exclusive Hands-on Tutorials With Code
- Real-World Interview Questions and Cases (From Real Companies)
- Full Math Explanations and Beautiful Visualizations
- Ad-Free Reading for an Uninterrupted Learning Experience

[Break into AI or sharpen your edge by joining today!](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=conformal-prediction-reliable-uncertainty-for-machine-learning)

---

### Example: Conformal Regression

Consider a regression problem, where the model predicts:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/81118306-5f82-4978-bb6b-3ae94eb1b873/image.png)

For each calibration example, we compute the absolute error:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/105f6751-55b4-4522-9cf7-a9e96d7dcc70/image.png)

These values form a list of calibration residuals. If we want 90% coverage, we take the 90th percentile of these residuals:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/e0a61460-a08a-4578-b703-b0175d9617d2/image.png)

Then for a new input (x), we output the prediction interval:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/47cc2348-7ffb-4201-9636-206c418fd7af/image.png)

This interval is guaranteed to contain the true value at least 90% of the time.

The key insight is that we do not need to assume anything about the distribution of errors.

### Conformal Prediction for Classification

In classification problems, the goal is slightly different. Instead of predicting a single label, we produce a set of plausible labels.

- For example, a prediction set = {dog, wolf}

This happens when the model cannot confidently distinguish between two classes.

The algorithm assigns a score to each label that measures how compatible it is with the input.

Then it constructs a prediction set containing all labels whose score passes a calibrated threshold.

As a result, ambiguous examples naturally produce larger prediction sets, while easy examples produce smaller sets. (Important)

### Adaptive Prediction Sets

A major improvement described in the paper is adaptive prediction sets.

Instead of using a fixed-size prediction set, the algorithm adapts the set size based on model confidence. For instance:

- High-confidence example: Prediction set = {cat}
- Ambiguous example: Prediction set = {cat, fox, raccoon}

This allows the system to communicate uncertainty more effectively.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/0a0c0236-23f7-4d2e-8d42-bdbbdb59714b/Adaptive__RAPS__vs_fixed_conformal_prediction.png)

Rather than forcing a single answer, the model acknowledges when multiple answers are plausible.

### Distribution-Free Guarantees

One of the most important properties of conformal prediction is its statistical guarantee.

If we request 90% coverage, the algorithm guarantees that:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/bdb82fd3-24b7-4e28-a509-fe1738df3cb6/image.png)

This guarantee holds without assuming:

- Gaussian noise
- Specific data distributions
- Correct model specification

This makes conformal prediction extremely robust in real-world applications where data distributions may be complex or unknown, which is usually the case.

The guarantee is also non-asymptotic, meaning it holds even with finite datasets.

### Applications of Conformal Prediction

Because conformal prediction can wrap around any model, it has broad applicability across machine learning.

#### Computer Vision

In image classification, conformal prediction can produce sets of possible labels like {cat, fox}.

If the image is blurry or ambiguous, the model expresses uncertainty instead of committing to one label.

#### Medical Diagnosis

In medical settings, uncertainty quantification is critical. A conformal system might produce a set of possible diagnoses instead of a single prediction.

Doctors can then use this information to guide further testing or analysis.

#### Autonomous Driving

Self-driving systems must detect when they are uncertain. Conformal prediction can provide uncertainty bounds for:

- Object detection
- Trajectory prediction
- Sensor interpretation

This allows the system to take safer actions when uncertainty is high.

#### Natural Language Processing

In NLP tasks such as translation or question answering, conformal prediction can produce sets of plausible outputs or confidence intervals.

This helps systems avoid overconfident incorrect predictions.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/6a156d35-8c27-4d8b-b329-e644856bd418/Conformal_Wrapper__Conformal_Prediction_.png)

The paper also explores several extensions of the basic framework, including:

### Conformalized Quantile Regression (CQR)

One limitation of standard conformal regression is that it produces symmetric prediction intervals of the form [f(x) - q, f(x) + q] as mentioned previously.

This implicitly assumes that prediction errors are uniform in all regions of the input space, which is rarely true in practice. In real-world datasets, noise is often heteroscedastic, meaning the uncertainty varies depending on the input.

For example:

- Predicting house prices in stable neighborhoods results in low uncertainty
- Predicting prices in volatile markets results in high uncertainty

A fixed-width interval fails to capture this variation. Conformalized Quantile Regression addresses this by combining:

1. Quantile Regression (to model input-dependent uncertainty)
2. Conformal Calibration (to guarantee coverage)

Instead of predicting a single value f(x), the model predicts two quantiles:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/0396ee95-419c-45f2-b053-5a05ab96ff54/image.png)

These define an initial interval:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/e1ae2974-a526-44b3-80e5-33254e597151/image.png)

However, this interval alone does not guarantee correct coverage. So we calibrate it using conformal methods.

On the calibration set, we compute nonconformity scores:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/8f745707-7ea9-462b-8ead-ff006f28c6ec/image.png)

This measures how much the true value falls outside the predicted interval.

We then compute a quantile q of these scores and adjust the interval:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/15da197d-562e-4150-862e-bdff180b27e0/image.png)

This results in intervals that are:

- Adaptive (wider where uncertainty is high, narrower where it is low)
- Statistically valid (coverage guarantee still holds)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2b7edf4d-cfd5-437e-9717-5e907c41b861/Conformalized_Quantile_Regression__CQR__Conformal_Prediction__2_.png)

In essence, CQR combines the flexibility of learned uncertainty with the rigor of conformal guarantees.

This is one of the most practically useful extensions, especially in regression-heavy domains like finance, energy forecasting, and time-series modeling.

### Why Conformal Prediction Matters

The growing deployment of machine learning in critical systems makes reliability increasingly important.

Traditional machine learning focuses primarily on accuracy. However, accuracy alone is insufficient when decisions have serious consequences.

Conformal prediction introduces a complementary goal: reliable uncertainty quantification.

Instead of pretending the model is always confident, conformal prediction explicitly measures and communicates uncertainty.

This makes machine learning systems more transparent, more trustworthy, and safer in real-world deployment.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/51fcd9aa-f086-41ff-8419-ef0e23d82e2c/Conformal_Prediction_Pipeline__1_.png)

### Conclusion

Conformal prediction provides a simple yet powerful framework for adding statistically rigorous uncertainty estimates to machine learning models.

By producing prediction sets rather than single predictions, the method guarantees that the true answer lies within the set with a user-specified probability.

The most important feature of conformal prediction is that it achieves this guarantee without relying on assumptions about the underlying data distribution.

And because it can be applied to any existing model, it serves as a flexible wrapper for improving reliability across a wide range of applications.

As machine learning continues to expand into safety-critical domains, techniques like conformal prediction will become increasingly important.

They shift the focus from merely producing accurate predictions to building systems that know when they might be wrong, enabling safer and more responsible deployment of artificial intelligence.

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=conformal-prediction-reliable-uncertainty-for-machine-learning):

- Sushant Waidande
- Sai Krishna Pashapu
- Bob Boncella
- Demet Yazici
- Yassine Ennaji

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Want to reach 50000+ ML engineers? Let’s work together:

[WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=conformal-prediction-reliable-uncertainty-for-machine-learning)

Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

If you like, you can also donate to help our team push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=conformal-prediction-reliable-uncertainty-for-machine-learning)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team