# Deep Evidential Regression, Simply Explained

#103 May 25 Theory

https://www.aibutsimple.com/p/deep-evidential-regression-simply-explained

AI systems are now integrated in high-stakes domains such as healthcare, security, autonomous vehicles, and finance, where incorrect predictions have severe consequences.

However, most machine learning models provide only a single prediction such as a class label or numerical value without conveying how confident the model actually is.

Feed models an input they have never seen before, and they will still output a number, with no indication of a bad answer.

This limitation has motivated the development of uncertainty quantification techniques, where ML/AI models are developed to be able to simultaneously compute confidence along with their predictions.

A couple weeks ago, we looked at [conformal prediction](https://newsletter.aibutsimple.com/p/conformal-prediction-reliable-uncertainty-for-machine-learning), an uncertainty quantification technique that creates guaranteed prediction sets based on quantiles.

Today, we’ll look at another helpful method from this field.

[Deep Evidential Regression](https://arxiv.org/abs/1910.02600?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=deep-evidential-regression-simply-explained) (Amini et al., 2020), published at NeurIPS by researchers at MIT and Harvard, proposes another method to quantify model uncertainty.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b4b0c195-68e2-49f4-8caa-930b553604e8/Evidential_Regression.png)

It trains a standard neural network to output not just a prediction, but uncertainty descriptors driven by probability.

The result is a model that is genuinely uncertain about doubtful outputs, calibrated enough to use in safety-critical settings, and fast enough to run in real time.

Deep Evidential Regression has the capability to understand two different types of uncertainties, which puts it a step above a method we have seen before: [Gaussian Process Regression](https://newsletter.aibutsimple.com/p/an-introduction-to-gaussian-process-regression).

This is considered important because it can provide a higher level of clarity about prediction confidence and, additionally, can also significantly benefit the training process.

---

### What You’ll Learn

1. The two kinds of uncertainty in machine learning
2. The evidential idea (what previous approaches lacked)
3. The crucial regularizer
4. Applications of evidential regression
5. Why evidential regression matters

### What’s Helpful to Know

- [Bayesian Inference](https://en.wikipedia.org/wiki/Bayesian_inference?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=deep-evidential-regression-simply-explained)
  - A statistical inference method that uses Bayes’ theorem to update a prior distribution (a model’s beliefs) given new observations.
- Prior, Posterior, Conjugate Distributions
  - Prior Distribution: The model's initial belief about a parameter before observing any new data.
  - Posterior Distribution: The updated distribution after considering new observations.
  - Conjugate Prior: A convenience where the prior and posterior distributions belong to the same probability family.
- Normal-Inverse-Gamma (NIG) Distribution
  - A higher-level, four-parameter probability distribution. Essentially a distribution over distributions.
- Student’s t-Distribution
  - A probability distribution that looks like a standard normal curve with "heavier tails."

---

### The Two Kinds of Uncertainty

Before understanding the method, it helps to be precise about what uncertainty means for a neural network.

There are two fundamentally different sources of uncertainty.

The first is called aleatoric uncertainty, which is an irreducible noise in the data itself.

For instance, if you are predicting depth from a photograph of a mirror, the mirror contains ambiguous information about depth.

- No amount of additional training data resolves this. The uncertainty is in the world and not in the model.

The second is called epistemic uncertainty. This is the model's own ignorance, stemming from limited training data or from encountering inputs it has never seen before.

If a depth estimation model trained on indoor scenes is suddenly shown an outdoor driving scene, it should be highly uncertain because it is operating outside its experience.

- This uncertainty can, in principle, be reduced by training on more diverse data.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/5a06fcb2-fe4c-4c6d-b7a0-01e7af86c6f2/Aleatoric_Uncertainty_vs._Epistemic_Uncertainty__1_.png)

Most neural networks collapse both uncertainties into a single output and cannot distinguish between them at all.

On the other hand, Deep Evidential Regression learns both from a single forward pass.

---

If you enjoy our content and custom visuals, consider sharing this newsletter with others or [upgrading](http://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=deep-evidential-regression-simply-explained) so we can keep doing what we do.

### [Become an ML Engineer With AI, But Simple PRO](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=deep-evidential-regression-simply-explained)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/963eba00-ab31-4912-8c1f-1470a52f9ecc/aibutsimple_academy__2_.png)

Supporting us by purchasing a PRO membership will get you:

- Exclusive Hands-on Tutorials With Code
- Real-World Interview Questions and Cases (From Real Companies)
- Full Math Explanations and Beautiful Visualizations
- Ad-Free Reading for an Uninterrupted Learning Experience

[Stay Ahead in AI With PRO](https://www.aibutsimple.com/pro?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=deep-evidential-regression-simply-explained)

---

### What Existing Approaches Lack?

The standard tool for epistemic uncertainty in neural networks is [Bayesian inference](https://en.wikipedia.org/wiki/Bayesian_inference?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=deep-evidential-regression-simply-explained).

Rather than learning a single set of weights, a Bayesian neural network maintains a probability distribution over all possible weight values.

Uncertainty in the prediction then comes from sampling many different weight configurations and seeing how much the outputs vary.

This is theoretically elegant but practically difficult.

Computing the true posterior distribution over weights is intractable, so practitioners approximate it either through dropout (randomly zeroing out neurons) or through ensembles (aggregating separate networks).

Both approaches require multiple forward passes to estimate uncertainty, which multiplies inference time by the number of samples. For a model running at 30 frames per second on a robot, that overhead matters.

There is also a deeper issue. When dropout or ensembles encounter out-of-distribution (OOD) inputs, their uncertainty estimates are unreliable.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/1a35462d-1180-4c75-acbf-f3ac0a1f5d45/Copy_of_The_Deeper_Problem.png)

They can be confidently wrong because the uncertainty mechanism depends on sampling from learned weights, which were themselves only trained on in-distribution data.

---

### The Evidential Idea

Deep Evidential Regression takes a different approach rooted in a concept from Bayesian statistics called “evidential priors.”

Rather than placing uncertainty over the network's weights, it places uncertainty directly over the output distribution.

A standard regression network assumes each prediction comes from a Gaussian with some mean and variance. It would then sample an output from this distribution.

However, those are point estimates, which are fixed numbers the network commits to.

Evidential regression asks a different question: what if the mean and variance are themselves uncertain?

What if, instead of committing to one Gaussian, the network learned a distribution over all possible Gaussians?

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/24bd8e58-456d-4453-ad45-82caf7c47247/NIG_Distribution__Evidential_Regression.png)

That higher-level distribution is called the Normal Inverse-Gamma (NIG) distribution.

It is parameterized by four numbers: gamma (γ), nu (ν), alpha (α), and beta (β).

Instead of training a network to make a direct prediction, evidential regression trains a network to output these four parameters.

From them, we can compute our uncertainties and the resulting regression prediction. The quantities follow analytically with a clean interpretation; no sampling required. Let’s get into the math below.

1. Prediction:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f4386acf-f81b-48cb-9a10-7d7ab3fba681/image.png)

The prediction is simply γ, the inferred mean.

2. Aleatoric Uncertainty:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/16965020-1a8c-4327-9262-d06c9aa15e66/image.png)

The aleatoric uncertainty is the expected variance of the data-generating Gaussian. It captures how noisy the observations themselves are.

3. Epistemic Uncertainty:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/41cd242d-9b8e-4831-ae96-d1e75ff7e812/image.png)

The epistemic uncertainty is the variance of the mean estimate. It captures how confident the model is in its own prediction.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c047668c-cfe8-4494-82d4-808def09122d/4_Parameters_of_Evidential_Regression.png)

The critical insight lies in the denominator of the epistemic term. The parameter ν plays the role of a virtual observation count.

- Higher ν means the model has accumulated more evidence in support of its prediction, so epistemic uncertainty falls.
- When the model encounters an out-of-distribution input, it has little evidence supporting any particular prediction (ν stays small), and epistemic uncertainty rises automatically, so no sampling is required.

---

### The Regularizer That Makes It Work

Having the network output four parameters instead of one is not enough on its own.

Without additional pressure during training, the network learns to assign high evidence to every prediction, even wrong ones.

A model that is always confident is just as unreliable as a model that is always uncertain.

The paper addresses this in two steps. First, it derives the correct likelihood term for training.

Because the NIG is a conjugate prior for the Gaussian likelihood, the marginal likelihood of an observation can be computed analytically, and it turns out to be a [Student-t distribution](https://en.wikipedia.org/wiki/Student%27s_t-distribution?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=deep-evidential-regression-simply-explained).

The resulting negative log-likelihood loss is:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2df429ef-e7fc-47fb-a0bd-5f80159e21c3/image.png)

Here, Ω = 2β(1 + ν). This loss maximizes the model evidence by fitting the observations to the evidential distribution.

But on its own it is not enough since the network can satisfy it by inflating evidence on every input.

The second step is a novel regularizer. For each training example, if the model's prediction is wrong, it is penalized in proportion to how much total evidence it claimed.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/fc2f759e-a06d-4eba-ba91-d2282f2f4eab/image.png)

Above, the quantity (2ν + α) is the total evidence, summing the virtual observation counts.

Large evidence of an accurate prediction incurs no penalty. Large evidence of an inaccurate prediction incurs a heavy one.

This forces the network to be stingy with its confidence, distributing evidence only where the data genuinely supports it.

Not only does this regularizer greatly improve generalization, achieving high regression performance, butit also is a non-trivial contribution.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/3531376d-7d17-4ffe-98c8-73d8f3ea61cf/Evidential_Regression_Regularizer__1_.png)

Prior evidential methods for classification had a clean mathematical handle for regularization: the KL divergence to a uniform Dirichlet prior.

For continuous regression, that clean method does not exist. The paper proves that the KL divergence between a NIG and a zero-evidence NIG prior is mathematically undefined, then introduces the direct error-weighted term for regularization.

The final training objective combines both terms:

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/5ed3aa77-9c04-4885-8b28-403996a55fe5/image.png)

The coefficient λ trades off tight data fitting against appropriate uncertainty inflation.

Setting λ to 0 produces an overconfident model. Setting it too high over-inflates uncertainty everywhere. In practice, λ = 0.01 works well for most regression tasks.

---

### Evidential Regression In Practice

The paper demonstrates the method on two types of tasks.

The first is a suite of nine standard regression benchmarks.

Evidential regression matches or outperforms dropout and ensembles on accuracy (RMSE), is the top performer across all nine datasets on calibration (negative log-likelihood), and runs at roughly one-quarter the inference time.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/0320b880-b760-498c-a07f-af3f9c07df9a/image.png)

The speed advantage is structural. Evidential regression requires only one forward pass versus five or more samples for the baselines.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/30df4950-4d33-401c-8e47-5cbe9f38bf7d/Evidential_Regression_Benchmarks.png)

The second is monocular depth estimation. Given a single RGB photograph of an indoor scene, the network predicts depth at every pixel simultaneously.

Evidential regression outputs four maps instead of one (one per NIG parameter), but the inference cost is still a single forward pass.

The results here are particularly revealing. When the depth model trained on indoor scenes is tested on outdoor driving footage from a completely different dataset, its epistemic uncertainty rises sharply and cleanly (it knows it’s lost).

Dropout and ensembles show the same general trend but with less separation between in-distribution and out-of-distribution uncertainty.

On adversarially perturbed inputs, epistemic uncertainty tracks the growing prediction error closely as perturbation strength increases.

Calibration tells the full story most directly.

A well-calibrated model is one where a stated 90% confidence interval actually contains the true value 90% of the time.

Dropout achieves a calibration error of 0.126. Ensembles improve to 0.048, and evidential regression reaches 0.033.

---

### Why Evidential Regression Matters

The appeal of deep evidential regression is in its structure. Every prediction comes with a principled decomposition into two different kinds of uncertainty, each with a clear interpretation and each useful for different downstream decisions.

Aleatoric uncertainty tells you where the data is noisy and no further training will help.

Epistemic uncertainty tells you where the model is ignorant and more data (or a different architecture) might fix things.

A robot that can tell the difference between these two situations is a robot that can fail more gracefully and recover more intelligently.

The method also fits naturally into existing deep learning infrastructure.

It requires no architectural changes beyond adding three output neurons per prediction target and adjusting the loss function.

Training time is comparable to standard regression. Inference time is strictly faster than any sampling-based alternative.

For neural networks being deployed in domains where a wrong confident answer is more dangerous than an honest admission of uncertainty, that combination of properties is exactly what is needed.

---

Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=deep-evidential-regression-simply-explained):

- Sushant Waidande
- Sai Krishna Pashapu (AI Developer, BTech, [LinkedIn](https://www.linkedin.com/in/sai-krishna-pashapu-997358283/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=deep-evidential-regression-simply-explained), [Github](https://github.com/saikrishna-pashapu?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=deep-evidential-regression-simply-explained))
- Bob Boncella
- Demet Yazici

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

Want to reach 8000+ ML engineers? Advertising Inquiries:

[WORK WITH US 📈](https://tally.so/r/A7Dgdk?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=deep-evidential-regression-simply-explained)

Feedback or inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com).

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team