1. Summary for basics of probability theory-Prior and Posterior Probability

Prior, likelihood and posterior describe how a belief about a hidden parameter is updated by observed data. Use this summary as the reference for Bayes rule before moving on to sampling, Markov chains and variational inference.

Reading time

2 min

Length

382 words

Updated

Sep 15, 2026

Total views

--

#math/bayes-theorem#math/probability

(1) Basic Concepts

Gaussian Distribution function :

$$\begin{array}{}\text{(1.1)}& f(x)=\frac{1}{\sigma \sqrt{2\pi }}\exp (-\frac{(x-\mu {)}^2}{2{\sigma }^2})\end{array}$$

Take we generate a random number $x$ from parameter $\theta$ as an example :

Posterior probability: If we know the result $x$ at first to guess the reason $\theta$ to generate $x$, we call this as the Posterior probability

$$\begin{array}{}\text{(1.2)}& P(\theta |x)\text{posterior}\end{array}$$

Prior probability : the probability to reach a result $x$ if we use parameter $\theta$.

$$\begin{array}{}\text{(1.3)}& P(x|\theta )\text{prior}\end{array}$$

Integration computation of the mean and variance :

$$\begin{array}{}\text{(1.4)}& \mu ={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}xf(x)dx{\sigma }^2=\text{Var}(x)={\int }_{-\mathrm{\infty }}^{+\mathrm{\infty }}(x-\mu {)}^{2}f(x)dx\end{array}$$

We also have following preferred computational formula :

$$\begin{array}{}\text{(1.5)}& {\sigma }^2=E[X^2]-(E[X]{)}^2={\int }_{-\mathrm{\infty }}^{+\mathrm{\infty }}{x}^{2}f(x)dx-{\mu }^2\end{array}$$

(2) Bayes Formula

From ^[1], we know the basic form of bayes formula is :

$$\begin{array}{}\text{(2.1)}& P(A|B)=\frac{P(B|A)P(A)}{P(B)}\end{array}$$

Then we got the relation between Prior and Posterior Probability :

$$\begin{array}{}\text{(2.2)}& \boxed{{\displaystyle P(\theta |x)=\frac{P(x|\theta )P(\theta )}{P(x)}=\frac{P(x|\theta )P(\theta )}{\sum _i(x|{\theta }_i)P({\theta }_i)}}}\end{array}$$

This is the formula for the calculation of posterior probability, Where $P(x|\theta )$ is get from the Similarity estimation, which is often referred as Similarity Function.

(3) Similarity Estimation for Parameters

1. ML (maximum likelihood/maximum prior)

Maximum likelihood estimation is to maximize $\prod _{i=1}^{n}p(x_i,\theta )$ or following : (see ^[2])

$$\begin{array}{}\text{(3.1.1)}& \boxed{{\displaystyle {\hat{\theta }}_{MLE}={\text{argmax}}_{\theta \in \mathrm{\Theta }}P(x|\theta )}}\end{array}$$

This is also maximum prior estimation.

2. MAP (maximum posterior)

If the prior probability is given, we can use

$$\begin{array}{}\text{(3.2.1)}& {\text{argmax}}_{\theta }P(\theta |x)={\text{argmax}}_{\theta }\frac{P(x|\theta )P(\theta )}{P(x)}\end{array}$$

In that case, since the $x$ is given, we simplify it to :

$$\begin{array}{}\text{(3.2.2)}& {\text{argmax}}_{\theta }P(x|\theta )P(x)\end{array}$$

from this comes often-used formula :

$$\begin{array}{}\text{(3.2.3)}& \text{posterior}\propto (\text{likelihood}\times \text{Prior})\end{array}$$

(4) Bayes Inference

refer to ^[3], Bayes estimation is for estimating the probability that new data appears.

Assume that we have the observed data $X$, and need to predict the appea probability of new data point $\tilde{x}$, then we have :

$$\begin{array}{}\text{(4.1)}& P(\tilde{x}|X)={\int }_{h\sim P(h)}p(\tilde{x}|\theta )p(\theta |X)dh\end{array}$$

(5) Normalizing Constant

For example, we have :

$$\begin{array}{}\text{(5.1)}& p(x)=e^{-\frac{x^2}{2}}x\in (-\mathrm{\infty },\mathrm{\infty })\end{array}$$

then we have

$$\begin{array}{}\text{(5.2)}& {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}p(x)dx=\sqrt{2\pi }\end{array}$$

so the Normalizing Constant (or Normalizing factor) for function (5.1) is $\sqrt{2\pi }$ so that we can reach :

$$\begin{array}{}\text{(5.3)}& \phi (x)=\frac{1}{\sqrt{2\pi }}p(x)=\frac{1}{\sqrt{2\pi }}{e}^{-\frac{x^2}{2}}\end{array}$$

---

1. Probability Theory, Chapter1, Section 6, Law of Total probability and Bayes Formula ↩︎

2. Probability Theory, Chapter7, Section 1, Maximum Similarly Estimation ↩︎

3. Bayes Estimation ↩︎