Recall Basic Probability

Overview

This note recall some basic stuffs of probability, include:

• Basic concepts of joint distribution and conditional distribution
• The propagation of uncertainty

Joint and Conditional Distribution

Consider the continuous single variable case, to describe the uncertain behavior, the concept of random variable is introduced, and CDF, a single variable function, is used to describe the probability of certain event happens between an interval. Then PDF is used to describe the density of the CDF, so that we could use calculus to form the whole theory of probability.

Move on to the multi-variable case, because more than one random event happens, we use joint distribution to describe it, namely a multi-variate function, e.g. instead of a density function with only one input $p(x)$, now we have $p(x,y)$ a density function with two inputs. The events themselves could affect each other or completely independent from each other. That’s why we introduce the idea of covariance, from $E[(x-E[x])(x-E[x]{)}^T]$ to $E[(x-E[x])(y-E[y]{)}^T]$.

Conditional distribution and joint distribution are quite similar, conditional distribution is still a multi-variate function, e.g. $p(x|y)$ is a density function with two inputs, but it would be normalized by a factor $p(y)$. But it’s nice to think it as a function of $x$ with some fixed $y$, that would show the intrinsic motivation of conditional probability, we would use this perspective to derive the conditional distribution of the Gaussian distribution.

The Propagation of Uncertainty

We have a random variable $X$ and $Y$, and we have $y=ax$. In the lens of probability, if we know $X\sim \mathcal{N}(\mu ,{\sigma }^2)$, how to calculate the probability of $Y$. Let’s substitute $x=y/a$ into the PDF of $X$.

$$\begin{array}{r}p(Y/a=y/a)=C\exp (-(y/a-{\mu }_x{)}^2/{\sigma }_x^2)\end{array}$$

here $C$ is the constant term, then rearrange it we have

$$\begin{array}{r}p(Y/a=y/a)=C\exp (-(y-{\mu }_y{)}^2/{\sigma }_y^2)\end{array}$$

where ${\mu }_y=a{\mu }_x$, ${\sigma }_y^2=a^2{\sigma }_x^2$. Extend to the multivariable case, $y=Ax$, assume $A$ to be invertible

$$\begin{array}{r}p(A^{-1}Y=A^{-1}y)=C\exp (-(A^{-1}y-{\mu }_x{)}^T{\Sigma }_x^{-1}(A^{-1}y-{\mu }_x))\end{array}$$

rearrange we have,

$$\begin{array}{r}p(A^{-1}Y=A^{-1}y)=C\exp (-(y-{\mu }_y{)}^T{\Sigma }_y^{-1}(y-{\mu }_y))\end{array}$$

where ${\mu }_y=A{\mu }_x$ and ${\Sigma }_y=A{\Sigma }_x{A}^T$. It’s easy to know that if we shift $y$ to $y=ax+b$, then only the mean value would change but not the covariance, namely the result would be ${\mu }_y=A{\mu }_x+b$.

Remark

We derive the result but what does the result tell us? First of all, Gaussian distribution comes from the study of the error of least square approach, mean is the prediction and covariance describe the property of the errors.

Let’s consider only the case $a>0$ or $A$ has only positive eigen values. Let’s say I want to measure the thickness of a book, and with the tape the thickness of one book follows the Gaussian distribution. And we want to know about the thickness of 10 books, we can think that our prediction is 10 * the thickness of one book, but the accumulated error would be 10 times bigger. Since covariance is somehow like the square of the error, so it’s 100 times bigger. But if we want to estimate half-of a book, the prediction would 0.5 * the thickness of one book, and the error would cut half.