The Law Of Large Number

11 May 2018

Prologue To The Law Of Large Number

Based on the result of the Chebyshev's inequality, the law of large number guarantees the precision of the averaged term quantity of interest would be well approximated to the mean of the term in the sample data.

The Chebyshev's Inequality

For $X$ to be any arbitrary random variable, and for ang given $a>0$:
$\;\;\;\;P(|X-E\lbrack X\rbrack|\ge a)\le \frac {1}{a^{2}}\cdot Var\lbrack X\rbrack$
proof::➀
Please go to the article Hoeffding Inequality v.s. Chebyshev’s Inequality
proof::➁
Or you can see it in Chebyshev’s Inequality

Theorem: The Law Of Large Number

Given random variables $X_1$,$X_2$,…,$X_n$, each is identically independent distributed with mean $\mu$ and variance $\sigma^{2}$.
$\;\;\;\;\lim_{n\rightarrow\infty}P(|\overline {X_n}-E\lbrack \overline {X_n}\rbrack|\ge \varepsilon)$=$0$

proof:
➀trivially, we know that $E\lbrack \overline {X_n}\rbrack$=$\mu$, $Var\lbrack \overline {X_n}\rbrack$=$\frac {\sigma^{2}}{n}$.
➁by using the Chebyshev's inequality, we have:
$P(|\overline {X_n}-E\lbrack \overline {X_n}\rbrack|\ge \varepsilon)$
=$P(|\overline {X_n}-\mu|\ge \varepsilon)$
$\le \frac {Var\lbrack \overline {X_n}\rbrack}{\varepsilon^{2}\cdot n}$
=$\frac {\sigma^{2}}{\varepsilon^{2}\cdot n}$, for any $\varepsilon>0$
➂$\lim_{n\rightarrow\infty}P(|\overline {X_n}-E\lbrack \overline {X_n}\rbrack|\ge \varepsilon)\le\lim_{n\rightarrow\infty}\frac {\sigma^{2}}{\varepsilon^{2}\cdot n}$, as $n\rightarrow\infty$, it holds that $\lim_{n\rightarrow\infty}P(|\overline {X_n}-E\lbrack \overline {X_n}\rbrack|\ge \varepsilon)$=$0$

Cautions must be made that the law of large number might fail if the expect value is infinite!!!

Example: Illustration Of The Law Of Large Number For Discrete Random Variable

The execution of a random variable would be the event.

In this example, we’d like to know the probability of this event occurrence.
➀if we treat $p$=$P(X\in C)$, where $C=(a,b]$ for $a<b$, we’d like to estimate this probability $p$. The most usual way is by using the relative frequency of $X_i\in C$ among $X_1$,$X_2$,…,$X_n$, that is the number of times $X_i$ hits $C$ divided by $n$.
➁we then define the random variable by
$Y_i$=$\left\{\begin{array}{l}1, for X_i\in C\\0, for X_i\not\in C\end{array}\right.$
; where above $X_i$ is just event, $Y_i$ just indicates whether $X_i$ hits $C$ for all $i$. Each execution of $Y_i$ would we obtain the event $X_i$ occurrence probability, and is only determined by $X_i$, therefore, $Y_i$ is identically independent distributed.
➂the expect value and variance of each $Y_i$:
$E\lbrack Y_i\rbrack$=$1\cdot P(X\in C)$+$0\cdot P(X\not\in C)$=$p$
$Var\lbrack Y_i\rbrack$=$E\lbrack (Y_i-E\lbrack Y_i\rbrack)^{2}\rbrack$=$E\lbrack Y_i^{2}\rbrack$-$E^{2}\lbrack Y_i\rbrack$=$p-p^{2}$=$p\cdot(1-p)$
➃the relative frequency is in this expression:
$\overline {Y_n}$=$\frac {Y_1+Y_2+…+Y_n}{n}$, then
$E\lbrack \overline {Y_n}\rbrack$=$\sum_{i=1}^{n}\frac {E\lbrack Y_i\rbrack}{n}$=$p$
$Var\lbrack \overline {Y_n}\rbrack$=$Var\lbrack \sum_{i=1}^{n}\frac {Y_i}{n}\rbrack$=$\frac {p\cdot(1-p)}{n}$
➄then, $\lim_{n\rightarrow\infty}P(|\overline {Y_n}-p|\ge \varepsilon)$=$0$, for any $\varepsilon>0$, since it is upper bounded to $\frac {p\cdot(1-p)}{\varepsilon^{2}\cdot n}$.

Example: Illustration Of The Law Of Large Number For Continuous Random Variable

We next have a look at the continuous case.
➀consider that $f$ is a PDF with respect to $F$, which is a CDF. Choose $C=(a-h,a+h]$, where $h$ is some very small positive value.
➁by above example equation, it holds to have:
$\overline {Y_n}\approx p$
=$P(X\in C)$
=$\int_{a-h}^{a+h}f(x)\operatorname dx$
$\approx 2\cdot h\cdot f(a)$
➂then, $f(a)\approx\frac {\overline {Y_n}}{2\cdot h}$, which is just the number of times $X_i$ hits $C$ for $i<n$, divided by the length of $C$.