Notes on Probability Primer 3: Random variables

(PP 3.1) Random Variables - Definition and CDF

“Random variables are where the rubber meets the road in probability.” – Mathematical Monk

“Random variable is not necessarily random and it’s not necesaarily a variable.”
– David Duvenaud (TWiML&AI: Composing Graphical Models With Neural Networks)

Definition. Given $(\Omega, \mathscr{A}, P)$, a random variable is a function

s.t. $[X \le x] := \{\omega \in \Omega : X(\omega) \le x\} \in \mathscr{A}$ for all $x \in \mathbb{R}$.

Remark.

1. $X : \Omega \rightarrow \mathbb{R}$ is a “measurable function.”
2. Notation: $X, Y$ random variables / $x, y$ values

Definition. The CDF of a random variable $X$ is the function

given by $F(x) = P[X \le x]$.

Exercise. Prove that $F_X$ is indeed a CDF.

Definition. The distribution of $X$ is the probability measure $P_X$ on $\mathbb{R}$ s.t.

for all $A \in \mathbb{B}(\mathbb{R})$.

Exercise. $P_X$ is a Borel probability measure.

Claim. $P_X$ is the probability measure induced by $F$.

Proof: If we denote by $Q$ the probability measure corresponding to $F$, then

Thus, $Q = P_X$.

(PP 3.2) Types of Random Variables

Definition. A RV $X$ is discrete if $X(\Omega) = \{ X(\omega) : \omega \in \Omega\}$ is countable.
(i.e. $X(\Omega) = \{x_1, x_2, \ldots \}$)

Definition. A RV $X$ has density $f$ if $F(x) = \int_{-\infty}^x f(\mu)d\mu$ for all $x \in \mathbb{R}$
(for some integrable $f: \mathbb{R} \rightarrow [0, \infty]$).

(PP 3.3) Discrete Randome Variables

Definition. The probability mass function (PMF) of a discrete random variable $X$ is the function

such that $p(x) = P[X = x]$.

Remark.

If we set $S = X(\Omega)$, $\sum_{x \in A \cap S} p(x)= P[X \in A] = P_X(A) = P_X(A \cap S) + P_X(A \cap S^c) = \sum_{x \in A \cap S} P[X = x].$ In particular, $P(\mathbb{R}) = \sum_{x \in S} p(x) = 1$.

Notation. When we simply write $X \sim p$, we’re assuming the underlying PMS $(\Omega, \mathscr{A}, P)$. Also write $X \sim F$ and $X \sim Q$, which may casue confusion in some situations.

Example.

1. $X \sim$ Bernoulli$(\alpha)$, $\alpha \in [0,1] , p(1) = \alpha, p(0) = 1- \alpha$
2. $X \sim$ Binomial$(n,\alpha)$, $\alpha \in [0,1], p(k) = \left(\begin{matrix} n \\ k \end{matrix}\right)\alpha^k(1-\alpha)^{n-k}$ where $k \in \{0, 1, \ldots \}$, $\left(\begin{matrix} n \\ k \end{matrix}\right) = \frac{n!}{k!(n-k)!}$.
3. $X \sim$ Geometric$(\alpha)$, $\alpha \in [0,1], p(k)=(1-\alpha)^{k-1}\alpha$, $k \in \{1, 2, \ldots\}$.
4. $X \sim$ Possion$(\lambda)$, $\lambda \ge 0, p(k) = e^{-\lambda}\frac{\lambda^k}{k!}$, $k \in \{0, 1, \ldots, \}$

(PP 3.4) Random Variables with Densities

$X$ has density if $F(x) = P(X \le x) = \int_{-\infty}^x f(u)du$

Notation.

1. We call $f$ the PDF of $X$ and write $X \sim f$.
2. On “notational chaos”
3. Indicator function of $A$:

Examples.
(1) $X \sim {\rm Uniform}(a, b)$ where $a < b$: $f(x) = \frac{1}{b-a}$ on $[a, b]$.
(2) $X \sim {\rm Exponential(\lambda)}$ where $\lambda > 0$: $f(x) = \lambda e^{-\lambda x}$ on $[0, \infty]$ .
(3) $X \sim {\rm Beta(\alpha, \beta)}$ where $\alpha, \beta > 0$: $f(x) = \frac{x^{\alpha-1}(1-x)}{B(\alpha, \beta)}$ on $[0,1]$.

Beta-Bernoulli model:
$X \sim {\rm Bern(\theta)}$ where $\theta$ governed by some Beta prior, i.e., $\theta \sim {\rm Beta}(\alpha, \beta)$ for some $\alpha, \beta$.