Class notes pgs 2-7 (1.1 - 1.3)

• Random vector theory
• Properties of Gaussian random vectors
• Concentration inequalities

Let $x\in {\mathbb{R}}^d$ be a random vector. We say that $x$ (or its law $\text{Law}(x)$) is orthogonally invariant if for each (deterministic) orthogonal matrix $Q\in \mathcal{O}(d)$ we have

The law is the distribution of the random vector.
We sometimes use "model" instead of "law" or "distribution" to remind us that our choice includes assumptions and judgements.

• We are modelling a situation we might encounter in applications

There exists a unique probability measure supported on ${\mathbb{S}}^{d-1}$ that is orthogonally invariant. We denote this $\text{Uniform}({\mathbb{S}}^{d-1})$

Suppose that $x$ is orthogonally invariant. Then

• $||x||$ is independent of $\frac{x}{||x||}$
• $\text{Law}\left(\frac{x}{||x||}\right)=\text{Uniform}({\mathbb{S}}^{d-1})$.

• What do the entries look like for a vector $x\sim \text{Uniform}({\mathbb{S}}^{d-1})$ ?
• What are the magnitude and direction of an iid random vector ?

The multivariate normal or Gaussian $\mathcal{N}(\mu ,\mathrm{\Sigma })$ for parameters $\mu \in {\mathbb{R}}^d$ and (positive semidefinite) $\mathrm{\Sigma }\in {\mathbb{R}}^{d\times d}$ is the probability measure with density

With respect to the Lebesgue measure on the row space of $\mathrm{\Sigma }$ and where

• ${\mathrm{\Sigma }}^{†}$ is the Moore-Penrose inverse
• $\stackrel{\ast }{det}$ is the product of all non-zero eigenvalues

When $\mathrm{\Sigma }$ is invertible,

• $\stackrel{\ast }{det}$ is the ordinary determinant
• ${\mathrm{\Sigma }}^{†}={\mathrm{\Sigma }}^{-1}$
• the Lebesgue measure is on all of ${\mathbb{R}}^d$

If a random vector $x\sim \mathcal{N}(\mu ,\mathrm{\Sigma })$, we call $x$ a Gaussian random vector. If $x\sim \mathcal{N}(0,I_d)$, then we call $x$ a standard Gaussian random vector.

Let $x\sim \mathcal{N}(\mu ,\mathrm{\Sigma })$ be a gaussian random vector. Then $\mu$ is the mean vector and $\mathrm{\Sigma }$ is the covariance matrix of $x$ and

ie, the law of a gaussian random vector is determined by its mean and covariance (or its linear and quadratic moments)

Let $x\in {\mathbb{R}}^d\sim (\mu ,\mathrm{\Sigma })$ be a gaussian random vector, $A\in {\mathbb{R}}^{d^{\prime }\times d}$, and $b\in {\mathbb{R}}^{d^{\prime }}$. Then $Ax+b$ is also a gaussian random vector with

ie, gaussian random vectors are closed under linear transformations.

Suppose $x\sim \mathcal{N}(0,I_d)$ is a standard gaussian random vector. For any $a\in {\mathbb{S}}^{d-1}$, we have

In particular, this does not depend on $a$ and $x$ is orthogonally invariant

For $Q\in \mathcal{O}(d)$, we have (from gaussian random vectors are closed under linear transformations) that

Now, consider a special case where $a$ is the first row of $Q$ (and we can find an orthogonal resulting $Q$ via Gram-Schmidt). Then we see that the law is independent of $a$ as desired.

If $x\sim \mathcal{N}(0,I_d)$, then $\text{Law}\left(\frac{x}{||x||}\right)=\text{Unif}({\mathbb{S}}^{d-1})$

The result follows immediately from standard gaussian random vectors are orthogonally invariant and the independence proposition for orthogonally invariant distribution on the unit sphere.

The direction of a gaussian random vector is uniformly distributed.

Let $x\sim \mathcal{N}(0,I_d)$ be a standard gaussian random vector and $t\ge 0$. Then

This proof uses the Chernoff Method, which is used to prove concentration inequalities like the Chebyshev, Hoeffding, Bernstein, Azuma-Hoeffding, and McDiarmid. It is a more general form of a "nonlinear Markov" inequality

We deal with only one side of the distribution; the desired inequality is achieved by simply multiplying by 2 to account for the other tail.

Note that $\mathbb{E}[x_i^2]=1$, so define $s_i=x_i^2-1$. Then

Where

• $(\ast )$ is from Markov's Inequality
• $(\ast \ast )$ is because the $x_i$ are independent

Now, define

As the cumulant generating function (wikipedia) of $x_1^2-1$ (which has expectation 0). Note that $\mathbb{E}[\exp (\lambda x_1^2)]$ is finite if and only if $\lambda <\frac{1}{2}$ and in this case

Where we get $(†)$ via Taylor expansion. This yields the bound $\psi (\lambda )\le 2{\lambda }^2$ for $\lambda \le \frac{1}{4}$

Assuming $\lambda \le \frac{1}{4}$, we have

Now, $2d{\lambda }^2-\lambda t$ is a convex quadratic function of $\lambda$, and is thus minimized at ${\lambda }^{\ast }=\frac{t}{4d}$. We want to set $\lambda ={\lambda }^{\ast }$, but we must ensure that $\lambda \le \frac{1}{4}$ to maintain our desired bound. Thus, we have 2 cases:

• If $t\le d$, then ${\lambda }^{\ast }\le \frac{1}{4}$ and we can safely set $\lambda ={\lambda }^{\ast }$.

• If $t\ge d$, then $\frac{t}{4d}\nleqq \frac{1}{4}$, so we instead set $\lambda =\frac{1}{4}$.

Which is the desired result

The important part of the proof is finding some $\psi (\lambda )=\mathcal{O}({\lambda }^2)$ for a bounded $\lambda$. This means that the random variance is subexponential (a weaker version of being subgaussian).

This result is characteristic of sums of $d$ iid subexponential random variables. Sums of this type have "Gaussian tails" scaled by $\exp (-\frac{c{t}^2}{d})$ up to a cutoff of $t\sim d$, after which they have "exponential tails" scaled by $\exp (-ct)$

• Bernstein's inequality is general tool for expressing this type of behavior (see notes pg. 7 for more + reference)

The {asa||direction||thing} of a {sas||gaussian random vector} is {asa||uniformly distributed on ${\mathbb{S}}^{d-1}$||characteristic of thing}