Lecture 19

[[lecture-data]]

2024-10-11

Readings

4. Chapter 4

Finishing up today

Corollary

Suppose $A, B \in M_{n}$ are hermitian. Then the vector of eigenvalues for $A + B$ , call this $λ (A + B)$ , majorizes the vector $λ (A) + λ (B)$ (the vector of eigenvalues of $A$ plus the vector of eigenvalues of $B$ ).

Proof

For all $k \in {1, 2, \dots, n}$ , we have

\begin{aligned} \sum_{i = 1}^{k} λ_{i} (A + B) & = min_{U \in M_{n, k} orthonormal} Tr U^{*} (A + B) U \\ = min_{U \in M_{n, k} orthonormal} [Tr (U^{*} A U) + Tr (U^{*} B U)] \\ \geq min_{U \in M_{n, k}} Tr (U^{*} A U) + min_{U \in M_{n, k}} Tr U^{*} B U \\ = \sum_{i = 1}^{k} λ_{i} (A) + \sum_{i = 1}^{k} λ_{i} (B) \\ = \sum_{i = 1}^{k} λ_{i} (A) + λ_{i} (B) \end{aligned}

When $n = k$ , we get precisely that

\sum_{i = 1}^{n} λ_{i} (A + B) = Tr (A + B) = Tr (A) + Tr (B) = \sum_{i = 1}^{n} λ_{i} (A) + λ_{i} (B)

(see the eigenvalues of the sum of hermitian matrices majorize the sum of their individual eigenvalues)

Theorem (Hadamard's Inequality)

Let $A \in M_{n}$ be positive semidefinite (aka hermitian). Then

det A \leq \prod_{i = 1}^{n} a_{i i}

Proof

For all $i$ , we have $0 \leq λ_{1} (A) \leq a_{i i}$ since $A$ is positive semidefinite and by interlacing 2. Thus we know $\prod_{i = 1}^{n} a_{i i} \geq 0$ . If $A$ is singular ie $det A = 0$ then the result is immediate.

If $A$ is not singular, then $A$ is positive definite and thus $0 < λ_{1} (A) \leq a_{i i}$ for all $i$ . Let $D$ be an (invertible) diagonal matrix with diagonal entries $d_{i i} = \frac{1}{\sqrt{a_{i i}}}$ .

Note that $D A D$ is hermitian and positive definite. For all $x \in C^{n} \neq 0$ , we have $x^{*} D A D x = x^{*} D^{*} A D x = y^{*} A y > 0$ since $A$ is positive definite.

\begin{aligned} \frac{det A}{\prod_{i = 1}^{n} a_{i i}} & = det (D^{2}) det A \\ = det D det D det A \\ = det D A D \\ = \prod_{i = 1}^{n} λ_{i} (D A D) \\ \leq {[\frac{1}{n} \sum_{i = 1}^{n} λ_{i} (D A D)]}^{n} (*) \\ = {[\frac{1}{n} Tr D A D]}^{n} (* *) \\ = {(\frac{n}{n})}^{n} \\ = 1 \end{aligned}

$(*)$ is due to the arithmetic-geometric mean inquality and $(* *)$ is because the eigenvalues of $D A D$ will be all 1s. This is because when we multiply $A$ by $D$ on each side, we get that the diagonals are equal to $1$ , and then
(see Hadamard's Inequality)

Arithmetic-Geometric Mean Inquality

Let $x_{i}, \dots, x_{n}$ be some collection of numbers.
The arithmetic mean is $μ_{A} = \frac{\sum_{i = 1}^{n} x_{i}}{n}$
The geometric mean is $μ_{G} = {(\prod_{i = 1}^{n} x_{i})}^{1 / n}$

And we have $μ_{A} \geq μ_{G}$ .
(see arithmetic-geometric mean inquality)

Drawing up a concept map.
When preparing for an exam, it is important to see all of the important ideas and try and see how they fit together.

In this chapter, Courant-Fisher is the main central theorem.

a special case is Rayleigh-Ritz, which we proved by using that the field of values for a normal matrix is the convex hull of the spectrum
this also gave us Weyl's Theorem, a continuity result about adding a perturbing matrix and how this changes the eigenvalues of the original matrix.
We also used this to get interlacing theorem 1 which gives us bounds on the eigenvalues of sums of hermitian matrices
and interlacing 2, which gives us bounds on the eigenvalues of principal submatrices.
- this gave us that the diagonal elements of a hermitian matrix majorize its eigenvalues
- and also the sum of the first least eigenvalues is the minimum of the trace of orthonormal multiplications
- this also gave us Hadamard's Inequality, which is the same as the above diagonal elements of a hermitian matrix majorize its eigenvalues

7. Chapter 7

Singular Value Decomposition

Let $A \in M_{m, n}$ such that $m \leq n$ . Then there exists $U \in M_{n}$ unitary, $Σ \in M_{m}$ diagonal, and $W \in M_{m, n}$ with orthonormal rows such that

A = U Σ W

WLOG, we assume the diagonal elements of $D$ are nonnegative and ordered in a non-increasing.

Proof

Note that $A A^{*}$ is hermitian and therefore positive semidefinite. So there exists $U \in M_{n}$ unitary and $D \in M_{n}$ diagonal such that $A A^{*} = U D U^{*}$ by the spectral theorem for hermitian matrices. Call the columns of $U = [u_{1} | u_{2} | \dots | u_{m}]$

Say each of the elements in $D$ are called $σ_{i i}^{2} \geq 0$ (we can do this because each eigenvector of $A A^{*}$ is real).

Say $A$ is rank $k$ . Then let $σ_{k} > 0, σ_{k + 1} = 0$ and define $Σ$ to be the $m \times m$ matrix containing the ordered $σ_{i}$ s.

For $i = 1, \dots, k$ , define the $i$ th row of $W$ to be $\frac{1}{σ_{i}} u_{i}^{*} A$ . These are orthonormal since for all $i, j$ we have $\frac{(}{1} σ_{i} u_{i}^{*} A) \frac{1}{σ_{j} u_{j}^{*} A} = u_{i}^{*} A A^{*} u_{j} = \frac{1}{σ_{i} σ_{j}} D_{i j}$ and this is $1$ if $i = j$ and $0$ otherwise.

For $i = k + 1, \dots, m$ , define the rows of $W$ any way you like, so long as they are orthonormal and orthogonal to the first $k$ rows of $W$ . And thus we are done! We have

U^{*} A = Σ W ⟹ A = U Σ W

For rows $1, \dots, k$ equality holds by construction. For rows $k + 1, \dots, m$ , we have $A A^{*} u_{i} = 0$ since each $u_{i}$ is an eigenvector with eigenvalues $0$ for $A A^{*}$ . Thus we have $u_{i}^{*} A A^{*} u_{i} = 0 = | | u_{i}^{*} A | |_{2}^{2} ⟹ u_{i}^{*} A = 0 =$ the $i$ th row of $W$ . So LHS = RHS = 0.

Note that if $A$ is real, we can take $U, Σ, W$ real.

(see singular value decomposition)