Lecture 19

[[lecture-data]]

2024-10-11

Readings

• a

4. Chapter 4

Finishing up today

Corollary

Suppose $A,B\in M_n$ are hermitian. Then the vector of eigenvalues for $A+B$, call this $\lambda (A+B)$, majorizes the vector $\lambda (A)+\lambda (B)$ (the vector of eigenvalues of $A$ plus the vector of eigenvalues of $B$).

$k\in \{1,2,\dots ,n\}$, we have

For all

\sum_{i=1}^k \lambda_{i}(A+B) &= \min_{U \in M_{n,k} \text{ orthonormal}} \mathrm{Tr}U^*(A+B)U \\ &= \min_{U \in M_{n,k} \text{ orthonormal}} [\mathrm{Tr}(U^*AU) + \mathrm{Tr}(U^*BU)] \\ & \geq \min_{U \in M_{n,k}} \mathrm{Tr}(U^*AU) + \min_{U \in M_{n,k}} \mathrm{Tr}U^*BU \\ &= \sum_{i=1}^k \lambda_{i}(A) + \sum_{i=1}^k \lambda_{i}(B) \\ &= \sum_{i=1}^k \lambda_{i}(A) + \lambda_{i}(B) \\ \end{aligned}$$ When $n=k$, we get precisely that $$\sum_{i=1}^n \lambda_{i}(A+B) = \mathrm{Tr}(A+B) = \mathrm{Tr}(A) + \mathrm{Tr}(B) = \sum_{i=1}^n \lambda_{i}(A) + \lambda_{i}(B)$$

(see the eigenvalues of the sum of hermitian matrices majorize the sum of their individual eigenvalues)

Theorem (Hadamard's Inequality)

Let $A\in M_n$ be positive semidefinite (aka hermitian). Then $\det A\le {\prod }_{i=1}^n{a}_{ii}$

$i$, we have $0\le {\lambda }_1(A)\le a_{ii}$ since $A$ is positive semidefinite and by interlacing 2. Thus we know ${\prod }_{i=1}^n{a}_{ii}\ge 0$. If $A$ is singular ie $\det A=0$ then the result is immediate.

For all

If $A$ is not singular, then $A$ is positive definite and thus $0<{\lambda }_1(A)\le a_{ii}$ for all $i$. Let $D$ be an (invertible) diagonal matrix with diagonal entries $d_{ii}=\frac{1}{\sqrt{a_{ii}}}$.

Note that $DAD$ is hermitian and positive definite. For all $x\in {\mathbb{C}}^n\ne 0$, we have $x^{\ast }DADx=x^{\ast }{D}^{\ast }ADx=y^{\ast }Ay>0$ since $A$ is positive definite.

\frac{\det A}{\prod_{i=1}^n a_{ii}} &= \det(D^2) \det A \\ &= \det D \det D\det A\\ &= \det DAD \\ &= \prod_{i=1}^n \lambda_{i}(DAD) \\ & \leq \left[ \frac{1}{n} \sum_{i=1}^n \lambda_{i}(DAD) \right]^n \;\;\;\;(*) \\ &= \left[ \frac{1}{n} \mathrm{Tr}DAD \right]^n \;\;\;\;(**) \\ &= \left( \frac{n}{n} \right)^n \\ &= 1 \end{aligned}$$ $(*)$ is due to the [[Concept Wiki/arithmetic-geometric mean inquality]] and $(**)$ is because the eigenvalues of $DAD$ will be all 1s. This is because when we multiply $A$ by $D$ on each side, we get that the diagonals are equal to $1$, and then

(see Hadamard’s Inequality)

Arithmetic-Geometric Mean Inquality

Let $x_i,\dots ,x_n$ be some collection of numbers. The arithmetic mean is ${\mu }_A=\frac{{\sum }_{i=1}^n{x}_i}{n}$ The geometric mean is ${\mu }_G={\left({\prod }_{i=1}^n{x}_i\right)}^{1/n}$

And we have ${\mu }_A\ge {\mu }_G$. (see arithmetic-geometric mean inquality)

Drawing up a concept map. When preparing for an exam, it is important to see all of the important ideas and try and see how they fit together.

In this chapter, Courant-Fisher is the main central theorem.

• a special case is Rayleigh-Ritz, which we proved by using that the field of values for a normal matrix is the convex hull of the spectrum
• this also gave us Weyl’s Theorem, a continuity result about adding a perturbing matrix and how this changes the eigenvalues of the original matrix.
• We also used this to get interlacing theorem 1 which gives us bounds on the eigenvalues of sums of hermitian matrices
• and interlacing 2, which gives us bounds on the eigenvalues of principal submatrices.
  • this gave us that the diagonal elements of a hermitian matrix majorize its eigenvalues
  • and also the sum of the first least eigenvalues is the minimum of the trace of orthonormal multiplications
  • this also gave us Hadamard’s Inequality, which is the same as the above diagonal elements of a hermitian matrix majorize its eigenvalues

7. Chapter 7

Singular Value Decomposition

Let $A\in M_{m,n}$ such that $m\le n$. Then there exists $U\in M_n$ unitary, $\Sigma \in M_m$ diagonal, and $W\in M_{m,n}$ with orthonormal rows such that $A=U\Sigma W$

WLOG, we assume the diagonal elements of $D$ are nonnegative and ordered in a non-increasing.

$A{A}^{\ast }$ is hermitian and therefore positive semidefinite. So there exists $U\in M_n$ unitary and $D\in M_n$ diagonal such that $A{A}^{\ast }=UD{U}^{\ast }$ by the spectral theorem for hermitian matrices. Call the columns of $U=[u_1|u_2|\dots |u_m]$

Note that

Say each of the elements in $D$ are called ${\sigma }_{ii}^2\ge 0$ (we can do this because each eigenvector of $A{A}^{\ast }$ is real).

Say $A$ is rank $k$. Then let ${\sigma }_k>0,{\sigma }_{k+1}=0$ and define $\Sigma$ to be the $m\times m$ matrix containing the ordered ${\sigma }_i$s.

For $i=1,\dots ,k$, define the $i$th row of $W$ to be $\frac{1}{{\sigma }_i}{u}_i^{\ast }A$. These are orthonormal since for all $i,j$ we have $\frac{(}{1}{\sigma }_i{u}_i^{\ast }A)\frac{1}{{\sigma }_j{u}_j^{\ast }A}=u_i^{\ast }A{A}^{\ast }{u}_j=\frac{1}{{\sigma }_i{\sigma }_j}{D}_{ij}$ and this is $1$ if $i=j$ and $0$ otherwise.

For $i=k+1,\dots ,m$, define the rows of $W$ any way you like, so long as they are orthonormal and orthogonal to the first $k$ rows of $W$. And thus we are done! We have $U^{\ast }A=\Sigma W⟹A=U\Sigma W$

For rows $1,\dots ,k$ equality holds by construction. For rows $k+1,\dots ,m$, we have $A{A}^{\ast }{u}_i=0$ since each $u_i$ is an eigenvector with eigenvalues $0$ for $A{A}^{\ast }$. Thus we have $u_i^{\ast }A{A}^{\ast }{u}_i=0=||u_i^{\ast }A|{|}_2^2⟹u_i^{\ast }A=0=$ the $i$th row of $W$. So LHS = RHS = 0.

Note that if $A$ is real, we can take $U,\Sigma ,W$ real.

(see singular value decomposition)