Lecture 20

[[lecture-data]]

2024-10-14

Readings

7. Chapter 7

Recall from singular value decomposition we have that for any matrix $A \in M_{m, n}$ with $m \leq n$

A = U Σ W

for some $U \in M_{m}$ unitary, $Σ \in M_{m}$ diagonal, and $W \in M_{m, n}$ with orthogonal columns. Recall that we call the entries of $Σ$ the singular values

Note

The values in $Σ$ for $A = U Σ W$ are uniquely determined.

Proof

$A A^{*} = U Σ W W^{*} Σ^{*} U^{*}$ and so the eigenvalues of $A A^{*}$ are the diagonal entries of $Σ Σ^{*}$ .

Theorem (Singular Value Decomposition)

For any $A \in M_{m, n}$ , there exists $U \in M_{m}$ , $V \in M_{n}$ both unitary and $Σ \in M_{m, n}$ "diagonal" with $min {m, n}$ real, nonnegative, nonincreasing, entries such that

A = U Σ V^{*}

Further, if $A$ is real-valued, we can take each of $U, Σ, V$ to be real.

(we take "diagonal" to mean that the only entries that can be nonzero have the same row and column index)

Note

We can think of this as a generalization of diagonalization/spectral decomposition, but where the loss is that $U$ and $V$ are distinct.
if $A$ is positive semidefinite, then the diagonalization is the singular value decomposition since we have that $A = U Σ U^{*}$ for some unitary $U$ and nonnegative $Σ$ all real.

Proof

If $m \leq n$ , then by the previous definition here we have $A = U Σ W$ where $U \in M_{n}$ unitary, $Σ \in M_{m}$ diagonal, and $W \in M_{m, n}$ with orthogonal columns. Since $m \leq n$ , we add $n - m$ columns of zeros to $Σ$ to make it size $m \times n$ . Then we can append $n - m$ orthogonal rows to $W$ with Gram-Schmidt to get a $V^{*}$

If $n > m$ , then $A^{*} = U Σ V^{*}$ is SVD by the above case. But then $A = V Σ^{T} U^{*}$ is an SVD of $A$ !

(see singular value decomposition)

Corollary - Polar Decomposition

For all $A \in M_{n}$ , there exists a hermitian, positive semidefinite $P \in M_{n}$ and a $W \in M_{n}$ unitary such that

A = P W

Proof

When $n = 1$ , we can write $a = α e^{i θ}$ where $α$ is a PSD 1x1 matrix and $e^{i θ}$ is some unitary matrix (it is a complex number with modulus 1!).

Let $A = U Σ V^{*}$ be its singular value decomposition. Then

A = U Σ U^{*} U V^{*} = (U Σ U^{*}) (U V^{*}) = P W

(see polar decomposition)

Notes on SVD

Say $A \in M_{m, n}$ and $A \in U Σ V^{*}$ its SVD. Suppose there are $k$ non-negative singular values call them $σ_{1}, \dots, σ_{k}$ . Denote $u_{1}, \dots u_{m}$ the columns of $U$ and $v_{1}^{*}, \dots, v_{n}^{*}$ the rows of $V^{*}$ . Then

$A = U Σ V^{*} = \sum_{i = 1}^{k} σ_{i} u_{i} v_{i}^{*}$
$rank (A) = rank (Σ) = k$
$range (A) = span {u_{1}, \dots, u_{k}}$ - easy to see from (1)
$null (A) = span (v_{k + 1}, \dots, v_{n})$ - easy to see from (1)
$range (A^{*}) = span {v_{1}, \dots, v_{k}}$
$null (A^{*}) = span {u_{k + 1}, \dots, u_{m}}$
For any $A \in M_{m, n}$ , we have $| | A | |_{2, 2} := max_{x \in C^{n} \neq 0} \frac{| | A x | |_{2}}{| | x | |_{2}} = σ_{1}$
$| | A | |_{F}^{2} = | | U Σ V^{*} | |_{F}^{2} = | | Σ | |_{F}^{2} = \sum_{i = 1}^{k} σ_{i}$

To see (5) and (6) , recall that $A^{*} = V Σ^{T} U^{*}$ and we simply apply (3) and (4) in this case.

To see (7), note that

| | A | |_{2, 2}^{2} = max_{x \neq 0} \frac{| | A x | |_{2}}{| | x | |_{2}} = max \frac{x^{*} A^{*} A x}{x^{*} x} =^{*, * *} σ_{1}^{2}

Note $*$ is by Rayleigh-Ritz that this equals the largest eigenvalue of $A^{*} A$ .
Then $* *$ is by the fact that the eigenvalues of $A^{*} A = (U Σ V^{*})^{*} (U Σ V^{*}) = V Σ Σ^{*} V^{*}$ - ie the eigenvalues are the squared singular values of $A$ .

recall that $range (A) = {x : A y = x}$

Generalized Inverse

Let $A \in M_{m, n}$ and $B \in M_{n, m}$ . Let $C =$ ${1, 2, 3, 4}$ . Then $B$ is a generalized inverse of $A$ precisely when

$1 \in C ⟹$ $A B A = A$
$2 \in C ⟹$ $A B$ is hermitian
$3 \in C ⟹$ $B A B = B$
$4 \in C ⟹$ $B A$ is hermitian

For each $i \in C$ , we call $B$ an " $i$ -generalized inverse"

if $B$ satisfies 1, 2, then $B$ is a 1-2-generalized inverse

Note

If $A$ is invertible, then $B$ is a 1-2-3-4-generalized inverse.

preview: every matrix has a unique 1-2-3-4-generalized inverse. We call this the Moore-Penrose inverse

Note

$0$ is a 2-3-4 generalized inverse for every matrix...

Note

If $B$ is a 1-3 for $A$ , then $A$ is a 2-4 for $B$ .

(see generalized inverse)

Proposition

Consider $A x = b$ . If $A$ is invertible, then $x = A^{- 1} b$ 🙂

But if $A$ is not square? What do we do? Suppose $B$ is a 1-generalized inverse of $A$ .

Let $A \in M_{m, n}, b \in C^{m}$ , and suppose $B$ is a 1-generalized inverse of $A$ . Then if $A x = b$ is consistent, then $x = B b$ is a solution.

Proof

Say $A z = b$ ie there is a solution for some $z \in C^{n}$ . Then

A (B b) = A B A z = A z = b

(see 1-generalized inverses give solutions to consistent linear systems)