Lecture 20

[[lecture-data]]

2024-10-14

Readings

• a

7. Chapter 7

Recall from singular value decomposition we have that for any matrix $A\in M_{m,n}$ with $m\le n$ $A=U\Sigma W$ for some $U\in M_m$ unitary, $\Sigma \in M_m$ diagonal, and $W\in M_{m,n}$ with orthogonal columns. Recall that we call the entries of $\Sigma$ the singular values

Note

The values in $\Sigma$ for $A=U\Sigma W$ are uniquely determined.

$A{A}^{\ast }=U\Sigma W{W}^{\ast }{\Sigma }^{\ast }{U}^{\ast }$ and so the eigenvalues of $A{A}^{\ast }$ are the diagonal entries of $\Sigma {\Sigma }^{\ast }$.

Theorem (Singular Value Decomposition)

For any $A\in M_{m,n}$, there exists $U\in M_m$, $V\in M_n$ both unitary and $\Sigma \in M_{m,n}$ “diagonal” with $\min \{m,n\}$ real, nonnegative, nonincreasing, entries such that $A=U\Sigma V^{\ast }$

Further, if $A$ is real-valued, we can take each of $U,\Sigma ,V$ to be real.

(we take “diagonal” to mean that the only entries that can be nonzero have the same row and column index)

Note

• We can think of this as a generalization of diagonalization/spectral decomposition, but where the loss is that $U$ and $V$ are distinct.
• if $A$ is positive semidefinite, then the diagonalization is the singular value decomposition since we have that $A=U\Sigma U^{\ast }$ for some unitary $U$ and nonnegative $\Sigma$ all real.

$m\le n$, then by the previous definition here we have $A=U\Sigma W$ where $U\in M_n$ unitary, $\Sigma \in M_m$ diagonal, and $W\in M_{m,n}$ with orthogonal columns. Since $m\le n$, we add $n-m$ columns of zeros to $\Sigma$ to make it size $m\times n$. Then we can append $n-m$ orthogonal rows to $W$ with Gram-Schmidt to get a $V^{\ast }$

If

If $n>m$, then $A^{\ast }=U\Sigma V^{\ast }$ is SVD by the above case. But then $A=V{\Sigma }^T{U}^{\ast }$ is an SVD of $A$!

(see singular value decomposition)

Corollary - Polar Decomposition

For all $A\in M_n$, there exists a hermitian, positive semidefinite $P\in M_n$ and a $W\in M_n$ unitary such that $A=PW$

$n=1$, we can write $a=\alpha e^{i\theta }$ where $\alpha$ is a PSD 1x1 matrix and $e^{i\theta }$ is some unitary matrix (it is a complex number with modulus 1!).

When

Let $A=U\Sigma V^{\ast }$ be its singular value decomposition. Then $A=U\Sigma U^{\ast }U{V}^{\ast }=(U\Sigma U^{\ast })(U{V}^{\ast })=PW$

(see polar decomposition)

Notes on SVD

Say $A\in M_{m,n}$ and $A\in U\Sigma V^{\ast }$ its SVD. Suppose there are $k$ non-negative singular values call them ${\sigma }_1,\dots ,{\sigma }_k$. Denote $u_1,\dots u_m$ the columns of $U$ and $v_1^{\ast },\dots ,v_n^{\ast }$ the rows of $V^{\ast }$. Then

1. $A=U\Sigma V^{\ast }={\sum }_{i=1}^k{\sigma }_i{u}_i{v}_i^{\ast }$
2. $\text{rank}(A)=\text{rank}(\Sigma )=k$
3. $\text{range}(A)=\text{span}\{u_1,\dots ,u_k\}$ - easy to see from (1)
4. $\text{null}(A)=\text{span}(v_{k+1},\dots ,v_n)$ - easy to see from (1)
5. $\text{range}(A^{\ast })=\text{span}\{v_1,\dots ,v_k\}$
6. $\text{null}(A^{\ast })=\text{span}\{u_{k+1},\dots ,u_m\}$
7. For any $A\in M_{m,n}$, we have $||A|{|}_{2,2}:={\max }_{x\in {\mathbb{C}}^n\ne 0}\frac{||Ax|{|}_2}{||x|{|}_2}={\sigma }_1$
8. $||A|{|}_F^2=||U\Sigma V^{\ast }|{|}_F^2=||\Sigma |{|}_F^2={\sum }_{i=1}^k{\sigma }_i$

To see (5) and (6) , recall that $A^{\ast }=V{\Sigma }^T{U}^{\ast }$ and we simply apply (3) and (4) in this case.

To see (7), note that $||A|{|}_{2,2}^2={\max }_{x\ne 0}\frac{||Ax|{|}_2}{||x|{|}_2}=\max \frac{x^{\ast }{A}^{\ast }Ax}{x^{\ast }x}{=}^{\ast ,\ast \ast }{\sigma }_1^2$

• Note $\ast$ is by Rayleigh-Ritz that this equals the largest eigenvalue of $A^{\ast }A$.
• Then $\ast \ast$ is by the fact that the eigenvalues of $A^{\ast }A=(U\Sigma V^{\ast }{)}^{\ast }(U\Sigma V^{\ast })=V\Sigma {\Sigma }^{\ast }{V}^{\ast }$ - ie the eigenvalues are the squared singular values of $A$.

recall that $\text{range}(A)=\{x:Ay=x\}$

Generalized Inverse

Let $A\in M_{m,n}$ and $B\in M_{n,m}$. Let \cal{C}=$$\{ 1,2,3,4 \}. Then $B$ is a generalized inverse of $A$ precisely when

• $1\in \mathcal{C}⟹$ $ABA=A$
• 2 \in \cal{C} \implies$$AB is hermitian
• $3\in \mathcal{C}⟹$ $BAB=B$
• 4 \in \cal{C} \implies$$BA is hermitian

For each $i\in \mathcal{C}$, we call $B$ an “$i$-generalized inverse”

• if $B$ satisfies 1, 2, then $B$ is a 1-2-generalized inverse

$A$ is invertible, then $B$ is a 1-2-3-4-generalized inverse.

If

• preview: every matrix has a unique 1-2-3-4-generalized inverse. We call this the Moore-Penrose inverse

$0$ is a 2-3-4 generalized inverse for every matrix...

$B$ is a 1-3 for $A$, then $A$ is a 2-4 for $B$.

If

(see generalized inverse)

Proposition

Consider $Ax=b$. If $A$ is invertible, then $x=A^{-1}b$ 🙂

But if $A$ is not square? What do we do? Suppose $B$ is a 1-generalized inverse of $A$.

Let $A\in M_{m,n},b\in {\mathbb{C}}^m$, and suppose $B$ is a 1-generalized inverse of $A$. Then if $Ax=b$ is consistent, then $x=Bb$ is a solution.

$Az=b$ ie there is a solution for some $z\in {\mathbb{C}}^n$. Then $A(Bb)=ABAz=Az=b$

Say

(see 1-generalized inverses give solutions to consistent linear systems)