Lecture 21

[[lecture-data]]

2024-10-16

Readings

• a

7. Chapter 7

We are talking about the singular value decomposition. Today: see how it can be used for generalized inverses

Recall, if we have a matrix $A\in M_{m,n}$ and $B\in M_{n,m}$ then we have generalized inverse when some of the conditions are met:

1. $ABA=A$
2. $AB$ is hermitian
3. $BAB=B$
4. $BA$ is hermitian

Note

if $A$ is square and invertible, then $B$ that satisfies all 4 conditions is the inverse of $A$ ie $A^{-1}$.

We can think of these in terms of the linear system $Ax=b$.

• if $A$ is square invertible, then $x=A^{-1}b$
• If $A$ is “tall and skinny” we want something to get the correct shape of $x$. If a system has a solution, then a 1-generalized inverse $B$ will give the solution $x=Bb$. And if $ABb=b$, then the system is consistent (ie a solution exists)

Theorem

Suppose that $A\in M_{m,n}$ and $b\in {\mathbb{C}}^m$ are given. Suppose $B\in M_{n,m}$ is a 1-2generalized inverse of $A$. Then $x=Bb$ solves $Ax=b$ in a least squares sense. ie, ${\min }_{x\in {\mathbb{C}}^n}||Ax-b|{|}_2$ has optimal solution $\hat{x}=Bb$.

${\mathbb{C}}^n$ can be expressed as $x=Bb+y$ for some $y\in {\mathbb{C}}^n$. We show that $||A(Bb+y)-b|{|}_2^2$ is minimized when $y=0$.

Any vector in

\lvert \lvert A(Bb+y) - b \rvert \rvert _{2}^2 &= [(AB-I)b + Ay]^*[(AB - I)b + Ay] \\ &= \lvert \lvert (AB-I)b \rvert \rvert _{2}^2 + \lvert \lvert Ay \rvert \rvert_{2}^2 + \lvert \lvert Ay \rvert \rvert_{2}^2 + y^*A^*(AB-I)b + b^*(AB-I)^* Ay \\ (*) &= \lvert \lvert (AB -I)b \rvert \rvert_{2}^2 + \lvert \lvert Ay \rvert \rvert_{2}^2 \end{aligned}$$ $(*)$ we see that $[A^*(AB-I)]^* = (AB-I)A = ABA-A =0$ since $B$ is a 1, 2 [[Concept Wiki/generalized inverse]] (ie, $AB$ is [[Concept Wiki/hermitian]] and $ABA=A$). ie, the expression depends only on the term $\lvert \lvert Ay \rvert \rvert_{2}^2$, which is minimized precisely when $\lvert \lvert Ay \rvert \rvert=0$ such as when $y =0$ for example. (the first term of the expression is constant). - thus the solutions to the least squares problem is the set $\{ Bb+y : y \in \text{Null}(A) \}$

(see a 1-2-generalized inverse gives a least squares optimal solution)

Theorem

There exists a unique 1-2-3-4 generalized inverse for every matrix $A\in M_{m,n}$ called the Moore-Penrose inverse (or pseudoinverse). And if $A$ is real valued, then this inverse is also real valued.

$B,C\in M_{n,m}$ are both 1-2-3-4 generalized inverses for $A$. NTS $B=C$.

(Uniqueness first). Suppose

• Claim 1: $AB=AC$
• $AB=ACAB$ since $A=ACA$ (1 generalized inverse)
• $AC$ and $AB$ are both hermitian. so we have
• $AB=C^{\ast }{A}^{\ast }{B}^{\ast }{A}^{\ast }=C^{\ast }(ABA{)}^{\ast }=C^{\ast }{A}^{\ast }$ since $ABA=A$
• But $AC$ is hermitian, so we get $AB=AC$.
• Claim 2: $BA=CA$ argued analogously to the first claim
• $BA=BACA$ since $A=ACA$
• Then $BA=BACA=A^{\ast }{B}^{\ast }{A}^{\ast }{C}^{\ast }$ since both $BA$ and $CA$ are hermitian
• $BA=(ABA{)}^{\ast }{C}^{\ast }=A^{\ast }{C}^{\ast }=CA$ since $CA$ is hermitian Consider $CAB$.
• By claim 2, we have $BAB=CAB$
• By claim 1, we have $CAB=CAC$.
• And by property 3 of the generalized inverses, we get $B=BAB=CAB=CAC=C$.

(Existence) Let us first consider a special case. Suppose $\Sigma$ is $M_{m,n}$ “diagonal”. Ie, $i\ne j,{\Sigma }_{ij}=0$. Define ${\Sigma }^{{}^{†}}\in M_{n,m}$ “diagonal”. And for all $i$, we have $({\Sigma }^{{}^{†}}{)}_{ii}=\frac{1}{{\Sigma }_{ii}}$ if ${\Sigma }_{ii}\ne 0$ and $0$ otherwise. Then ${\Sigma }^{{}^{†}}$ is a 1-2-3-4 generalized inverse for $\Sigma$. We can check this easily

• clearly $\Sigma {\Sigma }^{{}^{†}}$ is hermitian, same with ${\Sigma }^{{}^{†}}\Sigma$.
• $\Sigma {\Sigma }^{{}^{†}}\Sigma =\Sigma$
• ${\Sigma }^{{}^{†}}\Sigma {\Sigma }^{{}^{†}}={\Sigma }^{{}^{†}}$

Now, what happens if we have a matrix $Y\in M_{m,n}$ with a 1-2-3-4 generalized inverse $Z\in M_{n,m}$? Let $U\in M_m$ be unitary, $V\in M_n$ also unitary. Then $UY{V}^{\ast }$ has a 1-2-3-4 generalized inverse $VZ{U}^{\ast }$. We can show this easily:

• $[UY{V}^{\ast }][VZ{U}^{\ast }][UY{V}^{\ast }]=UYZY{V}^{\ast }$. Then since $Z$ is 1-2-3-4 generalized inverse we get $UYZY{V}^{\ast }=UY{V}^{\ast }$
• $[UY{V}^{\ast }][VZ{U}^{\ast }]=UYZ{U}^{\ast }$ is hermitian since $YZ$ is hermitian
• (And the other two conditions are shown exactly analogously)

Say $A=U\Sigma V^{\ast }$ is an SVD of $A$. Then the moore-penrose inverse of $A$ is $A^{{}^{†}}=V{\Sigma }^{{}^{†}}{U}^{\ast }$ by the two above facts.

• if $A$ is real, then the SVD is real and so the pseudoinverse is also real.

(see Moore-Penrose inverse)

Theorem

Let $A\in M_{m,n},b\in {\mathbb{C}}^m$ be given. Among the solutions to ${\min }_{x\in {\mathbb{C}}^n}||Ax-b|{|}_2$, we have that $A^{†}b$ is a unique solution of the minimum euclidian norm.

$A^{†}b+y:y\in \text{Null}(A)$. Let $A=U\Sigma V^{\ast }$ be an SVD and the rank of $A$ is $k$.

Recall that the solutions of the least squares problem are exactly

• $\text{Null}(A)=\text{span}\{v_{k+1},v_{k+2},\dots ,v_n\}$
• $\text{Range}(A^{†})=\text{span}\{v_0,v_1,\dots ,v_k\}$ since $A^{†}=V{\Sigma }^{†}{U}^{\ast }$ is (almost) an SVD. Almost because the sigmas in ${\Sigma }^{†}$ are not necessarily non-decreasing. (recall that for SVD we assume that the singular values are ordered)

Thus for all $y\in \text{Null}(A)$ we have $y\perp A^{†}b$ since we can see that $\text{range}(A^{†})\perp \text{Null}(A)$ from the above. Thus for any $y\in \text{Null}(A)$, we have $||A^{†}b+y|{|}_2^2=[A^{†}b+y{]}^{\ast }[A^{†}b+y]=||A^{†}b|{|}_2^2+||y|{|}_2^2+0+0⟹$

• the minimum occurs precisely when $y=0$!

(see the psuedoinverse gives the least norm solution to the least squares problem)