Definition 10.6.1.
Let \(A\) be an \(m\times n\) matrix. The singular values of \(A\) are the square roots of the positive eigenvalues of \(A^TA.\)
Section 10.6 SVD Decomposition
We begin this section with an important definition.
Singular Value Decomposition (SVD) can be thought of as a generalization of orthogonal diagonalization of a symmetric matrix to an arbitrary \(m\times n\) matrix. This decomposition is the focus of this section. The following is a useful result that will help when computing the SVD of matrices.
Lemma 10.6.2.
Let \(A\) be an \(m \times n\) matrix. Then \(A^TA\) and \(AA^T\) have the same \(\textbf{nonzero}\) eigenvalues.
Proof.
Suppose \(A\) is an \(m\times n\) matrix, and suppose that \(\lambda\) is a nonzero eigenvalue of \(A^TA\text{.}\) Then there exists a nonzero vector \(\mathbf{x} \in \R^n\) such that
\begin{equation}
(A^TA)\mathbf{x}=\lambda \mathbf{x}.\tag{10.6.1}
\end{equation}
Multiplying both sides of this equation by \(A\) yields:
\begin{align*}
A(A^TA)\mathbf{x} \amp = A\lambda \mathbf{x}, \\
(AA^T)(A\mathbf{x}) \amp = \lambda (A\mathbf{x}).
\end{align*}
Since \(\lambda\neq 0\) and \(\mathbf{x}\neq 0_n\text{,}\) \(\lambda \mathbf{x}\neq 0_n\text{,}\) and thus by (10.6.1), \((A^TA)\mathbf{x}\neq 0_m\text{;}\) thus \(A^T(A\mathbf{x})\neq 0_m\text{,}\) implying that \(A\mathbf{x}\neq 0_m\text{.}\)
Therefore \(A\mathbf{x}\) is an eigenvector of \(AA^T\) corresponding to eigenvalue \(\lambda\text{.}\) An analogous argument can be used to show that every nonzero eigenvalue of \(AA^T\) is an eigenvalue of \(A^TA\text{,}\) thus completing the proof.
Given an \(m\times n\) matrix \(A\text{,}\) we will see how to express \(A\) as a product \[ A=U\Sigma V^T\] where
- \(U\) is an \(m\times m\) orthogonal matrix whose columns are eigenvectors of \(AA^T\text{.}\)
- \(V\) is an \(n\times n\) orthogonal matrix whose columns are eigenvectors of \(A^TA\text{.}\)
- \(\Sigma\) is an \(m\times n\) matrix whose only nonzero values lie on its main diagonal, and are the singular values of \(A\text{.}\)
How can we find such a decomposition? We are aiming to decompose \(A\) in the following form:
\begin{equation*}
A=U\left[
\begin{array}{cc}
\sigma \amp 0 \\
0 \amp 0
\end{array}
\right] V^T
\end{equation*}
where \(\sigma \) is a block matrix of the form
\begin{equation*}
\sigma =\left[
\begin{array}{ccc}
\sigma _{1} \amp \amp 0 \\
\amp \ddots \amp \\
0 \amp \amp \sigma _{k}
\end{array}
\right]
\end{equation*}
Thus \(A^T=V\left[
\begin{array}{cc}
\sigma \amp 0 \\
0 \amp 0
\end{array}
\right] U^T\) and it follows that
\begin{equation*}
A^TA=V\left[
\begin{array}{cc}
\sigma \amp 0 \\
0 \amp 0
\end{array}
\right] U^TU\left[
\begin{array}{cc}
\sigma \amp 0 \\
0 \amp 0
\end{array}
\right] V^T=V\left[
\begin{array}{cc}
\sigma ^{2} \amp 0 \\
0 \amp 0
\end{array}
\right] V^T
\end{equation*}
and so \(A^TAV=V\left[
\begin{array}{cc}
\sigma ^{2} \amp 0 \\
0 \amp 0
\end{array}
\right] .\) Similarly,
\begin{equation*}
AA^TU=U\left[
\begin{array}{cc}
\sigma ^{2} \amp 0 \\
0 \amp 0
\end{array}
\right] .
\end{equation*}
Therefore, you would find an orthonormal basis of eigenvectors for \(AA^T\) make them the columns of a matrix such that the corresponding eigenvalues are decreasing. This gives \(U.\) You could then do the same for \(A^TA\) to get \(V\text{.}\)
We formalize this discussion in the following theorem.
Theorem 10.6.3. Singular Value Decomposition.
Let \(A\) be an \(m\times n\) matrix. Then there exist orthogonal matrices \(U\) and \(V\) of the appropriate size such that \(A= U \Sigma V^T\) where \(\Sigma\) is of the form
\begin{equation*}
\Sigma =
\left[
\begin{array}{cc}
\sigma \amp 0 \\
0 \amp 0
\end{array}
\right]
\end{equation*}
and \(\sigma \) is a block matrix of the form
\begin{equation*}
\sigma =\left[
\begin{array}{ccc}
\sigma _{1} \amp \amp 0 \\
\amp \ddots \amp \\
0 \amp \amp \sigma _{k}
\end{array}
\right]
\end{equation*}
for the \(\sigma _{i}\) the singular values of \(A.\)
Proof.
There exists an orthonormal basis, \(\left\{ \mathbf{v}_{1}, \dots, \mathbf{v}_n\right\}\) such that
\begin{equation*}
A^TA\mathbf{v}_{i}=\sigma _{i}^{2}\mathbf{v}_{i}
\end{equation*}
where \(\sigma
_{i}^{2}\gt 0\) for \(i=1,\dots ,k,\left( \sigma _{i}\gt 0\right) \) and equals zero if \(i\gt k.\) Thus for \(i\gt k,\) \(A\mathbf{v}_{i}=\mathbf{0}\) because
\begin{equation*}
A\mathbf{v}_{i}\cdot A\mathbf{v}_{i} = (A\mathbf{v}_i)^T(A\mathbf{v}_i)=\mathbf{v}_i^T(A^TA\mathbf{v}_i)=\mathbf{0}.
\end{equation*}
For \(i=1,\dots ,k,\) define \(\mathbf{u}_{i}\in \R^{m}\) by
\begin{equation*}
\mathbf{u}_{i}= \sigma _{i}^{-1}A\mathbf{v}_{i}.
\end{equation*}
Thus \(A\mathbf{v}_{i}=\sigma _{i}\mathbf{u}_{i}.\) Now,
\begin{align*}
\mathbf{u}_{i} \cdot \mathbf{u}_{j} \amp = \sigma _{i}^{-1}A
\mathbf{v}_{i} \cdot \sigma _{j}^{-1}A\mathbf{v}_{j} = \sigma_{i}^{-1}\mathbf{v}_{i}^T \sigma _{j}^{-1}A^TA\mathbf{v}_{j} \\
\amp = \sigma _{i}^{-1}\mathbf{v}_{i} \cdot \sigma _{j}^{-1}\sigma _{j}^{2} \mathbf{v}_{j} =
\frac{\sigma _{j}}{\sigma _{i}}\left( \mathbf{v}_{i} \cdot \mathbf{v}_{j}\right).
\end{align*}
This means that \(\mathbf{u}_{i} \cdot \mathbf{u}_{j}=1\) when \(i=j\) and \(\mathbf{u}_{i} \cdot \mathbf{u}_{j}=0\) when \(i\neq j\text{.}\) Thus \(\left\{ \mathbf{u}_{1}, \dots, \mathbf{u}_{k}\right\}\) is an orthonormal set of vectors in \(\R^{m}.\) Also,
\begin{equation*}
AA^T\mathbf{u}_{i}=AA^T\sigma _{i}^{-1}A\mathbf{v}_{i}=\sigma
_{i}^{-1}AA^TA\mathbf{v}_{i}=\sigma _{i}^{-1}A\sigma _{i}^{2}\mathbf{v}
_{i}=\sigma _{i}^{2}\mathbf{u}_{i}.
\end{equation*}
Using Gram-Schmidt, extend \(\left\{ \mathbf{u}_{1}, \dots, \mathbf{u}_{k}\right\}\) to an orthonormal basis for all of \(\R^{m},\left\{ \mathbf{u}_{1},\dots,\mathbf{u}_{m}\right\}\) and let
\begin{equation*}
U= \left[
\begin{array}{ccc}
\mathbf{u}_{1} \amp \cdots \amp \mathbf{u}_{m}
\end{array}
\right],
\end{equation*}
while \(V= \left[ \begin{array}{ccc} \mathbf{v}_{1} \amp \cdots \amp \mathbf{v}_{n}\end{array}\right] .\) Thus \(U\) is the matrix which has the \(\mathbf{u}_{i}\) as columns and \(V\) is defined as the matrix which has the \(\mathbf{v}_{i}\) as columns. Then
\begin{equation*}
U^TAV=\left[
\begin{array}{c}
\mathbf{u}_{1}^T \\
\vdots \\
\mathbf{u}_{k}^T \\
\vdots \\
\mathbf{u}_{m}^T
\end{array}
\right] A \left[ \begin{array}{ccc} \mathbf{v}_{1} \amp \cdots \amp \mathbf{v}_{n}\end{array}\right]
\end{equation*}
\begin{equation*}
=\left[
\begin{array}{c}
\mathbf{u}_{1}^T \\
\vdots \\
\mathbf{u}_{k}^T \\
\vdots \\
\mathbf{u}_{m}^T
\end{array}
\right] \left[
\begin{array}{cccccc}
\sigma _{1}\mathbf{u}_{1} \amp \cdots \amp \sigma _{k}\mathbf{u}_{k} \amp \mathbf{0}
\amp \cdots \amp \mathbf{0}
\end{array}
\right] =\left[
\begin{array}{cc}
\sigma \amp 0 \\
0 \amp 0
\end{array}
\right],
\end{equation*}
where \(\sigma \) is given in the statement of the theorem.
The SVD has as an immediate corollary which is given in the following interesting result.
Corollary 10.6.4.
Let \(A\) be an \(m\times n\) matrix. Then the rank of \(A\) (or of \(A^T\)) equals the number of singular values.
Let’s compute the SVD of a simple matrix.
Example 10.6.5.
Let
\begin{equation*}
A=\left[\begin{array}{rrr} 1 \amp -1 \amp 3 \\ 3 \amp 1 \amp 1 \end{array}\right].
\end{equation*}
Find the SVD of \(A\text{.}\)
Answer.
To begin, we compute \(AA^T\) and \(A^TA\text{.}\)
\begin{equation*}
AA^T = \left[\begin{array}{rrr} 1 \amp -1 \amp 3 \\ 3 \amp 1 \amp 1 \end{array}\right]
\left[\begin{array}{rr} 1 \amp 3 \\ -1 \amp 1 \\ 3 \amp 1 \end{array}\right]
= \left[\begin{array}{rr} 11 \amp 5 \\ 5 \amp 11 \end{array}\right],
\end{equation*}
\begin{equation*}
A^TA = \left[\begin{array}{rr} 1 \amp 3 \\ -1 \amp 1 \\ 3 \amp 1 \end{array}\right]
\left[\begin{array}{rrr} 1 \amp -1 \amp 3 \\ 3 \amp 1 \amp 1 \end{array}\right]
= \left[\begin{array}{rrr} 10 \amp 2 \amp 6 \\ 2 \amp 2 \amp -2\\
6 \amp -2 \amp 10 \end{array}\right].
\end{equation*}
Since \(AA^T\) is \(2\times 2\) while \(A^T A\) is \(3\times 3\text{,}\) and \(AA^T\) and \(A^TA\) have the same nonzero eigenvalues (by Lemma 10.6.2), we compute the characteristic polynomial \(c_{AA^T}(x)\) (because it is easier to compute than \(c_{A^TA}(x)\)).
\begin{align*}
c_{AA^T}(z)\amp = \det(zI-AA^T)= \det \left[\begin{array}{cc}
z-11 \amp -5 \\ -5 \amp z-11 \end{array}\right] \\
\amp = (z-11)^2 - 25 \\
\amp = z^2-22z+121-25 \\
\amp = z^2-22z+96 \\
\amp = (z-16)(z-6).
\end{align*}
Therefore, the eigenvalues of \(AA^T\) are \(\lambda_1=16\) and \(\lambda_2=6\text{.}\) The eigenvalues of \(A^TA\) are \(\lambda_1=16\text{,}\) \(\lambda_2=6\text{,}\) and \(\lambda_3=0\text{,}\) and the singular values of \(A\) are \(\sigma_1=\sqrt{16}=4\) and \(\sigma_2=\sqrt{6}\text{.}\) By convention, we list the eigenvalues (and corresponding singular values) in non increasing order (i.e., from largest to smallest).
\(\textbf{To find the matrix: }\)
To construct the matrix \(V\) we need to find eigenvectors for \(A^TA\text{.}\) Since the eigenvalues of \(AA^T\) are distinct, the corresponding eigenvectors are orthogonal, and we need only normalize them.
\(\lambda_1=16\text{:}\) solve \((16I-A^TA)\mathbf{x}_1= \mathbf{0}\text{.}\)
\begin{equation*}
\left[\begin{array}{rrr|r}
6 \amp -2 \amp -6 \amp 0 \\ -2 \amp 14 \amp 2 \amp 0 \\ -6 \amp 2 \amp 6 \amp 0
\end{array}\right]
\rightarrow
\left[\begin{array}{rrr|r}
1 \amp 0 \amp -1 \amp 0 \\ 0 \amp 1 \amp 0 \amp 0 \\ 0 \amp 0 \amp 0 \amp 0
\end{array}\right],
\end{equation*}
so
\begin{equation*}
\mathbf{x}_1 =\left[\begin{array}{r} t \\ 0 \\ t \end{array}\right]
=t\left[\begin{array}{r} 1 \\ 0 \\ 1 \end{array}\right],
t\in \R.
\end{equation*}
\(\lambda_2=6\text{:}\) solve \((6I-A^TA)\mathbf{x}_2= \mathbf{0}\text{.}\)
\begin{equation*}
\left[\begin{array}{rrr|r}
-4 \amp -2 \amp -6 \amp 0 \\ -2 \amp 4 \amp 2 \amp 0 \\ -6 \amp 2 \amp -4 \amp 0
\end{array}\right]
\rightarrow
\left[\begin{array}{rrr|r}
1 \amp 0 \amp 1 \amp 0 \\ 0 \amp 1 \amp 1 \amp 0 \\ 0 \amp 0 \amp 0 \amp 0
\end{array}\right],
\end{equation*}
so
\begin{equation*}
\mathbf{x}_2=\left[\begin{array}{r} -s \\ -s \\ s \end{array}\right]
=s\left[\begin{array}{r} -1 \\ -1 \\ 1 \end{array}\right],
s\in \R.
\end{equation*}
\(\lambda_3=0\text{:}\) solve \((-A^TA)\mathbf{x}_3= \mathbf{0}\text{.}\)
\begin{equation*}
\left[\begin{array}{rrr|r}
-10 \amp -2 \amp -6 \amp 0 \\ -2 \amp -2 \amp 2 \amp 0 \\ -6 \amp 2 \amp -10 \amp 0
\end{array}\right]
\rightarrow
\left[\begin{array}{rrr|r}
1 \amp 0 \amp 1 \amp 0 \\ 0 \amp 1 \amp -2 \amp 0 \\ 0 \amp 0 \amp 0 \amp 0
\end{array}\right],
\end{equation*}
so
\begin{equation*}
\mathbf{x}_3=\left[\begin{array}{r} -r \\ 2r \\ r \end{array}\right]
=r\left[\begin{array}{r} -1 \\ 2 \\ 1 \end{array}\right],
r\in \R.
\end{equation*}
With the eigenvectors found, let
\begin{equation*}
\mathbf{v}_1=\frac{1}{\sqrt{2}}\left[\begin{array}{r} 1\\ 0\\ 1 \end{array}\right],
\mathbf{v}_2=\frac{1}{\sqrt{3}}\left[\begin{array}{r} -1\\ -1\\ 1 \end{array}\right],
\mathbf{v}_3=\frac{1}{\sqrt{6}}\left[\begin{array}{r} -1\\ 2\\ 1 \end{array}\right]
\end{equation*}
Then
\begin{equation*}
V=\frac{1}{\sqrt{6}}\left[\begin{array}{rrr}
\sqrt 3 \amp -\sqrt 2 \amp -1 \\
0 \amp -\sqrt 2 \amp 2 \\
\sqrt 3 \amp \sqrt 2 \amp 1 \end{array}\right].
\end{equation*}
Also,
\begin{equation*}
\Sigma = \left[\begin{array}{rrr} 4 \amp 0 \amp 0 \\
0 \amp \sqrt 6 \amp 0 \end{array}\right]
\end{equation*}
and we use \(A\text{,}\) \(V^T\text{,}\) and \(\Sigma\) to find \(U\text{.}\) Since \(V\) is orthogonal and \(A=U\Sigma V^T\text{,}\) it follows that
\begin{equation*}
AV=U\Sigma.
\end{equation*}
Let \(V=\left[\begin{array}{ccc} \mathbf{v}_1 \amp \mathbf{v}_2 \amp \mathbf{v}_3 \end{array}\right]\text{,}\) and let \(U=\left[\begin{array}{cc} \mathbf{u}_1 \amp \mathbf{u}_2 \end{array}\right]\text{,}\) where \(\mathbf{u}_1\) and \(\mathbf{u}_2\) are the two columns of \(U\text{.}\) Then we have
\begin{align*}
A\left[\begin{array}{ccc} \mathbf{v}_1 \amp \mathbf{v}_2 \amp \mathbf{v}_3 \end{array}\right]
\amp = \left[\begin{array}{cc} \mathbf{u}_1 \amp \mathbf{u}_2 \end{array}\right]\Sigma
\left[\begin{array}{ccc} A\mathbf{v}_1 \amp A\mathbf{v}_2 \amp A\mathbf{v}_3 \end{array}\right] \\
\amp = \left[\begin{array}{ccc} \sigma_1\mathbf{u}_1 + 0\mathbf{u}_2 \amp
0\mathbf{u}_1 + \sigma_2 \mathbf{u}_2 \amp 0 \mathbf{u}_1 + 0\mathbf{u}_2 \end{array}\right] \\
= \left[\begin{array}{ccc} \sigma_1\mathbf{u}_1 \amp \sigma_2 \mathbf{u}_2 \amp
0 \end{array}\right]
\end{align*}
which implies that \(A\mathbf{v}_1=\sigma_1\mathbf{u}_1 = 4\mathbf{u}_1\) and \(A\mathbf{v}_2=\sigma_2\mathbf{u}_2 = \sqrt 6 \mathbf{u}_2\text{.}\) Thus,
\begin{equation*}
\mathbf{u}_1 = \frac{1}{4}A\mathbf{v}_1
= \frac{1}{4}
\left[\begin{array}{rrr} 1 \amp -1 \amp 3 \\ 3 \amp 1 \amp 1 \end{array}\right]
\frac{1}{\sqrt{2}}\left[\begin{array}{r} 1\\ 0\\ 1 \end{array}\right]
= \frac{1}{4\sqrt 2}\left[\begin{array}{r} 4\\ 4 \end{array}\right]
= \frac{1}{\sqrt 2}\left[\begin{array}{r} 1\\ 1 \end{array}\right]
\end{equation*}
and
\begin{align*}
\mathbf{u}_2 \amp = \frac{1}{\sqrt 6}A\mathbf{v}_2
= \frac{1}{\sqrt 6}
\left[\begin{array}{rrr} 1 \amp -1 \amp 3 \\ 3 \amp 1 \amp 1 \end{array}\right]
\frac{1}{\sqrt{3}}\left[\begin{array}{r} -1\\ -1\\ 1 \end{array}\right] \\
\amp =\frac{1}{3\sqrt 2}\left[\begin{array}{r} 3\\ -3 \end{array}\right]
=\frac{1}{\sqrt 2}\left[\begin{array}{r} 1\\ -1 \end{array}\right]
\end{align*}
Therefore,
\begin{equation*}
U=\frac{1}{\sqrt{2}}\left[\begin{array}{rr} 1 \amp 1 \\
1 \amp -1 \end{array}\right]
\end{equation*}
and
\begin{align*}
A \amp = \left[\begin{array}{rrr} 1 \amp -1 \amp 3 \\ 3 \amp 1 \amp 1 \end{array}\right] \\
\amp = \left(\frac{1}{\sqrt{2}}\left[\begin{array}{rr} 1 \amp 1 \\
1 \amp -1 \end{array}\right]\right)
\left[\begin{array}{rrr} 4 \amp 0 \amp 0 \\
0 \amp \sqrt 6 \amp 0 \end{array}\right]
\left(\frac{1}{\sqrt{6}}\left[\begin{array}{rrr}
\sqrt 3 \amp 0 \amp \sqrt 3 \\
-\sqrt 2 \amp -\sqrt 2 \amp \sqrt2 \\
-1 \amp 2 \amp 1 \end{array}\right]\right).
\end{align*}
Example 10.6.6.
Find an SVD for
\begin{equation*}
A=\left[\begin{array}{r} -1 \\ 2\\ 2 \end{array}\right]\text{.}
\end{equation*}
Answer.
Since \(A\) is \(3\times 1\text{,}\) \(A^T A\) is a \(1\times 1\) matrix whose eigenvalues are easier to find than the eigenvalues of the \(3\times 3\) matrix \(AA^T\text{.}\)
\begin{equation*}
A^TA=\left[\begin{array}{ccc} -1 \amp 2 \amp 2 \end{array}\right]
\left[\begin{array}{r} -1 \\ 2 \\ 2 \end{array}\right]
=\left[\begin{array}{r} 9 \end{array}\right].
\end{equation*}
Thus \(A^TA\) has eigenvalue \(\lambda_1=9\text{,}\) and the eigenvalues of \(AA^T\) are \(\lambda_1=9\text{,}\) \(\lambda_2=0\text{,}\) and \(\lambda_3=0.\text{.}\) Furthermore, \(A\) has only one singular value, \(\sigma_1=3\text{.}\)
\(\textbf{To find the matrix }:\)
To do so we find an eigenvector for \(A^TA\) and normalize it. In this case, finding a unit eigenvector is trivial: \(\mathbf{v}_1=\left[\begin{array}{r} 1 \end{array}\right]\text{,}\) and \(V=\left[\begin{array}{r} 1 \end{array}\right] \text{.}\) Also,
\begin{equation*}
\Sigma =\left[\begin{array}{r} 3 \\ 0\\ 0 \end{array}\right],
\end{equation*}
and we use \(A\text{,}\) \(V^T\text{,}\) and \(\Sigma\) to find \(U\text{.}\)
Now, \(AV=U\Sigma\) with \(V=\left[\begin{array}{r}\mathbf{v}_1 \end{array}\right]\text{,}\) and \(U=\left[\begin{array}{rrr} \mathbf{u}_1 \amp \mathbf{u}_2 \amp \mathbf{u}_3 \end{array}\right]\text{,}\) where \(\mathbf{u}_1\text{,}\) \(\mathbf{u}_2\text{,}\) and \(\mathbf{u}_3\) are the columns of \(U\text{.}\) Thus
\begin{align*}
A\left[\begin{array}{r} \mathbf{v}_1 \end{array}\right]
\amp = \left[\begin{array}{rrr} \mathbf{u}_1 \amp \mathbf{u}_2 \amp \mathbf{u}_3 \end{array}\right]\Sigma \\
\left[\begin{array}{r} A\mathbf{v}_1 \end{array}\right]
\amp = \left[\begin{array}{r} \sigma_1 \mathbf{u}_1+0\mathbf{u}_2+0\mathbf{u}_3 \end{array}\right] \\
\amp = \left[\begin{array}{r} \sigma_1 \mathbf{u}_1 \end{array}\right].
\end{align*}
This gives us \(A\mathbf{v}_1=\sigma_1 \mathbf{u}_1= 3\mathbf{u}_1\text{,}\) so
\begin{equation*}
\mathbf{u}_1 = \frac{1}{3}A\mathbf{v}_1
= \frac{1}{3}
\left[\begin{array}{r} -1 \\ 2 \\ 2 \end{array}\right]
\left[\begin{array}{r} 1 \end{array}\right]
= \frac{1}{3}
\left[\begin{array}{r} -1 \\ 2 \\ 2 \end{array}\right].
\end{equation*}
The vectors \(\mathbf{u}_2\) and \(\mathbf{u}_3\) are eigenvectors of \(AA^T\) corresponding to the eigenvalue \(\lambda_2=\lambda_3=0\text{.}\) Instead of solving the system \((0I-AA^T)\mathbf{x}= 0\) and then using the Gram-Schmidt process on the resulting set of two basic eigenvectors, the following approach may be used.
Find vectors \(\mathbf{u}_2\) and \(\mathbf{u}_3\) by first extending \(\{ \mathbf{u}_1\}\) to a basis of \(\R^3\text{,}\) then using the Gram-Schmidt algorithm to orthogonalize the basis, and finally normalizing the vectors. Starting with \(\{ 3\mathbf{u}_1 \}\) instead of \(\{ \mathbf{u}_1 \}\) makes the arithmetic a bit easier. It is easy to verify that
\begin{equation*}
\left\{ \left[\begin{array}{r} -1 \\ 2 \\ 2 \end{array}\right],
\left[\begin{array}{r} 1 \\ 0 \\ 0 \end{array}\right],
\left[\begin{array}{r} 0 \\ 1 \\ 0 \end{array}\right]\right\}
\end{equation*}
is a basis of \(\R^3\text{.}\) Set
\begin{equation*}
\mathbf{x}_1 = \left[\begin{array}{r} -1 \\ 2 \\ 2 \end{array}\right],
\mathbf{x}_2 = \left[\begin{array}{r} 1 \\ 0 \\ 0 \end{array}\right],
\mathbf{x}_3 =\left[\begin{array}{r} 0 \\ 1 \\ 0 \end{array}\right]
\end{equation*}
and apply the Gram-Schmidt algorithm to \(\{ \mathbf{x}_1, \mathbf{x}_2, \mathbf{x}_3\}\text{.}\) This gives us
\begin{equation*}
\mathbf{f}_1 = \left[\begin{array}{r} -1 \\ 2 \\ 2 \end{array}\right], \mathbf{f}_2 = \left[\begin{array}{r} 4 \\ 1 \\ 1 \end{array}\right]
\mbox{ and }
\mathbf{f}_3 = \left[\begin{array}{r} 0 \\ 1 \\ -1 \end{array}\right].
\end{equation*}
Therefore,
\begin{equation*}
\mathbf{u}_2 = \frac{1}{\sqrt{18}}
\left[\begin{array}{r} 4 \\ 1 \\ 1 \end{array}\right],
\mathbf{u}_3 = \frac{1}{\sqrt 2}
\left[\begin{array}{r} 0 \\ 1 \\ -1 \end{array}\right]
\end{equation*}
and
\begin{equation*}
U = \left[ \begin{array}{rrr} -\frac{1}{3} \amp \frac{4}{\sqrt{18}} \amp 0 \\
\frac{2}{3} \amp \frac{1}{\sqrt{18}} \amp \frac{1}{\sqrt 2} \\
\frac{2}{3} \amp \frac{1}{\sqrt{18}} \amp -\frac{1}{\sqrt 2} \end{array}\right]
\end{equation*}
Finally,
\begin{equation*}
A =
\left[\begin{array}{r} -1 \\ 2 \\ 2 \end{array}\right]
=
\left[ \begin{array}{rrr} -\frac{1}{3} \amp \frac{4}{\sqrt{18}} \amp 0 \\
\frac{2}{3} \amp \frac{1}{\sqrt{18}} \amp \frac{1}{\sqrt 2} \\
\frac{2}{3} \amp \frac{1}{\sqrt{18}} \amp -\frac{1}{\sqrt 2} \end{array}\right]
\left[\begin{array}{r} 3 \\ 0 \\ 0 \end{array}\right]
\left[\begin{array}{r} 1 \end{array}\right].
\end{equation*}
Example 10.6.7.
Find an SVD for the matrix
\begin{equation*}
A= \left[
\begin{array}{ccc}
\frac{2}{5}\sqrt{2}\sqrt{5} \amp \frac{4}{5}\sqrt{2}\sqrt{5} \amp 0 \\
\frac{2}{5}\sqrt{2}\sqrt{5} \amp \frac{4}{5}\sqrt{2}\sqrt{5} \amp 0
\end{array}
\right].
\end{equation*}
Answer.
First consider \(A^TA\)
\begin{equation*}
\left[
\begin{array}{ccc}
\frac{16}{5} \amp \frac{32}{5} \amp 0 \\
\frac{32}{5} \amp \frac{64}{5} \amp 0 \\
0 \amp 0 \amp 0
\end{array}
\right].
\end{equation*}
What are some eigenvalues and eigenvectors? Some computing shows that the eigenvalues are \(0\) and \(16\text{.}\) Furthermore, we can find a basis for each eigenspace.
\begin{equation*}
\mathcal{S}_0=\mbox{span}\left( \left[
\begin{array}{c}
0 \\
0 \\
1
\end{array}
\right] ,\left[
\begin{array}{c}
-\frac{2}{5}\sqrt{5} \\
\frac{1}{5}\sqrt{5} \\
0
\end{array}
\right] \right),
\quad\mathcal{S}_{16}=\mbox{span}\left( \left[
\begin{array}{c}
\frac{1}{5}\sqrt{5} \\
\frac{2}{5}\sqrt{5} \\
0
\end{array}
\right] \right).
\end{equation*}
Thus the matrix \(V\) is given by
\begin{equation*}
V=\left[
\begin{array}{ccc}
\frac{1}{5}\sqrt{5} \amp -\frac{2}{5}\sqrt{5} \amp 0 \\
\frac{2}{5}\sqrt{5} \amp \frac{1}{5}\sqrt{5} \amp 0 \\
0 \amp 0 \amp 1
\end{array}
\right].
\end{equation*}
Next consider \(AA^T\)
\begin{equation*}
\left[
\begin{array}{cc}
8 \amp 8 \\
8 \amp 8
\end{array}
\right].
\end{equation*}
Eigenvalues are \(0\) and \(16\text{,}\) and eigenspaces are
\begin{equation*}
\mathcal{S}_0=\mbox{span}\left(\left[
\begin{array}{c}
-\frac{1}{2}\sqrt{2} \\
\frac{1}{2}\sqrt{2}
\end{array}
\right] \right),\quad\mathcal{S}_{16}=\mbox{span}\left( \left[
\begin{array}{c}
\frac{1}{2}\sqrt{2} \\
\frac{1}{2}\sqrt{2}
\end{array}
\right] \right) .
\end{equation*}
Thus you can let \(U\) be given by
\begin{equation*}
U=\left[
\begin{array}{cc}
\frac{1}{2}\sqrt{2} \amp -\frac{1}{2}\sqrt{2} \\
\frac{1}{2}\sqrt{2} \amp \frac{1}{2}\sqrt{2}
\end{array}
\right].
\end{equation*}
Let’s check this. \(U^TAV=\)
\begin{equation*}
\left[
\begin{array}{cc}
\frac{1}{2}\sqrt{2} \amp \frac{1}{2}\sqrt{2} \\
-\frac{1}{2}\sqrt{2} \amp \frac{1}{2}\sqrt{2}
\end{array}
\right] \left[
\begin{array}{ccc}
\frac{2}{5}\sqrt{2}\sqrt{5} \amp \frac{4}{5}\sqrt{2}\sqrt{5} \amp 0 \\
\frac{2}{5}\sqrt{2}\sqrt{5} \amp \frac{4}{5}\sqrt{2}\sqrt{5} \amp 0
\end{array}
\right] \left[
\begin{array}{ccc}
\frac{1}{5}\sqrt{5} \amp -\frac{2}{5}\sqrt{5} \amp 0 \\
\frac{2}{5}\sqrt{5} \amp \frac{1}{5}\sqrt{5} \amp 0 \\
0 \amp 0 \amp 1
\end{array}
\right]
\end{equation*}
\begin{equation*}
=\left[
\begin{array}{ccc}
4 \amp 0 \amp 0 \\
0 \amp 0 \amp 0
\end{array}
\right].
\end{equation*}
This illustrates that if you have a good way to find the eigenvectors and eigenvalues for a Hermitian matrix which has nonnegative eigenvalues, then you also have a good way to find the SVD of an arbitrary matrix.
