Chapter 7 - Operators on Inner Product Spaces

From here on out, all $V$ and $W$ denote finite-dimensional inner product spaces over $F$ .

7.A: Self-Adjoint and Normal Operators

adjoint,

T^{*}

Suppose $T \in L (V, W)$ . The adjoint of $T$ is the function $T^{*} : W \to V$ such that:

⟨ T v, w ⟩ = ⟨ v, T^{*} w ⟩

for all $v \in V, w \in W$ .

In trying to construct this definition, this is the derivation. Suppose some $T \in L (V, W)$ . If you fix a $w \in W$ then consider the linear functional on $V$ that maps $v \in V$ to $⟨ T v, w ⟩$ , namely $φ \in V^{'}$ defined by:

φ (v) = ⟨ T v, w ⟩

where this works since $T, w$ are both fixed, while $v \in V$ is arbitrary. By Chapter 6 (cont.) - Finishing Inner Product Spaces#^a857c6, there exists a unique vector in $V$ , we call it $T^{*} w$ , that $⟨ T v, w ⟩ = ⟨ v, T^{*} w ⟩$ .

As an example, consider $T : R^{3} \to R^{2}$ by:

T (x_{1}, x_{2}, x_{3}) = (x_{2} + 3 x_{3}, 2 x_{1})

Here $T^{*}$ will be a function $T^{*} : R^{2} \to R^{3}$ . To find it, fix a point in the output $y = (y_{1}, y_{2}) \in R^{2}$ and then let $x = (x_{1}, x_{2}, x_{3}) \in R^{3}$ be arbitrary:

\begin{aligned} ⟨ x, T^{*} y ⟩ & = ⟨ T x, y ⟩ \\ = ⟨ T (x_{1}, x_{2}, x_{3}), y ⟩ \\ = ⟨ (x_{2} + 3 x_{3}, 2 x_{1}), (y_{1}, y_{2}) ⟩ \\ = x_{2} y_{1} + 3 x_{3} y_{1} + 2 x_{1} y_{2} \\ = ⟨ (x_{1}, x_{2}, x_{3}), (2 y_{2}, y_{1}, 3 y_{1}) ⟩ \\ = ⟨ x, (2 y_{2}, y_{1}, 3 y_{1}) ⟩ \end{aligned}

thus equate the right sides of the $⟨ ⟩$ and we see that $T^{*} (y) = (2 y_{2}, y_{1}, 3 y_{1})$ .

Notice if we want to algorithmize this process, fix $u \in V$ and $x \in W$ . Define $T \in L (V, W)$ by:

T v = ⟨ v, u ⟩ x

Then if we fix $w \in W$ then for any arbitrary $v \in V$ we have:

\begin{aligned} ⟨ v, T^{*} w ⟩ & = ⟨ T v, w ⟩ \\ = ⟨ ⟨ v, u ⟩ x, w ⟩ \\ = ⟨ v, u ⟩ ⟨ x, w ⟩ \\ = ⟨ v, ⟨ w, x ⟩ u ⟩ \end{aligned}

So:

T^{*} w = ⟨ w, x ⟩ u

The adjoint is a linear map

If $T \in L (V, W)$ then $T^{*} \in L (W, V)$ .

\begin{proof}
Suppose $T \in L (V, W)$ . Fix $w_{1}, w_{2} \in W$ . If $v \in V$ then:

\begin{aligned} ⟨ v, T^{*} (w_{1} + w_{2}) ⟩ & = ⟨ T v, w_{1} + w_{2} ⟩ \\ = ⟨ T v, w_{1} ⟩ + ⟨ T v, w_{2} ⟩ \\ = ⟨ v, T^{*} w_{1} ⟩ + ⟨ v, T^{*} w_{2} ⟩ \\ = ⟨ v, T^{*} w_{1} + T^{*} w_{2} ⟩ \end{aligned}

Fix $w \in W$ and $λ \in F$ . If $v \in V$ then:

\begin{aligned} ⟨ v, T^{*} (λ w) ⟩ & = ⟨ T v, λ w ⟩ \\ = \overset{―}{λ} ⟨ T v, w ⟩ \\ = \overset{―}{λ} ⟨ v, T^{*} w ⟩ \\ = ⟨ v, λ T^{*} w ⟩ \end{aligned}

\end{proof}

Properties of the adjoint

For all $S, T \in L (V, W)$ and $λ \in F$ :

$(S + T)^{*} = S^{*} + T^{*}$
$(λ T)^{*} = \overset{―}{λ} T^{*}$
$(T^{*})^{*} = T$
$I^{*} = I$ where $I$ is the identity operator on $V$ .
If instead $S \in L (W, U)$ , then $(S T)^{*} = T^{*} S^{*}$ . Here $U$ is an inner product space over $F$ .

The proofs are via 2015_Book_LinearAlgebraDoneRight#page=206 and are easy to see.

Null space and range of

T^{*}

Suppose $T \in L (V, W)$ . Then:

$null (T^{*}) = (range (T))^{⊥}$
$range (T^{*}) = (null (T))^{⊥}$
$null (T) = (range (T^{*}))^{⊥}$
$range (T) = (null (T^{*}))^{⊥}$

\begin{proof}
Prove the first one. Let $w \in W$ . Then:

\begin{aligned} w \in null (T^{*}) & \Leftrightarrow T^{*} w = 0 \\ \Leftrightarrow ⟨ v, T^{*} w ⟩ = 0 \forall v \in V \\ \Leftrightarrow ⟨ T v, w ⟩ = 0 \forall v \in V \\ \Leftrightarrow w \in (range (T))^{⊥} \end{aligned}

where the last step is because otherwise, the inner product would not be 0.

If we take the orthogonal complement of both sides of (a), we get (d), using Chapter 6 (cont.) - Finishing Inner Product Spaces#^0268c7. Replacing $T$ with $T^{*}$ in (a) gives (c), where we have used Chapter 7 - Operators on Inner Product Spaces#^232dd6 (c). Finally replacing $T$ with $T^{*}$ in (d) gives (b).
\end{proof}

conjugate transpose

The conjugate transpose of an $m \times n$ matrix is the $n \times m$ matrix obtained by interchanging the rows and columns and then taking the complex conjugate of each entry.

Using orthonormal bases

The next result only applies when we have an orthonormal basis. If you don't, then it's not necessarily true.

The matrix of

T^{*}

Let $T \in L (V, W)$ . Suppose $e_{1}, . . ., e_{n}$ is an orthonormal basis of $V$ and $f_{1}, . . ., f_{m}$ is an orthonormal basis of $W$ . Then:

M (T^{*}, (f_{1}, . . ., f_{m}), (e_{1}, . . ., e_{m}))

is the conjugate transpose of:

M (T, (e_{1}, . . ., e_{n}), (f_{1}, . . ., f_{m}))

\begin{proof}
We obtain the $k$ -th column of $M (T)$ by writing $T e_{k}$ as linear combination of the $f_{j}$ 's; the scalars used in this linear combination become the $k$ -th column of $M (T)$ . Because $f_{1}, . . ., f_{m}$ is an orthonormal basis of $W$ , we know how to write $T e_{k}$ as a linear combination of the $f_{j}$ 's:

T e_{k} = \sum_{i = 1}^{m} ⟨ T e_{k}, f_{i} ⟩ f_{i}

Thus the entry in row $j$ column $k$ of $M (T)$ is $⟨ T e_{k}, f_{j} ⟩$ .

Replacing $T$ with $T^{*}$ and interchanging the roles played by the $e$ 's and $f$ 's we see that row $j$ column $k$ of $M (T^{*})$ is $⟨ T^{*} f_{k}, e_{j} ⟩$ but here:

\begin{aligned} ⟨ T^{*} f_{k}, e_{j} ⟩ & = ⟨ f_{k}, T e_{j} ⟩ \\ = \overset{―}{⟨ T e_{j}, f_{k} ⟩} \\ = c_{k j}^{*} \in M (T) \end{aligned}

Thus $M (T)$ is the conjugate transpose of $M (T^{*})$ and vice versa.
\end{proof}

Self-Adjoint Operators

self-adjoint

An operator $T \in L (V)$ is called self-adjoint if $T = T^{*}$ . In other words, $T \in L (V)$ is self-adjoint iff:

⟨ T v, w ⟩ = ⟨ v, T w ⟩

for all $v, w \in V$

Eigenvalues of self-adjoint operators are real

Every eigenvalue of a self-adjoint operator is real

\begin{proof}
Suppose $T$ is a self-adjoint operator on $V$ . Let $λ$ be an eigenvalue of $T$ and let $v \neq \vec{0}$ be in $V$ where $T v = λ v$ . Then:

λ ‖ v ‖^{2} = ⟨ λ v, v ⟩ = ⟨ T v, v ⟩ = ⟨ v, T v ⟩ = ⟨ v, λ v ⟩ = \overset{―}{λ} ‖ v ‖^{2}

So $λ = \overset{―}{λ}$ thus $λ$ is real.
\end{proof}

Over

C

T v

is orthogonal to

v

for all

v

only for the

\vec{0}

operator

Suppose $V$ is a complex inner product space and $T \in L (V)$ . Suppose:

⟨ T v, v ⟩ = 0

for all $v \in V$ . Then $T = 0$ .

See the proof via 2015_Book_LinearAlgebraDoneRight#page=210.

Over

C

⟨ T v, v ⟩

is real for all

v

only for self-adjoint operators.

Suppose $V$ is a complex inner product space and $T \in L (V)$ . Then $T$ is self-adjoint iff:

⟨ T v, v ⟩ \in R

for every $v \in V$ .

\begin{proof}
Let $v \in V$ . Then:

⟨ T v, v ⟩ - \overset{―}{⟨ T v, v ⟩} = ⟨ T v, v ⟩ - ⟨ v, T v ⟩ = ⟨ T v, v ⟩ - ⟨ T^{*} v, v ⟩ = ⟨ (T - T^{*}) v, v ⟩

If $⟨ T v, v ⟩ \in R$ for every $v \in V$ then the LHS is 0, so the RHS is 0. Thus $T - T^{*} = 0$ so $T$ is self-adjoint.

Conversely if $T$ is self-adjoint then the RHS is 0, so $⟨ T v, v ⟩ = \overset{―}{T v, v}$ for every $v \in V$ . Thus $⟨ T v, v ⟩ \in R$ .
\end{proof}

T = T^{*}

and

⟨ T v, v ⟩ = 0

for all

v

, then

T = 0

Suppose $T$ is a self-adjoint operator on $V$ , such that:

⟨ T v, v ⟩ = 0

for all $v \in V$ . Then $T = 0$ .

\begin{proof}
See Chapter 7 - Operators on Inner Product Spaces#^bb6e99 for when $V$ is a complex inner product space. Assume $V$ is a real inner product space. If $u, w \in V$ then:

⟨ T u, w ⟩ = \frac{⟨ T (u + w), u + w ⟩ - ⟨ T (u - w), u - w ⟩}{4}

using the property that $T$ is self adjoint where:

⟨ T w, u ⟩ = ⟨ w, T u ⟩ = ⟨ T u, w ⟩

Each term on the RHS of the equation on top is of the form $⟨ T v, v ⟩$ , so $⟨ T u, w ⟩ = 0$ for all $u, w \in V$ , so taking $w = T u$ implies that $T = 0$ .
\end{proof}

Normal Operators

normal

An operator on an inner product space is called normal if it commutes with its adjoint.
$T \in L (V)$ is normal if $T T^{*} = T^{*} T$ .

Clearly every self-adjoint operator is normal.

T

is normal iff

‖ T v ‖ = ‖ T^{*} v ‖

for all

v

An operator $T \in L (V)$ is normal iff:

‖ T v ‖ = ‖ T^{*} v ‖

for all $v \in V$ .

\begin{proof}
Let $T \in L (V)$ :

\begin{aligned} T is normal & \Leftrightarrow T^{*} T - T T^{*} = 0 \\ \Leftrightarrow ⟨ (T^{*} T - T T^{*}) v, v ⟩ = 0 \forall v \in V \\ \Leftrightarrow ⟨ T^{*} T v, v ⟩ = ⟨ T T^{*} v, v ⟩ \forall v \in V \\ \Leftrightarrow ‖ T v ‖^{2} = ‖ T^{*} v ‖^{2} \forall v \in V \end{aligned}

using Chapter 7 - Operators on Inner Product Spaces#^427201 for the second equivalence.
\end{proof}
Compare the next lemma with that of HW 3 - Self-Adjoint and Normal Operators#2.

For

T

normal,

T, T^{*}

have the same eigenvectors

Suppose $T \in L (V)$ is normal and $v \in V$ is an eigenvector of $T$ with eigenvalue $λ$ . Then $v$ is also an eigenvector of $T^{*}$ with eigenvalue $\overset{―}{λ}$ .

\begin{proof}
Because $T$ is normal so is $T - λ I$ . Using Chapter 7 - Operators on Inner Product Spaces#^f09319, we have:

0 = ‖ (T - λ I) v ‖ = ‖ (T - λ I)^{*} v ‖ = ‖ (T^{*} - \overset{―}{λ} I) v ‖

so $v$ is an eigenvector of $T^{*}$ with eigenvalue $\overset{―}{λ}$ as desired.
\end{proof}

Orthogonal eigenvectors for normal operators

Suppose $T \in L (V)$ is normal. Then eigenvectors of $T$ corresponding to distinct eigenvalues are orthogonal.

\begin{proof}
Suppose $α, β$ are distinct eigenvalues of $T$ with corresponding eigenvectors $u, v$ . Thus $T u = α u$ and $T v = β v$ . From Chapter 7 - Operators on Inner Product Spaces#^d72e15, we have $T^{*} v = \overset{―}{β} v$ so then:

\begin{aligned} (α - β) ⟨ u, v ⟩ & = ⟨ α u, v ⟩ - ⟨ u, \overset{―}{β} v ⟩ \\ = ⟨ T u, v ⟩ - ⟨ u, T^{*} v ⟩ \\ = 0 \end{aligned}

so since $α \neq β$ then we must have $⟨ u, v ⟩$ as expected.
\end{proof}

7.B: Spectral Theorem

Recall that a diagonal matrix is a square matrix that is 0 except possibly on the diagonal. Recall that an operator on $V$ has a diagonal matrix w.r.t a basis iff the basis consists of eigenvectors of the operator via Chapter 5 - Eigenvalues, Eigenvectors, and Invariant Subspaces#^a68a4e.

The nicest operators on $V$ are those for which there is an orthonormal basis of $V$ with respect to which the operator has a diagonal matrix. These are precisely the operatrors $T \in L (V)$ such that there is an orthonormal basis of $V$ consisting of eigenvectors of $T$ .

The Complex Spectral Theorem

The key part of the Complex Spectral Theorem states that if $F = C$ and $T \in L (V)$ is normal, then $T$ has a diagonal matrix with respect to some orthonormal basis of $V$ .

For example, consider $T \in L (C^{2})$ whose matrix w.r.t the standard basis is:

(\begin{matrix} 2 & - 3 \\ 3 & 2 \end{matrix})

As you can see, $\frac{(i, 1)}{\sqrt{2}}, \frac{(- i, 1)}{\sqrt{2}}$ is an orthonormal basis of $C^{2}$ consisting of eigenvectors of $T$ , and with respect to this basis the matrix of $T$ is the diagonal matrix:

(\begin{matrix} 2 + 3 i & 0 \\ 0 & 2 - 3 i \end{matrix})

Complex Spectral Theorem

Suppose $F = C$ and $T \in L (V)$ . Then the following are equivalent:

$T$ is normal
$V$ has an orthonormal basis consisting of eigenvectors of $T$ .
$T$ has a diagonal matrix with respect to some orthonormal basis of $V$ .

\begin{proof}

(b) $=$ (c) via Chapter 5 - Eigenvalues, Eigenvectors, and Invariant Subspaces#^a68a4e.

Thus we only need to prove that (c) $\equiv$ (a) .

(c) $\to$ (a): Suppose (c), so $T$ has a diagonal matrix w.r.t. some orthonormal basis of $V$ . The matrix of $T^{*}$ w.r.t the same basis is obtained by taking the conjugate transpose of the matrix of $T$ ; hence $T^{*}$ also has a diagonal matrix. Any two diagonal matrices commute, so $T$ commutes with $T^{*}$ , so $T$ is normal showing (a)

(a) $\to$ (c): Suppose (a), so $T$ is normal. By Chapter 6 - Inner Product Spaces#^71c5e4, there is an orthonormal basis $e_{1}, . . ., e_{n}$ of $V$ w.r.t. which $T$ has an UT matrix. Thus:

M (T, (e_{1}, . . ., e_{n})) = [\begin{matrix} a_{11} & \dots & a_{1 n} \\ ⋮ & ⋱ & ⋮ \\ 0 & \dots & a_{n n} \end{matrix}]

We will show this matrix is actually diagonal. From above:

‖ T e_{1} ‖^{2} = | a_{11} |^{2}

and:

‖ T^{*} e_{1} ‖^{2} = \sum_{i = 1}^{n} | a_{1 i} |^{2}

Because $T$ is normal, then two lines above are actually equal. This implies that:

\sum_{i = 2}^{n} | a_{1 i} |^{2} = 0 \Leftrightarrow \forall i \in {2, . . ., n} (a_{1 i} = 0)

Similarly then:

‖ T e_{2} ‖^{2} = | a_{22} |^{2} = ‖ T^{*} e_{2} ‖^{2} = \sum_{i = 2}^{n} | a_{2 i} |^{2}

So then $a_{23}, . . ., a_{2 n} = 0$ .

Continue in this fashion to see that all non-diagonal entries in the matrix equal 0, so (c) holds.
\end{proof}

The Real Spectral Theorem

Let's get a few preliminary results.

Invertible quadratic expressions

Suppose $T \in L (V)$ is self-adjoint and $b, c \in R$ such that $b^{2} < 4 c$ . Then:

T^{2} + b T + c I

is invertible.

\begin{proof}
Let $v \neq \vec{0} \in V$ . Then:

\begin{aligned} ⟨ (T^{2} + b T + c I) v, v ⟩ & = ⟨ T^{2} v, v ⟩ + b ⟨ T v, v ⟩ + c ⟨ v, v ⟩ \\ = ⟨ T v, T v ⟩ + b ⟨ T v, v ⟩ + c ‖ v ‖^{2} \\ \geq ‖ T v ‖^{2} - | b | ‖ T v ‖ ‖ v ‖ + c ‖ v ‖^{2} \\ = {(‖ T v ‖ - \frac{| b | ‖ v ‖}{2})}^{2} + (c - \frac{b^{2}}{4}) ‖ v ‖^{2} \\ > 0 \end{aligned}

where the third line is via Cauchy-Schartz from Chapter 6 - Inner Product Spaces#^2ef5ba. The last inequality implies that $(T^{2} + b T + c I) v \neq 0$ so then $T^{2} + b T + c I$ is injective, implying invertibility via Chapter 3 - Linear Maps#^315386.
\end{proof}

Self-adjoint operators have eigenvalues

Suppose $V \neq {\vec{0}}$ and $T \in L (V)$ is a self-adjoint operator. Then $T$ has an eigenvalue.

\begin{proof}
We can assume $V$ is a real inner product space. Let $n = dim (V)$ and choose $v \in V$ with $v \neq 0$ . Then:

v, T v, T^{2} v, . . ., T^{n} v

cannot be linearly independent since $V$ has dimension $n$ and we have $n + 1$ vectors. Thus there exist real numbers $a_{i}$ not all 0 such that:

\sum_{i = 0}^{n} a_{i} T^{n} v = 0

Make the $a_{i}$ 's the coefficients of a polynomial, which can be written in factored form via Chapter 4 - Polynomials (short)#^63c018:

\begin{aligned} 0 & = \sum_{i = 0}^{n} a_{i} T^{n} v \\ = (\sum_{i = 0}^{n} a_{i} T^{n}) v \\ = c (T^{2} + b_{1} T + c_{1} I) \dots (T^{2} + b_{M} T + c_{M} I) (T - λ_{1} I) \dots (T - λ_{m} I) v \end{aligned}

Using Chapter 7 - Operators on Inner Product Spaces#^94dc7b, each $T^{2} + b_{j} T + c_{j} I$ is invertible. Recall also that $c \neq 0$ so then $m > 0$ and thus:

0 = (T - λ_{1} I) \dots (T - λ_{m} I) v

Hence $T - λ_{j} I$ is not injective for at least one $j$ so $T$ has an eigenvalue with eigenvector $v$ .
\end{proof}

Self-adjoint operators and invariant subspaces

Suppose $T \in L (V)$ is self-adjoint and $U$ is a subspace of $V$ that is invariant under $T$ . Then:

$U^{⊥}$ is invariant under $T$ .
$T |_{U} \in L (U)$ is self-adjoint
$T |_{U^{⊥}} \in L (U^{⊥})$ is self-adjoint.

\begin{proof}
(a): Suppose $v \in U^{⊥}$ . Let $u \in U$ . Then:

⟨ T v, u ⟩ = ⟨ v, T u ⟩ = 0

since $T = T^{*}$ , and the second equality comes from how $U$ is invariant under $T$ so $T u \in U$ and since $v \in U^{⊥}$ we get it equals 0. Because the equation holds for each $u \in U$ we conclude that $T v \in U^{⊥}$ so then $U^{⊥}$ is invariant under $T$ .

(b): Note if $u, v \in U$ then:

⟨ (T |_{U}) u, v ⟩ = ⟨ T u, v ⟩ = ⟨ u, T v ⟩ = ⟨ u, (T |_{U}) v ⟩

Thus $T |_{U}$ is self-adjoint.

(c): Replace $U$ with $U^{⊥}$ in (b), which is allowed via (a).
\end{proof}

Real Spectral Theorem

Suppose $F = R$ and $T \in L (V)$ . Then the following are equivalent:

$T$ is self-adjoint
$V$ has an orthonormal basis consisting of eigenvectors of $T$ .
$T$ has a diagonal matrix with respect to some orthonormal basis of $V$

\begin{proof}
(c) $\to$ (a): $T$ is self-adjoint iff $T$ has a diagonal matrix w.r.t. some orthonormal basis of $V$ . A diagonal matrix equals its transpose, so $T = T^{*}$ , thus $T$ is self-adjoint showing (a).

(a) $\to$ (b): Do induction over $dim (V)$ . If it's 1, then (a) implies (b) via a trivial case. Suppose that $dim (V) > 1$ and (a) implies (b) for all real inner product spaces of smaller dimension. Suppose (a), so then $T$ is self adjoint. Let $u$ be an eigenvector of $T$ with $‖ u ‖ = 1$ , which is guarunteed by Chapter 7 - Operators on Inner Product Spaces#^ea761b, where this eigenvector can be divided by its norm to produce a unit eigenvector.

Let $U = span (u)$ . Then $U$ is a 1-dimensional subspace of $V$ that is invariant under $T$ , so by Chapter 7 - Operators on Inner Product Spaces#^41adf0 (c) the operator $T |_{U^{⊥}} \in L (U^{⊥})$ is self-adjoint.

By the inductive hypothesis, there is an orthonormal basis of $U^{⊥}$ consisting of eigenvectors of $T |_{U^{⊥}}$ . Adjoining $u$ to this orthonormal basis of $U^{⊥}$ gives an orthonormal basis of $V$ consisting of eigenvectors of $T$ , completing the proof for (a) $\to$ (b).

(b) $\to$ (c): Trivial
\end{proof}

7.C: Positive Operators and Isometries

positive operator

An operator $T \in L (V)$ is called positive if $T$ is self-adjoint and:

⟨ T v, v ⟩ \geq 0

for all $v \in V$ .

If $V$ is a complex vector space then the requirement that $T$ is self-adjoint can be dropped from the definition by Chapter 7 - Operators on Inner Product Spaces#^a30225. We require it for real vector spaces though.

As an example, if $U$ is a subspace of $V$ then the orthogonal projection $P_{U}$ is a positive operator.

As another example, if $T \in L (V)$ is self-adjoint and $b, c \in R$ are such that $b^{2} < 4 c$ then $T^{2} + b T + c I$ is a positive operator via the proof for Chapter 7 - Operators on Inner Product Spaces#^94dc7b.

square root

An operator $R$ is called a square root of an operator $T$ if $R^{2} = T$ .

As an example, if $T \in L (F^{3})$ is defined by $T (z_{1}, z_{2}, z_{3}) = (z_{3}, 0, 0)$ then the operator $R \in L (F^{3})$ defined by $R (z_{1}, z_{2}, z_{3}) = (z_{2}, z_{3}, 0)$ is a square root of $T$ .

Notice that we said a square root. The square root is only unique in very specific circumstances. Further, the characterizations of the positive operators in the next result correspond to the characterizations of the nonnegative numbers among $C$ . Specifically:

$z \in C$ is nonnegative iff $z$ has a nonnegative square root.
$z$ is nonnegative iff it has a real square root
$z$ is nonnegative iff there exists a complex number $w$ where $z = \overset{―}{w} w$ .

These conditions are similar to the one below:

Characterization of positive operators

Let $T \in L (V)$ . Then the following are equivalent:

$T$ is positive
$T$ is self-adjoint and all eigenvalues of $T$ are non-negative
$T$ has a positive square root
$T$ has a self-adjoint square root
$\exists R \in L (V)$ s.t. $T = R^{*} R$ .

\begin{proof}
We prove (a) $\to$ (b) $\to$ (c) $\to$ (d) $\to$ (e) $\to$ (a).

(a) $\to$ (b): Suppose $T$ is positive. $T$ is self-adjoint from the definition. To show all eigenvalues are non-negative, suppose $λ$ is an eigenvalue of $T$ and $v$ is it's eigenvector:

0 \leq ⟨ T v, v ⟩ = ⟨ λ v, v ⟩ = λ ⟨ v, v ⟩

So since $⟨ v, v ⟩ \geq 0$ then $λ \geq 0$ .

(b) $\to$ (c): Suppose $T$ is self-adjoint and all $λ$ 's are non-negative. By the Chapter 7 - Operators on Inner Product Spaces#^6995f7 or Chapter 7 - Operators on Inner Product Spaces#^d93d1a, there is an orthonormal basis $e_{1}, . . ., e_{n}$ of $V$ all eigenvectors. Let $λ_{1}, . . ., λ_{n}$ be these eigenvalues. Thus $λ_{j}$ is non-negative. Let $R$ be the linear map from $V \to V$ such that:

R e_{j} = \sqrt{λ_{j}} e_{j}

for $j = 1, . . ., n$ via Chapter 3 - Linear Maps#^9a80a8. Then $R$ is a positive operator through verification. Furthermore, $R^{2} e_{j} = λ_{j} e_{j} = T e_{j}$ for each $j$ . Thus $R^{2} = T$ . Thus $R$ is a positive square root of $T$ showing (c).

(d) $\to$ (e): There is some self-adjoint operator $R$ on $V$ where $R^{2} = T$ . Then $T = R^{*} R$ because $R = R^{*}$ , so $R$ is self-adjoint.

(e) $\to$ (a): Let $R \in L (V)$ be such that $T = R^{*} R$ . Then $T^{*} = (R^{*} R)^{*} = R^{*} R = T$ so $T$ is self-adoint. To show positivity:

⟨ T v, v ⟩ = ⟨ R^{*} R v, v ⟩ = ⟨ R v, R v ⟩ = ‖ R v ‖^{2} \geq 0

thus $T$ is positive.
\end{proof}

Each positive operator has only one positive square root

Every positive operator on $V$ has a unique positive square root.

\begin{proof}
Suppose $T \in L (V)$ is positive. Suppose $v \in V$ is an eigenvector of $T$ . There exists $λ \geq 0$ such that $T v = λ v$ .

Let $R$ be a positive square root of $T$ . We'll show $R v = \sqrt{λ} v$ , implying that the behavior of the eigenvectors of $R$ is uniquely determined. Because there is a basis of $V$ consisting of eigenvectors of $T$ (by the Chapter 7 - Operators on Inner Product Spaces#^6995f7) then $R$ is uniquely determined.

To prove that $R v = \sqrt{λ} v$ , use the Spectral Theorem to say that there is an orthonormal basis $e_{1}, . . ., e_{n}$ of $V$ of eigenvectors from $R$ . Because $R$ is a positive operator, all its eigenvalues are nonnegative via Chapter 7 - Operators on Inner Product Spaces#^aaa8f3. Thus there exist nonnegative number $λ_{1}, . . ., λ_{n}$ such that $R e_{j} = \sqrt{λ_{j}} e_{j}$ for $j = 1, . . ., n$ .

Because $e_{1}, . . ., e_{n}$ is a basis of $V$ we can write:

v = \sum_{i = 1}^{n} α_{i} e_{i}

where $α_{i} \in F$ . Thus:

R v = \sum_{i = 1}^{n} α_{i} \sqrt{λ_{i}} e_{i} \Rightarrow R^{2} v = \sum_{i = 1}^{n} α_{i} λ_{i} e_{i}

Thus the equation for $R^{2} v$ implies that $α_{i} (λ - λ_{i}) = 0$ for all $i$ . Hence ignoring where $α = 0$ :

v = \sum_{j : λ_{j} = λ} α_{j} e_{j}

thus:

R v = \sum_{j : λ_{j} = λ} a_{j} \sqrt{λ} e_{j} = \sqrt{λ} v

\end{proof}

Isometries

isometry

An operator $S \in L (V)$ is called an isometry if:

‖ S v ‖ = ‖ v ‖

for all $v \in V$ .

In other words, an operator is an isometry if it preserves norms.

For example, $λ I$ is an isometry if $λ \in F$ satisfies $| λ | = 1$ .

As an example, let $λ_{i}$ all be scalars with absolute value 1 and $S \in L (V)$ satisfies $S e_{j} = λ_{j} e_{j}$ for some orthonormal basis $e_{1}, . . ., e_{n}$ of $V$ . $S$ is an isometry since for any $v \in V$ :

v = \sum_{i = 1}^{n} ⟨ v, e_{i} ⟩ e_{i}

thus:

‖ v ‖^{2} = \sum_{i = 1}^{n} | ⟨ v, e_{i} ⟩ |^{2}

where we have used Chapter 6 - Inner Product Spaces#^ef9c3b. Applying $S$ to both sides of the first equation gives:

\begin{aligned} S v & = \sum_{i = 1}^{n} ⟨ v, e_{i} ⟩ S e_{i} \\ = \sum_{i = 1}^{n} λ_{i} ⟨ v, e_{i} ⟩ e_{i} \end{aligned}

Using the fact that $λ_{i}$ has magnitude 1:

‖ S v ‖^{2} = \sum_{i = 1}^{n} | ⟨ v, e_{i} ⟩ |^{2} = ‖ v ‖^{2}

Thus $S$ is an isometry.

For the next lemma:

(a) and (b) shows that an isometry always preserves inner produces
(a) and (c)/(d) shows an operator is an isometry iff the list of columns of its matrix with respect to every basis is orthonormal.

Characterization of isometries

$S$ is an isometry
$⟨ S u, S v ⟩ = ⟨ u, v ⟩$ for all $u, v \in V$
$S e_{1}, . . ., S e_{n}$ is orthonormal for every orthonormal list of vectors $e_{1}, . . ., e_{n}$ in $V$
$\exists$ an orthonormal basis $e_{1}, . . ., e_{n}$ of $V$ such that $S e_{1}, . . ., S e_{n}$ is orthonormal.
$S^{*} S = I$
$S S^{*} = I$
$S^{*}$ is an isometry
$S$ is invertible and $S^{- 1} = S^{*}$ .

\begin{proof}
We go (a) $\to$ (b) $\to$ $\dots$ $\to$ (h) $\to$ (a)

(a) $\to$ (b): Suppose $S$ is an isometry. Using HW 6 - Finishing UT Matrices, Eigenspaces and Diagonal Matrices#20 and 19 to get that inner product can be computer from norms. Because $S$ preserves norms, then $S$ preserves inner products, so (b). In actually calculating this:

\begin{aligned} ⟨ S u, S v ⟩ & = (‖ S u + S v ‖^{2} - ‖ S u - S v ‖^{2}) / 4 & 6.A Exercise 19 \\ = (‖ S (u + v) ‖^{2} - ‖ S (u - v) ‖^{2}) / 4 & S \in L (V) \\ = (‖ u + v ‖^{2} - ‖ u - v ‖^{2}) / 4 & S is an isometry \\ = ⟨ u, v ⟩ & 6.A Exercise 19 \end{aligned}

When $V$ is a complex inner space, HW 6 - Finishing UT Matrices, Eigenspaces and Diagonal Matrices#20 will give you the same answer. Either case, (b) holds.

(b) $\to$ (c): $S$ preserves inner products. Suppose $e_{1}, . . ., e_{n}$ is an orthonormal list of vectors in $V$ . Then we see that the list $S e_{1}, . . ., S e_{n}$ is orthonormal because $⟨ S e_{j}, S e_{k} ⟩ = ⟨ e_{j}, e_{k} ⟩$ by definition. Thus (c) holds.

(d) $\to$ (e): Let $e_{1}, . . ., e_{n}$ be an orthonormal basis of $V$ such that $S e_{1}, . . ., S e_{n}$ is orthonormal. Thus:

⟨ S^{*} S e_{j}, e_{k} ⟩ = ⟨ e_{j}, e_{k} ⟩

for $j, k = 1, . . ., n$ (because the left term equals $⟨ S e_{j}, S e_{k} ⟩$ and $S e_{1}, . . ., S e_{n}$ is orthonormal). All vectors $u, v \in V$ can be written as a linear combination of $e_{1}, . . ., e_{n}$ and thus the equation above implies $⟨ S^{*} S u, v ⟩ = ⟨ u, v ⟩$ . Hence $S^{*} S = I$ so (e).

(e) $\to$ (f): $S S^{*} = I$ . In general an operator $S$ need not commute with $S^{*}$ but $S^{*} S = I$ iff $S S^{*} = I$ , a special case of 3.D Exercise 10. Thus $S S^{*} = I$ showing (f).

(f) $\to$ (g): If $v \in V$ :

‖ S^{*} v ‖^{2} = ⟨ S^{*} v, S^{*} v ⟩ = ⟨ S S^{*} v, v ⟩ = ⟨ v, v ⟩ = ‖ v ‖^{2}

So $S^{*}$ is an isometry (g).

(g) $\to$ (h): $S^{*}$ is an isometry. Using (a) $\to$ (e) and (a) $\to$ (f), using $S$ replaced with $S^{*}$ (and using $(S^{*})^{*} = S$ ) then $S S^{*} = S^{*} S = I$ . Thus $S$ is invertible and $S^{- 1} = S^{*}$ , showing (h).

(h) $\to$ (a): $S$ is invertible and $S^{- 1} = S^{*}$ . Thus $S^{*} S = I$ . If $v \in V$ :

‖ S v ‖^{2} = ⟨ S v, S v ⟩ = ⟨ S^{*} S v, v ⟩ = ⟨ v, v ⟩ = ‖ v ‖^{2}

So $S$ is an isometry, showing (a).
\end{proof}
Since every isometry is normal using the previous lemma, then we can use these characterizations to describe isometries:

Description of isometries when

F = C

Suppose $V$ is a complex inner product space and $S \in L (V)$ . Then the following are equivalent:

$S$ is an isometry
There is an orthonormal basis of $V$ consisting of eigenvectors of $S$ whose corresponding eigenvalues all have absolute value of 1.

\begin{proof}
We have shown that (b) implies (a) via our example near the top of Chapter 7 - Operators on Inner Product Spaces#Isometries.

To prove (a) $\to$ (b), suppose $S$ is an isometry. By the Chapter 7 - Operators on Inner Product Spaces#^6995f7, there is an orthonormal basis $e_{1}, . . ., e_{n}$ of $V$ consisting of eigenvectors of $S$ . For $j = 1, . . ., n$ let $λ_{j}$ be the eigenvalue corresponding to $e_{j}$ . Then:

| λ_{j} | = ‖ λ_{j} e_{j} ‖ = ‖ S e_{j} ‖ = ‖ e_{j} ‖ = 1

Each $| λ_{j} | = 1$ as desired.
\end{proof}

7.D: Polar Decomposition and Singular Value Decomposition

Polar Decomposition

Recall the analogy between $C$ and $L (V)$ :

A complex number $z$ corresponds to some operator $T$
$\overset{―}{z}$ corresponds to $T^{*}$
Real numbers, so $z = \overset{―}{z}$ corresponds to self-adjoint operators where $T = T^{*}$
Non-negative reals correspond to the positive operators

Another interesting characteristic of $C$ is the unit circle, $| z | = 1 = z \overset{―}{z}$ . Under our analogy this implies that $T T^{*}$ is "on" the unit circle, which is corresponds to our isometry definition. So $C$ corresponds to isometries

All complex numbers can be written as:

z = \frac{z}{| z |} | z | = \frac{z}{| z |} \sqrt{\overset{―}{z} z}

Which we do similarly with operators:

\sqrt{T}

If $T$ is a positive operator then $\sqrt{T}$ denotes the unique positive square root of $T$ .

Polar Decomposition

Suppose $T \in L (V)$ . Then $\exists$ an isometry $S \in L (V)$ such that:

T = S \sqrt{T^{*} T}

\begin{proof}
While the proof in the book is alright, I think the lecture proof we did is a bit better at understanding what's going on. The proofs are essentially identical, but this one is more intuitive.
\end{proof}
The Polar Decomposition Theorem states that each $T \in L (V)$ is the product of an isometry and a positive operator. This allows us to turn a complicated operator into two, easy to work with operators.

Using Spectral Theorem under $C = F$ , then $T = S \sqrt{T^{*} T}$ is a polar decomposition of $T \in L (V)$ where $S$ is an isometry. So there is an orthonormal basis of $V$ consisting of eigenvectors of $S$ whose corresponding eigenvalues all have magnitude 1 (and thus there's a diagonal matrix), and there is an orthonormal basis of $V$ with respect to $\sqrt{T^{*} T}$ has a diagonal matrix.

Possible difference in ONEB's

Even though what's said above is true, we may have different orthonormal eigenbases

Singular Value Decomposition

While the eigenvalues of an operator tell us something about the behavior of an operator, singular values are essentially the square roots of the eigenvalues of $T$ 's positive operator decomposition (essentially doing the same thing):

singular values

Suppose $T \in L (V)$ . The singular values of $T$ are the eigenvalues of $\sqrt{T^{*} T}$ with each eigenvalues $λ$ repeated $dim (E (λ, \sqrt{T^{*} T}))$ time.

The singular values of $T$ must be all non-negative, because they are eigenvalues of a positive operator $\sqrt{T^{*} T}$ .

Example

Consider $T \in L (F^{4})$ where:

T (z_{1}, z_{2}, z_{3}, z_{4}) = (0, 3 z_{1}, 2 z_{2}, - 3 z_{4})

To find the singular values of $T$ , calculate $T^{*} T$ first. Notice:

T (e_{1}) = (0, 3, 0, 0), T (e_{2}) = (0, 0, 2, 0), T (e_{3}) = (0, 0, 0, 0), T (e_{4}) = (0, 0, 0, - 3)

Thus fix $y \in F^{4}$ and let $x \in F^{4}$ be arbitrary:

\begin{aligned} ⟨ x, T^{*} y ⟩ & = ⟨ T x, y ⟩ \\ = ⟨ (0, 3 x_{1}, 2 x_{2}, - 3 x_{4}), (y_{1}, y_{2}, y_{3}, y_{4}) ⟩ \\ = 3 x_{1} y_{2}^{*} + 2 x_{2} y_{3}^{*} - 3 x_{4} y_{4}^{*} \\ = 3 x_{1} y_{2}^{*} + 2 x_{2} y_{3}^{*} + 0 x_{3} - 3 x_{4} y_{4}^{*} \\ = ⟨ x, (3 y_{2}^{*}, 2 y_{3}^{*}, 0, - 3 y_{4}^{*}) ⟩ \end{aligned}

Thus $T^{*} (x) = (3 x_{2}^{*}, 2 x_{3}^{*}, 0, - 3 x_{4}^{*})$ . So:

T^{*} T (x) = T^{*} (0, 3 x_{1}, 2 x_{2}, - 3 x_{4}) = (9 x_{1}, 4 x_{2}, 0, 9 x_{4})

So then $\sqrt{T^{*} T}$ are just the square-rooted eigenvalues of $T^{*} T$ :

\sqrt{T^{*} T} (x) = (3 x_{1}, 2 x_{2}, 0, 3 x_{4})

With eigenvalues $3, 2, 0$ and:

dim (E (3, \sqrt{T^{*} T})) = 2, dim (E (2, \sqrt{T^{*} T})) = 1, dim (E (0, \sqrt{T^{*} T})) = 1

so our singular values are $3, 3, 2, 0$ .

Note

While the eigenvalues of $T$ are only $3, 0$ , the singular values incorporate the missing $2$ singular value.

Applying Spectral Theorem and diagonalizability conditions (e), each $T \in L (V)$ has $dim (V)$ singular values (apply to $\sqrt{T^{*} T}$ ). The operator from the prior example has 4 singular values since $dim (F^{4}) = 4$ .

Singular-Value Decomposition

Suppose $T \in L (V)$ has singular values $s_{1}, . . ., s_{n}$ . Then there exist orthonormal bases $e_{1}, . . ., e_{n}$ and $f_{1}, . . ., f_{n}$ such that:

T v = s_{1} ⟨ v, e_{1} ⟩ f_{1} + \dots + s_{n} ⟨ v, e_{n} ⟩ f_{n}

for every $v \in V$ .

\begin{proof}
Apply the lecture proof.
\end{proof}
The idea in the proof seen above is that by using two different bases for our $M (T)$ , we can always get a diagonal matrix! In other words, every operator on $V$ has a diagonal matrix w.r.t. some orthonormal bases of $V$ , assuming we can use different bases $e_{i}$ and $f_{i}$ .

To compute singular values for computational linear algebra:

Compute $T^{*} T$
Compute approximations to eigenvalues of $T^{*} T$
Square roots of these approximations are our singular values

We don't have to compute $\sqrt{T^{*} T}$ !

Singular values without taking square root of an operator

Suppose $T \in L (V)$ . Then the singular values of $T$ are the nonnegative square roots of the eigenvalues of $T^{*} T$ with each eigenvalue $λ$ repeated $dim (E (λ, T^{*} T))$ times.

\begin{proof}
The Spectral Theorem implies there is an orthonormal basis $e_{1}, . . ., e_{n}$ and nonnegative numbers $λ_{1}, . . ., λ_{n}$ such that $T^{*} T e_{j} = λ_{j} e_{j}$ for all $j$ . Clearly $\sqrt{T^{*} T} e_{j} = \sqrt{λ_{j}} e_{j}$ for all $j$ via Chapter 7 - Operators on Inner Product Spaces#^1a3d65, which gives the desired result.
\end{proof}