Chapter 6 (cont.) - Finishing Inner Product Spaces

We continue from the discussion of linear functions via Chapter 6 - Inner Product Spaces#^4afaec.

Linear Functionals on Inner Product Spaces

See our definitions of linear functional via 3.F in:

linear functional

A linear functional of vector space $V$ is an element of $L (V, F)$ .

But notice that now inner products, over some constant $v \in V$ can be formed via:

ϕ : V \to F, ϕ (v_{i n}) = ⟨ v_{i n}, v ⟩

where $ϕ \in L (V, F)$ . For example, consider $φ : P_{2} (R) \to R$ given by:

φ (p) = \int_{- 1}^{1} p (t) (\cos (π t)) d t

is a linear functional. But it is not obvious that for all $p \in P_{2} (R)$ that:

φ (p) = ⟨ p, u ⟩

because notice that $u = \cos (π t) \notin P_{2} (R)$ .

Riesz Representation Theorem

Suppose $V$ is finite-dimensional and $φ$ is a linear functional on $V$ . Then there is a unique vector $u \in V$ such that:

φ (v) = ⟨ v, u ⟩

for every $v \in V$ .

\begin{proof}
First we show there exists a vector $u \in V$ such that $φ (v) = ⟨ v, u ⟩$ for every $v \in V$ . Let $e_{1}, . . ., e_{n}$ be an orthonormal basis of $V$ . Then:

\begin{aligned} φ (v) & = φ (\sum_{i = 1}^{n} ⟨ v, e_{i} ⟩ e_{i}) \\ = \sum_{i = 1}^{n} ⟨ v, e_{i} ⟩ φ (e_{i}) \\ = ⟨ v, \sum_{i = 1}^{n} \overset{―}{φ (e_{i})} e_{i} ⟩ \end{aligned}

Where the first equality comes from Chapter 6 - Inner Product Spaces#^ef9c3b. Thus, setting:

u = \sum_{i = 1}^{n} \overset{―}{φ (e_{i})} e_{i}

Then $φ (v) = ⟨ v, u ⟩$ for every $v \in V$ as desired.

Now we prove that only one vector $u \in V$ has the desired behavior. Suppose $u_{1}, u_{2} \in V$ are such that:

⟨ v, u_{1} ⟩ = ⟨ v, u_{2} ⟩ = φ (v)

for every $v \in V$ . Then:

0 = ⟨ v, u_{1} ⟩ - ⟨ v, u_{2} ⟩ = ⟨ v, u_{1} - u_{2} ⟩

for every $v \in V$ . Taking $v = u_{1} - u_{2}$ shows that then $u_{1} - u_{2} = 0$ so then $u_{1} = u_{2}$ , showing uniqueness.
\end{proof}
As an example, let's find $u \in P_{2} (R)$ such that:

\int_{- 1}^{1} p (t) (\cos (π t)) d t = \int_{- 1}^{1} p (t) u (t) d t

for all $p \in P_{2} (R)$ . The way to do this is to let:

φ (p) = \int_{- 1}^{1} p (t) \cos (π t) d t

Where clearly $φ$ is linear here. As such, then apply Chapter 6 (cont.) - Finishing Inner Product Spaces#^a857c6, by applying:

$u = \sum_{i = 1}^{n} \overset{―}{φ (e_{i})} e_{i}$

Thus then using the orthonormal basis from an earlier example:

\begin{aligned} u (x) & = \sum_{i = 1}^{3} \overset{―}{φ (e_{i})} e_{i} \\ = (\int_{- 1}^{1} \sqrt{\frac{1}{2}} \cos (π t) d t) \sqrt{\frac{1}{2}} + (\int_{- 1}^{1} \sqrt{\frac{3}{2}} t \cos (π t) d t) \sqrt{\frac{3}{2}} x \\ + (\int_{- 1}^{1} \sqrt{\frac{45}{8}} (t^{2} - 1 / 3) \cos (π t) d t) \sqrt{\frac{45}{8}} (x^{2} - 1 / 3) \end{aligned}

where doing a bit of simplification shows that:

u (x) = \frac{- 45}{2 π^{2}} (x^{2} - 1 / 3)

Notice that our formula for $u$ is dependent on both our orthonormal basis $e_{1}, . . ., e_{n}$ as well as our chosen $φ$ . However, by Chapter 6 (cont.) - Finishing Inner Product Spaces#^a857c6, then $u$ is uniquely determined by $φ$ , so the RHS of our definition of $u$ is the same regardless of which orthonormal basis $e_{1}, . . ., e_{n}$ of $V$ that is chosen.

6.C: Orthogonal Complements and Minimization Problems

orthogonal complement,

U^{⊥}

If $U$ is a subset of $V$ , then the orthogonal complement of $U$ , denoted $U^{⊥}$ , is the set of all vectors in $V$ that are orthogonal to every vector in $U$ :

U^{⊥} = {v \in V : ⟨ v, u ⟩ = 0 \forall u \in U}

For instance, if $U$ is a line in $R^{3}$ then $U^{⊥}$ is the plane containing the origin that is perpendicular to $U$ . If $U$ is a plane in $R^{3}$ , then $U^{⊥}$ is the line containing the origin that is perpendicular to $U$ .

Basic properties of the orthogonal complement

If $U$ is a subset of $V$ , then $U^{⊥}$ is a subspace of $V$
${\vec{0}}^{⊥} = V$
$V^{⊥} = {\vec{0}}$
If $U$ is a subset of $V$ , then $U \cap U^{⊥} \subseteq {\vec{0}}$
If $U, W$ are subsets of $V$ and $U \subseteq W$ then $W^{⊥} \subseteq U^{⊥}$ .

\begin{proof}
(a) Suppose $U$ is a subset of $V$ . Then $⟨ 0, u ⟩ = 0$ for every $u \in U$ . So $\vec{0} \in U^{⊥}$ . Suppose $v, w \in U^{⊥}$ . If $u \in U$ then:

⟨ v + w, u ⟩ = ⟨ v, u ⟩ + ⟨ v, w ⟩ = 0 + 0 = 0

Thus $v + w \in U^{⊥}$ . If $λ \in F$ then:

⟨ λ v, u ⟩ = λ ⟨ v, u ⟩ = λ 0 = 0

Thus $λ v \in U^{⊥}$ . Thus $U^{⊥}$ is a subspace of $V$ .

(b) Suppose $v \in V$ . Then $⟨ v, 0 ⟩ = 0$ implying that $v \in {\vec{0}}^{⊥}$ so ${\vec{0}}^{⊥} = V$ .
(c) Suppose $v \in V^{⊥}$ . Then $⟨ v, v ⟩ = 0$ so then $v = \vec{0}$ thus $V^{⊥} = {\vec{0}}$ .
(d) Suppose $U \subseteq V$ and $v \in U \cap U^{⊥}$ . Then $⟨ v, v ⟩ = 0$ so $v = \vec{0}$ so $U \cap U^{⊥} \subseteq {\vec{0}}$ .
(e) Suppose $U, W \subseteq V$ and $U \subseteq W$ . Suppose $v \in W^{⊥}$ , then $⟨ v, u ⟩ = 0$ for all $u \in W$ . So $⟨ v, u ⟩ = 0$ for all $u \in U$ , so $v \in U^{⊥}$ thus $W^{⊥} \subseteq U^{⊥}$ .
\end{proof}
Recall that if $U, W$ are subspaces then $V$ is a direct sum $V = U \oplus W$ iff $U \cap W = {\vec{0}}$ . In a similar manner:

Direct sum of a subspace and its orthogonal complement

Suppose $U$ is a finite-dimensional subspace of $V$ . Then:

V = U \oplus U^{⊥}

\begin{proof}
$U$ is a subset of $V$ , so then $U \cap U^{⊥} \subseteq {\vec{0}}$ . Since $\vec{0} \in U, U^{⊥}$ then we have equality. Thus, if $V = U + U^{⊥}$ then we are good. Notice that we can always do that because if $v \in V$ then $e_{1}, . . ., e_{m}$ is an orthonormal basis of $U$ :

v = \underset{u}{\underset{⏟}{\sum_{i = 1}^{m} ⟨ v, e_{i} ⟩ e_{i}}} + \underset{w}{\underset{⏟}{v - \sum_{i = 1}^{m} ⟨ v, e_{i} ⟩ e_{i}}}

choose $u, w$ as defined above. Clearly $u \in U$ since $e_{1}, . . ., e_{m}$ is an orthonormal list. For each $j = 1, . . ., m$ we have:

⟨ w, e_{j} ⟩ = ⟨ v, e_{j} ⟩ - ⟨ v, e_{j} ⟩ = 0

so $w$ is orthogonal to each vector in $span (e_{1}, . . ., e_{m})$ , so $w \in U^{⊥}$ . Thus $v = u + w$ showing the sum $V = U + U^{⊥}$ .
\end{proof}

Dimension of the orthogonal complement

Suppose $V$ is finite-dimensional and $U$ is a subspace of $V$ . Then:

dim (U^{⊥}) = dim (V) - dim (U)

\begin{proof}
Apply Chapter 3 (cont.) - Products and Quotients of Vector Spaces#^5193d7.
\end{proof}

The orthogonal complement of the orthogonal complement

Suppose $U$ is finite dimensional subspace of $V$ . Then:

U = (U^{⊥})^{⊥}

\begin{proof}
First we show ( $\subseteq$ ). Suppose $u \in U$ . Then $⟨ u, v ⟩ = 0$ for every $v \in U^{⊥}$ . Because $u$ is orthogonal to every vector in $U^{⊥}$ then $u \in (U^{⊥})^{⊥}$ .

For the other direction, suppose $v \in (U^{⊥})^{⊥}$ . By Chapter 6 (cont.) - Finishing Inner Product Spaces#^8fdc3c, we can write $v = u + w$ where $u \in U$ and $w \in U^{⊥}$ . We have $v - u = w \in U^{⊥}$ . Because $v \in (U^{⊥})^{⊥}$ and $u \in (U^{⊥})^{⊥}$ (from using $\subseteq$ ) then $w = v - u \in (U^{⊥})^{⊥}$ by closure under v. addition. Thus $w = v - u \in U^{⊥} \cap (U^{⊥})^{⊥}$ , so then $v - u$ is orthogonal to itself, so $v - u = \vec{0}$ , so $v = u$ so $v \in U$ .
\end{proof}

orthogonal projection,

P_{U}

Suppose $U$ is a finite-dimensional subspace of $V$ . The orthogonal projection of $V$ onto $U$ is the operator $P_{U} \in L (V)$ defined as follows. For $v \in V$ write $v = u + w$ where $u \in U$ and $w \in U^{⊥}$ . Then $P_{U} v = u$ .

$P_{U}$ is well defined since we can use Chapter 6 (cont.) - Finishing Inner Product Spaces#^8fdc3c.

Properties of the orthogonal projection

P_{U}

$P_{U} \in L (V)$
$P_{U} u = u$ for all $u \in U$ .
$P_{U} w = 0$ for all $w \in U^{⊥}$ .
$range (P_{U}) = U$ .
$null (P_{U}) = U^{⊥}$ .
$v - P_{U} v \in U^{⊥}$
$P_{U}^{2} = P_{U}$
$‖ P_{U} v ‖ \leq ‖ v ‖$
For every orthonormal basis $e_{1}, . . ., e_{m}$ of $U$ :

P_{U} v = \sum_{i = 1}^{m} ⟨ v, e_{i} ⟩ e_{i}

\begin{proof}
The proof of these is very easy to do, and is found in 2015_Book_LinearAlgebraDoneRight#page=197.
\end{proof}

Minimization Problems

The following problem is really common: given a subspace $U$ of $V$ and a point $v \in V$ , find a point $u \in U$ such that $‖ v - u ‖$ is as small as possible. The next lemma shows that this problem is solved by taking $u = P_{U} v$ .

Minimizing the distance to a subspace

Suppose $U$ is a finite-dimensional subspace of $V$ , $v \in V$ and $u \in U$ . Then:

‖ v - P_{U} v ‖ \leq ‖ v - u ‖

Furthermore the inequality above is an equality iff $u = P_{U} v$ .

\begin{proof}

\begin{aligned} ‖ v - P_{U} v ‖^{2} & \leq ‖ v - P_{U} v ‖^{2} + ‖ P_{U} v - u ‖^{2} & (0 \leq ‖ P_{U} v - u ‖^{2}) \\ = ‖ (v - P_{U} v) + (P_{U} v - u) ‖^{2} & Pythagorean Theorem \\ = ‖ v - u ‖^{2} \end{aligned}

Here we only get equality if the top line is an equality, iff $‖ P_{U} v - u ‖ = 0$ iff $u = P_{U} v$ .
\end{proof}
Let's do a cool example! Let's find a polynomial $u$ with real coefficients and degree at most 5 that approximates $\sin (x)$ as well as possible on the interval $[- π, π]$ in the sense that:

\int_{- π}^{π} | \sin (x) - u (x) |^{2} d x

is as small as possible. We'll compare to the Taylor Series approximation. Let $C_{R} [- π, π]$ denote the real inner product space of continuous real-valued functions on $[- π, π]$ with inner product:

\int_{- π}^{π} f (x) g (x) d x

Let $v \in C_{R} [- π, π]$ be the function defined by $v (x) = \sin (x)$ . Let $U$ denote the subspace of $C_{R} [- π, π]$ consisting of the polynomials with real coefficients and degree at most 5. Our problem can be reformulated as:

Find $u \in U$ such that $‖ v - u ‖$ is as small as possible.

To compute this, apply Gram-Schmidt via Chapter 6 - Inner Product Spaces#^d43651 to the basis $β_{s} = {1, x, x^{2}, x^{3}, x^{4}, x^{5}}$ of $U$ , producing an orthonormal basis $e_{1}, . . ., e_{6}$ of $U$ . Then, again using the inner product given by the Gram-Schmidt Procedure, compute $P_{U} v$ using Chapter 6 (cont.) - Finishing Inner Product Spaces#^3861ef part (i), where for every orthonormal basis and using $m = 6$ in this case:

P_{U} v = \sum_{i = 1}^{6} ⟨ v, e_{i} ⟩ e_{i}

doing this here shows that $P_{U} v$ is the function $u$ defined by:

u (x) = 0.987862 x - 0.155271 x^{3} + 0.00564312 x^{5}

as a close approximation. And to see how good it is, check it out:

desmos-graph(1).png

Can you see the red sine wave? That's $\sin (x)$ ! That's right, this sine wave is really well approximated by the blue $u (x)$ curve.

Compare that with the 5-th degree Taylor polynomial over the same interval

desmos-graph(2).png

Look at how much better this approximation is! That's because taylor polynomials really are only good near $| x | < 2$ . Other than that, they start to really suck.