Lecture 20 - Principal Component Analysis (Application)

The goal for this application is:

Given a "centered" data ${\vec{x}}_{1}, . . ., {\vec{x}}_{n} \in R^{d}$
A given $k$ where $1 \leq k \leq d$

Find the $k$ -dimensional subspace that minimizes the average square distance between the points ${\vec{x}}_{i}$ and the subspace

In reference to the picture above, we want to equivalently maximize the average squared distance of the projection.

centered data

Data is centered if $\sum_{i = 1}^{n} {\vec{x}}_{i} = \vec{0}$

If the data is not centered, you can just subtract the average from each ${\vec{x}}_{i}$ and then get a centered data. Specifically:

\overset{―}{x} = \frac{1}{n} \sum_{i = 1}^{n} {\vec{x}}_{i}

Then let ${\vec{y}}_{i} = {\vec{x}}_{i} - \overset{―}{x}$ . This new data ${\vec{y}}_{i}$ is now centered.

How do we Find the Subspace?

We really want to find some orthonormal basis for the subspace $U$ . Namely a $v_{1}, . . ., v_{k}$ such that it maximizes the projections from $U$ onto $R^{d}$ , or:

\frac{1}{n} \sum_{i = 1}^{n} \sum_{j = 1}^{k} {⟨ {\vec{x}}_{i}, {\vec{v}}_{j} ⟩}^{2}

is the value we want to maximize.

Define the matrix $X$ such that the rows of the matrix are the data:

X = [\begin{matrix} \dots & {\vec{x}}_{1} & \dots \\ \dots & {\vec{x}}_{2} & \dots \\ \dots & {\vec{x}}_{3} & \dots \\ ⋮ & ⋮ & ⋮ \end{matrix}]

Given any unit vector $\vec{u}$ then:

X_{\vec{u}} = [\begin{matrix} ⟨ {\vec{x}}_{1}, \vec{u} ⟩ \\ ⟨ {\vec{x}}_{2}, \vec{u} ⟩ \\ ⋮ \\ ⟨ {\vec{x}}_{n}, \vec{u} ⟩ \end{matrix}] = ‖ X_{\vec{u}} ‖^{2}

We want to maximize $‖ X_{\vec{u}} ‖^{2}$ . Notice:

‖ X_{\vec{u}} ‖^{2} = ⟨ X u, X u ⟩ = (X u)^{T} X u = u^{T} \underset{Symmetric, Set to A}{\underset{⏟}{X^{T} X}} u

Thus we want to diagonalize $A$ , $A = Q D Q^{T}$ :

We can use SVD to determine that $X = U S V^{T}$ :

A = X^{T} X = (U S V^{T})^{T} U S V^{T} = (V S^{T} U^{T}) (U S V^{T}) = V S^{T} S V^{T}

Recall that $S^{T} S$ was a diagonal matrix, so $S$ contains the singular values for $A$ , with right singular vectors from $V^{T}$ . So the right singular vectors are eigenvalues of $X^{T} X$ .

Recall that $X = U S V^{T}$ . So then since $V$ is a unitary matrix:

\begin{aligned} X V & = U S \\ [\begin{array}{c} ⟨ x_{1}, v_{1} ⟩ & ⟨ x_{1}, v_{2} ⟩ & \dots & ⟨ x_{1}, v_{d} ⟩ \\ ⋮ & ⋮ & ⋱ & ⋮ \\ ⟨ x_{n}, v_{1} ⟩ & ⟨ x_{n}, v_{2} ⟩ & \dots & ⟨ x_{n}, v_{d} ⟩ \end{array}] & = \end{aligned}

What we can do is take any row and chop it off before the $d$ -th entry to obtain the projection of one $x_{i}$ onto the subspace we found:

{\vec{x}}_{i} = \sum_{j = 1}^{k} ⟨ x_{i}, v_{j} ⟩