Page 444 - Jolliffe I. Principal Component Analysis

A.1. Numerical Calculation of Principal Components
Finding PCs reduces to finding the eigenvalues and eigenvectors of a
positive-semidefinite matrix. We now look briefly at some of the possible
algorithms that can be used to solve such an eigenproblem. 409
The Power Method
A form of the power method was described by Hotelling (1933) in his
original paper on PCA, and an accelerated version of the technique was
presented in Hotelling (1936). In its simplest form, the power method is a
technique for finding the largest eigenvalue and the corresponding eigen-
vector of a (p × p) matrix T. The idea is to choose an initial p-element
vector u 0 , and then form the sequence
u 1 = Tu 0
2
u 2 = Tu 1 = T u 0
. .
. .
. .
r
u r = Tu r−1 = T u 0
. .
. .
. .
lf α 1 , α 2 ,..., α p are the eigenvectors of T, then they form a basis for
p-dimensional space, and we can write, for arbitrary u 0 ,

p

u 0 = κ k α k
k=1
for some set of constants κ 1 ,κ 2 ,...,κ p . Then
p p

u 1 = Tu 0 = κ k Tα k = κ k λ k α k ,
k=1 k=1
where λ 1 ,λ 2 ,...,λ p are the eigenvalues of T. Continuing, we get for r =
2, 3,...
p

u r = r
κ k λ α k
k
k=1
and

r r
κ 2 λ 2 κ p λ p
= α 1 + α 2 + ··· + α p .
u r
(κ 1 λ ) κ 1 λ 1 κ 1 λ 1
r
1
Assuming that the first eigenvalue of T is distinct from the remaining
eigenvalues, so that λ 1 >λ 2 ≥ ··· ≥ λ p , it follows that a suitably nor-
malized version of u r → α 1 as r →∞. It also follows that the ratios of
corresponding elements of u r and u r−1 → λ 1 as r →∞.
The power method thus gives a simple algorithm for finding the first
(largest) eigenvalue of a covariance or correlation matrix and its corre-
sponding eigenvector, from which the first PC and its variance can be