Page 49 - Jolliffe I. Principal Component Analysis

2. Properties of Population Principal Components
18
2.2 Geometric Properties of Population Principal
Components
It was noted above that Property A5 can be interpreted geometrically, as
well as algebraically, and the discussion following Property A4 shows that
A4, too, has a geometric interpretation. We now look at two further, purely
geometric, properties.
Property G1. Consider the family of p-dimensional ellipsoids
x Σ −1 x =const. (2.2.1)

The PCs define the principal axes of these ellipsoids.

Proof. The PCs are defined by the transformation (2.1.1) z = A x,and
since A is orthogonal, the inverse transformation is x = Az. Substituting
into (2.2.1) gives
(Az) Σ −1 (Az) = const = z A Σ −1 Az.

It is well known that the eigenvectors of Σ −1 are the same as those of Σ,
and that the eigenvalues of Σ −1 are the reciprocals of those of Σ, assuming
that they are all strictly positive. It therefore follows, from a corresponding
result to (2.1.3), that AΣ −1 A = Λ −1 and hence

z Λ −1 z =const.
This last equation can be rewritten

p 2
z
= const (2.2.2)
k
k=1 λ k
and (2.2.2) is the equation for an ellipsoid referred to its principal axes.
Equation (2.2.2) also implies that the half-lengths of the principal axes are
1/2 1/2 1/2
proportional to λ 1 , λ 2 ,...,λ p .
This result is statistically important if the random vector x has a mul-
tivariate normal distribution. In this case, the ellipsoids given by (2.2.1)
define contours of constant probability for the distribution of x. The first
(largest) principal axis of such ellipsoids will then define the direction in
which statistical variation is greatest, which is another way of expressing
the algebraic definition of the first PC given in Section 1.1. The direction
of the first PC, defining the first principal axis of constant probability el-
lipsoids, is illustrated in Figures 2.1 and 2.2 in Section 2.3. The second
principal axis maximizes statistical variation, subject to being orthogonal
to the first, and so on, again corresponding to the algebraic definition. This
interpretation of PCs, as defining the principal axes of ellipsoids of constant
density, was mentioned by Hotelling (1933) in his original paper.
It would appear that this particular geometric property is only of direct
statistical relevance if the distribution of x is multivariate normal, whereas