Page 142 - Jolliffe I. Principal Component Analysis

6

Choosing a Subset of Principal
Components or Variables

In this chapter two separate, but related, topics are considered, both of
which are concerned with choosing a subset of variables. In the first section,
the choice to be examined is how many PCs adequately account for the
total variation in x. The major objective in many applications of PCA is
to replace the p elements of x by a much smaller number m of PCs, which
nevertheless discard very little information. It is crucial to know how small
m can be taken without serious information loss. Various rules, many ad
hoc, have been proposed for determining a suitable value of m, and these
are discussed in Section 6.1. Examples of their use are given in Section 6.2.
Using m PCs instead of p variables considerably reduces the dimension-
ality of the problem when m p, but usually the values of all p variables
are still needed in order to calculate the PCs, as each PC is likely to be
a function of all p variables. It might be preferable if, instead of using m
PCs we could use m, or perhaps slightly more, of the original variables,
to account for most of the variation in x. The question arises of how to
compare the information contained in a subset of variables with that in
the full data set. Different answers to this question lead to different criteria
and different algorithms for choosing the subset. In Section 6.3 we concen-
trate on methods that either use PCA to choose the variables or aim to
reproduce the PCs in the full data set with a subset of variables, though
other variable selection techniques are also mentioned briefly. Section 6.4
gives two examples of the use of variable selection methods.
All of the variable selection methods described in the present chapter
are appropriate when the objective is to describe variation within x as
well as possible. Variable selection when x is a set of regressor variables