Page 500 - Elementary_Linear_Algebra_with_Applications_Anton__9_edition

It follows from the Best Approximation Theorem (6.4.1) that the closest vector in W to b is the orthogonal projection of b on

W. Thus, for a vector x to be a least squares solution of , this vector must satisfy

(2)

One could attempt to find least squares solutions of by first calculating the vector and then solving 2;

however, there is a better approach. It follows from the Projection Theorem (6.3.4) and Formula 5 of Section 6.3 that

is orthogonal to W. But W is the column space of A, so it follows from Theorem 6.2.6 that lies in the nullspace of .

Therefore, a least squares solution of must satisfy

or, equivalently,

(3)

This is called the normal system associated with , and the individual equations are called the normal equations

associated with . Thus the problem of finding a least squares solution of has been reduced to the problem of

finding an exact solution of the associated normal system.

Note the following observations about the normal system:

The normal system involves n equations in n unknowns (verify).

The normal system is consistent, since it is satisfied by a least squares solution of .

The normal system may have infinitely many solutions, in which case all of its solutions are least squares solutions of
.

From these observations and Formula 2, we have the following theorem.
THEOREM 6.4.2

For any linear system , the associated normal system

is consistent, and all solutions of the normal system are least squares solutions of .
Moreover, if W is the column space of A, and x is any least squares solution of , then the
orthogonal projection of b on W is

Uniqueness of Least Squares Solutions

Before we examine some numerical examples, we shall establish conditions under which a linear system is guaranteed to
have a unique least squares solution. We shall need the following theorem.