Econometrics2011

4.4Illustration

We illustrate the least-squares estimator in practice with the data set used to generate the estimates from Chapter 3. This is the March 2009 Current Population Survey, which has extensive information on the U.S. population. This data set is described in more detail in Section ? For this illustration, we use the sub-sample of non-white married non-military female wages earners with 12 years potential work experience. This sub-sample has 61 observations. Let yi be log wages and xi be an intercept and years of education. Then

We often write the estimated equation using the format

An interpretation of the estimated equation is that each year of education is associated with an 16% increase in mean wages.

Equation (4.9) is called a bivariate regression as there are only two variables. A multivariate regression has two or more regressors, and allows a more detailed investigation. Let’s redo the example, but now including all levels of experience. This expanded sample includes 2454 observations. Including as regressors years of experience and its square (experience2=100) (we divide by 100 to simplify reporting), we obtain the estimates

These estimates suggest a 12% increase in mean wages per year of education, holding experience constant.

4.5Least Squares Residuals

We make a distinction between the error ei and the residual e^i: The error ei is unobservable while the residual e^i is a by-product of estimation. These two variables are frequently mislabeled, which can cause confusion.

Sample sums can also be written in matrix notation. For example

Xn

xix0i = X0X

i=1

Xn

xiyi = X0y:

i=1

Using matrix notation we have simple expressions for most estimators. This is particularly convenient for computer programming, as most languages allow matrix notation and manipulation.

Important Matrix Expressions

Early Use of Matrices

The earliest known treatment of the use of matrix methods to solve simultaneous systems is found in Chapter 8 of the Chinese text The Nine Chapters on the Mathematical Art, written by several generations of scholars from the 10th to 2nd century BCE.

4.7Projection Matrix

This is a property of a projection matrix. More generally, for any matrix Z which can be written as Z = X for some matrix (we say that Z lies in the range space of X); then

P Z = P X = X X0X 1 X0X = X = Z:

As an important example, if we partition the matrix X into two matrices X1 and X2 so that

X = [X1 X2] ;

then P X1 = X1.

The matrix P is symmetric and idempotent1. To see that it is symmetric,

P 0 = X X0X 1 X0 0

=X0 0 X0X 1 0 (X)0

=X X0X 0 1 X0

=X (X)0 X0 0 1 X0

=P :

To establish that it is idempotent, the fact that P X = X implies that

P P = P X X0X 1 X0

=X X0X 1 X0

=P :

The matrix P has the property that it creates the …tted values in a least-squares regression:

P y = X X0X 1 X0y = X = y^:

tr P = tr X X0X 1 X0

=tr X0X 1 X0X

=tr (Ik)

=k:

1 A matrix P is symmetric if P 0 = P : A matrix P is idempotent if P P = P : See Appendix A.8.

4.9 Regression Components

The OLS estimator of = ( 01; 02)0 is obtained by regression of y on X = [X1 X2] and can be written as

Now

4.10Residual Regression

As …rst recognized by Ragnar Frisch, expressions (4.29) and (4.30) can be used to show that the least-squares estimators 1 and 2 can be found by a two-step regression procedure.

and

e~1 = M1y:

b

Thus the coe¢ cient estimate 2 is algebraically equal to the least-squares regression of e~1 on

f

X2: Notice that these two are y and X2, respectively, premultiplied by M1. But we know that multiplication by M1 is equivalent to creating least-squares residuals. Therefore e~1 is simply the

f

least-squares residual from a regression of y on X1; and the columns of X2 are the least-squares residuals from the regressions of the columns of X2 on X1:

We have proven the following theorem.

4.11Prediction Errors

The least-squares residual e^i are not true prediction errors, as they are constructed based on the full sample including yi. A proper prediction for yi should be based on estimates constructed only using the other observations. We can do this by de…ning the leave-one-out OLS estimator of as that obtained from the sample of n 1 observations excluding the i’th observation:

A convenient feature of this expression is that it shows that computation of e~i is based on a simple linear operation, and does not really require n separate estimations.

One use of the prediction errors is to estimate the out-of-sample mean squared error

p

This is also known as the mean squared prediction error. Its square root ~ = ~2 is the prediction standard error.

Proof of Equation (4.32). The Sherman–Morrison formula (A.3) from Appendix A.5 states that for nonsingular A and vector b

A bb0 1 = A 1 + 1 b0A 1b 1 A 1bb0A 1:

This implies

X0X xix0i 1 = X0X 1 + (1 hii) 1 X0X 1 xix0i X0X 1