Handbook_of_statistical_analysis_using_SAS

Display 4.2

particularly strong, apart perhaps from Ex0 and Ex1. The scatterplot matrix also clearly highlights the very strong relationship between these two variables. Highly correlated explanatory variables, multicollinearity, can cause several problems when applying the multiple regression model, including:

1.It severely limits the size of the multiple correlation coefficient R because the explanatory variables are primarily attempting to explain much of the same variability in the response variable (see Dizney and Gromen [1967] for an example).

2.It makes determining the importance of a given explanatory variable (see later) difficult because the effects of explanatory variables are confounded due to their intercorrelations.

3.It increases the variances of the regression coefficients, making use of the predicted model for prediction less stable. The parameter estimates become unreliable.

Spotting multicollinearity amongst a set of explanatory variables might not be easy. The obvious course of action is to simply examine the correlations between these variables, but whilst this is often helpful, it is

©2002 CRC Press LLC

by no means foolproof — more subtle forms of multicollinearity may be missed. An alternative and generally far more useful approach is to examine what are known as the variance inflation factors of the explanatory variables. The variance inflation factor VIFj for the jth variable is given by

where Rj2 is the square of the multiple correlation coefficient from the regression of the jth explanatory variable on the remaining explanatory variables. The variance inflation factor of an explanatory variable indicates the strength of the linear relationship between the variable and the remaining explanatory variables. A rough rule of thumb is that variance inflation factors greater than 10 give some cause for concern.

How can multicollinearity be combatted? One way is to combine in some way explanatory variables that are highly correlated. An alternative is simply to select one of the set of correlated variables. Two more complex possibilities are regression on principal components and ridge regression, both of which are described in Chatterjee and Price (1991).

The analysis of the crime rate data begins by looking at the variance inflation factors of the 13 explanatory variables, obtained using the following SAS instructions:

proc reg data=uscrime; model R= Age--X / vif;

run;

The vif option in the model statement requests that variance inflation factors be included in the output shown in Display 4.3.

The REG Procedure

Model: MODEL1

Dependent Variable: R

Analysis of Variance

©2002 CRC Press LLC

Display 4.3

Concentrating for now on the variance inflation factors in Display 4.3, we see that those for Ex0 and Ex1 are well above the value 10. As a consequence, we simply drop variable Ex0 from consideration and now regress crime rate on the remaining 12 explanatory variables using the following:

proc reg data=uscrime;

model R= Age--Ed Ex1--X / vif; run;

The output is shown in Display 4.4. The square of the multiple correlation coefficient is 0.75, indicating that the 12 explanatory variables account for 75% of the variability in the crime rates of the 47 states. The variance inflation factors are now all less than 10.

©2002 CRC Press LLC

Display 4.4

The adjusted R2 statistic given in Display 4.4 is the square of the multiple correlation coefficient adjusted for the number of parameters in the model. The statistic is calculated as:

©2002 CRC Press LLC

where n is the number of observations used in fitting the model and i is an indicator variable that is 1 if the model includes an intercept and 0 otherwise.

The main features of interest in Display 4.4 are the analysis of variance table and the parameter estimates. In the former, the F-test is for the hypothesis that all the regression coefficients in the regression equation are zero. Here, the evidence against this hypothesis is very strong (the relevant P-value is 0.0001). In general, however, this overall test is of little real interest because it is most unlikely in general that none of the explanatory variables will be related to the response. The more relevant question is whether a subset of the regression coefficients is zero, implying that not all the explanatory variables are informative in determining the response. It might be thought that the nonessential variables can be identified by simply examining the estimated regression coefficients and their standard errors as given in Display 4.4, with those regression coefficients significantly different from zero identifying the explanatory variables needed in the derived regression equation, and those not different from zero corresponding to variables that can be omitted. Unfortunately, this very straightforward approach is not in general suitable, simply because the explanatory variables are correlated in most cases. Consequently, removing a particular explanatory variable from the regression will alter the estimated regression coefficients (and their standard errors) of the remaining variables. The parameter estimates and their standard errors are conditional on the other variables in the model. A more involved procedure is thus necessary for identifying subsets of the explanatory variables most associated with crime rate. A number of methods are available, including:

Forward selection. This method starts with a model containing none of the explanatory variables and then considers variables one by one for inclusion. At each step, the variable added is one that results in the biggest increase in the regression sum of squares. An F-type statistic is used to judge when further additions would not represent a significant improvement in the model.

Backward elimination. This method starts with a model containing all the explanatory variables and eliminates variables one by one, at each stage choosing the variable for exclusion as the one leading to the smallest decrease in the regression sum of squares. Once again, an F-type statistic is used to judge when further exclusions would represent a significant deterioration in the model.

©2002 CRC Press LLC

Stepwise regression. This method is, essentially, a combination of forward selection and backward elimination. Starting with no variables in the model, variables are added as with the forward selection method. Here, however, with each addition of a variable, a backward elimination process is considered to assess whether variables entered earlier might now be removed because they no longer contribute significantly to the model.

In the best of all possible worlds, the final model selected by each of these procedures would be the same. This is often the case, but it is in no way guaranteed. It should also be stressed that none of the automatic procedures for selecting subsets of variables are foolproof. They must be used with care, and warnings such as the following given in Agresti (1996) must be noted:

Computerized variable selection procedures should be used with caution. When one considers a large number of terms for potential inclusion in a model, one or two of them that are not really important may look impressive simply due to chance. For instance, when all the true effects are weak, the largest sample effect may substantially overestimate its true effect. In addition it often makes sense to include certain variables of special interest in a model and report their estimated effects even if they are not statistically significant at some level.

In addition, the comments given in McKay and Campbell (1982a;b) concerning the validity of the F-tests used to judge whether variables should be included or eliminated, should be considered.

Here, we apply a stepwise procedure using the following SAS code:

proc reg data=uscrime;

model R= Age--Ed Ex1--X / selection=stepwise sle=.05 sls=.05;

plot student.*(ex1 x ed age u2); plot student.*predicted. cookd.*obs.; plot npp.*residual.;

run;

The proc, model, and run statements specify the regression analysis and produce the output shown in Display 4.5. The significance levels required for variables to enter and stay in the regression are specified with the sle and sls options, respectively. The default for both is P = 0.15. (The plot statements in this code are explained later.)

©2002 CRC Press LLC

Display 4.5 shows the variables entered at each stage in the variable selection procedure. At step one, variable Ex1 is entered. This variable is the best single predictor of the crime rate. The square of the multiple correlation coefficient is observed to be 0.4445. The variable Ex1 explains 44% of the variation in crime rates. The analysis of variance table shows both the regression and residual or error sums of squares. The F-statistics is highly significant, confirming the strong relationship between crime rate and Ex1. The estimated regression coefficient is 0.92220, with a standard error of 0.15368. This implies that a unit increase in Ex1 is associated with an estimated increase in crime rate of 0.92. This appears strange but perhaps police expenditures increase as the crime rate increases.

At step two, variable X is entered. The R-square value increases to 0.5550. The estimated regression coefficient of X is 0.42312, with a standard error of 0.12803. In the context of regression, the type II sums of squares and F-tests based on them are equivalent to type III sums of squares described in Chapter 6.

In this application of the stepwise option, the default significance levels for the F-tests used to judge entry of a variable into an existing model and to judge removal of a variable from a model are each set to 0.05. With these values, the stepwise procedure eventually identifies a subset of five explanatory variables as being important in the prediction of the crime rate. The final results are summarised at the end of Display 4.5. The selected five variables account for just over 70% of the variation in crime rates compared to the 75% found when using 12 explanatory variables in the previous analysis. (Notice that in this example, the stepwise procedure gives the same results as would have arisen from using forward selection with the same entry criterion value of 0.05 because none of the variables entered in the “forward” phase are ever removed.)

The statistic Cp was suggested by Mallows (1973) as a possible alternative criterion useful for selecting informative subsets of variables. It is defined as:

where s2 is the mean square error from the regression, including all the explanatory variables available; and SSEp is the error sum of squares for a model that includes just a subset of the explanatory variable. If Cp is plotted against p, Mallows recommends accepting the model where Cp first approaches p (see Exercise 4.2).

(The Bounds on condition number given in Display 4.5 are fully explained in Berk [1977]. Briefly, the condition number is the ratio of the largest and smallest eigenvalues of a matrix and is used as a measure of the numerical stability of the matrix. Very large values are indicative of possible numerical problems.)

©2002 CRC Press LLC

The REG Procedure

Model: MODEL1

Dependent Variable: R

Stepwise Selection: Step 1

Variable Ex1 Entered: R-Square = 0.4445 and C(p) = 33.4977

Analysis of Variance

Bounds on condition number: 1, 1

----------------------------------------------------------------------------------------------

Stepwise Selection: Step 2

Variable X Entered: R-Square = 0.5550 and C(p) = 20.2841

Analysis of Variance

Bounds on condition number: 1.7244, 6.8978

----------------------------------------------------------------------------------------------

©2002 CRC Press LLC

Stepwise Selection: Step 3

Variable Ed Entered: R-Square = 0.6378 and C(p) = 10.8787

The REG Procedure

Model: MODEL1

Dependent Variable: R

Stepwise Selection: Step 3

Bounds on condition number: 3.1634, 21.996

----------------------------------------------------------------------------------------------

Stepwise Selection: Step 4

Variable Age Entered: R-Square = 0.6703 and C(p) = 8.4001

Analysis of Variance

©2002 CRC Press LLC

Bounds on condition number: 3.5278, 38.058

----------------------------------------------------------------------------------------------

Stepwise Selection: Step 5

Variable U2 Entered: R-Square = 0.7049 and C(p) = 5.6452

Analysis of Variance

The REG Procedure

Model: MODEL1

Dependent Variable: R

Stepwise Selection: Step 5

Bounds on condition number: 3.5289, 57.928

----------------------------------------------------------------------------------------------

©2002 CRC Press LLC