Curvy models: logistic regression

The problem is that the large variability in age, in both the malignant and benign groups, obscures the difference in age (if any) between them. However, if you group the age data: 40-49, 50-59, etc., and then calculate the proportion of women with a malignant diagnosis (i.e. with Y = 1) in each group, this will reduce the variability, but preserve the underlying relationship between the two variables. The results of doing this are shown in Table 18.1.

Notice that I’ve labelled the first column as the ‘Proportion with Y = 1, or the Probability that Y = 1, written as P(Y = 1)’. Here’s why. In linear regression, you will recall that the dependent variable is the mean of Y for a given X. But what about a binary dependent variable? Can we find something analogous to the mean? As it happens, the mean of a set of binary, zero or one, values is the same as the proportion of ones,1 so an appropriate equivalent version of the binary dependent variable would seem to be ‘Proportion of (Y = 1)s’.

But proportions can be interpreted as probabilities (see Chapter 8). So the dependent variable becomes the ‘Probability that Y = 1’, or P(Y = 1), for a given value of X. For example the probability of a malignant diagnosis (Y = 1) for all of those women aged 40, which we can write as, P(Y = 1) given X = 40.

You can see in Table 18.1, the proportion with malignant breast lumps (the probability that Y = 1) increases with age, but does it increase linearly? A scatterplot of the proportion with malignant lumps, Y = 1, against group age midpoints is shown in Figure 18.2, which does suggest some sort of relationship between the two variables. But it’s definitely not linear, so a linear regression model won’t work. In fact, the curve has more of an elongated S shape, so what we need is a mathematical equation that will give such an S-shaped curve.

There are several possibilities, but the logistic model is the model of choice. Not only because it produces an S-shaped curve, which we want, but, critically, it has a meaningful clinical interpretation. Moreover, the value of P(Y = 1) is restricted by the maths of the logistic model to lie between zero and one, which is what we want, since it’s a probability.

The logistic regression model

The simple2 population logistic regression equation is:

1For example, the mean of the five values: 0, 1, 1, 0, 0 is 2/5 = 0.4, which is the same as the proportion of 1s, i.e. 2 in 5 or 0.4.

2‘Simple’ because there is only one independent variable – so far.

Age_midpt

Figure 18.2 Scatterplot of the proportion of women with a malignant diagnosis (Y = 1) against midpoints of age group

which we estimate with the sample logistic regression equation:

by determining the values of the estimators b0 and b1. We’ll come back to this problem in a moment. Note that e is the exponential operator, equal to 2.7183, and has nothing to do with the residual term in linear regression. As you can see, the logistic regression model is mathematically a bit more complicated than the linear regression model.

The outcome variable, P(Y = 1), is the probability that Y = 1 (the lump is malignant), for some given value of the independent variable X. There is no restriction on the type of independent variable, which can be nominal, ordinal or metric.

As an example, let’s return to our breast cancer study (Figure 1.6). Our outcome variable is diagnosis, where Y = 1 (malignant) or Y = 0 (benign). We’ll start with one independent variable – ever used an oral contraceptive pill (OCP), Yes = 1, or No = 0. We are going to treat OCP use as a possible risk factor for receiving a malignant diagnosis. This gives us the sample regression model:

So all we’ve got to do to determine the probability that a woman picked at random from the sample will get a malignant diagnosis (Y = 1), with or without OCP use, is to calculate

the values of b0 and b1 somehow, and then put them in the logistic regression equation, with OCP = 0, or OCP = 1.

Estimating the parameter values

Whereas the linear regression models use the method of ordinary least squares to estimate the regression parameters β0 and β1, logistic regression models use what is called maximum likelihood estimation. Essentially this means choosing the population which is most likely to have generated the sample results observed. Figure 18.3 and Figure 18.4, respectively, show the output from SPSS’s and Minitab’s logistic regression program for the above OCP model.

SPSS’s and Minitab’s logistic regression program both give b0 = −0.2877 and b1 = −0.9507. If we substitute these values into the logistic regression model of Equation (3), we get:3

if OCP = 0 (has never used OCP), P(Y = 1) = 0.4286

if OCP = 1 (has used OCP), then P(Y = 1) = 0.2247

So a woman who has never used an oral contraceptive pill has a probability of getting a malignant diagnosis nearly twice that of a woman who has used an oral contraceptive. Rather than being a risk factor for a malignant diagnosis, in this sample the use of oral contraceptives seems to confer some protection against a breast lump being malignant.

Figure 18.3 Abbreviated output from SPSS for a logistic regression with diagnosis as the dependent variable, and use of oral contraceptive pill (OCP) as the independent variable or risk factor

3You’ll first need to work out the values of (b0 + b1 × OCP), then (1 + b0 + b1 × OCP), then raise e to each of these powers. Then divide the former by the latter.

Figure 18.6 The output from SPSS for the regression of diagnosis on body mass index (some columns are missing)

The multiple logistic regression model

In my explanation of the odds ratio above I used a simple logistic regression model, i.e. one with a single independent variable (OCP), because this offers the simplest treatment. However, the result we got, that the odds ratio is equal to eb1 , applies to each coefficient if there is more than one independent variable, i.e. eb2 , eb3 , etc. The usual situation is to have a risk factor variable plus a number of confounder variables (the usual suspects – age, sex, etc.). Suppose, for example, that you decided to include age and body mass index (BMI) along with OCP as independent variables. Equation (1) would then become:

P(Y = 1) = (eβ0+β1 ×OCP+β2 ×age+β3 ×BMI)/(1 + eβ0+β1 ×OCP+β2 ×age+β3 ×BMI)

P(Y = 1) is still of course the probability that the woman will receive a malignant diagnosis, Y = 1. The odds ratio for age is eb2; the odds ratio for BMI is eb3. Moreover, as with linear regression, each of these odds ratios is adjusted for any possible interaction between the independent variables.

As an example, output from Minitab for the above multiple regression model of diagnosis against use of oral contraceptives (OCP), age and body mass index (BMI), is shown in Figure 18.7.

Exercise 18.4 Comment on what is revealed in the output in Figure 18.7 about the relationship between diagnosis and the three independent variables shown.

Building the model

The strategy for model building in the logistic regression model is similar to that for linear regression:

Make a list of candidate independent variables.

For any nominal or ordinal variables in the list construct a contingency table and perform a chi-squared test.5 Make a note of the p-value.

5Provided the number of categories isn’t too big for the size of your sample – you don’t want any empty cells or low expected values (see Chapter 14)

Figure 18.7 Minitab output for the model diagnosis against OCP, age and BMI

For any metric variables, perform either a two-sample t test, or a univariate logistic regression; note the p-value in either case.

Pick out all those variables in the list whose p-value is 0.25 or less. Select the variable with the smallest p-value (if there is more than one with the smallest p-value pick one arbitrarily) to be your first independent variable. This is your starting model.

Finally, add variables to your model one at a time, each time examining the p-values for statistical significance. If a variable, when added to the model, is not statistically significant, drop it, unless there are noticeable changes in coefficient values, which is indicative of confounding.

Goodness-of-fit

In the linear regression model you used R2 to measure goodness-of-fit. In the logistic regression model measuring goodness-of-fit is much more complicated, and can involve graphical as well as numeric measures. Two numeric measures that can be used are the deviance coefficient and the Hosmer-Lemeshow statistic. Very briefly, both of these have a chi-squared distribution, and we can use the resulting p-value to reject, or not, the null hypothesis that the model provides a good fit. The graphical methods are quite complex and you should consult more specialist

sources for further information on this and other aspects of this complex procedure. Hosmer and Lemeshow (1989) is an excellent source.

Exercise 18.5 Use the Hosmer-Lemeshow goodness-of-fit statistic in the output of Figure 18.7 to comment on the goodness-of-fit of the model shown.

Linear and logistic regression modelling are two methods from a more general class of methods known collectively as multivariable statistics. Multivariate statistics on the other hand, is a set of procedures applicable where there is more than one dependent variable, and includes methods such as principal components analysis, multidimensional scaling, cluster and discriminant analysis, and more. Of these, principal components analysis appears most often in the clinical literature, but even so is not very common. Unfortunately, there is no space to discuss any of these methods.

IX

Two More Chapters