1manly_b_f_j_statistics_for_environmental_science_and_managem

116 Statistics for Environmental Science and Management, Second Edition

(1984). In this case, let wi be the weight for the ith study, and define the test statistic

If the null hypothesis is true for all studies, then this will follow a standard normal distribution. If it is significantly low in comparison with the standard normal distribution, then this is evidence that the null hypothesis is not always true.

Meta-analysis, as generally understood, involves more than just combining the p-values from several sets of data. In fact, the usual approach is to take a series of studies, and for each one to calculate an estimated effect size, which is often just the mean difference between the treated and control groups in units of the estimated standard deviation of individual observations. Questions of interest are:

•How large is the effect overall?

•Is it generally a positive or a negative effect, and is it usually different from zero?

•Are the effect sizes similar for all studies?

•If there is variation between studies, can this be related to the different types of studies involved?

There is much literature on this type of meta-analysis. Comprehensive sources for more information are the books by Hedges and Olkin (1985) and Sutton et al. (2000), while the introductions to the topic provided by Gurevitch and Hedges (1993, 1999) will be useful for beginners in this area. Also, a special feature in the journal Ecology gives a review of applications of metaanalysis in this area (Osenberg et al. 1999).

Example 4.4:  The Exxon Valdez Oil Spill and Intertidal Sites

Table 4.3 summarizes the results from part of the Oil Spill Trustees’ Coastal Habitat Injury Assessment study of the shoreline impact of the Exxon Valdez oil spill that was discussed in Example 1.1. The source of these results is the paper by McDonald et al. (1995), who also provide the example calculations considered here. They come from comparing the biomass of barnacles (the family Balanomorpha) at some paired intertidal sites, of which one was oiled and the other not. The pairing was designed to ensure that the sites were similar in all respects except the oiling.

One way of looking at these results is to regard each pair of sites as a quasi-experiment on the effects of oiling. Each experiment can then be summarized by its p-value, which gives the probability of observing a positive difference between the biomass for the control and oiled sites as large as that observed. This is the p-value for a one-sided test because the assumption is that, if anything, oiling will reduce the biomass. The p-values are 0.374, 0.015, 0.167, 0.052, and 0.148. They are all fairly small,

This analysis is still not without problems. In particular, the z values are only a very indirect measure of the difference between oiled and unoiled sites, and all that can be said if there is an effect of oiling is that these z values will tend to be negative. An analysis based on a more meaningful measure of the effect of oiling would therefore have been preferable.

4.11  Bayesian Inference

So far, the methods discussed in this book have all been based on a traditional view of statistics, with tests of significance and confidence intervals being the main tools for inference, justified either by the study design (for design-based inference) or an assumed model (with model-based inference). There is, however, another fundamentally different approach to inference that is being used increasingly in recent times because certain computational difficulties that used to occur have now been overcome.

This alternative approach is called Bayesian inference because it is based on a standard result in probability theory called Bayes’s theorem. To see what this theorem says, consider the situation where it is possible to state that a certain parameter θ must take one, and only one, of a set of n specific values denoted by θ1, θ2, …, θn, and where—before any data are collected—it is known that the prior probability of θi (i.e., the probability of this being the

correct value for θ) is P(θi), with P(θ1) + P(θ2) + … + P(θn) = 1. Some data that are related to θ are then collected. Under these conditions, Bayes’s theorem

states that the posterior probability that θi (i.e., the probability that this is the correct value of θ, given the evidence in the data) is

where the summation is for k = 1 to n, and P(data|θi) is the probability of observing the data if θ = θi.

This result offers a way to calculate the probability that a particular value is the correct one for θon the basis of the data, which is the best that can be hoped for in terms of inferences about θ. The stumbling block is that, to do this, it is necessary to know the prior probabilities before the data are collected.

There are two approaches used to determine prior probabilities when, as is usually the case, these are not really known. The first approach uses the investigator’s subjective probabilities, based on general knowledge about the situation. The obvious disadvantage of this is that another investigator will likely not have the same subjective probabilities, so that the conclusions from the data will depend to some extent at least on who does the analysis. It is also very important that the prior probabilities not be determined after the data have been examined, because equation (4.8) does not apply if the prior probabilities depend on the data. Thus inferences based on equation (4.8)

120 Statistics for Environmental Science and Management, Second Edition

with the prior probabilities depending partly on the data are not Bayesian inferences. In fact, they are not justified at all.

The second approach is based on choosing prior probabilities that are uninformative, so that they do not have much effect on the posterior probabilities. For example, the n possible values of θ can all be given the prior probability 1/n. One argument for this approach is that it expresses initial ignorance about the parameter in a reasonable way and that, provided that there is enough data, the posterior probabilities do not depend very much on whatever is assumed for the prior probabilities.

Equation (4.8) generalizes in a straightforward way to situations where there are several or many parameters of interest, and where the prior distributions for these parameters are discrete or continuous. For many purposes, all that needs to be known is that

P(parameters|data) P(data|parameters)P(parameters)

i.e., the probability of a set of parameter values given the data is proportional to the probability of the data given the parameter values, multiplied by the prior probability of the set of parameter values. This result can be used to generate posterior probability distributions using possibly very complicated models when the calculations are done using a special technique called Markov-chain Monte Carlo.

There is one particular aspect of Bayesian inference that should be appreciated. It is very much model-based in the sense discussed in Section 4.4. This means that it is desirable, with any serious study, that the conclusions from an analysis should be quite robust to both the assumptions made about prior distributions and the assumptions made about the other components in the model. Unfortunately, these types of assessments are often either not done or are not done very thoroughly.

This brief introduction to Bayesian inference has been included here because it seems likely that environmental scientists will be seeing more of these methods in the future as a means of drawing conclusions from data. More information about them with the emphasis on Markov-chain Monte Carlo methods is provided in the books by Manly (2007) and Gilks et al. (1996). For more on Bayesian data analysis in general, see the book by Gelman et al. (2003).

4.12  Chapter Summary

•In drawing conclusions from data, it is important to distinguish between observational and experimental studies. In general, observational studies are more likely to be affected by uncontrolled factors, leading to incorrect conclusions.

122 Statistics for Environmental Science and Management, Second Edition

can be made in this respect: (a) inferences can be extended outside the population actually sampled, and (b) observations that are correlated because they are close in space or time are analyzed without taking error (a) into account.

•When several related hypothesis tests are carried out at the same time and all the null hypotheses are true, the probability of at least one significant result increases with the number of tests. This is a well-known problem that has led to the development of a range of procedures to take into account the multiple testing, including using a Bonferroni adjustment for the significance level used with individual tests and Holm’s stepwise method for adjusting these levels.

•Meta-analysis is concerned with the problem of combining the results from a number of different studies on the same subject. This can be done by combining the p-values obtained from different studies using Fisher’s method or the Stouffer method. The Stouffer method also has a variation called the Liptak-Stouffer method, which allows the results of different studies to receive different weights. Alternatively, rather than using p-values, an effect size is estimated for each study, and these effect sizes are examined in terms of the overall effect, the extent to which the effect varies from study to study, and whether the variation between studies can be explained by differences between the types of studies used.

•Bayesian inference is different from conventional statistical inference, and it is becoming more widely used as the cost of computing power decreases. With this approach, a prior distribution assumed for a parameter of interest is changed, using Bayes’s theorem, into a posterior distribution, given the information from some new data. Modern computing methods, particularly Markov-chain Monte Carlo, make the Bayesian approach much easier to use than was the case in the past. However, it is cautioned that Bayesian inference is very much model-based, with all the potential problems that this implies.

Exercises

Exercise 4.1

Suppose that a pill is marketed with the claim that if one pill a day is taken, then it will lead to a significant weight loss for male and female adults. Describe how this claim might be tested using the experimental designs described in Section 4.3 (pre-experimental designs, a quasiexperimental design, and two proper experimental designs). Explain the potential problems with the pre-experimental designs and the quasiexperimental design, and how these can be overcome with the proper designs.