vstatmp_engl

138 6 Parameter Inference I

that compared to the direct probability where the parameter is known and the chances of an event are described, we are in the inverse position where we have observed the event and want to associate probabilities to the various causes that could have led to the observation.

As already stated above, the likelihood of a certain hypothesis is large if the observation is probable for this hypothesis. It measures how strongly a hypothesis is supported by the data. If an observation is very unlikely the validity of the hypothesis is doubtful – however this classification applies only when there is an alternative hypothesis with larger likelihood. Only relations between likelihoods make sense.

Usually experiments provide a sample of N independent observations xi which all follow independently the same p.d.f. f(x|θ) which depends on the unknown parameter

˜|

=f(x1,x2, . . . , xN θ)

YN

=f(xi|θ)

i=1

YN

=L(θ|xi) .

i=1

The same relation also holds for discrete variates:

When we have a sample of independent observations, it is convenient to consider the logarithm of the likelihood. It is called log-likelihood . It is equal to

for continuous variates. A corresponding relation holds for discrete variates.

Fig. 6.3. Likelihood of three observations and two hypotheses with di erent p.d.f.s.

Fig. 6.3 illustrates the notion of likelihood in a concrete case of two hypotheses. For two given hypotheses and a sample of three observations we present the values of the likelihood, i.e. the products of the three corresponding p.d.f. values. The broad p.d.f. in the right hand picture matches better. Its likelihood is about thirty times higher than that of the left hand hypothesis.

So far we have considered the likelihood of samples of i.i.d. variates. Also the case where two independent experiments A, B measure the same quantity x is of considerable interest. The combined likelihood L is just the product of the individual

likelihoods LA(θ|x1) = fA(x1|θ) and LB(θ|x2) = fB(x2|θ) as is obvious from the definition:

f(x1, x2|θ) = fA(x1|θ)fB(x2|θ) , L(θ) = f(x1, x2|θ) ,

hence

L = LALB ,

ln L = ln LA + ln LB .

We state: The likelihood of several independent observations or experiments is equal to the product of the individual likelihoods. Correspondingly, the log-likelihoods add up.

Y

L = Li ,

X

ln L = ln Li .

140 6 Parameter Inference I

6.4 The Likelihood Ratio

To discriminate between hypotheses, we use the likelihood ratio. According to a lemma of Neyman and Pearson there is no other more powerful quantity. This means that classifying according to the likelihood ratio, we can obtain the smallest number of false decisions (see Chap. 10). When we have to choose between more than two hypotheses, there are of course several independent ratios.

Example 73. Likelihood ratio of Poisson frequencies

We observe 5 decays and want to compute the relative probabilities for three hypotheses. Prediction H1 assumes a Poisson distribution with expectation value 2, H2 and H3 have expectation values 9 and 20, respectively. The likelihoods following from the Poisson distribution Pλ(k) are:

L1 = P2(5) ≈ 0.036 ,

L2 = P9(5) ≈ 0.061 ,

L3 = P20(5) ≈ 0.00005 .

We can form di erent likelihood ratios. If we are interested for example in hypothesis 2, then the quotient L2/(L1 + L2 + L3) ≈ 0.63 is relevant4. If we observe in a second measurement in the same time interval 8 decays, we obtain:

L1 = P2(5)P2(8) = P4(13) ≈ 6.4 · 10−3 ,

L2 = P9(5)P9(8) = P18(13) ≈ 5.1 · 10−2 ,

L3 = P20(5)P20(8) = P40(13) ≈ 6.1 · 10−7 .

The likelihood ratio L2/(L1 +L2 +L3) ≈ 0.89 (for H1 and H3 correspondingly 0.11 and 10−5) now is much more significant. The fact that all values Li are small is unimportant because one of the three hypotheses has to be valid.

We now apply the same procedure to hypotheses with probability densities.

Example 74. Likelihood ratio of normal distributions

We compare samples drawn from one out of two alternative normal distributions with di erent expectation values and variances (Fig. 6.4)

a) Initially the sample consists of a single observation at x = 0, for both cases one standard deviation o the mean values of the two distributions (Fig. 6.4a):

L1 = 2 e−1/2 = 2 .

L2 e−4/8

4We remember that the likelihood ratio is not the ratio of the probabilities. The latter depends on prior probabilities of the hypotheses.

Fig. 6.4. Likelihood ratio for two normal distributions. Top: 1 observation, bottom: 5 observations.

b) Now we place the observation at x = 2, the maximum of the second distribution (Fig. 6.4b):

L1 = 2 e−1/2 = 1.2 .

L2 e−0

c) We now consider five observations which have been taken from distribution f1 (Fig. 6.4c) and distribution f2, respectively (Fig. 6.4d). We obtain the likelihood ratios

It turns out that small distributions are easier to exclude than broad ones. On the other hand we get in case b) a preference for distribution 1 even though the observation is located right at the center of distribution 2.

Example 75. Likelihood ratio for two decay time distributions

A sample of N decay times ti has been recorded in the time interval tmin < t < tmax. The times are expected to follow either an exponential distribution f1(t) e−t/τ (hypothesis 1), or an uniform distribution f2(t) = const. (hypothesis 2). How likely are H1, H2? First we have to normalize the p.d.f.s:

The likelihoods are equal to the product of the p.d.f.s at the observations:

P

With t = ti/N the mean value of the times, we obtain the likelihood ratio

6.5 The Maximum Likelihood Method for Parameter Inference

In the previous examples we have compared a sample with di erent hypotheses which di ered only in the value of a parameter but corresponded to the same distribution. We now allow for an infinite number of hypotheses by varying the value of a parameter. As in the discrete case, in the absence of a given prior probability, the only

available piece of information which allows us to judge di erent parameter values is the likelihood function. A formal justification for this assertion is given by the likelihood principle (LP) which states that the likelihood function exhausts all the information contained in the observations related to the parameters and which we will discuss in the following chapter. It is then plausible to choose the parameter such that the likelihood is as large as possible. This is the maximum likelihood estimate (MLE). When we are interested in a parameter range, we will choose the interval such that the likelihood outside is always less than inside.

Remark that the MLE as well as likelihood intervals are invariant against transformations of the parameter. The likelihood is not a p.d.f. but a function of the parameter and therefore L(θ) = L′(θ′) for θ′(θ). Thus a likelihood analysis estimating, for example, the mass of a particle will give the same result as that inferring the mass squared, and estimates of the decay rate γ and mean life τ = 1/γ will be consistent.

Here and in the following sections we assume that the likelihood function is continuous and di erentiable and has exactly one maximum inside the valid range of the parameter. This condition is fulfilled in the majority of all cases.

Besides the maximum likelihood (ML) method, invented by Fisher, there exist a number of other methods of parameter estimation. Popular is especially the method of least squares (LS) which was first proposed by Gauß5. It is used to adjust parameters of curves which are fixed by some measured points and will be discussed in the next chapter. It can be traced back to the ML method if the measurement errors are normally distributed.

5Carl Friedrich Gauß (1777-1855), German mathematician, astronomer and physicist.

144 6 Parameter Inference I

In most cases the likelihood function resembles a bell shaped Gaussian and ln L(θ) approximately a downwards open parabola (see Fig. 6.5). This approximation is especially good for large samples.

To find the maximum of L (ln L and L have their maxima at the same location), we derive the log-likelihood6 with respect to the parameter and set the derivative

equal to zero. The value ˆ, that satisfies the equation which we obtain in this way is

θ

the MLE of θ.

Since only the derivative of the likelihood function is of importance, factors in the likelihood or summands in the log-likelhood which are independent of θ can be omitted.

The estimate ˆ is a function of the sample values , and consequently a statistic.

θ xi

The point estimate has to be accompanied by an error interval. Point estimate and error interval form an ensemble and cannot be discussed separately. Choosing as point estimate the value that maximizes the likelihood function it is natural to include inside the error limits parameter values with higher likelihood than all parameters that are excluded. This prescription leads to so-called likelihood ratio error intervals.

We will discuss the error interval estimation in a separate chapter, but fix the error limit already now by definition:

Definition: The limits of a standard error interval are located at the parameter values where the likelihood function has decreased from its maximum by a factor e1/2. For two and three standard deviations the factors are e2 and e4.5. This choice corresponds to di erences for the log-likelihood of 0.5 for one, of 2 for two and of 4.5 for three standard error intervals as illustrated in Fig. 6.5. For the time being we assume that these limits exist inside the parameter range.

The reason for this definition is the following: As already mentioned, asymptotically, when the sample size N tends to infinity, under very general conditions the likelihood function approaches a Gaussian and becomes proportional to the probability density of the parameter (for a proof, see Appendix 13.3). Then our error limit corresponds exactly to the standard deviation of the p.d.f., i.e. the square root of the variance of the Gaussian. We keep the definition also for non normally shaped likelihood functions and small sample sizes. Then we usually get asymmetric error limits.

6.5.2 Examples

Example 76. Maximum likelihood estimate (MLE) of the mean life of an unstable particle

Given be N decay times ti of an unstable particle with unknown mean life τ. For an exponential decay time distribution

f(t|γ) = γe−γt

with γ = 1/τ the likelihood is

6The advantage of using the log-likelihood compared to the normal likelihood is that we do not need to derive a product but a sum which is much more convenient.

Fig. 6.6. Log-likelihood functions for the parameters of a normal distribution: a) for the mean µ with known width (solid curve) and unknown width (dashed curve), b) for the

width σ with known mean (solid curve) and unknown mean (dashed curve). The position p

of the maximum is σˆ = σ N/(N − 1) = 2.108 for both curves.

The log-likelihood function is a parabola. It is shown in Fig. 6.6a for σ = 2. Deriving it with respect to the unknown parameter µ and setting the result equal to zero, we get

N (x − µˆ) = 0 , σ2

µˆ = x .

The likelihood estimate µˆ for the expectation value of the normal distribution is equal to the arithmetic mean x of the sample. It is independent of σ, but

σ determines the width of the likelihood function and the standard error

√

δµ = σ/ N.

Example 78. MLE of the width of a normal distribution with given mean (case Ib)