vstatmp_engl

Here Pi(x) is the Legendre polynomial in the usual form. The first parameter θ0 is fixed, θ0 = 1, and the other parameters are restricted such that gk is positive. The user has to choose the parameter k which limits the degree of the polynomials. If the alternative hypothesis is suspected to contain narrow structures, we have to admit large k. The test with k = 1 rejects a linear contribution, k = 2 in addition a quadratic component and so on. Obviously, the null hypothesis H0 corresponds to

θ1 = · · · = θk = 0, or equivalently to Pk θ2 = 0. We have to look for a test statistic

i=1 i

which increases with the value of this sum.

For a sample of size N the test statistic proposed by Neyman is

This choice is plausible, because a large absolute value of ti is due to a strong contribution of the polynomial πi to the observed distribution and thus also to a large value of θi2, while under H0 we have for i ≥ 1

htii = Nhπi(z)i = 0 ,

Asymptotically, N → ∞, under H0 the test statistic rk2 follows a χ2 distribution with k degrees of freedom (see 3.6.7). This is a consequence of the orthonormality of the polynomials πi and the central limit theorem: We have

√

and as a sum of N random variables the statistic ti/ N is normally distributed for large N, with expectation value zero and variance one. Due to the orthogonality of the πi, the ti are uncorrelated. For small N the distribution of the test statistic rk2 has to be obtained by a Monte Carlo simulation.

In any case, large values of rk2 indicate bad agreement of the data with H0, but for a fixed value of k the smooth test is not consistent6. Its power approaches unity for N → ∞ only for the class of alternatives Hk having a PIT which is represented by an expansion in Legendre polynomials up to order k. Hence with respect to these, while usually uninteresting, restricted alternatives it is consistent. Thus for large samples and especially for the exclusion of narrow structures k should not be chosen too small. The value of k in the smooth test corresponds roughly to the number of bins in the χ2-test.

The smooth test is in most cases superior to the χ2 test. This can be understood in the following way: The smooth test scrutinizes not only for structures of a fixed

6For k = 1, for instance, the test cannot exclude distributions concentrated near z = 1/2.

10.3.9 Comparing a Data Sample to a Monte Carlo Sample and the Metric

We know turn to tests where we compare our sample not to an analytic distribution but to a Monte Carlo simulation of f0. This is not a serious restriction because anyhow acceptance and resolution e ects have to be taken into account in the majority of all experiments. Thus the null hypothesis is usually represented by a simulation sample.

To compare two samples we have to construct a relation between observations of the samples which in the multi-dimensional case has to depend in some way on the distance between them. We can define the distance in the usual way using the standard Euclidian metric but since the di erent dimensions often represent completely di erent physical quantities, e.g. spatial distance, time, mass etc., we have considerable freedom in the choice of the metric and we will try to adjust the metric such that the test is powerful.

We usually want that all coordinates enter with about equal weight into the test. If, for example, the distribution is very narrow in x but wide in y, then the distance r of points is almost independent of y and it is reasonable to stretch the distribution in the x direction before we apply a test. Therefore we propose for the general case to scale linearly all coordinates such that the empirical variances of the sample are the same in all dimensions. In addition we may want to get rid of correlations when for instance a distribution is concentrated in a narrow band along the x-y diagonal.

Instead of transforming the coordinates we can use the Mahalanobis distance7 in order to normalize distances between observations (x1, . . . , xN } with sample mean x. (The bold-face symbols here denote P -dimensional vectors describing di erent features measured on each of the N sampled objects.)

The Mahalanobis distance dM of two observations x and x′ is q

dM = (x − x′)T C−1(x − x′) ,

with

XN

Cij = (xni − xi)(xnj − xj )/N .

n=1

It is equivalent to the Euclidian distance after a linear transformation of the vector components which produces a sample with unity covariance matrix. If the covariance matrix is diagonal, then the resulting distance is the normalized Euclidean distance in the P -dimensional space:

In the following tests the choice of the metric is up to the user. In many situations it is reasonable to use the Mahalanobis distance, even though moderate variations of the metric normally have little influence on the power of a test.

7This is a distance measure introduced by P. C. Mahalanobis in 1936.

The term φ3 is independent of the data and can be omitted but is normally included to reduce statistical fluctuations.

The distance function R relates sample points and simulated points of the null hypothesis to each other. Proven useful have the functions

The small positive constant ε suppresses the pole of the logarithmic distance function. Its value should be chosen approximately equal to the experimental resolution8 but variations of ε by a large factor have no sizable influence on the result. With the function R1 = 1/r we get the special case of electrostatics. With the Gaussian distance function Rs the test is very similar to the χ2 test with bin width 2s but avoids the arbitrary binning of the latter. For slowly varying functions we propose to use the logarithmic distance function.

Empirical studies have shown that the test statistic follows to a very good approximation a distribution from the family of the extreme value distributions,

with parameters µ, σ, ξ which we have discussed in Sect. 3.6.12. We have to compute this distribution f0(φ) by a Monte Carlo simulation. To get a su cient precision for small p-values we have to generate a very large sample. We gain computing time when we extract the first three moments of a relatively modest Monte Carlo sample and compute from those the parameters of the extreme value distribution.

The energy test is consistent [63]. It is quite powerful in many situations and has the advantage that it is not required to sort the sample elements.

The energy test with Gaussian distance function is completely equivalent to the L2 test. It is more general than the latter in that it allows to use various distance functions.

10.3.12 Tests Designed for Specific Problems

The power of tests depends on the alternatives. If we have an idea of it, even if it is crude, we can design a GOF test which is especially sensitive to the deviations from H0 which we have in mind. The distribution of the test statistic has to be produced by a Monte Carlo simulation.

Example 130. Designed test: three region test

8Distances between two points that are smaller than the resolution are accidental and thus insignificant.

Fig. 10.12. Di erent admixtures to a uniform distribution.

Experimental distributions often show a local excess of observations which are either just a statistical fluctuation or stem from a physical process. To check whether an experimental sample is compatible with the absence of a bump caused by a physical process, we propose the following three region test. We subdivide the domain of the variable in three regions with expected numbers of observations n10, n20, n30 and look for di erences to the corresponding experimental numbers n1, n2, n3. The subdivision is chosen such that the sum of the di erences is maximum. The test statistic R3 is

R3 = sup [(n1 − n10) + (n2 − n20) + (n3 − n30)] .

n1,n2

Notice, that n3 = N − n1 − n2 is a function of n1 and n2. The generalization to more than three regions is trivial. Like in the χ2 test we could also divide the individual squared di erences by their expected value:

10.3.13 Comparison of Tests

Univariate Distributions

Whether a test is able to detect deviations from H0 depends on the distribution f0 and on the kind of distortion. Thus there is no test which is most powerful in all situations.

To get an idea of the power of di erent tests, we consider six di erent admixtures to a uniform distribution and compute the fraction of cases in which the distortion of the uniform distribution is detected at a significance level of 5%. For each distribution constructed in this way, we generate stochastically 1000 mixtures with 100 observations each. The distributions which we add are depicted in Fig. 10.12. One of them is linear, two are normal with di erent widths, and three are parabolic. The χ2 test was performed with 12 bins following the prescription of Ref. [58], the parameter of Neyman’s smooth test was k = 2 and the width of the Gaussian of the energy test was s = 1/8. The sensitivity of di erent tests is presented in Fig. 10.13.

The histogram of Fig. 10.13 shows that none of the tests is optimum in all cases. The χ2 test performs only mediocrely. Probably a lower bin number would improve the result. The tests of Neyman, Anderson–Darling and Kolmogorov–Smirnov are sensitive to a shift of the mean value while the Anderson–Darling test reacts especially to changes at the borders of the distribution. The tests of Watson and Kuiper detect preferentially variations of the variance. Neyman’s test and the energy test with logarithmic distance function are rather e cient in most cases.

274 10 Hypothesis Tests

Fig. 10.13. Power (fraction of identified distortions) of di erent tests.

Multivariate Distributions

The goodness-of-fit of multivariate distributions cannot be tested very well with simple tests. The χ2 test often su ers from the small number of entries per bin. Here the k-nearest neighbor test and the energy test with the long range logarithmic distance function are much more e cient.

Example 131. GOF test of a two-dimensional sample

Figure 10.14 shows a comparison of a sample H1 with a two-dimensional normal distribution (H0). H1 corresponds to the distribution of H0 but contains an admixture of a linear distribution. The p-value of the energy test is 2%.