Теория информации / Cover T.M., Thomas J.A. Elements of Information Theory. 2006., 748p

From (11.110), it follows that P has the form

to estimate the probability of observing more than 700 heads in a series of 1000 tosses. The problem is like Example 11.5.1. The probability is

where P is the (0.7, 0.3) distribution and Q is the (0.5, 0.5) distribution. In this case, D(P ||Q) = 1 − H (P ) = 1 − H (0.7) = 0.119. Thus, the probability of 700 or more heads in 1000 trials is approximately 2−119.

Example 11.5.3 (Mutual dependence) Let Q(x, y) be a given joint distribution and let Q0(x, y) = Q(x)Q(y) be the associated product distribution formed from the marginals of Q. We wish to know the likelihood that a sample drawn according to Q0 will “appear” to be jointly distributed according to Q. Accordingly, let (Xi , Yi ) be i.i.d. Q0(x, y) = Q(x)Q(y). We define joint typicality as we did in Section 7.6; that is, (xn, yn) is jointly typical with respect to a joint distribution Q(x, y) iff the sample entropies are close to their true values:

We wish to calculate the probability (under the product distribution) of seeing a pair (xn, yn) that looks jointly typical of Q [i.e., (xn, yn)

E

P

P *

Q

FIGURE 11.5. Pythagorean theorem for relative entropy.

number of terms. Since the number of terms is polynomial in the length of the sequences, the sum is equal to the largest term to first order in the exponent.

We now strengthen the argument to show that not only is the probability of the set E essentially the same as the probability of the closest type P but also that the total probability of other types that are far away from P is negligible. This implies that with very high probability, the type observed is close to P . We call this a conditional limit theorem.

Before we prove this result, we prove a “Pythagorean” theorem, which gives some insight into the geometry of D(P ||Q). Since D(P ||Q) is not a metric, many of the intuitive properties of distance are not valid for D(P ||Q). The next theorem shows a sense in which D(P ||Q) behaves like the square of the Euclidean metric (Figure 11.5).

Theorem 11.6.1 For a closed convex set E P and distribution Q / E, let P E be the distribution that achieves the minimum distance to Q; that is,

for all P E.

368 INFORMATION THEORY AND STATISTICS

Note. The main use of this theorem is as follows: Suppose that we have a sequence Pn E that yields D(Pn||Q) → D(P ||Q). Then from the Pythagorean theorem, D(Pn||P ) → 0 as well.

Then Pλ → P as λ → 0. Also, since E is convex, Pλ E for 0 ≤ λ ≤ 1. Since D(P ||Q) is the minimum of D(Pλ||Q) along the path P → P , the derivative of D(Pλ||Q) as a function of λ is nonnegative at λ = 0. Now

Note that the relative entropy D(P ||Q) behaves like the square of the Euclidean distance. Suppose that we have a convex set E in Rn. Let A be a point outside the set, B the point in the set closest to A, and C any

A

B

C

FIGURE 11.6. Triangle inequality for distance squared.

other point in the set. Then the angle between the lines BA and BC must be obtuse, which implies that lAC2 ≥ lAB2 + lBC2 , which is of the same form as Theorem 11.6.1. This is illustrated in Figure 11.6.

We now prove a useful lemma which shows that convergence in relative entropy implies convergence in the L1 norm.

Definition The L1 distance between any two distributions is defined as

370 INFORMATION THEORY AND STATISTICS

Also note that

X

The left-hand side of (11.137) is called the variational distance between P1 and P2.

Lemma 11.6.1

Proof: We first prove it for the binary case. Consider two binary distributions with parameters p and q with p ≥ q. We will show that

Define a new binary random variable Y = φ (X), the indicator of the set A,

and let ˆ1 and ˆ2 be the distributions of . Thus, ˆ1 and ˆ2 correspond

P P Y P P

to the quantized versions of P1 and P2. Then by the data-processing

372 INFORMATION THEORY AND STATISTICS

where λ is chosen to satisfy P (a)a2 = α. Thus, the conditional distribution on X1 given a constraint on the sum of the squares is a (normalized) product of the original probability mass function and the maximum entropy probability mass function (which in this case is Gaussian).

Proof of Theorem: Define the sets

374 INFORMATION THEORY AND STATISTICS

which goes to 0 as n → ∞. Hence the conditional probability of B goes to 0 as n → ∞, which implies that the conditional probability of A goes to 1.

We now show that all the members of A are close to P in relative entropy. For all members of A,

since D(P ||Q) = D . Thus, Px A implies that D(Px||Q) ≤ D + 2δ, and therefore that D(Px||P ) ≤ 2δ. Consequently, since Pr{PXn A|PXnE} → 1, it follows that

as n → ∞. By Lemma 11.6.1, the fact that the relative entropy is small implies that the L1 distance is small, which in turn implies that maxa X

|PXn (a) − P (a)| is small. Thus, Pr(|PXn (a) − P (a)| ≥ |PXn E) →

0 as n → ∞. Alternatively, this can be written as

In this theorem we have only proved that the marginal distribution goes to P as n → ∞. Using a similar argument, we can prove a stronger version of this theorem:

Pr(X1 = a1, X2 = a2, . . . , Xm

i=1

This holds for fixed m as n → ∞. The result is not true for m = n, since there are end effects; given that the type of the sequence is in E, the last elements of the sequence can be determined from the remaining elements, and the elements are no longer independent. The conditional limit