Теория информации / Cover T.M., Thomas J.A. Elements of Information Theory. 2006., 748p

Finally, relative entropy is stronger than the L1 norm in the following

The relative entropy between two probability mass functions P (x) and Q(x) is zero when P = Q. Around this point, the relative entropy has a quadratic behavior, and the first term in the Taylor series expansion of the relative entropy D(P ||Q) around the point P = Q is the chi-squared

17.4INEQUALITIES FOR TYPES

The method of types is a powerful tool for proving results in large deviation theory and error exponents. We repeat the basic theorems.

666 INEQUALITIES IN INFORMATION THEORY

Theorem 17.4.2 (Theorem 11.1.2) If X1, X2, . . . , Xn are drawn i.i.d. according to Q(x), the probability of xn depends only on its type and is given by

17.5COMBINATORIAL BOUNDS ON ENTROPY

Lemma 17.5.1 For 0 < p < 1, q = 1 − p, such that np is an integer,

Proof: We begin with a strong form of Stirling’s approximation [208], which states that

Applying this to find an upper bound, we obtain

The lower bound is obtained similarly. Using Stirling’s formula, we

and the lower bound follows directly from substituting this into the equation. The exceptions to this condition are the cases where np = 1, nq = 1 or 2, and np = 2, nq = 2 (the case when np ≥ 3, nq = 1 or 2 can be handled by flipping the roles of p and q). In each of these cases

Thus, even in these special cases, the bound is valid, and hence the lower bound is valid for all p =0, 1. Note that the lower bound blows up when p = 0 or p = 1, and is therefore not valid.

17.6ENTROPY RATES OF SUBSETS

We now generalize the chain rule for differential entropy. The chain rule provides a bound on the entropy rate of a collection of random variables in terms of the entropy of each random variable:

n

h(X1, X2, . . . , Xn) ≤ h(Xi ). (17.50)

i=1

for all k ≤ n by first conditioning on a k-element subset, and then taking

the expectation over all k-element subsets chosen uniformly from the n

Proof: Starting from (17.54), we multiply both sides by r, exponentiate, and then apply the arithmetic mean geometric mean inequality, to obtain

e n1 rh(X1, X2, . . . , Xn)

which is equivalent to tn(n) ≤ tn(n)−1. Now we use the same arguments as

in Theorem 17.6.1, taking an average over all subsets to prove the result that for all k ≤ n, tk(n) .

Definition The average conditional entropy rate per element for all subsets of size k is the average of the above quantities for k-element subsets of {1, 2, . . . , n}:

Here gk (S) is the entropy per element of the set S conditional on the elements of the set Sc. When the size of the set S increases, one can expect a greater dependence among the elements of the set S, which explains Theorem 17.6.1.

672 INEQUALITIES IN INFORMATION THEORY

Theorem 17.7.1 (Theorem 11.10.1: Cram´er – Rao inequality) The mean-squared error of any unbiased estimator T (X) of the parameter θ is lower bounded by the reciprocal of the Fisher information:

We now prove a fundamental relationship between the differential entropy and the Fisher information:

Theorem 17.7.2 (de Bruijn’s identity: entropy and Fisher information ) Let X be any random variable with a finite variance with a density f (x). Let Z be an independent normally distributed random variable with zero

where he is the differential entropy to base e. In particular, if the limit exists as t → 0,

−∞

674 INEQUALITIES IN INFORMATION THEORY

(17.84) goes to 0 at both limits and the theorem is proved. In the proof, we have exchanged integration and differentiation in (17.74), (17.76), (17.78), and (17.82). Strict justification of these exchanges requires the application of the bounded convergence and mean value theorems; the details may be found in Barron [30].

This theorem can be used to prove the entropy power inequality, which gives a lower bound on the entropy of a sum of independent random variables.

Theorem 17.7.3 (Entropy power inequality) If X and Y are independent random n-vectors with densities, then

We outline the basic steps in the proof due to Stam [505] and Blachman [61]. A different proof is given in Section 17.8.

Stam’s proof of the entropy power inequality is based on a perturbation argument. Let n = 1. Let Xt = X + √f (t)Z1, Yt = Y + √g(t)Z2, where Z1 and Z2 are independent N(0, 1) random variables. Then the entropy power inequality for n = 1 reduces to showing that s(0) ≤ 1, where we define

If f (t) → ∞ and g(t) → ∞ as t → ∞, it is easy to show that s(∞) = 1. If, in addition, s (t) ≥ 0 for t ≥ 0, this implies that s(0) ≤ 1. The proof of the fact that s (t) ≥ 0 involves a clever choice of the functions f (t) and g(t), an application of Theorem 17.7.2 and the use of a convolution inequality for Fisher information,

The entropy power inequality can be extended to the vector case by induction. The details may be found in the papers by Stam [505] and Blachman [61].

17.8 ENTROPY POWER INEQUALITY AND BRUNN–MINKOWSKI INEQUALITY

The entropy power inequality provides a lower bound on the differential entropy of a sum of two independent random vectors in terms of their individual differential entropies. In this section we restate and outline an