Теория информации / Cover T.M., Thomas J.A. Elements of Information Theory. 2006., 748p

8.2 AEP FOR CONTINUOUS RANDOM VARIABLES 245

8.2AEP FOR CONTINUOUS RANDOM VARIABLES

One of the important roles of the entropy for discrete random variables is in the AEP, which states that for a sequence of i.i.d. random variables, p(X1, X2, . . . , Xn) is close to 2−nH (X) with high probability. This enables us to define the typical set and characterize the behavior of typical sequences.

We can do the same for a continuous random variable.

Theorem 8.2.1 Let X1, X2, . . . , Xn be a sequence of random variables drawn i.i.d. according to the density f (x). Then

Proof: The proof follows directly from the weak law of large numbers.

The properties of the typical set for continuous random variables parallel those for discrete random variables. The analog of the cardinality of the typical set for the discrete case is the volume of the typical set for continuous random variables.

Definition The volume Vol(A) of a set A Rn is defined as

Vol(A) = dx1 dx2 · · · dxn. (8.12)

A

Theorem 8.2.2 The typical set A(n) has the following properties:

1.Pr A(n) > 1 − for n sufficiently large.

2.Vol A(n) ≤ 2n(h(X)+ ) for all n.

3.Vol A(n) ≥ (1 − )2n(h(X)− ) for n sufficiently large.

248 DIFFERENTIAL ENTROPY

since f (xi ) = f (x) = 1. If f (x) log f (x) is Riemann integrable (a condition to ensure that the limit is well defined [556]), the first term in (8.29) approaches the integral of −f (x) log f (x) as → 0 by definition of Riemann integrability. This proves the following.

Theorem 8.3.1 If the density f (x) of the random variable X is Riemann integrable, then

Thus, the entropy of an n-bit quantization of a continuous random variable X is approximately h(X) + n.

Example 8.3.1

In general, h(X) + n is the number of bits on the average required to describe X to n-bit accuracy.

The differential entropy of a discrete random variable can be considered to be −∞. Note that 2−∞ = 0, agreeing with the idea that the volume of the support set of a discrete random variable is zero.

8.4JOINT AND CONDITIONAL DIFFERENTIAL ENTROPY

As in the discrete case, we can extend the definition of differential entropy of a single random variable to several random variables.

Definition The differential entropy of a set X1, X2, . . . , Xn of random variables with density f (x1, x2, . . . , xn) is defined as

h(X1, X2, . . . , Xn) = − f (xn) log f (xn) dxn. (8.31)

Definition If X, Y have a joint density function f (x, y), we can define the conditional differential entropy h(X|Y ) as

Since in general f (x|y) = f (x, y)/f (y), we can also write

But we must be careful if any of the differential entropies are infinite.

The next entropy evaluation is used frequently in the text.

Theorem 8.4.1 (Entropy of a multivariate normal distribution) Let X1, X2, . . . , Xn have a multivariate normal distribution with mean µ and covariance matrix K. Then

where |K| denotes the determinant of K.

Definition The relative entropy (or Kullback–Leibler distance) D(f ||g) between two densities f and g is defined by

Note that D(f ||g) is finite only if the support set of f is contained in the support set of g. [Motivated by continuity, we set 0 log 00 = 0.]

Definition The mutual information I (X; Y ) between two random variables with joint density f (x, y) is defined as

I (X; Y ) = f (x, y) log f (x)f (y) dx dy. (8.47)

From the definition it is clear that

I (X; Y ) = h(X) − h(X|Y ) = h(Y ) − h(Y |X) = h(X) + h(Y ) − h(X, Y )

(8.48)

The properties of D(f ||g) and I (X; Y ) are the same as in the discrete case. In particular, the mutual information between two random variables is the limit of the mutual information between their quantized

More generally, we can define mutual information in terms of finite partitions of the range of the random variable. Let X be the range of a random variable X. A partition P of X is a finite collection of disjoint sets Pi such that i Pi = X. The quantization of X by P (denoted [X]P) is the discrete random variable defined by

For two random variables X and Y with partitions P and Q, we can calculate the mutual information between the quantized versions of X and Y using (2.28). Mutual information can now be defined for arbitrary pairs of random variables as follows:

252 DIFFERENTIAL ENTROPY

Definition The mutual information between two random variables X and Y is given by

where the supremum is over all finite partitions P and Q.

This is the master definition of mutual information that always applies, even to joint distributions with atoms, densities, and singular parts. Moreover, by continuing to refine the partitions P and Q, one finds a monotonically increasing sequence I ([X]P ; [Y ]Q) I .

By arguments similar to (8.52), we can show that this definition of mutual information is equivalent to (8.47) for random variables that have a density. For discrete random variables, this definition is equivalent to the definition of mutual information in (2.28).

Example 8.5.1 (Mutual information between correlated Gaussian ran-

If ρ = 0, X and Y are independent and the mutual information is 0. If ρ = ±1, X and Y are perfectly correlated and the mutual information is infinite.

8.6 PROPERTIES OF DIFFERENTIAL ENTROPY, RELATIVE ENTROPY, AND MUTUAL INFORMATION

We have equality iff we have equality in Jensen’s inequality, which occurs iff f = g a.e.

Corollary I (X; Y ) ≥ 0 with equality iff X and Y are independent.

Corollary h(X|Y ) ≤ h(X) with equality iff X and Y are independent.

Theorem 8.6.2 (Chain rule for differential entropy )

with equality iff X1, X2, . . . , Xn are independent.

Proof: Follows directly from Theorem 8.6.2 and the corollary to Theorem 8.6.1.

Application (Hadamard’s inequality ) If we let X N(0, K) be a multivariate normal random variable, calculating the entropy in the above inequality gives us

i=1

which is Hadamard’s inequality. A number of determinant inequalities can be derived in this fashion from information-theoretic inequalities (Chapter 17).

Theorem 8.6.3

Translation does not change the differential entropy.

Proof: Follows directly from the definition of differential entropy.

Similarly, we can prove the following corollary for vector-valued random variables.

Corollary

We now show that the multivariate normal distribution maximizes the entropy over all distributions with the same covariance.

Theorem 8.6.5 Let the random vector X Rn have zero mean and covariance K = EXXt (i.e., Kij = EXi Xj , 1 ≤ i, j ≤ n). Then h(X) ≤ 12 log(2π e)n|K|, with equality iff X N(0, K).