Теория информации / Cover T.M., Thomas J.A. Elements of Information Theory. 2006., 748p

It would be useful computationally to have upper and lower bounds converging to the limit from above and below. We can halt the computation when the difference between upper and lower bounds is small, and we

H (Yn|Yn−1, . . . , Y1, X1). This is a neat trick based on the idea that X1 contains as much information about Yn as Y1, Y0, Y−1, . . . .

where (a) follows from that fact that Y1 is a function of X1, and (b) follows from the Markovity of X, (c) follows from the fact that Yi is a function of Xi , (d) follows from the fact that conditioning reduces entropy, and (e) follows by stationarity. Since the inequality is true for all k, it is true in the limit. Thus,

The next lemma shows that the interval between the upper and the lower bounds decreases in length.

Lemma 4.5.2

and I (X1; Y1, Y2, . . . , Yn) increases with n. Thus, lim I (X1; Y1, Y2, . . . , Yn) exists and

i=1

Since this infinite sum is finite and the terms are nonnegative, the terms must tend to 0; that is,

Combining Lemmas 4.5.1 and 4.5.2, we have the following theorem.

Theorem 4.5.1 If X1, X2, . . . , Xn form a stationary Markov chain, and

Yi = φ (Xi ), then

and

lim H (Yn|Yn−1, . . . , Y1, X1) = H (Y) = lim H (Yn|Yn−1, . . . , Y1). (4.74)

In general, we could also consider the case where Yi is a stochastic function (as opposed to a deterministic function) of Xi . Consider a Markov

SUMMARY 87

process X1, X2, . . . , Xn, and define a new process Y1, Y2, . . . , Yn, where each Yi is drawn according to p(yi |xi ), conditionally independent of all the other Xj , j =i; that is,

Such a process, called a hidden Markov model (HMM), is used extensively in speech recognition, handwriting recognition, and so on. The same argument as that used above for functions of a Markov chain carry over to hidden Markov models, and we can lower bound the entropy rate of a hidden Markov model by conditioning it on the underlying Markov state. The details of the argument are left to the reader.

SUMMARY

Entropy rate. Two definitions of entropy rate for a stochastic process are

Second law of thermodynamics. For a Markov chain:

1.Relative entropy D(µn||µn) decreases with time

2.Relative entropy D(µn||µ) between a distribution and the stationary distribution decreases with time.

3.Entropy H (Xn) increases if the stationary distribution is uniform.

PROBLEMS 89

In other words, the present has a conditional entropy given the past equal to the conditional entropy given the future. This is true even though it is quite easy to concoct stationary random processes for which the flow into the future looks quite different from the flow into the past. That is, one can determine the direction of time by looking at a sample function of the process. Nonetheless, given the present state, the conditional uncertainty of the next symbol in the future is equal to the conditional uncertainty of the previous symbol in the past.

4.3Shuffles increase entropy . Argue that for any distribution on shuffles T and any distribution on card positions X that

if X and T are independent.

4.4Second law of thermodynamics . Let X1, X2, X3, . . . be a station-

ary first-order Markov chain. In Section 4.4 it was shown that H (Xn | X1) ≥ H (Xn−1 | X1) for n = 2, 3, . . . . Thus, conditional uncertainty about the future grows with time. This is true although the unconditional uncertainty H (Xn) remains constant. However, show by example that H (Xn|X1 = x1) does not necessarily grow with n for every x1.

4.5Entropy of a random tree. Consider the following method of generating a random tree with n nodes. First expand the root node:

Then expand one of the two terminal nodes at random:

At time k, choose one of the k − 1 terminal nodes according to a uniform distribution and expand it. Continue until n terminal nodes

90 ENTROPY RATES OF A STOCHASTIC PROCESS

have been generated. Thus, a sequence leading to a five-node tree might look like this:

Surprisingly, the following method of generating random trees yields the same probability distribution on trees with n terminal nodes. First choose an integer N1 uniformly distributed on {1, 2, . . . , n − 1}. We then have the picture

{1, 2, . . . , N1 − 1}, and independently choose another integer N3 uniformly over {1, 2, . . . , (n − N1) − 1}. The picture is now

Continue the process until no further subdivision can be made. (The equivalence of these two tree generation schemes follows, for example, from Polya’s urn model.)

Now let Tn denote a random n-node tree generated as described. The probability distribution on such trees seems difficult to describe, but we can find the entropy of this distribution in recursive form.

First some examples. For n = 2, we have only one tree. Thus, H (T2) = 0. For n = 3, we have two equally probable trees:

Thus, H (T3) = log 2. For n = 4, we have five possible trees, with

probabilities 13 , 16 , 16 , 16 , 16 .

Now for the recurrence relation. Let N1(Tn) denote the number of terminal nodes of Tn in the right half of the tree. Justify each of the steps in the following:

for appropriately defined cn. Since cn = c < ∞, you have proved that n1 H (Tn) converges to a constant. Thus, the expected number of bits necessary to describe the random tree Tn grows linearly with n.

4.6Monotonicity of entropy per element . For a stationary stochastic process X1, X2, . . . , Xn, show that

(a)

92ENTROPY RATES OF A STOCHASTIC PROCESS

4.7Entropy rates of Markov chains

(a)Find the entropy rate of the two-state Markov chain with tran-

sition matrix

(b)What values of p01, p10 maximize the entropy rate?

(c)Find the entropy rate of the two-state Markov chain with tran-

sition matrix

(d)Find the maximum value of the entropy rate of the Markov

chain of part (c). We expect that the maximizing value of p

should be less than 12 , since the 0 state permits more information to be generated than the 1 state.

(e)Let N (t ) be the number of allowable state sequences of length t for the Markov chain of part (c). Find N (t ) and calculate

1

H0 = lim log N (t ).

t →∞ t

[Hint: Find a linear recurrence that expresses N (t ) in terms of N (t − 1) and N (t − 2). Why is H0 an upper bound on the entropy rate of the Markov chain? Compare H0 with the maximum entropy found in part (d).]

4.8Maximum entropy process . A discrete memoryless source has the alphabet {1, 2}, where the symbol 1 has duration 1 and the sym-

bol 2 has duration 2. The probabilities of 1 and 2 are p1 and p2, respectively. Find the value of p1 that maximizes the source entropy

per unit time H (X) = H (X) . What is the maximum value H (X)?

ET

4.9Initial conditions . Show, for a Markov chain, that

H (X0|Xn) ≥ H (X0|Xn−1).

Thus, initial conditions X0 become more difficult to recover as the future Xn unfolds.

4.10Pairwise independence. Let X1, X2, . . . , Xn−1 be i.i.d. random variables taking values in {0, 1}, with Pr{Xi = 1} = 12 . Let Xn = 1

PROBLEMS 93

(a) Show that Xi and Xj are independent for i =j , i, j {1, 2,

.. . , n}.

(b)Find H (Xi , Xj ) for i =j .

(c)Find H (X1, X2, . . . , Xn). Is this equal to nH (X1)?

4.11Stationary processes . Let . . . , X−1, X0, X1, . . . be a stationary (not necessarily Markov) stochastic process. Which of the following statements are true? Prove or provide a counterexample.

(a)H (Xn|X0) = H (X−n|X0) .

(b)H (Xn|X0) ≥ H (Xn−1|X0) .

(c)H (Xn|X1, X2, . . . , Xn−1, Xn+1) is nonincreasing in n.

(d)H (Xn|X1, . . . , Xn−1, Xn+1, . . . , X2n) is nonincreasing in n.

4.12Entropy rate of a dog looking for a bone. A dog walks on the integers, possibly reversing direction at each step with probability p = 0.1. Let X0 = 0. The first step is equally likely to be positive or negative. A typical walk might look like this:

(X0, X1, . . .) = (0, −1, −2, −3, −4, −3, −2, −1, 0, 1, . . .).

(a)Find H (X1, X2, . . . , Xn).

(b)Find the entropy rate of the dog.

(c)What is the expected number of steps that the dog takes before reversing direction?

4.13The past has little to say about the future. For a stationary stochastic process X1, X2, . . . , Xn, . . . , show that

Thus, the dependence between adjacent n-blocks of a stationary process does not grow linearly with n.

4.14Functions of a stochastic process

(a)Consider a stationary stochastic process X1, X2, . . . , Xn, and let Y1, Y2, . . . , Yn be defined by

for some function φ. Prove that