Теория информации / Cover T.M., Thomas J.A. Elements of Information Theory. 2006., 748p

with initial conditions X(1) = [1 1 0]t .

Using the eigenvalue decomposition of A for the case of distinct

where Y1, Y2, Y3 do not depend on n. For large n, this sum is dominated by the largest term. Therefore, argue that for i = 1, 2, 3, we have

where λ is the largest (positive) eigenvalue. Thus, the number of sequences of length n grows as λn for large n. Calculate λ for the matrix A above. (The case when the eigenvalues are not distinct can be handled in a similar manner.)

(d) We now take a different approach. Consider a Markov chain whose state diagram is the one given in part (a), but with arbitrary transition probabilities. Therefore, the probability transition matrix of this Markov chain is

Show that the stationary distribution of this Markov chain is

PROBLEMS 97

(b) What is the entropy rate?

(c) Find the mutual information I (Xn+1; Xn) assuming that the process is stationary.

4.20 Random walk on chessboard . Find the entropy rate of the Markov chain associated with a random walk of a king on the 3 × 3 chess-

rates H (X), H (Y), H (Z), and H (V) of the processes {Xi },{Yi }, {Zi }, and {Vi }. What is the inequality relationship ≤, =, or ≥

between each of the pairs listed below?

(a)H (X)<H (Y).

(b)H (X)<H (Z).

(c)H (X)<H (V).

(d)H (Z)<H (X).

4.25Monotonicity

(a)Show that I (X; Y1, Y2, . . . , Yn) is nondecreasing in n.

98ENTROPY RATES OF A STOCHASTIC PROCESS

(b)Under what conditions is the mutual information constant for all n?

4.26Transitions in Markov chains . Suppose that {Xi } forms an irreducible Markov chain with transition matrix P and stationary distri-

bution µ. Form the associated “edge process” {Yi } by keeping track only of the transitions. Thus, the new process {Yi } takes values in X × X, and Yi = (Xi−1, Xi ). For example,

Xn = 3, 2, 8, 5, 7, . . .

becomes

Y n = ( , 3), (3, 2), (2, 8), (8, 5), (5, 7), . . . .

Find the entropy rate of the edge process {Yi }.

4.27Entropy rate. Let {Xi } be a stationary {0, 1}-valued stochastic process obeying

Xk+1 = Xk Xk−1 Zk+1,

where {Zi } is Bernoulli(p)and denotes mod 2 addition. What is the entropy rate H (X)?

4.28Mixture of processes . Suppose that we observe one of two stochastic processes but don’t know which. What is the entropy rate? Specifically, let X11, X12, X13, . . . be a Bernoulli process with

parameter p1, and let X21, X22, X23, . . . be Bernoulli(p2). Let

and let Yi = Xθ i , i = 1, 2, . . . , be the stochastic process observed. Thus, Y observes the process {X1i } or {X2i }. Eventually, Y will know which.

(a)Is {Yi } stationary?

(b)Is {Yi } an i.i.d. process?

(c)What is the entropy rate H of {Yi }?

(d)Does

1

− n log p(Y1, Y2, . . . Yn) −→ H ?

(e) Is there a code that achieves an expected per-symbol description length n1 ELn −→ H ?

PROBLEMS 99

Now let θi be Bern( 12 ). Observe that

Zi = Xθi i , i = 1, 2, . . . .

Thus, θ is not fixed for all time, as it was in the first part, but is chosen i.i.d. each time. Answer parts (a), (b), (c), (d), (e) for the process {Zi }, labeling the answers (a ), (b ), (c ), (d ), (e ).

4.29Waiting times . Let X be the waiting time for the first heads to appear in successive flips of a fair coin. For example, Pr{X = 3} = ( 12 )3. Let Sn be the waiting time for the nth head to appear. Thus,

S0 = 0

Sn+1 = Sn + Xn+1,

where X1, X2, X3, . . . are i.i.d according to the distribution above.

(a)Is the process {Sn} stationary?

(b)Calculate H (S1, S2, . . . , Sn).

(c)Does the process {Sn} have an entropy rate? If so, what is it? If not, why not?

(d)What is the expected number of fair coin flips required to generate a random variable having the same distribution as Sn?

4.30Markov chain transitions

Let X1 be distributed uniformly over the states {0, 1, 2}. Let {Xi }∞1 be a Markov chain with transition matrix P ; thus, P (Xn+1 = j |Xn = i) = Pij , i, j {0, 1, 2}.

(a)Is {Xn} stationary?

(b)Find limn→∞ n1 H (X1, . . . , Xn).

Now consider the derived process Z1, Z2, . . . , Zn, where

Z1 = X1

Zi = Xi − Xi−1 (mod 3), i = 2, . . . , n.

Thus, Zn encodes the transitions, not the states.

(c)Find H (Z1, Z2, . . . , Zn).

(d)Find H (Zn) and H (Xn) for n ≥ 2.

100 ENTROPY RATES OF A STOCHASTIC PROCESS

(e)Find H (Zn|Zn−1) for n ≥ 2.

(f)Are Zn−1 and Zn independent for n ≥ 2?

4.31Markov . Let {Xi } Bernoulli(p). Consider the associated

Markov chain {Yi }ni=1, where

Yi = (the number of 1’s in the current run of 1’s). For example, if Xn = 101110 . . . , we have Y n = 101230 . . . .

(a)Find the entropy rate of Xn.

(b)Find the entropy rate of Y n.

4.32Time symmetry . Let {Xn} be a stationary Markov process. We condition on (X0, X1) and look into the past and future. For what index k is

H (X−n|X0, X1) = H (Xk |X0, X1)?

Give the argument.

4.33Chain inequality . Let X1 → X2 → X3 → X4 form a Markov chain. Show that

I (X1; X3) + I (X2; X4) ≤ I (X1; X4) + I (X2; X3). (4.108)

4.34Broadcast channel . Let X → Y → (Z, W ) form a Markov chain [i.e., p(x, y, z, w) = p(x)p(y|x)p(z, w|y) for all x, y, z, w]. Show that

4.35Concavity of second law . Let {Xn}∞−∞ be a stationary Markov process. Show that H (Xn|X0) is concave in n. Specifically, show that

H (Xn|X0) − H (Xn−1|X0) − (H (Xn−1|X0) − H (Xn−2|X0))

Thus, the second difference is negative, establishing that H (Xn|X0) is a concave function of n.

HISTORICAL NOTES

The entropy rate of a stochastic process was introduced by Shannon [472], who also explored some of the connections between the entropy rate of the process and the number of possible sequences generated by the process. Since Shannon, there have been a number of results extending the basic

theorems of information theory to general stochastic processes. The AEP for a general stationary stochastic process is proved in Chapter 16.

Hidden Markov models are used for a number of applications, such as speech recognition [432]. The calculation of the entropy rate for constrained sequences was introduced by Shannon [472]. These sequences are used for coding for magnetic and optical channels [288].

104 DATA COMPRESSION

For example, C(red) = 00, C(blue) = 11 is a source code for X = {red, blue} with alphabet D = {0, 1}.

Definition The expected length L(C) of a source code C(x) for a random variable X with probability mass function p(x) is given by

where l(x) is the length of the codeword associated with x.

Without loss of generality, we can assume that the D-ary alphabet is

D= {0, 1, . . . , D − 1}.

Some examples of codes follow.

Example 5.1.1 Let X be a random variable with the following distribution and codeword assignment:

The entropy H (X) of X is 1.75 bits, and the expected length L(C) = El(X) of this code is also 1.75 bits. Here we have a code that has the same average length as the entropy. We note that any sequence of bits can be uniquely decoded into a sequence of symbols of X. For example, the bit string 0110111100110 is decoded as 134213.

Example 5.1.2 Consider another simple example of a code for a random variable:

Just as in Example 5.1.1, the code is uniquely decodable. However, in this case the entropy is log 3 = 1.58 bits and the average length of the encoding is 1.66 bits. Here El(X) > H (X).

Example 5.1.3 (Morse code) The Morse code is a reasonably efficient code for the English alphabet using an alphabet of four symbols: a dot,