Теория информации / Cover T.M., Thomas J.A. Elements of Information Theory. 2006., 748p

66ASYMPTOTIC EQUIPARTITION PROPERTY

(c)Show that |An ∩ Bn| ≤ 2n(H + ) for all n.

(d)Show that |An ∩ Bn| ≥ 12 2n(H − ) for n sufficiently large.

3.5Sets defined by probabilities . Let X1, X2, . . . be an i.i.d. sequence of discrete random variables with entropy H (X). Let

Cn(t ) = {xn Xn : p(xn) ≥ 2−nt }

denote the subset of n-sequences with probabilities ≥ 2−nt .

(a)Show that |Cn(t )| ≤ 2nt .

(b)For what values of t does P ({Xn Cn(t )}) → 1?

3.6AEP-like limit . Let X1, X2, . . . be i.i.d. drawn according to probability mass function p(x). Find

1

lim (p(X1, X2, . . . , Xn)) n .

n→∞

3.7AEP and source coding . A discrete memoryless source emits a sequence of statistically independent binary digits with probabilities p(1) = 0.005 and p(0) = 0.995. The digits are taken 100 at a time and a binary codeword is provided for every sequence of 100 digits containing three or fewer 1’s.

(a)Assuming that all codewords are the same length, find the minimum length required to provide codewords for all sequences with three or fewer 1’s.

(b)Calculate the probability of observing a source sequence for which no codeword has been assigned.

(c)Use Chebyshev’s inequality to bound the probability of observing a source sequence for which no codeword has been assigned. Compare this bound with the actual probability computed in part (b).

3.8Products . Let

Let X1, X2, . . . be drawn i.i.d. according to this distribution. Find the limiting behavior of the product

1

(X1X2 · · · Xn) n .

PROBLEMS 67

3.9AEP . Let X1, X2, . . . be independent, identically distributed random variables drawn according to the probability mass function

(a)Evaluate lim − n1 log q(X1, X2, . . . , Xn), where X1, X2, . . . are i.i.d. p(x).

odds favoring q are exponentially small when p is true.

3.10Random box size.

An n-dimensional rectangular box with sides X1, X2, X3, . . . , Xn is

of a n-cube with the same volume as the random box is l = Vn . Let X1, X2, . . . be i.i.d. uniform random variables over the unit

the expected edge length does not capture the idea of the volume of the box. The geometric mean, rather than the arithmetic mean, characterizes the behavior of products.

3.11Proof of Theorem 3.3.1. This problem shows that the size of the smallest “probable” set is about 2nH . Let X1, X2, . . . , Xn be i.i.d.

p(x). Let Bδ(n) Xn such that Pr(Bδ(n)) > 1 − δ. Fix < 12 .

(a)Given any two sets A, B such that Pr(A) > 1 − 1 and Pr(B) >

1 − 2, show that Pr(A ∩ B) > 1 − 1 − 2. Hence, Pr(A(n) ∩

Bδ(n)) ≥ 1 − − δ.

(b)Justify the steps in the chain of inequalities

CHAPTER 4

ENTROPY RATES

OF A STOCHASTIC PROCESS

The asymptotic equipartition property in Chapter 3 establishes that nH (X) bits suffice on the average to describe n independent and identically distributed random variables. But what if the random variables are dependent? In particular, what if the random variables form a stationary process? We will show, just as in the i.i.d. case, that the entropy H (X1, X2, . . . , Xn) grows (asymptotically) linearly with n at a rate H (X), which we will call the entropy rate of the process. The interpretation of H (X) as the best achievable data compression will await the analysis in Chapter 5.

4.1MARKOV CHAINS

A stochastic process {Xi } is an indexed sequence of random variables. In general, there can be an arbitrary dependence among the random variables. The process is characterized by the joint probability mass functions

Pr{(X1, X2, . . . , Xn) = (x1, x2, . . . , xn)} = p(x1, x2, . . . , xn), (x1, x2, . . . , xn) Xn for n = 1, 2, . . . .

Definition A stochastic process is said to be stationary if the joint distribution of any subset of the sequence of random variables is invariant with respect to shifts in the time index; that is,

Pr{X1 = x1, X2 = x2, . . . , Xn = xn}

= Pr{X1+l = x1, X2+l = x2, . . . , Xn+l = xn} (4.1) for every n and every shift l and for all x1, x2, . . . , xn X.

Elements of Information Theory, Second Edition, By Thomas M. Cover and Joy A. Thomas Copyright 2006 John Wiley & Sons, Inc.

71

72 ENTROPY RATES OF A STOCHASTIC PROCESS

A simple example of a stochastic process with dependence is one in which each random variable depends only on the one preceding it and is conditionally independent of all the other preceding random variables. Such a process is said to be Markov.

Definition A discrete stochastic process X1, X2, . . . is said to be a

Markov chain or a Markov process if for n = 1, 2, . . . ,

for all x1, x2, . . . , xn, xn+1 X.

In this case, the joint probability mass function of the random variables

Definition The Markov chain is said to be time invariant if the conditional probability p(xn+1|xn) does not depend on n; that is, for n =

1, 2, . . . ,

Pr{Xn+1 = b|Xn = a} = Pr{X2 = b|X1 = a} for all a, b X. (4.4)

We will assume that the Markov chain is time invariant unless otherwise stated.

If {Xi } is a Markov chain, Xn is called the state at time n. A timeinvariant Markov chain is characterized by its initial state and a probability

transition matrix P = [Pij ], i, j {1, 2, . . . , m}, where Pij = Pr{Xn+1 = j |Xn = i}.

If it is possible to go with positive probability from any state of the Markov chain to any other state in a finite number of steps, the Markov chain is said to be irreducible. If the largest common factor of the lengths of different paths from a state to itself is 1, the Markov chain is said to aperiodic.

If the probability mass function of the random variable at time n is

A distribution on the states such that the distribution at time n + 1 is the same as the distribution at time n is called a stationary distribution. The

stationary distribution is so called because if the initial state of a Markov chain is drawn according to a stationary distribution, the Markov chain forms a stationary process.

If the finite-state Markov chain is irreducible and aperiodic, the stationary distribution is unique, and from any starting distribution, the distribution of Xn tends to the stationary distribution as n → ∞.

Example 4.1.1 Consider a two-state Markov chain with a probability transition matrix

as shown in Figure 4.1.

Let the stationary distribution be represented by a vector µ whose components are the stationary probabilities of states 1 and 2, respectively. Then the stationary probability can be found by solving the equation µP = µ or, more simply, by balancing probabilities. For the stationary distribution, the net probability flow across any cut set in the state transition graph is zero. Applying this to Figure 4.1, we obtain

Since µ1 + µ2 = 1, the stationary distribution is

If the Markov chain has an initial state drawn according to the stationary distribution, the resulting process will be stationary. The entropy of the

FIGURE 4.1. Two-state Markov chain.