Теория информации / Gray R.M. Entropy and information theory. 1990., 284p

3.2Stationary Ergodic Sources

This section is devoted to proving the entropy ergodic theorem for the special case of stationary ergodic sources. The result was originally proved by Breiman [19]. The original proof flrst used the martingale convergence theorem to infer the convergence of conditional probabilities of the form m(X0jX¡1; X¡2; ¢ ¢ ¢ ; X¡k) to m(X0jX¡1; X¡2; ¢ ¢ ¢). This result was combined with an an extended form of the ergodic theorem stating that if gk ! g as k ! 1 and if gk is L1-dominated

(supk jgkj is in L1), then 1=n Pn¡1 gkT k has the same limit as 1=n Pn¡1 gT k.

k=0 k=0

Combining these facts yields that that

n¡1

n1 ln m(Xn) = n1 X ln m(XkjXk)

k=0

n¡1

= n1 X ln m(X0jX¡k k)T k

k=0

has the same limit as

n¡1

1 X ln m(X0jX¡1; X¡2; ¢ ¢ ¢)T k n

k=0

which, from the usual ergodic theorem, is the expectation

E(ln m(X0jX¡) · E(ln m(X0jX¡1; X¡2; ¢ ¢ ¢)):

As suggested at the end of the preceeding chapter, this should be minus the conditional entropy H(X0jX¡1; X¡2; ¢ ¢ ¢) which in turn should be the entropy

„

rate HX . This approach has three shortcomings: it requires a result from martingale theory which has not been proved here or in the companion volume [50], it requires an extended ergodic theorem which has similarly not been proved here, and it requires a more advanced deflnition of entropy which has not yet been introduced. Another approach is the sandwich proof of Algoet and Cover [7]. They show without using martingale theory or the extended ergodic theo-

= ¡H(X0jX¡1; X¡2; ¢ ¢ ¢):

By showing that these two limits are arbitrarily close as k ! 1, the result is proved. The drawback of this approach for present purposes is that again the

more advanced notion of conditional entropy given the inflnite past is required. Algoet and Cover’s proof that the above two entropies are asymptotically close involves martingale theory, but this can be avoided by using Corollary 5.2.4 as will be seen.

The result can, however, be proved without martingale theory, the extended ergodic theorem, or advanced notions of entropy using the approach of Ornstein and Weiss [117], which is the approach we shall take in this chapter. In a later chapter when the entropy ergodic theorem is generalized to nonflnite alphabets and the convergence of entropy and information densities is proved, the sandwich approach will be used since the appropriate general deflnitions of entropy will have been developed and the necessary side results will have been proved.

Lemma 3.2.1: Given a flnite alphabet source fXng with a stationary ergodic distribution m, we have that

lim ¡ ln m(Xn) = h; m ¡ a:e:;

n!1 n

where h(x) is the invariant function deflned by

„

h(x) = Hm(X):

Proof: Deflne

hn(x) = ¡ ln m(Xn)(x) = ¡ ln m(xn)

Since m((x0; ¢ ¢ ¢ ; xn¡1)) • m((x1; ¢ ¢ ¢ ; xn¡1)), we have that

hn(x) ‚ hn¡1(T x):

Dividing by n and taking the limit inflmum of both sides shows that h(x) ‚ h(T x). Since the n¡1hn are nonnegative and uniformly integrable (Lemma 2.3.6), we can use Fatou’s lemma to deduce that h and hence also hT are integrable with respect to m. Integrating with respect to the stationary measure m yields

Z Z

dm(x)h(x) = dm(x)h(T x)

which can only be true if

h(x) = h(T x); m ¡ a:e:;

that is, if h is an invariant function with m-probability one. If h is invariant almost everywhere, however, it must be a constant with probability one since m is ergodic (Lemma 6.7.1 of [50]). Since it has a flnite integral (bounded by

„

Hm(X)), h must also be flnite. Henceforth we consider h to be a flnite constant.

We now proceed with steps that resemble those of the proof of the ergodic theorem in Section 7.2 of [50]. Fix † > 0. We also choose for later use a – > 0 small enough to have the following properties: If A is the alphabet of X0 and jjAjj is the flnite cardinality of the alphabet, then

The latter property is possible since h2(–) ! 0 as – ! 0.

Deflne the random variable n(x) to be the smallest integer n for which n¡1hn(x) • h + †. By deflnition of the limit inflmum there must be inflnitely many n for which this is true and hence n(x) is everywhere flnite. Deflne the set of \bad" sequences by B = fx : n(x) > N g where N is chosen so large that m(B) < –=2. Still mimicking the proof of the ergodic theorem, we deflne a bounded modiflcation of n(x) by

‰

n~(x) =

n(x) x 62B 1 x 2 B

so that n~(x) • N for all x 2 Bc. We now parse the sequence into variable-length blocks. Iteratively deflne nk(x) by

n0(x) = 0

n1(x) = n~(x)

n2(x) = n1(x) + n~(T n1(x)x) = n1(x) + l1(x)

.

.

.

nk+1(x) = nk(x) + n~(T nk (x)x) = nk(x) + lk(x);

where lk(x) is the length of the kth block:

lk(x) = n~(T nk (x)x):

We have parsed a long sequence xL = (x0; ¢ ¢ ¢ ; xL¡1), where L >> N ,

length lk(x) for k = 0; 1; ¢ ¢ ¢. We refer to this parsing as the block decomposition of a sequence. The kth block, which begins at time nk(x), must either have sample entropy satisfying

or it must consist of only a single symbol. Blocks having length 1 (lk = 1) could have the correct sample entropy, that is,

or they could be bad in the sense that they are the flrst symbol of a sequence with n > N ; that is,

n(T nk (x)x) > N;

or, equivalently,

T nk (x)x 2 B:

Except for these bad symbols, each of the blocks by construction will have a probability which satisfles the above bound.

Deflne for nonnegative integers n and positive integers l the sets

S(n; l) = fx : m(Xnl (x)) ‚ e¡l(h+†)g;

that is, the collection of inflnite sequences for which (3.2.2) and (3.2.3) hold for a block starting at n and having length l. Observe that for such blocks there cannot be more than el(h+†) distinct l-tuples for which the bound holds (lest the probabilities sum to something greater than 1). In symbols this is

The ergodic theorem will imply that there cannot be too many single symbol blocks with n(T nk (x)x) > N because the event has small probability. These facts will be essential to the proof.

Even though we write n~(x) as a function of the entire inflnite sequence, we can determine its value by observing only the preflx xN of x since either there

N -tuples x that are preflxes of bad inflnite sequences, sequences x for which n(x) > N . Thus in particular,

Now recall that we parse sequences of length L >> N and deflne the set GL of \good" L-tuples by

that is, GL is the collection of all L-tuples which have fewer than –(L¡N ) • –L time slots i for which xNi is a preflx of a bad inflnite sequence. From (3.12) and

the ergodic theorem for stationary ergodic sources we know that m-a.e. we get an x for which

From the deflnition of a limit, this means that with probability 1 we get an x for which there is an L0 = L0(x) such that

This follows simply because if the limit is less than –=2, there must be an L0 so large that for larger L the time average is at least no greater than 2–=2 = –. We can restate (3.14) as follows: with probability 1 we get an x for which xL 2 GL for all but a flnite number of L. Stating this in negative fashion, we have one of the key properties required by the proof: If xL 2 GL for all but a flnite number of L, then xL cannot be in the complement GcL inflnitely often, that is,

We now change tack to develop another key result for the proof. For each L we bounded above the cardinality jjGLjj of the set of good L-tuples. By construction there are no more than –L bad symbols in an L-tuple in GL and these can occur in any of at most

µ¶

k•–L

places, where we have used Lemma 2.3.5. Eq. (3.16) provides an upper bound on the number of ways that a sequence in GL can be parsed by the given rules. The bad symbols and the flnal N symbols in the L-tuple can take on any of the jjAjj difierent values in the alphabet. Eq. (3.11) bounds the number of flnite length sequences that can occur in each of the remaining blocks and hence for any given block decomposition, the number of ways that the remaining blocks blocks can be fllled is bounded above by

regardless of the details of the parsing. Combining these bounds we have that

jjGLjj • eh2(–)L £ jjAjj–L £ jjAjjN £ eL(h+†) = eh2(–)L+(–L+N) ln jjAjj+L(h+†)

or

jjGLjj • eL(h+†+h2(–)+(–+ NL ) ln jjAjj):

Since – satisfles (3.7){(3.8), we can choose L1 large enough so that N ln jjAjj=L1 •

† and thereby obtain

This bound provides the second key result in the proof of the lemma. We now combine (3.18) and (3.15) to complete the proof.

Let BL denote a collection of L-tuples that are bad in the sense of having too large a sample entropy or, equivalently, too small a probability; that is if

xL 2 BL, then

m(xL) • e¡L(h+5†)

• jjGLjje¡L(h+5†) • e¡†L:

Recall now that the above bound is true for a flxed † > 0 and for all L ‚ L1. Thus

lim sup hL(x) • h + 5†:

L!1

Since † is arbitrary we have proved that the limit supremum of the sample entropy ¡n¡1 ln m(Xn) is less than or equal to the limit inflmum and therefore that the limit exists and hence with m-probability 1

Since the terms on the left in (3.19) are uniformly integrable from Lemma 2.3.6, we can integrate to the limit and apply Lemma 2.4.1 to flnd that

which completes the proof of the lemma and hence also proves Theorem 3.1.1 for the special case of stationary ergodic measures. 2

3.3Stationary Nonergodic Sources

Next suppose that a source is stationary with ergodic decomposition fm‚; ‚ 2 ⁄g and ergodic component function ˆ as in Theorem 1.8.3. The source will produce with probability one under m an ergodic component m‚ and Lemma 3.2.2 will hold for this ergodic component. In other words, we should have that

This argument is made rigorous in the following lemma.

Lemma 3.3.1: Suppose that fXng is a stationary not necessarily ergodic source with ergodic component function ˆ. Then

From the ergodic decomposition (e.g., Theorem 1.8.3 or [50], Theorem 8.5.1) and (1.26)

Z

m(G) = dPˆ(‚)m‚(G);

where

\

m‚(G) = m(Gjˆ = ‚) = m(G fx : ˆ(x) = ‚gjˆ = ‚) = m(G‚jˆ = ‚) = m‚(G‚)

which is 1 for all ‚ from the stationary ergodic result. Thus

Z

m(G) = dPˆ(‚)m‚(G‚) = 1:

It is straightforward to verify that all of the sets considered are in fact measurable. 2

Unfortunately it is not the sample entropy using the distribution of the ergodic component that is of interest, rather it is the original sample entropy

for which we wish to prove convergence. The following lemma shows that the two sample entropies converge to the same limit and hence Lemma 3.3.1 will also provide the limit of the sample entropy with respect to the stationary measure.

Lemma 3.3.2: Given a stationary source fXng, let fm‚; ‚ 2 ⁄g denote the ergodic decomposition and ˆ the ergodic component function of Theorem

1.8.3. Then

lim 1 ln mˆ(Xn) = 0; m ¡ a:e:

n!1 n

Proof: First observe that if m(an) is 0, then from the ergodic decomposition with probability 1 mˆ(an) will also be 0. One part is easy. For any † > 0 we have from the Markov inequality that

Z

=dPˆ(‚)m(A(n‚)) • 1;

We now turn to the converse inequality. For any positive integer k, we can construct a stationary k-step Markov approximation to m as in Section 2.6, that is, construct a process m(k) with the conditional probabilities

m(k)(Xn 2 F jXn) = m(k)(Xn 2 F jXnk¡k)

= m(Xn 2 F jXnk¡k)

and the same kth order distributions m(k)(Xk 2 F ) = m(Xk 2 F ). Consider the probability

• Em( m(k)(Xn) )e¡n†:

The expectation is evaluated as

X m(k)(xn) m(xn) = 1 m(xn)

xn

and hence we again have using Borel-Cantelli that

lim sup 1 ln m(k)(Xn) • 0:

n!1 n

We can apply the usual ergodic theorem to conclude that with probability 1 under m

This bound holds for any integer k and hence it must also be true that m-a.e. the following holds: