Теория информации / Gray R.M. Entropy and information theory. 1990., 284p

Using the deflnition of the measure S and iterated expectation we have that

ZZ

dp‡n • dMXknjXk¡ dPXk¡ fXknjXk (Xn+k):

Z

=dMXknjXk dPXk¡ fXknjXk (Xn+k):

Since the integrand is now measurable with respect to ¾(Xn+k), this reduces

l=k

Invoking the ergodic theorem for the rightmost terms and the fact that the middle term converges to 0 almost everywhere since ln fXk is flnite almost everywhere implies that

Combining this with (8.3) completes the sandwich and proves the theorem.

2

8.3Stationary Nonergodic Sources

Next suppose that the source p is stationary with ergodic decomposition fp‚; ‚ 2 ⁄g and ergodic component function ˆ as in Theorem 1.8.3. We flrst require some technical details to ensure that the various Radon-Nikodym derivatives are well deflned and that the needed chain rules for densities hold.

Lemma 8.3.1: Given a stationary source fXng, let fp‚; ‚ 2 ⁄g denote the ergodic decomposition and ˆ the ergodic component function of Theorem 1.8.3. Let Pˆ denote the induced distribution of ˆ. Let PXn and PX‚ n denote the induced marginal distributions of p and p‚. Assume that fXng has the flnite-gap information property of (6.13); that is, there exists a K such that

This will be the case, for example, if (8.9) holds for K = 0. Let m be a K- step Markov process such that MXn >> PXn for all n. (Observe that such a process exists since from (8.9) the Kth order Markov approximation p(K) su–ces.) Deflne MXn;ˆ = MXn £ Pˆ. Then

and hence PXn £ Pˆ >> PXn;ˆ (since otherwise I(Xn; ˆ) would be inflnite for some n and hence inflnite for all larger n since it is increasing with n). This proves the right-most absolute continuity relation of (8.11). This in turn implies that MXn £ Pˆ >> PXn;ˆ. The lemma then follows from Theorem 5.3.1 with X = Xn, Y = ˆ and the chain rule for Radon-Nikodym derivatives. 2

We know that the source will produce with probability one an ergodic component p‚ and hence Theorem 8.2.1 will hold for this ergodic component. In other words, we have for all ‚ that

Making this step precise generalizes Lemma 3.3.1.

Lemma 8.3.2: Suppose that fXng is a stationary not necessarily ergodic source with ergodic component function ˆ. Then (8.15) holds.

Proof: The proof parallels that for Lemma 3.3.1. Observe that if we have two random variables U; V (U = X0; X1; ¢ ¢ ¢ and Y = ˆ above) and a sequence

¡1 nj „

of functions gn(U; V ) (n fXnjˆ(X ˆ)) and a function g(V ) (Hpˆ (X)) with the property

lim gn(U; v) = g(v); PUjV =v ¡ a:e:;

n!1

then also

lim gn(U; V ) = g(V ); PUV ¡ a:e:

n!1

since deflning the (measurable) set G = fu; v : limn!1 gn(u; v) = g(v)g and its section Gv = fu : (u; v) 2 Gg, then from (1.26)

Z

PUV (G) = PUjV (Gvjv)dPV (v) = 1

if PUjV (Gvjv) = 1 with probability 1. 2

It is not, however, the relative entropy density using the distribution of the ergodic component that we wish to show converges. It is the original sample density fXn . The following lemma shows that the two sample entropies converge to the same thing. The lemma generalizes Lemma 3.3.1 and is proved by a

sandwich argument analogous to Theorem 8.2.1. The result can be viewed as an almost everywhere version of (8.14).

Theorem 8.3.1: Given a stationary source fXng, let fp‚; ‚ 2 ⁄g denote the ergodic decomposition and ˆ the ergodic component function of Theorem 1.8.3. Assume that the flnite-gap information property (8.9) is satisfled and that (8.10) holds for some n. Then

Proof: From Theorem 5.4.1 we have immediately that

which provides half of the sandwich proof.

To develop the other half of the sandwich, for each k ‚ K let p(k) denote the k-step Markov approximation of p. Exactly as in the proof of Theorem 8.2.1, it follows that (8.1) holds. Now, however, the Markov approximation relative entropy density converges instead as

From Lemma 7.4.1, the right hand side is just Ipˆ (Xk; X¡jXk) which from

„

Corollary 7.4.2 is just Hpjjp(k) (X). Since the bound holds for all k, we have that

Using the ergodic decompostion of relative entropy rate (Corollary 7.5.1) that and the fact that Markov approximations are asymptotically accurate (Corollary 7.4.3) we have further that

ZZ

„

dPˆ‡ = dPˆ inf Hpˆ jjp(k) (X)

k

which with (8.16) completes the sandwich proof. 2

Simply restating the theorem yields and using (8.15) the ergodic theorem for relative entropy densities in the general stationary case.

Corollary 8.3.1: Given the assumptions of Theorem 8.3.1,

The corollary states that the sample relative entropy density of a process satisfying (8.9) converges to the conditional relative entropy rate with respect to the underlying ergodic component. This is a slight extension and elaboration

essary for the flnite-gap information property of (8.9). In particular, the flnite gap information property implies that

„ ¡j k 1

Hpjjp(k) (X) = Ip(Xk; X X ) < ;

Algoet and Cover [7] do not characterize the limiting density as the entropy rate of the ergodic component, instead they efiectively show that the limit is Epˆ (ln fX0jX¡ (X0jX¡)). This, however, is equivalent since it follows from the ergodic decomposition (see speciflcally Lemma 8.6.2 [50]) that fX0jX¡ = fX0jX¡;ˆ with probability one since the ergodic component ˆ can be determined from the inflnite past X¡.

8.4 AMS Sources

The following lemma is a generalization of Lemma 3.4.1. The result is due to Barron [9], who proved it using martingale inequalities and convergence results.

Lemma 8.4.1: Let fXng be an AMS source with the property that for

This implies that

PXk £(Xk +l;¢¢¢)jXkl >> PX0;X1;::::

156 CHAPTER 8. ERGODIC THEOREMS FOR DENSITIES

Again applying Markov’s inequality and the Borel-Cantelli lemma as previously we have that

and hence also that

p(lim inf 1 fXn (Xn) ‚ h) ‚ †:

n!1 n

Since † can be made arbitrarily small, this proves that p-a.e. lim inf n¡1hn ‚ h, which completes the proof of the lemma. 2

We can now extend the ergodic theorem for relative entropy densities to the general AMS case.

Corollary 8.4.1: Given the assumptions of Theorem 8.4.1,

where p„ˆ is the ergodic component of the stationary mean p„ of p.

Proof: The proof follows immediately from Theorem 8.4.1 and Corollary 8.3.1, the ergodic theorem for the relative entropy density for the stationary mean. 2

8.5Ergodic Theorems for Information Densities.

As an application of the general theorem we prove an ergodic theorem for mutual information densities for stationary and ergodic sources. The result can be extended to AMS sources in the same manner that the results of Section 8.3 were extended to those of Section 8.4. As the stationary and ergodic result su–ces for the coding theorems and the AMS conditions are messy, only the stationary case is considered here. The result is due to Barron [9].

Theorem 8.5.1: Let fXn; Yng be a stationary ergodic pair random process with standard alphabet. Let PXnY n , PXn , and PY n denote the induced distributions and assume that for all n PXn £ PY n >> PXnY n and hence the information densities

in(Xn; Y n) = dPXnY n d(PXn £ PY n )

are well deflned. Assume in addition that both the fXng and fYng processes have the flnite-gap information property of (8.9) and hence by the comment

following Corollary 7.3.1 there is a K such that both processes satisfy the K-

gap property

I(XK ; X¡jXK ) < 1; I(YK ; Y ¡jY K ) < 1:

the Kth order Markov approximations of fXng and fYng, respectively. The flnite-gap approximation implies as in Section 8.3 that the densities

Deflne the measures MZn by MXn £ MY n . Then this is a K-step Markov source and since

MXn £ MY n >> PXn £ PY n >> PXn;Y n = PZn ;

the density

fZn = dMZn

is well deflned and from Theorem 8.2.1 has a limit

lim 1 ln fZn (Zn) = Hpjjm(Z0jZ¡):

n!1 n

If the density in(Xn; Y n) is inflnite for any n, then it is inflnite for all larger n and convergence is trivially to the inflnite information rate. If it is flnite, the

The limit is not indeterminate ( of the form 1¡1) because the two subtracted terms are flnite. Since convergence is to a constant, the constant must also be

„

the limit of the expected values of n¡1in(Xn; Y n), that is, I(X; Y ). 2