Теория информации / Gray R.M. Entropy and information theory. 1990., 284p

exactly as in the proof of Theorem 6.4.2 and the same argument then shows that the limit is I¡(X; Y ), which completes the proof. 2

One result developed in the proofs of Theorems 6.4.2 and 6.4.3 will be important later in its own right and hence we isolate it as a corollary. The result is just (6.7), which remains valid under the more general conditions of Theorem 6.4.3, and the fact that the Cesµaro mean of converging terms has the same limit.

Corollary 6.4.1: If a process fXng has the flnite-gap information property

I(XK ; X¡jXK ) < 1

for some K, then

lim I(Xn; X¡jXn) = 0

n!1

and

lim 1 I(Xn; X¡) = 0:

n!1 n

The corollary can be interpreted as saying that if a process has the the flnite gap information property, then the mutual information between a single sample and the inflnite past conditioned on the intervening samples goes to zero as the number of intervening samples goes to inflnity. This can be interpreted as a form of asymptotic independence property of the process.

Corollary 6.4.2: If a one-sided stationary source fXng is such that for some K, I(Xn; Xn¡K jXnK¡K ) is bounded uniformly in n, then it has the flnite-gap property and hence

„ ⁄

I(X; Y ) = I (X; Y ):

Proof: Simply imbed the one-sided source into a two-sided stationary source with the same probabilities on all flnite-dimensional events. For that source

Thus if the terms are bounded, the conditions of Theorem 6.4.2 are met for the two-sided source. The one-sided equality then follows. 2

The above results have an information theoretic implication for the ergodic decomposition, which is described in the next theorem.

Theorem 6.4.4: Suppose that fXng is a stationary process with the flnitegap property (6.13). Let ˆ be the ergodic component function of Theorem 1.8.3

(This will be the case, for example, if the flnite-gap information property holds for 0 gap, that is, I(X0; X¡) < 1 since ˆ can be determined from X¡ and information is decreased by taking a function.) Then

lim 1 I(Xn; ˆ) = 0:

n!1 n

Comment: For discrete alphabet processes this theorem is just the ergodic decomposition of entropy rate in disguise (Theorem 2.4.1). It also follows for flnite alphabet processes from Lemma 3.3.1. We shall later prove a corresponding almost everywhere convergence result for the corresponding densities. All of these results have the interpretation that the per-symbol mutual information between the outputs of the process and the ergodic component decreases with time because the ergodic component in efiect can be inferred from the process output in the limit of an inflnite observation sequence. The flniteness condition on some I(Xn; ˆ) is necessary for the nonzero flnite-gap case to avoid cases such as where Xn = ˆ for all n and hence

I(Xn; ˆ) = I(ˆ; ˆ) = H(ˆ) = 1;

in which case the theorem does not hold.

Proof:

Deflne ˆn = ˆ for all n. Since ˆ is invariant, fXn; ˆng is a stationary

„ ⁄ process. Since Xn satisfles the given conditions, however, I(X; ˆ) = I (X; ˆ).

„ ⁄

But for any scalar quantizer q, I(q(X); ˆ) is 0 from Lemma 3.3.1. I (X; ˆ) is

deflned by

fXn (Xn)(x) · fXn (Xn(x)) = fXn (xn)

for n = 1; 2; ¢ ¢ ¢. Similarly the entropy densities can be deflned on the common

space (›; B) by

hXn (Xn) = ln fXn (Xn):

The reader is warned of the potentially confusing dual use of Xn in this notation: the subscript is the name of the random variable Xn and the argument is the random variable Xn itself. To simplify notation somewhat, we will often abbreviate the previous (unconditional) densities to

fn = fXn (Xn); hn = hXn (Xn):

For n = 1; 2; ¢ ¢ ¢ deflne the relative entropy by

provided fXn 6= 0 and fXnjXn = 1 otherwise. As for unconditional densities we change the notation when we wish to emphasize that the densities can all be deflned on a common underlying sequence space. For example, we follow the notation for ordinary conditional probability density functions and deflne the

on (›; B). These densities will not have a simple abbreviation as do the unconditional densities.

All of the above deflnitions are immediate applications of deflnitions of Chapter 5 to the random variables Xn and Xn. The difierence is that these are now deflned for all samples of a random process, that is, for all n = 1; 2; ¢ ¢ ¢. The focus of this chapter is the interrelations of these entropy measures and on some of their limiting properties for large n.

For convenience deflne

Dn = Hpjjm(XnjXn); n = 1; 2; ¢ ¢ ¢ ;

and D0 = Hpjjm(X0). From Theorem 5.3.1 this quantity is nonnegative and

Dn + D(PXn jjMXn ) = D(PXn+1 jjMXn+1 ):

If D(PXn jjMXn ) < 1, then also

Dn = D(PXn+1 jjMXn+1 ) ¡ D(PXn jjMXn ):

We can write Dn as a single divergence if we deflne as in Theorem 5.3.1 the distribution SXn+1 by

Z

SXn+1 (F £ G) = MXnjXn (F jxn) dPXn (xn); F 2 BA; G 2 BAn : (7.3)

F

Recall that SXn+1 combines the distribution PXn on Xn with the conditional distribution MXnjXn giving the conditional probability under M for Xn given Xn. We shall abbreviate this construction by

Note that SXn+1 is not in general a consistent family of measures in the sense of the Kolmogorov extension theorem since its form changes with n, the flrst n samples being chosen according to p and the flnal sample being chosen using the conditional distribution induced by m given the flrst n samples. Thus, in particular, we cannot infer that there is a process distribution s which has SXn ; ; n = 1; 2; ¢ ¢ ¢ as its vector distributions.

We immediately have a chain rule for densities

i=0

and a corresponding chain rule for conditional relative entropies similar to that for ordinary entropies:

l=k

Proof: If n = k + 1, then the results follow from Corollary 5.3.3 and Lemma 5.5.4 with X = Xn, Z = Xk, and Y = Xk. Now proceed by induction and assume that the results hold for n. Consider the distribution QX(n+1) specifled by QXn = PXn and QXnjXn = PXnjXnk¡k . In other words,

QXn+1 = PXnjXnk¡k PXn

Application of Corollary 5.3.1 with Z = Xn¡k, Y = Xnk¡k, and X = Xn implies that MXn+1 >> QXn+1 >> PXn+1 and that

Eq. (7.12) then follows by taking expectations. Eq. (7.13) follows from (7.11)

and

n¡1

Y

fXn = fXk fXl jXl ;

l=k

whence (7.14) follows by taking expectations. If m = p(k), then the claims follow from (5.27){(5.28). 2

Corollary 7.3.1: Given a stationary source p, suppose that for some K there exists a K-step Markov source m with distributions MXn >> PXn , n = 1; 2; ¢ ¢ ¢. Then for all k ‚ K (7.8){(7.10) hold.

Proof: If m is a K-step Markov source with the property MXn >> PXn , n = 1; 2; ¢ ¢ ¢, then it is also a k-step Markov source with this property for all k ‚ K. The corollary then follows from the theorem. 2

Comment: The corollary implies that if any K-step Markov source dominates p on its flnite dimensional distributions, then for all k ‚ K the k-step Markov approximations p(k) also dominate p on its flnite dimensional distributions.

The following variational corollary follows from Theorem 7.3.1.

Corollary 7.3.2: For a flxed k let Let Mk denote the set of all k-step Markov distributions. Then infM2Mk D(PXn jjM ) is attained by P (k), and

Since the divergence can be thought of as a distance between probability distributions, the corollary justifles considering the k-step Markov process with the same kth order distributions as the k-step Markov approximation or model for the original process: It is the minimum divergence distribution meeting the k-step Markov requirement.

7.4Stationary Processes

Several of the previous results simplify when the processes m and p are both stationary. We can consider the processes to be two-sided since given a stationary one-sided process, there is always a stationary two-sided process with the same probabilities on all positive time events. When both processes are stationary, the densities fXmn and fXn satisfy

and have the same expectation for any integer m. Similarly the conditional densities fXnjXn , fXk jXkn¡n , and fX0jX¡1;X¡2;¢¢¢;X¡n satisfy

i=0

Using the construction of Theorem 5.3.1 we have also that

Di = Hpjjm(XijXi) = Hpjjm(X0jX¡1; ¢ ¢ ¢ ; X¡i)

Z

SX0;X¡1;¢¢¢;X¡i (F £ G) = MX0jX¡1;¢¢¢;X¡i (F jxi) dPX¡1;¢¢¢;X¡i (xi); F

F 2 BA; G 2 BAi :

As before the SXn distributions are not in general consistent. For example, they can yield difiering marginal distributions SX0 . As we saw in the flnite case, general conclusions about the behavior of the limiting conditional relative entropies cannot be drawn for arbitrary reference measures. If, however, we assume as in the flnite case that the reference measures are Markov, then we can proceed.

Suppose now that under m the process is a k-step Markov process. Then for

any n ‚ k (X¡n; ¢ ¢ ¢ ; X¡k¡2; X¡k¡1) ! X¡k k ! X0 is a Markov chain under m and Lemma 5.5.4 implies that

Hpjjm(X0jX¡1; ¢ ¢ ¢ ; X¡n) = Hpjjm(XkjXk) + Ip(Xk; (X¡1; ¢ ¢ ¢ ; X¡n)jXk) (7.23)

and hence from (7.21)

We also have, however, that X¡ ! Xk ! Xk is a Markov chain under m and hence a second application of Lemma 5.5.4 implies that

Putting these facts together and using (7.2) yields the following lemma.

Lemma 7.4.1: Let fXng be a two-sided process with a standard alphabet and let p and m be stationary process distributions such that MXn >> PXn all n and m is kth order Markov. Then the relative entropy rate exists and

Corollary 7.4.1: Given the assumptions of Lemma 7.4.1,

Hpjjm(XN jX¡) = N Hpjjm(X0jX¡):