Теория информации / Gray R.M. Entropy and information theory. 1990., 284p

k=0

We now apply Fatou’s lemma, a change of variables, and Lemma 11.2.1 to

obtain

–(R; „T ¡i) • ‰(CK (i); „T ¡i)

Thus if J and hence K are chosen large enough to ensure that N=K • †=2, then

–(R; „T ¡i) • –(R; „);

which proves that –(R; „T ¡i) • –(R; „). The reverse implication is found in a similar manner: Let CN be a codebook for „T ¡i and construct a codebook CK (N ¡i) for use on „. By arguments nearly identical to those above the reverse inequality is found and the proof completed. 2

Corollary 11.2.4: Let „ be an AMS source with a reference letter. Fix N and let „ and „N denote the stationary and N -stationary means. Then for

R > 0

–(R; „) = –(R; „N T ¡i); i = 0; 1; ¢ ¢ ¢ ; N ¡ 1:

Proof: It follows from the previous lemma that the –(R; „N T ¡i) are all equal and hence it follows from Lemma 11.2.2, Theorem 7.3.1 of [50], and Corollary 7.3.1 of [50] that

N¡1

–(R; „) ‚ N1 X –(R; „N T ¡i) = –(R; „N ):

i=0

To prove the reverse inequality, take „ = „N in the previous lemma and

N¡1

•N1 X ‰(CK (i); „N T ¡i) • KN ‰⁄ + ‰(CN ; „N )

i=0

and hence as before

1

–(R + JN log N; „) • –(R; „N ):

From Corollary 11.2.1 –(R; „) is continuous in R for R > 0 since „ is stationary. Hence taking J large enough yields –(R; „) • –(R; „N ). This completes the proof since from the lemma –(R; „N T ¡i) = –(R; „N ). 2

We are now prepared to demonstrate the fundamental fact that the block source coding OPTA function for an AMS source with an additive fldelity criterion is the same as that of the stationary mean process. This will allow us to assume stationarity when proving the actual coding theorems.

Theorem 11.2.2: If „ is an AMS source and f‰ng an additive fldelity criterion with a reference letter, then for R > 0

–(R; „) = –(R; „):

Proof: We have from Corollaries 11.2.1 and 11.2.4 that

–(R; „) • –(R; „N ) • –N (R; „N ) = –N (R; „):

Taking the inflmum over N yields

–(R; „) • –(R; „):

Conversely, flx † > 0 let CN be a block length N codebook for which ‰(CN ; „)

• –(R; „) + †. From Lemma 11.2.1, Corollary 11.2.1, and Lemma 11.2.4

N¡1

‚N1 X –(R; „T ¡i) = –(R; „);

i=0

which completes the proof since † is arbitrary. 2

Since the OPTA functions are the same for an AMS process and its stationary mean, this immediately yields the following corollary from Corollary 11.2.2:

Corollary 11.2.5: If „ is AMS, then –(R; „) is a convex function of R and hence a continuous function of R for R > 0.

11.3Block Coding Stationary Sources

We showed in the previous section that when proving block source coding theorems for AMS sources, we could conflne interest to stationary sources. In this section we show that in an important special case we can further conflne interest to only those stationary sources that are ergodic by applying the ergodic decomposition. This will permit us to assume that sources are stationary and ergodic in the next section when the basic Shannon source coding theorem is proved and then extend the result to AMS sources which may not be ergodic.

As previously we assume that we have a stationary source fXng with distribution „ and we assume that f‰ng is an additive distortion measure and there exists a reference letter. For this section we now assume in addition that the alphabet A is itself a Polish space and that ‰1(r; y) is a continuous function of

2 ^

r for every y A. If the underlying alphabet has a metric structure, then it is reasonable to assume that forcing input symbols to be very close in the underlying alphabet should force the distortion between either symbol and a flxed output to be close also. The following theorem is the ergodic decomposition of the block source coding OPTA function.

Theorem 11.3.1: Suppose that „ is the distribution of a stationary source and that f‰ng is an additive fldelity criterion with a reference letter. Assume also that ‰1(¢; y) is a continuous function for all y. Let f„xg denote the ergodic decomposition of „. Then

Z

–(R; „) = d„(x)–(R; „x);

that is, –(R; „) is the average of the OPTA of its ergodic components.

Proof: Analogous to the ergodic decomposition of entropy rate of Theorem 2.4.1, we need to show that –(R; „) satisfles the conditions of Theorem 8.9.1 of [50]. We have already seen (Corollary 11.2.3) that it is an a–ne function. We next see that it is upper semicontinuous. Since the alphabet is Polish, choose a distance dG on the space of stationary processes having this alphabet with the property that G is constructed as in Section 8.2 of [50]. Pick an N large enough and a length N codebook C so that

†

–(R; „) ‚ –N (R; „) ¡ 2 ‚ ‰N (C; „) ¡ †:

‰N (xN ; y) is by assumption a continuous function of xN and hence so is ‰N (xN ; C) = miny2C ‰(xN ; y). Since it is also nonnegative, we have from Lemma 8.2.4 of [50] that if „n ! „ then

lim sup E„n ‰N (XN ; C) • E„‰N (XN ; C):

n!1

The left hand side above is bounded below by

lim sup –N (R; „n) ‚ lim sup –(R; „n):

Thus since † is arbitrary,

lim sup –(R; „n) • –(R; „)

n!1

and hence –(R; „) upper semicontinuous in „ and hence also measurable. Since the process has a reference letter, –(R; „x) is integrable since

–(R; „X ) • –N (R; „x) • E„x ‰1(X0; a⁄)

which is integrable if ‰1(x0; a⁄) is from the ergodic decomposition theorem. Thus Theorem 8.9.1 of [50] yields the desired result. 2

The theorem was flrst proved by Kiefier [76] for bounded continuous additive distortion measures. The above extension removes the requirement that ‰1 be bounded.

11.4Block Coding AMS Ergodic Sources

We have seen that the block source coding OPTA of an AMS source is given by that of its stationary mean. Hence we will be able to concentrate on stationary sources when proving the coding theorem.

Theorem 11.4.1: Let „ be an AMS ergodic source with a standard alphabet and f‰ng an additive distortion measure with a reference letter. Then

–(R; „) = D(R; „);

where „ is the stationary mean of „.

Proof: From Theorem 11.2.2 –(R; „) = –(R; „) and hence we will be done if we can prove that

–(R; „) = D(R; „):

This will follow if we can show that –(R; „) = D(R; „) for any stationary ergodic source with a reference letter. Henceforth we assume that „ is stationary and ergodic.

We flrst prove the negative or converse half of the theorem. First suppose that we have a codebook C such that

Taking the limits as N ! 1 proves the easy half of the theorem:

–(R; „) ‚ D(R; „):

(Recall that both OPTA and distortion rate functions are given by limits if the source is stationary.)

The fundamental idea of Shannon’s positive source coding theorem is this: for a flxed block size N , choose a code at random according to a distribution implied by the distortion-rate function. That is, perform 2NR independent random selections of blocks of length N to form a codebook. This codebook is then used to encode the source using a minimum distortion mapping as above. We compute the average distortion over this double-random experiment (random codebook selection followed by use of the chosen code to encode the random source). We will flnd that if the code generation distribution is properly chosen, then this average will be no greater than D(R; „) + †. If the average over all randomly selected codes is no greater than D(R; „) + †, however, than there must be at least one code such that the average distortion over the source distribution for that one code is no greater than D(R; „) + †. This means that there exists at least one code with performance not much larger than D(R; „). Unfortunately the proof only demonstrates the existence of such codes, it does not show how to construct them.

To flnd the distribution for generating the random codes we use the ergodic process deflnition of the distortion-rate function. From Theorem 10.6.1 (or Lemma 10.6.3) we can select a stationary and ergodic pair process with distribution p which has the source distribution „ as one coordinate and which has

(and hence information densities converge in L1 from Theorem 6.3.1). Denote the implied vector distributions for (XN ; Y N ), XN , and Y N by pN , „N , and ·N , respectively.

For any N we can generate a codebook C at random according to ·N as described above. To be precise, consider the random codebook as a large random vector C = (W0; W1; ¢ ¢ ¢ ; WM ), where M = beN(R+†)c (where natural logarithms are used in the deflnition of R), where W0 is the flxed reference vector a⁄N and where the remaining Wn are independent, and where the marginal distributions for the Wn are given by ·N . Thus the distribution for the randomly selected code can be expressed as

M

PC = £ ·N :

i=1

This codebook is then used with the optimal encoder and we denote the resulting average distortion (over codebook generation and the source) by

Z

¢N = E‰(C; „) = dPC(W)‰(W; „) (11.8)

224 CHAPTER 11. SOURCE CODING THEOREMS

and where

‰N (xN ; C) = min ‰N (xN ; y):

y2C

Choose – > 0 and break up the integral over x into two pieces: one over a set GN = fx : N ¡1‰N (xN ; a⁄N ) • ‰⁄ + –g and the other over the complement of this set. Then

where we have used the fact that ‰N (xN ; mW) • ‰N (xN ; a⁄N ). Fubini’s theorem implies that because

Z

d„N (xN )‰N (xN ; a⁄N ) < 1

and

‰N (xN ; W) • ‰N (xN ; a⁄N );

the limits of integration in the second integral of (11.9) can be interchanged to obtain the bound

Z Z

= 1 d„N (xN )[ dPC(W)‰N (xN ; W)

N GN C:‰N (xN ;C)•N(D+–)

+ 1 Z dPC(W)‰N (xN ; W)]

N W:‰N (xN ;W)>N(D+–)

Z Z

• d„N (xN )[D + – + 1 (‰⁄ + –) dpC(W)]

GN N W:‰N (xN ;W)>N(D+–)

where we have used the fact that for x 2 G the maximum distortion is given by ‰⁄ + –. Deflne the probability

Z

P (N ¡1‰N (xN ; C) > D + –jxN ) = dpC(W)

W:‰N (xN ;W)>N(D+–)

The remainder of the proof is devoted to proving that the two integrals above go to 0 as N ! 1 and hence

We shall see that this integral goes to zero as an easy application of the ergodic theorem. The integrand is dominated by N ¡1‰N (xN ; a⁄N ) which is uniformly integrable (Lemma 4.7.2 of [50]) and hence the integrand is itself uniformly integrable (Lemma 4.4.4 of [50]). Thus we can invoke the extended Fatou lemma to conclude that

We have, however, that lim supN!1 1GcN (xN ) is 0 unless xN 2 GcN i.o. But this set has measure 0 since with „N probability 1, an x is produced so that

exists and hence with probability one one gets an x which can yield

N ¡1‰N (xN ; a⁄N ) > ‰⁄ + –

at most for a flnite number of N . Thus the above integral of the product of a function that is 0 a.e. with a dominated function must itself be 0 and hence

Recall that P (‰N (xN ; C) > D+–jxN ) is the probability that for a flxed input block xN , a randomly selected code will result in a minimum distortion codeword larger than D + –. This is the probability that none of the M words (excluding the reference code word) selected independently at random according to to the distribution ·N lie within D + – of the flxed input word xN . This probability is bounded above by

Now mutual information comes into the picture. The above probability can be bounded below by adding a condition:

the Radon-Nikodym derivative of pN with respect to the product measure „N £ ·N . Thus we require both the distortion and the sample information be less than slightly more than their limiting value. Thus we have in the region of integration that

N1 iN (xN ; yN ) = N1 ln fN (xN ; yN ) • R + –

and hence

Z

·N (‰N (xN ; Y N ) • D + –) ‚ d·N (yN )

yN :‰N (xN ;yN )•D+–;fN (xN ;yN )•eN (R+–)

Z

‚ e¡N(R+–) d·N (yN )fN (xN ; yN )

yN :‰N (xN ;yN )•D+–;fN (xN ;yN )•eN (R+–)

which yields the bound

P ( N1 ‰N (xN ; C) > D + –jxN ) • [1 ¡ ·N ( N1 ‰N (xN ; Y N ) • D + –)]M

Z

• [1 ¡ e¡N(R+–) d·N (yN )fN (xN ; yN )]M ;

yN : N1 ‰N (xN ;yN )•D+–; N1 iN (xN ;yN )•R+–

= 1 + e¡Me¡N (R+–)

Since M is bounded below by eN(R+†) ¡ 1, the exponential term is bounded

above by

e[¡e(N (R+†)e¡N (R+–)+e¡N (R+–)] = e[¡eN (†¡–)+e¡N (R+–)]:

If † > –, this term goes to 0 as N ! 1.

The probability term in (11.14) goes to 1 from the mean ergodic theorem applied to ‰1 and the mean ergodic theorem for information density since mean convergence (or the almost everywhere convergence proved elsewhere) implies convergence in probability. This implies that

lim sup bN = 0

n!1

which with (11.13) gives (11.12). Choosing an N so large that ¢N • –, we have proved that there exists a block code C with average distortion less than D(R; „) + – and rate less than R + † and hence

Since † and – can be chosen as small as desired and since D(R; „) is a continuous function of R (Lemma 10.6.1), the theorem is proved. 2

The source coding theorem is originally due to Shannon [129] [130], who proved it for discrete i.i.d. sources. It was extended to stationary and ergodic discrete alphabet sources and Gaussian sources by Gallager [43] and to stationary and ergodic sources with abstract alphabets by Berger [10] [11], but an error in the information density convergence result of Perez [123] (see Kiefier [74]) left a gap in the proof, which was subsequently repaired by Dunham [35]. The result was extended to nonergodic stationary sources and metric distortion measures and Polish alphabets by Gray and Davisson [53] and to AMS ergodic processes by Gray and Saadat [61]. The method used here of using a stationary and ergodic measure to construct the block codes and thereby avoid the block ergodic decomposition of Nedoma [106] used by Gallager [43] and Berger [11] was suggested by Pursley and Davisson [29] and developed in detail by Gray and Saadat [61].

11.5Subadditive Fidelity Criteria

In this section we generalize the block source coding theorem for stationary sources to subadditive fldelity criteria. Several of the interim results derived previously are no longer appropriate, but we describe those that are still valid in the course of the proof of the main result. Most importantly, we now consider only stationary and not AMS sources. The result can be extended to AMS sources in the two-sided case, but it is not known for the one-sided case. Source coding theorems for subadditive fldelity criteria were flrst developed by Mackenthun and Pursley [96].

Theorem 11.5.1: Let „ denote a stationary and ergodic distribution of a source fXng and let f‰ng be a subadditive fldelity criterion with a reference

⁄ 2 ^

letter, i.e., there is an a A such that

E‰1(X0; a⁄) = ‰⁄ < 1:

Then the OPTA for the class of block codes of rate less than R is given by the Shannon distortion-rate function D(R; „).

Proof: Suppose that we have a block code of length N , e.g., a block encoder

the induced input/output distribution is then N -stationary and the performance resulting from using this code on a source „ is