Теория информации / Gray R.M. Entropy and information theory. 1990., 284p

in the source codebook is assigned a channel codeword as index. The source is encoded by selecting the minimum distortion word in the codebook and then inserting the resulting channel codeword into the channel. The decoder then uses its decoding sets to decide which channel codeword was sent and then puts out the corresponding reproduction vector. Since the indices of the source code words are accurately decoded by the receiver with high probability, the reproduction vector should yield performance near that of –((K=N )(C ¡†); „). Since

† is arbitrary and –(R; „) is a continuous function of R, this implies that the OPTA for block coding „ for ” is given by –((K=N )C; „), that is, by the OPTA for block coding a source evaluated at the channel capacity normalized to bits or nats per source symbol. Making this argument precise yields the block joint source and channel coding theorem.

A joint source and channel (N; K) block code consists of an encoder fi :

correspond to K channel time units. The block code yields sequence coders

T ! T „ ^T ! ^T

fi„ : A B and fl : B A deflned by

fi„(x) = ffi(xNiN ); all ig

„ f N g

fl(x) = fl(xiN ); all i :

Let E denote the class of all such codes (all N and K consistent with the physical stationarity requirement). Let ¢⁄(„; ”; E) denote the block coding OPTA function and D(R; „) the distortion-rate function of the source with respect to an additive fldelity criterion f‰ng. We assume also that ‰n is bounded, that is, there is a flnite value ‰max such that

n1 ‰n(xn; x^n) • ‰max

for all n. This assumption is an unfortunate restriction, but it yields a simple proof of the basic result.

Theorem 12.7: Let fXng be a stationary source with distribution „ and

„

let ” be a stationary and ergodic d-continuous channel with channel capacity C. Let f‰ng be a bounded additive fldelity criterion. Given † > 0 there exists for su–ciently large N and K (where K channel time units correspond to N

and choose N large enough to ensure the existence of a source codebook C of length N and rate Rsource = (K=N )(C ¡ °) with performance

†

‰(C; „) • –(Rsource; „) + 3 :

We also assume that N and hence K is chosen large enough so that for a suitably small – (to be specifled later) there exists a channel (beKRc; K; –) code, with R = C ¡ °=2. Index the beNRsource c words in the source codebook by the beK(C¡°=2c channel codewords. By construction there are more indices than source codewords so that this is possible. We now evaluate the performance of this code.

Suppose that there are M words in the source codebook and hence M of the channel words are used. Let x^i and wi denote corresponding source and channel codewords, that is, if x^i is the minimum distortion word in the source codebook for an observed vector, then wi is transmitted over the channel. Let ¡i denote the corresponding decoding region. Then

MM Z

The flrst term is bounded above by –(Rsource; „) + †=3 by construction. The second is bounded above by ‰max times the channel error probability, which is less than – by assumption. If – is chosen so that ‰max– is less than †=2, the theorem is proved. 2

Theorem 12.7.2: Let fXng be a stationary source source with distribution „ and let ” be a stationary channel with channel capacity C. Let f‰ng be a bounded additive fldelity criterion. For any block stationary communication system („; f; ”; g), the average performance satisfles

Z

¢(„; f; ”; g) • d„(x)D(C; „x);

x

where „ is the stationary mean of „ and f„xg is the ergodic decomposition of „, C is the capacity of the channel, and D(R; „) the distortion-rate function.

for n ‚ L. Then if the ergodic component „x is in efiect, the performance can be no better than

which when integrated yields a lower bound of

Z

K

d„(x)D( N C + †; „x) ¡ –:

Since – and † are arbitrary, the lemma follows from the continuity of the distortion rate function. 2

Combining the previous results yields the block coding OPTA for stationary

„

sources and stationary and ergodic d-continuous channels.

Corollary 12.7.1: Let fXng be a stationary source with distribution „ and

„

let ” be a stationary and ergodic d-continuous process with channel capacity C. Let f‰ng be a bounded additive fldelity criterion. The block coding OPTA function is given by

Z

¢⁄(„; ”; E; D) = d„(x)D(C; „x):

12.8Synchronizing Block Channel Codes

As in the source coding case, the flrst step towards proving a sliding block coding theorem is to show that a block code can be synchronized, that is, that the decoder can determine (at least with high probability) where the block code words begin and end. Unlike the source coding case, this cannot be accomplished by the use of a simple synchronization sequence which is prohibited from appearing within a block code word since channel errors can cause the appearance of the sync word at the receiver by accident. The basic idea still holds, however, if the codes are designed so that it is very unlikely that a non-sync word can be con-

„

verted into a valid sync word. If the channel is d-continuous, then good robust Feinstein codes as in Corollary 12.5.1 can be used to obtain good codebooks

. The basic result of this section is Lemma 12.8.1 which states that given a sequence of good robust Feinstein codes, the code length can be chosen large enough to ensure that there is a sync word for a slightly modifled codebook; that is, the synch word has length a specifled fraction of the codeword length

262 CHAPTER 12. CODING FOR NOISY CHANNELS

and the sync decoding words never appear as a segment of codeword decoding words. The technique is due to Dobrushin [33] and is an application of Shannon’s random coding technique. The lemma originated in [59].

The basic idea of the lemma is this: In addition to a good long code, one selects a short good robust Feinstein code (from which the sync word will be chosen) and then performs the following experiment. A word from the short code and a word from the long code are selected independently and at random. The probability that the short decoding word appears in the long decoding word is shown to be small. Since this average is small, there must be at least one short word such that the probability of its decoding word appearing in the decoding word of a randomly selected long code word is small. This in turn implies that if all long decoding words containing the short decoding word are removed from the long code decoding sets, the decoding sets of most of the original long code words will not be changed by much. In fact, one must remove a bit more from the long word decoding sets in order to ensure the desired properties are preserved when passing from a Feinstein code to a channel codebook.

Lemma 12.8.1: Assume that † • 1=4 and fCn; n ‚ n0g is a sequence of

-robust f ( ) 2g Feinstein codes for a „-continuous channel having

† ¿; M n ; n; †= d ” capacity C > 0. Assume also that h(2†) + 2† log(jjBjj ¡ 1) < C, where B is the channel output alphabet. Let – 2 (0; 1=4). Then there exists an n1 such that for all n ‚ n1 the following statements are true.

(A)If Cn = fvi; ¡i; i = 1; ¢ ¢ ¢ ; M (n)g, then there is a modifled codebook Wn = fwi; Wi; i = 1; ¢ ¢ ¢ ; K(n)g and a set of K(n) indices Kn = fk1; ¢ ¢ ¢ ; kK(n) ‰ f1; ¢ ¢ ¢ ; M (n)g such that wi = vki , Wi ‰ (¡i)†2 ; i = 1; ¢ ¢ ¢ ; K(n), and

(B)There is a sync word ¾ 2 Ar, r = r(n) = d–ne = smallest integer larger than –n, and a sync decoding set S 2 BBr such that

x2c(¾)

and such that no r-tuple in S appears in any n-tuple in Wi; that is, if

G(br) = fyn : yir = br some i = 0; ¢ ¢ ¢ ; n ¡ rg and G(S) = Sbr 2S G(br), then

The modifled code Wn has fewer words than the original code Cn, but (12.27) ensures that Wn cannot be much smaller since

proving (12.25). Next observe that if yn 2 (G((Sj )†)c)†–, then there is a bn 2 G((Sj )†)c such that dn(yn; bn) • †– and thus for i = 0; 1; ¢ ¢ ¢ ; n ¡r we have that

dr(yir; bri ) • nr †2– • 2† :

Since bn 2 G((Sj )†)c, it has no r-tuple within † of an r-tuple in Sj and hence

the r-tuples yir are at least †=2 distant from Sj and hence yn 2 H((S)†=2)c). We have therefore that (G((Sj )†)c)†– ‰ G((Sj )†)c and hence

\ \ \

G(S) Wi = G((Sj )†) (¡ki G((Sj )†)c)–†

\

‰ G((Sj )†=2) (G((Sj )†)c)–† = ;;

completing the proof. 2

Combining the preceding lemma with the existence of robust Feinstein codes at rates less than capacity (Lemma 12.6.1) we have proved the following synchronized block coding theorem.

„

Corollary 12.8.1: Le ” be a stationary ergodic d-continuous channel and flx † > 0 and R 2 (0; C). Then there exists for su–ciently large blocklength

N , a length N codebook f¾ £ wi; S £ Wi; i = 1; ¢ ¢ ¢ ; M g, M ‚ 2NR, ¾ 2 Ar, wi 2 An, r + n = N , such that

sup ”xr(Sc) • †;

x2c(¾)

max ”xn(Wjc) • †;

i•j•M

\

Wj G(S) = ;:

Proof: Choose – 2 (0; †=2) so small that C ¡ h(2–) ¡ 2– log(jjBjj ¡ 1) > (1 +

–)R(1 ¡ log(1 ¡ –2)) and choose R0 2 ((1 + –)R(1 ¡ log(1 ¡ –2)), C ¡ h(2–) ¡ 2– log(jjBjj ¡ 1). From Lemma 12.6.1 there exists an n0 such that for n ‚ n0 there exist –-robust (¿; „; n; –) Feinstein codes with M (n) ‚ 2nR0 . From Lemma 12.8.1 there exists a codebook fwi; Wi; i = 1; ¢ ¢ ¢ ; K(n)g, a sync word ¾ 2 Ar, and a sync decoding set S 2 BBr , r = d–ne such that

T

G(S) Wj = ;; j = 1; ¢ ¢ ¢ ; K(n), and from (12.28)

M = K(n) ‚ (1 ¡ –2)M (n):

Therefore for N = n + r

N ¡1 log M ‚ (ndn–e)¡1 log((1 ¡ –2)2nR0)

‚ R0 + log(1 ¡ –2) ‚ R; 1 + –

completing the proof. 2

12.9Sliding Block Source and Channel Coding

Analogous to the conversion of block source codes into sliding block source codes, the basic idea of constructing a sliding block channel code is to use a punctuation sequence to stationarize a block code and to use sync words to locate the blocks in the decoded sequence. The sync word can be used to mark the beginning of a codeword and it will rarely be falsely detected during a codeword. Unfortunately, however, an r-tuple consisting of a segment of a sync and a segment of a codeword may be erroneously detected as a sync with nonnegligible probability. To resolve this confusion we look at the relative frequency of sync-detects over a sequence of blocks instead of simply trying to flnd a single sync. The idea is that if we look at enough blocks, the relative frequency of the sync-detects in each position should be nearly the probability of occurrence in that position and these quantities taken together give a pattern that can be used to determine the true sync location. For the ergodic theorem to apply, however, we require that blocks be ergodic and hence we flrst consider totally ergodic sources and channels and then generalize where possible.

Totally Ergodic Sources

„

Lemma 12.9.1: Let ” be a totally ergodic stationary d-continuous channel. Fix †; – > 0 and assume that CN = f¾ £ wi; S £ Wi; i = 1; ¢ ¢ ¢ ; Kg is a preflxed codebook satisfying (12.24){(12.26). Let °n : GN ! CN assign an N -tuple in the preflxed codebook to each N -tuple in GN and let [G; „; U ] be an N - stationary, N -ergodic source. Let c(an) denote the cylinder set or rectangle of all sequences u = (¢ ¢ ¢ ; u¡1; u0; u1; ¢ ¢ ¢) for which un = an. There exists for su–ciently large L (which depends on the source) a sync locating function

s : BLN ! f0; 1; ¢ ¢ ¢ ; N ¡ 1g and a set ' 2 BGm, m = (L + 1)N , such that if um 2 ' and °N (ULNN ) = ¾ £ wi, then

inf ”x(y : s(yLN ) = µ; µ = 0; ¢ ¢ ¢ ; N ¡1; yLN 2 S£Wi) ‚ 1¡3†: (12.31)

x2c(°m(um))

Comments: The lemma can be interpreted as follows. The source is block encoded using °N . The decoder observes a possible sync word and then looks \back" in time at previous channel outputs and calculates s(yLN ) to obtain the exact sync location, which is correct with high probability. The sync locator function is constructed roughly as follows: Since „ and ” are N -stationary and N -ergodic, if °„ : A1 ! B1 is the sequence encoder induced by the length N block code °N , then the encoded source „°„¡1 and the induced channel

thermore, zj can have no smaller period than N since from (12.24){(12.26) ·(T j c(S)) • †, j = r + 1; ¢ ¢ ¢ ; n ¡ r and ·(c(S)) ‚ 1 ¡ †. Thus deflning the sync pattern fzj ; j = 0; 1; ¢ ¢ ¢ ; N ¡ 1g, the pattern is distinct from any cyclic shift of itself of the form fzk; ¢ ¢ ¢ ; zN¡1; z0; ¢ ¢ ¢ ; xk¡1g, where k • N ¡ 1. The sync locator computes the relative frequencies of the occurrence of S at intervals of length N for each of N possible starting points to obtain, say, a vector z^N = (^z0; z^1; ¢ ¢ ¢ ; z^N¡1). The ergodic theorem implies that the z^i will be near their expectation and hence with high probability (^z0; ¢ ¢ ¢ ; z^N¡1) = (zµ; zµ+1; ¢ ¢ ¢ ; zN¡1; z0; ¢ ¢ ¢ ; zµ¡1), determining µ. Another way of looking at the result is to observe that the sources ·T j ; j = 0; ¢ ¢ ¢ ; N ¡ 1 are each N -ergodic and N -stationary and hence any two are either identical or orthogonal in the sense that they place all of their measure on disjoint N -invariant sets. (See, e.g., Exercise 1, Chapter 6 of [50].) No two can be identical, however, since if ·T i = ·T j for i 6= j; 0 • i; j • N ¡ 1, then · would be periodic with period ji ¡ jj strictly less than N , yielding a contradiction. Since membership in any set can be determined with high probability by observing the sequence for a long enough time, the sync locator attempts to determine which of the N distinct sources ·T j is being observed. In fact, synchronizing the output is exactly equivalent to forcing the N sources ·T j ; j = 0; 1; ¢ ¢ ¢ ; N ¡ 1 to be distinct N -ergodic sources. After this is accomplished, the remainder of the

„

proof is devoted to using the properties of d-continuous channels to show that synchronization of the output source when driven by „ implies that with high probability the channel output can be synchronized for all flxed input sequences in a set of high „ probability.

The lemma is stronger (and more general) than the similar results of Nedoma [107] and Vajda [141], but the extra structure is required for application to sliding block decoding.

We show that s is well deflned by showing that ˆ(µ) ‰ ˆ(µ; 4‡), which sets are disjoint for µ = 0; 1; ¢ ¢ ¢ ; N ¡ 1 from (12.32). If yLN 2 ˆ(µ), there is a bLN 2 ˆ(µ; ‡) for which dLN (yLN ; bLN ) • 2‡=N and hence for any j 2 f0; 1; ¢ ¢ ¢ ; N ¡1g at most LN (2‡=N ) = 2‡L of the consecutive nonoverlapping N -tuples yjN+iN ,

i = 0; 1; ¢ ¢ ¢ ; L ¡ 2, can difier from the corresponding bNj+iN and therefore

LN 2 ~ µ £ £ N¡µ 2 Bm

and hence y ˆ(µ; 4‡). If ˆ(µ) is deflned to be B ˆ(µ) B B , then we also have that