Теория информации / Gray R.M. Entropy and information theory. 1990., 284p

Corollary 12.3.1: Suppose that a channel [A; ”; B] is input memoryless and input nonanticipatory (see Section 9.4). Then a („; M; n; †)-Feinstein code for some channel input process „ is also an (M; n; †)-code.

Proof: Immediate since for a channel without input memory and anticipation we have that ”xn(F ) = ”un(F ) if xn = un. 2

The principal idea of constructing channel codes from Feinstein codes for more general channels will be to place assumptions on the channel which ensure that for su–ciently large n the channel distribution ”xn and the induced flnite dimensional channel ”^n(¢jxn) are close. This general idea was proposed by McMillan [103] who suggested that coding theorems would follow for channels that were su–ciently continuous in a suitable sense.

The previous results did not require stationarity of the channel, but in a sense stationarity is implicit if the channel codes are to be used repeatedly (as they will be in a communication system). Thus the immediate applications of the Feinstein results. will be to stationary channels.

The following is a rephrasing of Feinstein’s theorem that will be useful. Corollary 12.3.2: Suppose that [A £B; „”] is an AMS and ergodic hookup

„ ⁄ „ information rate and assume that I„” = I„” is flnite. Then for any R < I„” and

any † > 0 there exists an n0 such that for all n ‚ n0 there are („; benRc; n; †)- Feinstein codes.

As a flnal result of the Feinstein variety, we point out a variation that applies to nonergodic channels.

Corollary 12.3.3: Suppose that [A £ B; „”] is an AMS hookup of a source „ and channel ”. Suppose also that the information density converges a.e. to a limiting density

i1 = lim 1 in(Xn; Y n):

n!1 n

(Conditions for this to hold are given in Theorem 8.5.1.) Then given † > 0 and

– > 0 there exists for su–ciently large n a [„; M; n; †+„”(i1 • R+–)] Feinstein code with M = benRc.

Proof: Follows from the lemma and from Fatou’s lemma which implies that

lim sup p( 1 in(Xn; Y n) • a) • p(i1 • a): 2

n!1 n

12.4Channel Capacity

The form of the Feinstein lemma and its corollaries invites the question of how large R (and hence M ) can be made while still getting a code of the desired form. From Feinstein’s theorem it is seen that for an ergodic channel R can be

„

any number less than I(„”) which suggests that if we deflne the quantity

AMS and ergodic „

and hence it follows as in the proof of Corollary 2.4.1 that h1(„) is an upper

„

semicontinuous function of „. It is also a–ne because H·(Y ) is an a–ne function of · (Lemma 2.4.2) which is in turn a linear function of „. Thus from Theorem 8.9.1 of [50]

Z

h1(„) = d„(x)h1(„x):

„

h2(„) is also a–ne in „ since h1(„) is a–ne in „ and I„” is a–ne in „ (since it is a–ne in „” from Lemma 6.2.2). Hence we will be done if we can show that h2(„) is upper semicontinuous in „ since then Theorem 8.9.1 of [50] will imply that

Z

h2(„) = d„(x)h2(„x)

which with the corresponding result for h1 proves the lemma. To see this observe that if „k ! „ on flnite dimensional rectangles, then

Next observe that for stationary processes

H(Y njXn) • H(Y mjXn) + H(Ymn¡mjXn)

• H(Y mjXm) + H(Ymn¡mjXmn¡m) = H(Y mjXm) + H(Y n¡mjXn¡m)

which as in Section 2.4 implies that H(Y njXn) is a subadditive sequence and hence

Coupling this with (12.22) proves upper semicontinuity exactly as in the proof of Corollary 2.4.1, which completes the proof of the lemma. 2

Lemma 12.4.2: If a channel ” has a flnite alphabet and is stationary, then all of the above information rate capacities are equal.

„ ⁄

Proof: From Theorem 6.4.1 I = I for flnite alphabet processes and hence from Lemma 6.6.2 and Lemma 9.3.2 we have that if „ is AMS with stationary mean „, then

„ „ „

I„” = I„” = I„”

and thus the supremum over AMS sources must be the same as that over stationary sources. The fact that Cs • Cs; e follows immediately from the previous lemma since the best stationary source can do no better than to put all of its measure on the ergodic component yielding the maximum information rate. Combining these facts with (12.19){(12.20) proves the lemma. 2

Because of the equivalence of the various forms of information rate capacity for stationary channels, we shall use the symbol C to represent the information rate capacity of a stationary channel and observe that it can be considered as the solution to any of the above maximization problems.

Shannon’s original deflnition of channel capacity applied to channels without input memory or anticipation. We pause to relate this deflnition to the process

deflnitions. Suppose that a channel [A; ”; B] has no input memory or anticipation and hence for each n there are regular conditional probability measures ”^n(Gjxn); x 2 An, G 2 BBn , such that

”xn(G) = ”^n(Gjxn):

Deflne the flnite-dimensional capacity of the ”^n by

Cn(^”n) = sup I„n”^n (Xn; Y n);

„n

where the supremum is over all vector distributions „n on An. Deflne the Shannon capacity of the channel „ by

CShannon = lim 1 Cn(^”n)

n!1 n

if the limit exists. Suppose that the Shannon capacity exists for a channel ” without memory or anticipation. Choose N large enough so that CN is very close to CShannon and let „N approximately yield CN . Then construct a block memoryless source using „N . A block memoryless source is AMS and hence if the channel is AMS we must have an information rate

Since the input process is block memoryless, we have from Lemma 9.4.2 that

k

X

I(XkN ; Y kN ) ‚ I(XiNN ; YiNN ):

i=0

If the channel is stationary then fXn; Yng is N -stationary and hence if

N1 I„N ”^N (XN ; Y N ) ‚ CShannon ¡ †;

then

kN1 I(XkN ; Y kN ) ‚ CShannon ¡ †:

Taking the limit as k ! 1 we have that

and hence

C ‚ CShannon:

Conversely, pick a stationary source „ which nearly yields C = Cs, that is,

„ ‚ ¡

I„” Cs †:

This implies, however, that for n ‚ n0

Cn ‚ Cs ¡ 2†;

and hence application of the previous lemma proves the following lemma. Lemma 12.4.3: Given a flnite alphabet stationary channel ” with no input

memory or anticipation,

C = CAMS = Cs = Cs; e = CShannon:

The Shannon capacity is of interest because it can be numerically computed while the process deflnitions are not always amenable to such computation.

With Corollary 12.3.2 and the deflnition of channel capacity we have the following result.

Lemma 12.4.4: If ” is an AMS and ergodic channel and R < C, then there is an n0 su–ciently large to ensure that for all n ‚ n0 there exist („; benRc; n; †) Feinstein codes for some channel input process „.

Corollary 12.4.1: Suppose that [A; ”; B] is an AMS and ergodic channel with no input memory or anticipation. Then if R < C, the information rate capacity or Shannon capacity, then for † > 0 there exists for su–ciently large n a (benRc; n; †) channel code.

Proof: Follows immediately from Corollary 12.3.3 by choosing a stationary

„ 2

and ergodic source „ with I„” (R; C). 2

There is another, quite difierent, notion of channel capacity that we introduce for comparison and to aid the discussion of nonergodic stationary channels. Deflne for an AMS channel ” and any ‚ 2 (0; 1) the quantile

C⁄(‚) = sup supfr : „”(i1 • r) < ‚)g;

AMS „

where the supremum is over all AMS channel input processes and i1 is the limiting information density (which exists because „” is AMS and has flnite alphabet). Deflne the information quantile capacity C⁄ by

C⁄ = lim C⁄(‚):

‚!0

The limit is well deflned since the C⁄(‚) are bounded and nonincreasing. The information quantile capacity was introduced by Winkelbauer [149] and its properties were developed by him and by Kiefier [75]. Fix an R < C⁄ and deflne

– = (C⁄¡R)=2. Given † > 0 we can flnd from the deflnition of C⁄ an AMS channel input process „ for which „”(i1 • R + –) • †. Applying Corollary 12.3.3 with this – and †=2 then yields the following result for nonergodic channels.

Lemma 12.4.5: If ” is an AMS channel and R < C⁄, then there is an n0 su–ciently large to ensure that for all n ‚ n0 there exist („; f enRf; n; †) Feinstein codes for some channel input process „.

We close this section by relating C and C⁄ for AMS channels. Lemma 12.4.6: Given an AMS channel ”,

C ‚ C⁄:

Proof: Fix ‚ > 0. If r < C⁄(‚) there is a „ such that ‚ > „”(i1 • r) =

¡ ‚ „

1 „”(i1 > r) 1I„” =r, where we have used the Markov inequality. Thus for

⁄ „ ‚ ¡ •

all r < C we have that I„” r(1 „”(i1 r)) and hence

‚ „ ‚ ⁄ ¡ ! ⁄

C I„” C (‚)(1 ‚) C : 2

‚!0

It can be shown that if a stationary channel is also ergodic, then C = C⁄ by using the ergodic decomposition to show that the supremum deflning C(‚) can be taken over ergodic sources and then using the fact that for ergodic „ and ”,

„

i1 equals I„” with probability one. (See Kiefier [75].)

12.5Robust Block Codes

Feinstein codes immediately yield channel codes when the channel has no input memory or anticipation because the induced vector channel is the same with respect to vectors as the original channel. When extending this technique to channels with memory and anticipation we will try to ensure that the induced channels are still reasonable approximations to the original channel, but the approximations will not be exact and hence the conditional distributions considered in the Feinstein construction will not be the same as the channel conditional distributions. In other words, the Feinstein construction guarantees a code that works well for a conditional distribution formed by averaging the channel over its past and future using a channel input distribution that approximately yields channel capacity. This does not in general imply that the code will also work well when used on the unaveraged channel with a particular past and future input sequence. We solve this problem by considering channels for which the two distributions are close if the block length is long enough.

In order to use the Feinstein construction for one distribution on an actual channel, we will modify the block codes slightly so as to make them robust in the sense that if they are used on channels with slightly difierent conditional distributions, their performance as measured by probability of error does not change much. In this section we prove that this can be done. The basic technique is due to Dobrushin [33] and a similar technique was studied by Ahlswede and G¶acs [4]. (See also Ahlswede and Wolfowitz [5].) The results of this section are due to Gray, Ornstein, and Dobrushin [59].

A channel block length n code fwi; ¡i; i = 1; 2; ¢ ¢ ¢ ; M will be called –- robust (in the Hamming distance sense) if the decoding sets ¡i are such that the

i=0

and dH (a; b) is the Hamming distance (1 if a 6= b and 0 if a = b). Thus the code is – robust if received n-tuples in a decoding set can be changed by an average Hamming distance of up to – without falling in a difierent decoding set. We show that by reducing the rate of a code slightly we can always make a Feinstein

code robust.

Lemma 12.5.1: Let fwi0; ¡0i; i = 1; 2; ¢ ¢ ¢ ; M 0g be a („; enR0 ; n; †)-Feinstein code for a channel ”. Given – 2 (0; 1=4) and

R < R0 ¡ h2(2–) ¡ 2– log(jjBjj ¡ 1);

where as before h2(a) is the binary entropy function ¡a log a ¡(1¡a) log(1¡a), there exists a –-robust („; benRc; n; †n)-Feinstein code for ” with

n-tuples bn 2 Bn such that n¡1dn(yn; bn) • 2–. Set p = 1 ¡ 1=jjBjj and apply Lemma 2.3.5 to the sum to obtain the bound

jjBjj

= ¡h2(–) + 2– ln jjBjj ¡ 1 + (1 ¡2–) ln jjBjj = ¡h2(–) + ln jjBjj¡2– ln(jjBjj¡1):

Combining this bound with the fact that the ¡i are disjoint we have that

Set M = benRc and select 2M subscripts k1; ¢ ¢ ¢ ; k2M from f1; ¢ ¢ ¢ ; M 0g by random equally likely independent selection without replacement so that each index pair (kj ; km); j; m = 1; ¢ ¢ ¢ ; 2M ; j 6= m, assumes any unequal pair with probability (M 0(M 0 ¡ 1))¡1. We then have that

a random coding argument, since the above expectation is less than ‚n, there

Since no more than half of the above indices can exceed twice the expected value, there must exist indices k1; ¢ ¢ ¢ ; kM 2 fj1; ¢ ¢ ¢ ; j2M g for which

M

X\

m=1;m6=j

Deflne the code fwi; ¡i; i = 1; ¢ ¢ ¢ ; M g by wi = wk0 i and

=1;m=i

The (¡i)– are obviously disjoint since we have removed from ¡0ki all words within 2– of a word in any other decoding set. Furthermore, we have for all i = 1; 2; ¢ ¢ ¢ ; M that

X\

which proves the lemma. 2

Corollary 12.5.1: Let ” be a stationary channel and let Cn be a sequence of („n; benR0 c; n; †=2) Feinstein codes for n ‚ n0. Given an R > 0 and – > 0 such that R < R0 ¡h2(2–)¡2– log(jjBjj¡1), there exists for n1 su–ciently large a sequence Cn0 ; n ‚ n1, of –-robust („n; benRc; n; †) Feinstein codes.

Proof: The corollary follows from the lemma by choosing n1 so that

e¡n1(R0¡R¡h2(2–)¡2– ln(jjBjj¡1)¡3=n1) • 2† : 2

Note that the sources may be difierent for each n and that n1 does not depend on the channel input measure.

12.6Block Coding Theorems for Noisy Channels

„

Suppose now that ” is a stationary flnite alphabet d-continuous channel. Suppose also that for n ‚ n1 we have a sequence of –-robust („n; benRc; n; †) Feinstein codes fwi; ¡ig as in the previous section. We now quantify the performance of these codes when used as channel block codes, that is, used on the actual channel ” instead of on an induced channel. As previously let ”^n be the n-dimensional channel induced by „n and the channel ”, that is, for „nn(an) > 0

We use the same codewords wi for the channel code, but we now use the expanded regions (¡i)– for the decoding regions. Since the Feinstein codes were

–-robust, these sets are disjoint and the code well deflned. Since the channel is

Suppose that we have a Feinstein code such that for the induced channel

”^(¡ijwi) ‚ 1 ¡ †:

Then if the conditions of Lemma 10.5.1 are met and „n is the channel input source of the Feinstein code, then

and hence

inf ”xn((¡i)–) ‚ ”^n(¡ijwi) ¡ – ‚ 1 ¡ † ¡ –:

x2c(wi)

Thus if the channel block code is constructed using the expanded decoding sets, we have that

that is, the code fwi; (¡i)–g is a (benRc; n; † + –) channel code. We have now

Combining the lemma with Lemma 12.4.4 and Lemma 12.4.5 yields the following theorem.

„

Theorem 12.6.1: Let ” be an AMS ergodic d-continuous channel. If R < C then given † > 0 there is an n0 such that for all n ‚ n0 there exist (benRc; n; †) channel codes. If the channel is not ergodic, then the same holds true if C is replaced by C⁄.

Up to this point the channel coding theorems have been \one shot" theorems in that they consider only a single use of the channel. In a communication system, however, a channel will be used repeatedly in order to communicate a sequence of outputs from a source.

12.7Joint Source and Channel Block Codes

We can now combine a source block code and a channel block code of comparable rates to obtain a block code for communicating a source over a noisy channel. Suppose that we wish to communicate a source fXng with a distri-

„ ^ bution „ over a stationary and ergodic d-continuous channel [B; ”; B]. The

channel coding theorem states that if K is chosen to be su–ciently large, then we can reliably communicate length K messages from a collection of beKRc messages if R < C. Suppose that R = C ¡ †=2. If we wish to send the given source across this channel, then instead of having a source coding rate of (K=N ) log jjBjj bits or nats per source symbol for a source (N; K) block code, we reduce the source coding rate to slightly less than the channel coding rate R, say Rsource = (K=N )(R ¡ †=2) = (K=N )(C ¡ †). We then construct a block source codebook C of this rate with performance near –(Rsource; „). Every codeword