Теория информации / Gray R.M. Entropy and information theory. 1990., 284p

on individual sequences, then the set Q can be divided into N disjoint sets of equal probability, say W0; W1; ¢ ¢ ¢ ; WN¡1. Deflne the set FQ by

FQ contains (K ¡2) N shifts of W0, of T W1; ¢ ¢ ¢ of T lWl; ¢ ¢ ¢ and of T N¡1WN¡1. Because it only takes N -shifts of each small set and because it does not include the top N levels of the original column, shifting FQ fewer than N times causes no overlap, that is, T lFQ are disjoint for j = 0; 1; ¢ ¢ ¢ ; N ¡1. The union of these sets contains all of the original column of the tower except possibly portions of the top and bottom N ¡1 levels (which the construction may not include). The new base F is now deflned to be the union of all of the FQk T S . The sets T lF

are then disjoint (since all the pieces are) and contain all of the levels of the original tower except possibly the top and bottom N ¡ 1 levels. Thus

‚ K ¡ 2 1KN¡ † = 1N¡ † ¡ KN2 :

by choosing † = –=2 and K large this can be made larger than 1 ¡ –. Thus the new tower satisfles conditions (A)-(C) and we need only verify the new condition (D), that is, (9.11). We have that

„(PijT lF ) = „(Pi T T lF ) :

Since the denominator does not depend on l, we need only show the numerator does not depend on l. From (9.14) applied to the original tower we have that

that is, the sum over all column levels (old tower) labeled i of the probability of the intersection of the column level and the lth shift of the new base F . The intersection of a column level in the jth level of the original tower with any shift of F must be an intersection of that column level with the jth shift of one of the sets W0; ¢ ¢ ¢ ; WN¡1 (which particular set depends on l). Whichever set is chosen, however, the probability within the sum has the form

\ \ \ \

„(T j (Qk S) T lF ) = „(T j (Qk S) T j Wm)

190 CHAPTER 9. CHANNELS AND CODES

\ \

= „((Qk S) Wm) = „(Wm);

where the flnal step follows since Wm was originally chosen as a subset of Qk S.

which proves (9.11) since there is no dependence on l. This completes the proof of the lemma. 2

†It should be tractable so that one can do useful theory.

†It should be computable so that it can be measured in real systems.

†It should be subjectively meaningful in the sense that small (large) distortion corresponds to good (bad) perceived quality.

Unfortunately these requirements are often inconsistent and one is forced to compromise between tractability and subjective signiflcance in the choice of distortion measures. Among the most popular choices for distortion measures are metrics or distances, but many practically important distortion measures are not metrics, e.g., they are not symmetric in their arguments or they do not satisfy a triangle inequality. An example of a metric distortion measure that will often be emphasized is that given when the input space A is a Polish space, a complete separable metric space under a metric ‰, and B is either A itself or a Borel subset of A. In this case the distortion measure is fundamental to the structure of the alphabet and the alphabets are standard since the space is Polish.

Suppose next that we have a sequence of product spaces An and Bn for n = 1; 2; ¢ ¢ ¢. A fldelity criterion ‰n, n = 1; 2; ¢ ¢ ¢ is a sequence of distortion measures on An £ Bn. If one has a pair random process, say fXn; Yng, then it will be of interest to flnd conditions under which there is a limiting per symbol distortion in the sense that

exists. As one might guess, the distortion measures in the sequence often are interrelated. The simplest and most common example is that of an additive or single-letter fldelity criterion which has the form

n¡1

X

‰n(xn; yn) = ‰1(xi; yi):

i=0

Here if the pair process is AMS, then the limiting distortion will exist and it is invariant from the ergodic theorem. By far the bulk of the information theory literature considers only single-letter fldelity criteria and we will share this emphasis. We will point out, however, other examples where the basic methods and results apply. For example, if ‰n is subadditive in the sense that

‰n(xn; yn) • ‰k(xk; yk) + ‰n¡k(xnk¡k; ykn¡k);

then stationarity of the pair process will ensure that n¡1‰n converges from the subadditive ergodic theorem. For example, if d is a distortion measure on A£B,

then

ˆn¡1 !1=p

X

‰n(xn; yn) = d(xi; yi)p

i=0

for p > 1 is subadditive from Minkowski’s inequality.

As an even simpler example, if d is a distortion measure on A £ B, then the following fldelity criterion converges for AMS pair processes:

n¡1

n1 ‰n(xn; yn) = n1 X f (d(xi; yi)):

i=0

This form often arises in the literature with d being a metric and f being a nonnegative nondecreasing function (sometimes assumed convex).

The fldelity criteria introduced here all are context-free in that the distortion between n successive input/output samples of a pair process does not depend on samples occurring before or after these n-samples. Some work has been done on context-dependent distortion measures (see, e.g., [93]), but we do not consider their importance su–cient to merit the increased notational and technical di–culties involved. Hence we shall consider only context-free distortion measures.

10.3Performance

As a flrst application of the notion of distortion, we deflne a performance measure of a communication system. Suppose that we have a communication system

f ^ g

[„; f; ”; g] such that the overall input/output process is Xn; Xn . For the moment let p denote the corresponding distribution. Then one measure of the quality (or rather the lack thereof) of the communication system is the long term time average distortion per symbol between the input and output as determined by the fldelity criterion. Given two sequences x and x^ and a fldelity criterion ‰n; n = 1; 2; ¢ ¢ ¢, deflne the limiting sample average distortion or sequence distortion by

‰1(x; y) = lim sup 1 ‰n(xn; yn):

n!1 n

Deflne the performance of a communication system by the expected value of the limiting sample average distortion:

We will focus on two important special cases. The flrst is that of AMS systems and additive fldelity criteria. A large majority of the information theory literature is devoted to additive distortion measures and this bias is re°ected here. We also consider the case of subadditive distortion measures and systems that are either two-sided and AMS or are one-sided and stationary. Unhappily the overall AMS one-sided case cannot be handled as there is not yet a subadditive ergodic theorem for that case. In all of these cases we have that if ‰1 is

When a system and fldelity criterion are such that (10.2) and (10.3) are satisfled we say that we have a convergent fldelity criterion. We henceforth make this assumption.

Since ‰1 is invariant, we have from Lemma 6.3.1 of [50] that

If the fldelity criterion is additive, then we have from the stationarity of p„ that the performance is given by

Assume for the remainder of this section that ‰n is an additive fldelity crite-

rion. Suppose now that we now that p is N -stationary; that is, if T = TA £ TA^

We will have this N stationarity, for example, if the source and channel are stationary and the coders are N -stationary, e.g., are length N -block codes. More generally, the source could be N -stationary, the flrst sequence coder (N; K)- stationary, the channel K-stationary (e.g., stationary), and the second sequence coder (K; N )-stationary.

We can also consider the behavior of the N -shift more generally when the system is only AMS This will be useful when considering block codes. Suppose now that p is AMS with stationary mean p„. Then from Theorem 7.3.1 of [50], p is also T N -AMS with an N -stationary mean, say p„N . Applying the ergodic theorem to the N shift then implies that if ‰N is p„N -integrable, then

Given a notion of the performance of a communication system, we can now deflne the optimal performance achievable for trying to communicate a given source fXng with distribution „ over a channel ”: Suppose that E is some class of sequence coders f : AT ! BT . For example, E might consist of all sequence coders generated by block codes with some constraint or by flnite-length sliding

Deflne the optimal performance theoretically achievable or OPTA function for the source „, channel ”, and code classes E and D by

The goal of the coding theorems of information theory is to relate the OPTA function to (hopefully) computable functions of the source and channel.

10.4The rho-bar distortion

In the previous sections it was pointed out that if one has a distortion measure ‰ on two random objects X and Y and a joint distribution on the two random objects (and hence also marginal distributions for each), then a natural notion of the difierence between the processes or the poorness of their mutual approximation is the expected distortion E‰(X; Y ). We now consider a difierent question: What if one does not have a joint probabilistic description of X and Y , but instead knows only their marginal distributions. What then is a natural notion of the distortion or poorness of approximation of the two random objects? In other words, we previously measured the distortion between two random variables whose stochastic connection was determined, possibly by a channel, a code, or a communication system. We now wish to flnd a similar quantity for the case when the two random objects are only described as individuals. One possible deflnition is to flnd the smallest possible distortion in the old sense consistent with the given information, that is, to minimize E‰(X; Y ) over all joint distributions consistent with the given marginal distributions. Note that this will necessarily give a lower bound to the distortion achievable when any speciflc joint distribution is specifled.

To be precise, suppose that we have random variables X and Y with distributions PX and PY and alphabets A and B, respectively. Let ‰ be a distortion measure on A £ B. Deflne the ‰„-distortion (pronounced ‰-bar) between the random variables X and Y by

‰„(PX ; PY ) = inf Ep‰(X; Y );

p2P

Where P = P(PX ; PY ) is the collection of all measures on (A £B; BA £BB ) with PX and PY as marginals; that is,

p(A £ F ) = PY (F ); F 2 BB ;

and

p(G £ B) = PX (G); G 2 BA:

Note that P is not empty since, for example, it contains the product measure

PX £ PY .

Levenshtein [94] and Vasershtein [144] studied this quantity for the special case where A and B are the real line and ‰ is the Euclidean distance. When as in their case the distortion is a metric or distance, the ‰„-distortion is called the ‰„-distance. Ornstein [116] developed the distance and many of its properties for the special case where A and B were common discrete spaces and ‰ was the

„

Hamming distance. In this case the ‰„-distance is called the d-distance. R. L. Dobrushin has suggested that because of the common su–x in the names of its originators, this distance between distributions should be called the shtein or stein distance.

The ‰„-distortion can be extended to processes in a natural way. Suppose now that fXng is a process with process distribution mX and that fYng is a process with process distribution mY . Let PXn and PY n denote the induced flnite dimensional distributions. A fldelity criterion provides the distortion ‰n between these n dimensional alphabets. Let ‰„n denote the corresponding ‰„ distortion between the n dimensional distributions. Then

1

‰„(mX ; mY ) = supn n ‰„n(PXn ; PY n );

that is, the ‰„-distortion between two processes is the maximum of the ‰„-distortions per symbol between n-tuples drawn from the process. The properties of the ‰„ distance are developed in [57] [119] and a detailed development may be found in [50] . The following theorem summarizes the principal properties.

Theorem 10.4.1: Suppose that we are given an additive fldelity criterion ‰n with a pseudo-metric per-letter distortion ‰1 and suppose that both distributions mX and mY are stationary and have the same standard alphabet. Then

(a)limn!1 n¡1‰„n(PXn ; PY n ) exists and equals supn n¡1‰„n(PXn ; PY n ).

(b)‰„n and ‰„ are pseudo-metrics. If ‰1 is a metric, then ‰„n and ‰„ are metrics.

(c)If mX and mY are both i.i.d., then ‰„(mX ; mY ) = ‰„1(PX0 ; PY0 ).

(d)Let Ps = Ps(mX ; mY ) denote the collection of all stationary distributions pXY having mX and mY as marginals, that is, distributions on fXn; Yng

with coordinate processes fXng and fYng having the given distributions. Deflne the process distortion measure ‰„0

‰„0(mX ; mY ) = inf EpXY ‰(X0; Y0):

pXY 2Ps

Then

‰„(mX ; mY ) = ‰„0(mX ; mY );

that is, the limit of the flnite dimensional minimizations is given by a minimization over stationary processes.

(e) Suppose that mX and mY are both stationary and ergodic. Deflne Pe = Pe(mX ; mY ) as the subset of Ps containing only ergodic processes, then

‰„(mX ; mY ) = inf EpXY ‰(X0; Y0);

pXY 2Pe

(f) Suppose that mX and mY are both stationary and ergodic. Let GX denote a collection of generic sequences for mX in the sense of Section 8.3 of [50]. Generic sequences are those along which the relative frequencies of a set of generating events all converge and hence by measuring relative frequencies on generic sequences one can deduce the underlying stationary and ergodic measure that produced the sequence. An AMS process produces generic sequences with probability 1. Similarly let GY denote a set of generic sequences for mY . Deflne the process distortion measure

Then

‰„(mX ; mY ) = ‰„00(mX ; mY );

that is, the ‰„ distance gives the minimum long term time average distortion obtainable between generic sequences from the two sources.

(g)The inflma deflning ‰„n and ‰„0 are actually minima.

10.5d-bar Continuous Channels

We can now generalize some of the notions of channels by using the ‰„-distance to weaken the deflnitions. The flrst deflnition is the most important for channel coding applications. We now conflne interest to the d-bar distance, the ‰-distance for the special case of the Hamming distance:

restriction of the channel to Bn, that is, the output distribution on Y n given

sequences are very close provided that the input sequences are identical over the

same time period and that n is large. This generalizes the notions of 0 or flnite input memory and anticipation since the distributions need only approximate each other and do not have to be exactly the same.

More generally we could consider ‰„-continuous channels in a similar manner,

„

but we will focus on the simpler discrete d-continuous channel.

„

d-continuous channels possess continuity properties that will be useful for proving block and sliding block coding theorems. They are \continuous" in the sense that knowing the input with su–ciently high probability for a su–ciently long time also specifles the output with high probability. The following two lemmas make these ideas precise.

Lemma 10.5.1: Suppose that x, x„ 2 c(an) and

„ n; ”n) • –2: d(”x x„

„

This is the case, for example, if the channel is d continuous and n is chosen su–ciently large. Then

„

Proof: From Theorem 10.4.1 the inflma deflning the d distance are actually minima and hence there is a pmf p on Bn £ Bn such that

X

p(yn; bn) = ”xn(yn)

bn2Bn

and

‚ ”x„n(G) ¡ –

proving the flrst statement. The second statement follows from the flrst. 2

Next suppose that [G; „; U ] is a stationary source, f is a stationary encoder which could correspond to a flnite length sliding block encoder or to an inflnite