Теория информации / Gray R.M. Entropy and information theory. 1990., 284p

which with (10.12)-(10.14) proves the lemma. 2

The following corollary states that the probability of error using sliding block

„

codes over a d-continuous channel is a continuous function of the encoder as measured by the metric on encoders given by the probability of disagreement of the outputs of two encoders.

„

Corollary 10.5.1: Given a stationary d-continuous channel ” and a flnite length decoder gm : Bm ! A, then given † > 0 there is a – > 0 so that if f and ` are two stationary encoders such that Pr(f 6= g) • –, then

jPe(„; ”; f; g) ¡ Pe(„; ”; `; g)j • †:

Proof: Fix † > 0 and choose r so large that

mr • 3† ;

and choose – = †=(3r). Then Lemma 10.5.2 implies that

jPe(„; ”; f; g) ¡ Pe(„; ”; `; g)j • †: 2

Given an arbitrary channel [A; ”; B], we can deflne for any block length

Na closely related CBI channel [A; ”;~ B] as the CBI channel with the same probabilities on output N -blocks, that is, the same conditional probabilities for YkNN given x, but having conditionally independent blocks. We shall call ”~ the

N-CBI approximation to ”. A channel ” is said to be conditionally almost block independent or CABI if given † there is an N0 such that for any N ‚ N0 there is an M0 such that for any x and any N -CBI approximation ”~ to ”

where ”xM denotes the restriction of ”x to BBN , that is, the output distribution on Y N given x. A CABI channel is one such that the output distribution is close (in

„

a d sense) to that of the N -CBI approximation provided that N is big enough. CABI channels were introduced by Neuhofi and Shields [110] who provided

several examples alternative characterizations of the class. In particular they

„

showed that flnite memory channels are both d-continuous and CABI. Their

„

principal result, however, requires the notion of the d distance between channels.

0 „

Given two channels [A; ”; B] and [A; ” ; B], deflne the d distance between the

channels to be

„ 0 „ n 0N d(”; ” ) = lim sup sup d(”x ; ” x ):

n!1 x

Neuhofi and Shields [110] showed that the class of CABI channels is exactly

„

the class of primitive channels together with the d limits of such channels.

10.6The Distortion-Rate Function

We close this chapter on distortion, approximation, and performance with the introduction and discussion of Shannon’s distortion-rate function. This function (or functional) of the source and distortion measure will play a fundamental role in evaluating the OPTA functions. In fact, it can be considered as a form of information theoretic OPTA. Suppose now that we are given a source [A; „]

the reproduction alphabet. Then the Shannon distortion rate function (DRF) is deflned in terms of a nonnegative parameter called rate by

If RN (R; „N ) is empty, then DN (R; „N ) is 1. DN is called the Nth order

Proof: Nonnegativity is obvious from the nonnegativity of distortion. Suppose that pi 2 RN (Ri; „N ); i = 1; 2 yields

Epi ‰N (XN ; Y N ) • DN (Ri; „) + †:

Ip„ • ‚Ip1 + (1 ¡ ‚)Ip2 • ‚R1 + (1 ¡ ‚)R2

and hence p„ 2 RN (‚R1 + (1 ¡ ‚)R2) and therefore

DN (‚R1 + (1 ¡ ‚)R2) • Ep„‰N (XN ; Y N )

= ‚Ep1 ‰N (XN ; Y N ) + (1 ¡ ‚)Ep2 ‰N (XN ; Y N )

• ‚DN (R1; „) + (1 ¡ ‚)DN (R2; „):

Since D(R; „) is the limit of DN (R; „), it too is convex. It is well known from real analysis that convex functions are continuous except possibly at their end points. 2

The following lemma shows that when the underlying source is stationary and the fldelity criterion is subadditive (e.g., additive), then the limit deflning D(R; „) is an inflmum.

Lemma 10.6.2: If the source „ is stationary and the fldelity criterion is

pn together with „n implies a regular conditional probability q(F jxn), F 2 Bn .

^

A

Similarly pN¡n and „N¡n imply a regular conditional probability r(GjxN¡n). Deflne now a regular conditional probability t(¢jxN ) by its values on rectangles as

t(F £ GjxN ) = q(F jxn)r(GjxNn ¡n); F 2 Bn^; G 2 BN^ ¡n:

A A

Note that this is the flnite dimensional analog of a block memoryless channel with two blocks. Let pN = „N t be the distribution induced by „ and t. Then exactly as in Lemma 9.4.2 we have because of the conditional independence that

IpN (XN ; Y N ) • IpN (Xn; Y n) + IpN (XnN¡n; YnN¡n)

and hence from stationarity

IpN (XN ; Y N ) • Ipn (Xn; Y n) + IpN ¡n (XN¡n; Y N¡n)

• Dn(R; „n) + DN¡n(R; „N¡n) + †:

Thus since † is arbitrary we have shown that if dn = Dn(R; „n), then

dN • dn + dN¡n; n • N ;

that is, the sequence dn is subadditive. The lemma then follows immediately from Lemma 7.5.1 of [50]. 2

As with the ‰„ distance, there are alternative characterizations of the distortionrate function when the process is stationary. The remainder of this section is devoted to developing these results. The idea of an SBM channel will play an important role in relating nth order distortion-rate functions to the process deflnitions. We henceforth assume that the input source „ is stationary and we conflne interest to additive fldelity criteria based on a per-letter distortion

‰ = ‰1.

The basic process DRF is deflned by

input distribution and having mutual information rate Ip = Ip(X; Y ) • R. The

original idea of a process rate-distortion function was due to Kolmogorov and his colleagues [87] [45] (see also [23]). The idea was later elaborated by Marton [101] and Gray, Neuhofi, and Omura [55].

Recalling that the L1 ergodic theorem for information density holds when

„ ⁄

Ip = Ip ; that is, the two principal deflnitions of mutual information rate yield the same value, we also deflne the process DRF

Ds⁄(R; „) = inf Ep‰(X0; Y0);

p2R⁄s (R;„)

where R⁄s (R; „) is the collection of all stationary processes p having „ as an

„ • „ ⁄ input distribution, having mutual information rate Ip R, and having Ip = Ip .

If „ is both stationary and ergodic, deflne the corresponding ergodic process

where Re(R; „) is the subset of Rs(R; „) containing only ergodic measures and R⁄e (R; „) is the subset of R⁄s (R; „) containing only ergodic measures.

and mutual information for vectors and for SBM channels. These are stated and proved in the following lemma, the proof of which is straightforward but somewhat tedious. The theorem is proved after the lemma.

If ” is an (N; –) SBM channel induced by q as in Example 9.4.11 and if p = „” is the resulting hookup and fXn; Yng the input/output pair process, then

that is, the resulting mutual information rate of the induced stationary process satisfles the same inequality as the vector mutual information and the resulting distortion approximately satisfles the vector inequality provided – is su–ciently small. Observe that if the fldelity criterion is additive, the (10.18) becomes

Ep‰1(X0; Y0) • D + ‰⁄–:

Proof: We flrst consider the distortion as it is easier to handle. Since the SBM channel is stationary and the source is stationary, the hookup p is stationary and

where pz is the conditional distribution of fXn; Yng given fZng. Note that the above formula reduces to Ep‰(X0; Y0) if the fldelity criterion is additive because of the stationarity. Given z, deflne J0n(z) to be the collection of indices of zn for which zi is not in an N -cell. (See the discussion in Example 9.4.11.) Let J1n(z) be the collection of indices for which zi begins an N -cell. If we deflne the event G = fz : z0 begins an N ¡ cellg, then i 2 J1n(z) if T iz 2 G. From Corollary 9.4.3 mZ (G) • N ¡1. Since „ is stationary and fXng and fZng are mutually independent,

Since mZ is stationary, integrating the above we have that

Ep‰1(X0; Y0) = ‰⁄mZ (Gc) + N mZ (G)EpN ‰N

• ‰⁄– + EpN ‰N ;

proving (10.18).

^

Let rm and tm denote asymptotically accurate quantizers on A and A; that is, as in Corollary 6.2.1 deflne

Since Zn has a flnite alphabet, the limits of n¡1I(Zn; (rm(X)n; tm(Y )n)) are

„ ⁄

the same regardless of the order from Theorem 6.4.1. Thus I will equal I if we can show that

Given z, let J0n(z) be as before. Let J2n(z) denote the collection of all indices i of zi for which zi begins an N cell except for the flnal such index (which may begin an N -cell not completed within zn). Thus J2n(z) is the same as J1n(z) except that the largest index in the latter collection may have been removed