Теория информации / Cover T.M., Thomas J.A. Elements of Information Theory. 2006., 748p

5.35Generating random variables. One wishes to generate a random variable X

You are given fair coin flips Z1, Z2, . . . . Let N be the (random) number of flips needed to generate X. Find a good way to use Z1, Z2, . . . to generate X. Show that EN ≤ 2.

5.36Optimal word lengths.

(a)Can l = (1, 2, 2) be the word lengths of a binary Huffman code. What about (2,2,3,3)?

(b)What word lengths l = (l1, l2, . . .) can arise from binary Huffman codes?

5.37Codes. Which of the following codes are

(a)Uniquely decodable?

(b)Instantaneous?

C1 = {00, 01, 0}

C2 = {00, 01, 100, 101, 11}

C3 = {0, 10, 110, 1110, . . .}

C4 = {0, 00, 000, 0000}

5.38Huffman. Find the Huffman D-ary code for (p1, p2, p3, p4, p5, p6) = ( 256 , 256 , 254 , 254 , 253 , 252 ) and the expected word length

(a)For D = 2.

(b)For D = 4.

156 DATA COMPRESSION

5.39Entropy of encoded bits. Let C : X −→ {0, 1} be a nonsingular but nonuniquely decodable code. Let X have entropy H (X).

(a)Compare H (C(X)) to H (X).

(b)Compare H (C(Xn)) to H (Xn).

5.40Code rate. Let X be a random variable with alphabet {1, 2, 3} and distribution

The data compression code for X assigns codewords

Let X1, X2, . . . be independent, identically distributed according to this distribution and let Z1Z2Z3 · · · = C(X1)C(X2) · · · be the string of binary symbols resulting from concatenating the corresponding codewords. For example, 122 becomes 01010.

(a)Find the entropy rate H (X) and the entropy rate H (Z) in bits per symbol. Note that Z is not compressible further.

(b)Now let the code be

and find the entropy rate H (Z).

5.41Optimal codes. Let l1, l2, . . . , l10 be the binary Huffman codeword lengths for the probabilities p1 ≥ p2 ≥ · · · ≥ p10. Suppose that we get a new distribution by splitting the last probability

mass. What can you say about the optimal binary codeword lengths

l˜1, l˜2, . . . , l11˜ for the probabilities p1, p2, . . . , p9, αp10, (1 − α)p10, where 0 ≤ α ≤ 1.

5.42Ternary codes. Which of the following codeword lengths can be the word lengths of a 3-ary Huffman code, and which cannot?

(a)(1, 2, 2, 2, 2)

(b)(2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3)

5.43Piecewise Huffman. Suppose the codeword that we use to describe a random variable X p(x) always starts with a symbol chosen from the set {A, B, C}, followed by binary digits {0, 1}. Thus, we have a ternary code for the first symbol and binary thereafter. Give the optimal uniquely decodable code (minimum expected number of symbols) for the probability distribution

5.44 Huffman. Find the word lengths of the optimal binary encoding

expected number of wrong objects agreeing with the answers?

(d) Use Markov’s inequality Pr{X ≥ t µ} ≤ 1t , to show that the probability of error (one or more wrong object remaining) goes to zero as n −→ ∞.

HISTORICAL NOTES

The foundations for the material in this chapter can be found in Shannon’s original paper [469], in which Shannon stated the source coding

158 DATA COMPRESSION

theorem and gave simple examples of codes. He described a simple code construction procedure (described in Problem 5.5.28), which he attributed to Fano. This method is now called the Shannon–Fano code construction procedure.

The Kraft inequality for uniquely decodable codes was first proved by McMillan [385]; the proof given here is due to Karush [306]. The Huffman coding procedure was first exhibited and proved to be optimal by Huffman [283].

In recent years, there has been considerable interest in designing source codes that are matched to particular applications, such as magnetic recording. In these cases, the objective is to design codes so that the output sequences satisfy certain properties. Some of the results for this problem are described by Franaszek [219], Adler et al. [5] and Marcus [370].

The arithmetic coding procedure has its roots in the Shannon–Fano code developed by Elias (unpublished), which was analyzed by Jelinek [297]. The procedure for the construction of a prefix-free code described in the text is due to Gilbert and Moore [249]. The extension of the Shannon–Fano–Elias method to sequences is based on the enumerative methods in Cover [120] and was described with finite-precision arithmetic by Pasco [414] and Rissanen [441]. The competitive optimality of Shannon codes was proved in Cover [125] and extended to Huffman codes by Feder [203]. Section 5.11 on the generation of discrete distributions from fair coin flips follows the work of Knuth and Yao[317].

160 GAMBLING AND DATA COMPRESSION

We assume that the gambler distributes all of his wealth across the horses. Let bi be the fraction of the gambler’s wealth invested in horse i, where bi ≥ 0 and bi = 1. Then if horse i wins the race, the gambler will receive oi times the amount of wealth bet on horse i. All the other bets are lost. Thus, at the end of the race, the gambler will have multiplied his wealth by a factor bi oi if horse i wins, and this will happen with probability pi . For notational convenience, we use b(i) and bi interchangeably throughout this chapter.

The wealth at the end of the race is a random variable, and the gambler wishes to “maximize” the value of this random variable. It is tempting to bet everything on the horse that has the maximum expected return (i.e., the one with the maximum pi oi ). But this is clearly risky, since all the money could be lost.

Some clarity results from considering repeated gambles on this race. Now since the gambler can reinvest his money, his wealth is the product of the gains for each race. Let Sn be the gambler’s wealth after n races. Then

i=1

where S(X) = b(X)o(X) is the factor by which the gambler’s wealth is multiplied when horse X wins.

Definition The wealth relative S(X) = b(X)o(X) is the factor by which the gambler’s wealth grows if horse X wins the race.

Definition The doubling rate of a horse race is

k=1

The definition of doubling rate is justified by the following theorem.

Theorem 6.1.1 Let the race outcomes X1, X2, . . . be i.i.d. p(x). Then the wealth of the gambler using betting strategy b grows exponentially at rate W (b, p); that is,

Proof: Functions of independent random variables are also independent, and hence log S(X1), log S(X2), . . . are i.i.d. Then, by the weak law of large numbers,

Now since the gambler’s wealth grows as 2nW (b,p), we seek to maximize the exponent W (b, p) over all choices of the portfolio b.

Definition The optimum doubling rate W (p) is the maximum doubling rate over all choices of the portfolio b:

We maximize W (b, p) as a function of b subject to the constraintbi = 1. Writing the functional with a Lagrange multiplier and changing the base of the logarithm (which does not affect the maximizing b), we have

162 GAMBLING AND DATA COMPRESSION

second derivatives. Instead, we use a method that works for many such problems: Guess and verify. We verify that proportional gambling b = p is optimal in the following theorem. Proportional gambling is known as

Kelly gambling [308].

and is achieved by the proportional gambling scheme b = p.

Proof: We rewrite the function W (b, p) in a form in which the maximum is obvious:

with equality iff p = b (i.e., the gambler bets on each horse in proportion to its probability of winning).

Example 6.1.1 Consider a case with two horses, where horse 1 wins with probability p1 and horse 2 wins with probability p2. Assume even odds (2-for-1 on both horses). Then the optimal bet is proportional betting (i.e., b1 = p1, b2 = p2). The optimal doubling rate is W (p) =pi log oi − H (p) = 1 − H (p), and the resulting wealth grows to infinity at this rate:

Thus, we have shown that proportional betting is growth rate optimal for a sequence of i.i.d. horse races if the gambler can reinvest his wealth and if there is no alternative of keeping some of the wealth in cash.

We now consider a special case when the odds are fair with respect to

some distribution (i.e., there is no track take and 1 = 1). In this case,

oi

we write ri = 1 , where ri can be interpreted as a probability mass function

oi

over the horses. (This is the bookie’s estimate of the win probabilities.) With this definition, we can write the doubling rate as

This equation gives another interpretation for the relative entropy distance: The doubling rate is the difference between the distance of the bookie’s estimate from the true distribution and the distance of the gambler’s estimate from the true distribution. Hence, the gambler can make money only if his estimate (as expressed by b) is better than the bookie’s.

An even more special case is when the odds are m-for-1 on each horse. In this case, the odds are fair with respect to the uniform distribution and the optimum doubling rate is

In this case we can clearly see the duality between data compression and the doubling rate.

Thus, the sum of the doubling rate and the entropy rate is a constant.

Every bit of entropy decrease doubles the gambler’s wealth. Low entropy races are the most profitable.

In the analysis above, we assumed that the gambler was fully invested. In general, we should allow the gambler the option of retaining some of his wealth as cash. Let b(0) be the proportion of wealth held out as cash, and b(1), b(2), . . . , b(m) be the proportions bet on the various horses. Then at the end of a race, the ratio of final wealth to initial wealth (the wealth relative) is

Now the optimum strategy may depend on the odds and will not necessarily have the simple form of proportional gambling. We distinguish three subcases:

164 GAMBLING AND DATA COMPRESSION

organizers of the race track take a cut of all the bets. In this case it is optimal to bet only some of the money and leave the rest aside as cash. Proportional gambling is no longer log-optimal. A parametric form for the optimal strategy can be found using Kuhn – Tucker conditions (Problem 6.6.2); it has a simple “water-filling” interpretation.

6.2GAMBLING AND SIDE INFORMATION

Suppose the gambler has some information that is relevant to the outcome of the gamble. For example, the gambler may have some information about the performance of the horses in previous races. What is the value of this side information?

One definition of the financial value of such information is the increase in wealth that results from that information. In the setting described in Section 6.1 the measure of the value of information is the increase in the doubling rate due to that information. We will now derive a connection between mutual information and the increase in the doubling rate.

To formalize the notion, let horse X {1, 2, . . . , m} win the race with probability p(x) and pay odds of o(x) for 1. Let (X, Y ) have joint

probability mass function p(x, y). Let b(x|y) ≥ 0, x b(x|y) = 1 be an arbitrary conditional betting strategy depending on the side information