Теория информации / Cover T.M., Thomas J.A. Elements of Information Theory. 2006., 748p

b , 376 An, 376 Bn, 376 φA(), 376

Pλ, 380 λ , 380

C(P1, P2), 386 ψ(s), 392

T (X1, X2, . . . , Xn), 393

V , 394

J (θ ), 394

bT (θ ), 396 Jij (θ ), 397 R(k), 415

S(λ), 415

ˆ

R(k), 415

K(n), 416 σ∞2 , 417

Kp, 420(u), 422

pθ , 428

R(pθ , q), 429

R , 429

qπ , 430

A(n, k), 434

q 1 (xn), 436

2

FU (u), 437

Rn(X0, X1, . . . , Xn−1), 444 Qu(i), 445

Aj k , 445 c(n), 450

nk , 450 cls , 452

{Xi }∞−∞, 455 AD , 460

U, 466 U(p), 466 p, 466

KU (x), 466

KU (x|l(x)), 467

K(x|l(x)), 468

log n, 469

H0(p), 470

K(n), 471

PU (x), 481

, 484

n, 484

Kk (xn|n), 496 k , 497

p , 497 S , 497 S , 497

CI, 535

R(S), 543

Sc, 543

β p1, 569 b, 613

S, 613

W (b, F ), 615 W (F ), 615 b , 615

Sn , 615 W , 615

B, 617W , 622

W∞, 624

U , 628

Sn (xn), 630

Vn, 631 j n, 634

w(j n), 634

K, 636

726 LIST OF SYMBOLS

(m), 638

B(λ1, λ2), 642 ( , B, P ), 644

X(ω), 644

H k , 645 H ∞, 645

pˆn, 660(z), 662

ψ(z), 662

γ , 662

X(S), 668

h(n)k , 668 tk(n), 669 gk(n), 669 fk(n), 671

J (X), 671

gt (y), 672 V (A), 675

f g, 676

Lr , 676

||f ||r , 676

Cp, 676

hr (X), 676 Vr (X), 677

Pk , 680

Sk(n), 680

K(i1, i2, . . . , ik ), 680

Qk , 681 σi2, 681

Kk , 681

Abrahams, J., 689 Abramson, N.M., xxiii, 689 Abu-Mostafa, Y.S., 689 acceptance region, 376, 383

achievable rate, 195, 268, 306, 538, 546, 550, 598

achievable rate distortion pair, 306 achievable rate region, 514, 550 Aczel,´ J., 690

Adams, K., xxiii

adaptive dictionary compression algorithms, 441

additive channel, 229

additive noise channel, 280, 287, 296 additive white Gaussian noise (AWGN),

289

Adler, R.L., 158, 689

AEP, xix, xx, 6, 12, 57, 58, 64, 69, 77, 101, 168, 219, 220, 222, 223, 347, 356, 381, 382, 409, 460, 531, 554, 560, 566, 644, 645, 649, 656, 686, see also Shannon-McMillan-Breiman theorem

continuous random variables, 245 discrete random variables, 58– 62 distortion typical, 319

growth rate, 650 joint, 196, 223 products, 66

relative entropy, 380 sandwich proof, 644 – 649

stationary ergodic processes, 644 – 649 Ahlswede, R., 11, 609, 610, 689, 690, 712 Akaike, H., 690

Algoet, P., xxiii, 69, 626, 645, 656, 690

algorithm, 442

alternating minimization, 332

arithmetic coding, see arithmetic coding Blahut-Arimoto, see Blahut-Arimoto

algorithm Durbin, 419 Frank-Wolfe, 191 gradient search, 191 greedy, 127

Huffman coding, 127, 154, see Huffman coding

iterative, 332

Lempel-Ziv, see Lempel-Ziv coding Levinson, 419

online, 427

algorithmic complexity, 1, 3, 463, 464, 466, 507, 508, see Kolmogorov

complexity

algorithmically random, 477, 486, 502, 507

almost sure, 58, 69 alphabet, 13, 347

discrete, 13

effective size, 46, 256 effective support set, 256 input, 183

output, 183 alphabetic codes, 121 Altria, 643

Amari, S.I., 55, 690, 703 Anantharam, V., 690 antenna, 292, 611

Arimoto, S., 191, 240, 335, 346, 690, see also Blahut-Arimoto algorithm

arbitrarily varying channel, 689, 690, 697

728 INDEX

arithmetic coding, 130, 158, 171, 427, 428, 435 – 440, 461

finite precision arithmetic, 439

arithmetic mean geometric mean inequality, 669

ASCII, 466 Ash, R.B., 690

asymmetric distortion, 337

asymptotic equipartition property, see AEP asymptotic optimality,

log-optimal portfolio, 619 ATM networks, 218 atmosphere, 412

atom, 137 – 140, 257, 404 autocorrelation, 415, 420 autoregressive process, 416 auxiliary random variable, 565, 569

average codeword length, 124, 129, 148 average description length, 103

average distortion, 318, 324, 325, 329, 344 average power, 261, 295, 296, 547

average probability of error, 199, 201, 202, 204, 207, 231, 297, 532, 554

AWGN (Additive white Gaussian noise), 289

axiomatic definition of entropy, 14, 54

Baer, M., xxi

Bahl, L.R., xxiii, 690, 716 bandlimited, 272 bandpass filter, 270

bandwidth, 272 – 274, 289, 515, 547, 606 Barron, A.R., xxi, xxiii, 69, 420, 508, 656,

674, 691, 694 base of logarithm, 14 baseball, 389

Baum, E.B., 691

Bayesian error exponent, 388 Bayesian hypothesis testing, 384 Bayesian posterior probability, 435 Bayesian probability of error, 385, 399 BCH (Bose-Chaudhuri-Hocquenghem)

codes, 214 Beckenbach, E.F., 484 Beckner, W., 691 Bell, R., 182, 656, 691 Bell, T.C., 439, 691 Bell’s inequality, 56 Bellman, R., 691

Bennett, C.H., 241, 691 Bentley, J., 691 Benzel, R., 692

Berger, T., xxiii, 325, 345, 610, 611, 692, 720

Bergmans, P., 609, 692 Bergstrøm’s inequality, 684 Berlekamp, E.R., 692, 715 Bernoulli, J., 182

Bernoulli distribution, 434

Bernoulli process, 237, 361, 437, 476, 484, 488

Bernoulli random variable, 63 Bernoulli source, 307, 338

rate distortion, 307 Berrou, C., 692 Berry’s paradox, 483 Bertsekas, D., 692

beta distribution, 436, 661 beta function, 642

betting, 162, 164, 167, 173, 174, 178, 487, 626

horse race, 160 proportional, 162, 626

bias, 393, 402 Bierbaum, M., 692 Biglieri, E., 692

binary entropy function, 15, 49 graph, 15, 16

binary erasure channel, 188, 189, 218, 227, 232, 457

binary multiplier channel, 234, 602 binary rate distortion function, 307 binary source, 308, 337

binary symmetric channel (BSC), 8, 187 – 189, 210, 215, 222, 224, 225, 227 – 229, 231, 232, 237, 238, 262, 308, 568, 601

capacity, 187 binning, 609

random, 551, 585 bioinformatics, xv bird, 97

Birkhoff’s ergodic theorem, 644 bishop, 80

bit, 14

Blachman, N., 674, 687, 692 Blackwell, D., 692

Blahut, R.E., 191, 240, 335, 346, 692, see also Blahut-Arimoto algorithm

Blahut-Arimoto algorithm, 191, 334 block code, 226

block length, 195, 204

block Markov encoding, 573 Boltzmann, L., 11, 55, 693, see

also Maxwell-Boltzmann distribution bone, 93

bookie, 163

Borel-Cantelli lemma, 357, 621, 649 Bose, R.C., 214, 693

bottleneck, 47, 48

bounded convergence theorem, 396, 647 bounded distortion, 307, 321

brain, 465 Brascamp, H.J., 693 Brassard, G., 691

Breiman, L., 69, 655, 656, 692, 693, see also Shannon-McMillan-Breiman theorem

Brillouin, L., 56, 693

broadcast channel, 11, 100, 515, 518, 533, 563, 560 – 571, 593, 595, 599, 601, 604, 607, 609, 610

capacity region, 564 convexity, 598

common information, 563 converse, 599

degraded, 599, 609 achievablity, 565 converse, 599

with feedback, 610 Gaussian, 515, 610 physically degraded, 564 stochastically degraded, 564

Brunn-Minkowski inequality, xx, 657, 674 – 676, 678, 679, 687

BSC, see binary symmetric channel (BSC) Bucklew, J.A., 693

Burg, J.P., 416, 425, 693 Burg’s algorithm, 416

Burg’s maximum entropy theorem, 417 Burrows, M., 462, 693

burst error correcting code, 215 Buzo, A., 708

Caire, G., 693 cake, 65

INDEX 729

calculus, 103, 111, 410 Calderbank, A.R., 693 Canada, 82

capacity, 223, 428

channel, see channel capacity capacity region, 11, 509 – 610

degraded broadcast channel, 565 multiple access channel, 526

capacity theorem, xviii, 215

CAPM (Capital Asset Pricing Model), 614 Caratheodory’s´ theorem, 538

cardinality, 245, 538, 542, 565, 569 cards, 55, 84, 167, 608

Carleial, A.B., 610, 693 cascade, 225

cascade of channels, 568 Castelli, V., xxiii Cauchy distribution, 661

Cauchy-Schwarz inequality, 393, 395 causal, 516, 623

causal investment strategy, 619, 623, 635 causal portfolio, 620, 621, 624, 629 – 631,

635, 639, 641, 643 universal, 651

CCITT, 462

CDMA (Code Division Multiple Access), 548

central limit theorem, xviii, 261, 361 centroid, 303, 312

Cesaro´ mean, 76, 625, 682

chain rule, xvii, 35, 43, 49, 287, 418, 540, 541, 555, 578, 584, 590, 624, 667

differential entropy, 253 entropy, 23, 31, 43, 76 growth rate, 650

mutual information, 24, 25, 43 relative entropy, 44, 81

Chaitin, G.J., 3, 4, 484, 507, 508, 693, 694 Chang, C.S., xxi, 694

channel,

binary erasure, 188, 222

binary symmetric, see binary symmetric channel (BSC)

broadcast, see broadcast channel cascade, 225, 568

discrete memoryless, see discrete memoryless channel

exponential noise, 291 extension, 193

730 INDEX

channel (continued ) feedback, 216

Gaussian, see Gaussian channel interference, see interference channel multiple access, see multiple access

channel

noiseless binary, 7, 184 parallel, 224, 238 relay, see relay channel

symmetric, 189, 190, 222 time-varying, 226, 294 two-way, see two-way channel union, 236

weakly symmetric, 190 with memory, 224, 240 Z-channel, 225

channel capacity, xv, xviii, xix, 1, 3, 7 – 9, 11, 38, 183 – 241, 291, 297, 298, 307, 333, 427, 432, 458, 461, 544 – 546, 548, 592, 604, 608, 686

achievability, 200 computation, 332 feedback, 223 information, 184, 263 operational definition, 184 properties, 222

zero-error, 226

channel capacity theorem, 38 channel code, 193, 220

channel coding, 207, 219, 220, 318, 324, 325, 344

achievability, 195

channel coding theorem, 9, 199, 207, 210, 223, 230, 321, 324, 347

achievability, 200 converse, 207

channel transition matrix, 190, 234, 428, 433, 509

channels with memory, 224, 277 characteristic function, 422 Chellappa, R., 694

Cheng, J.F., 709

Chernoff, H., 694 Chernoff bound, 392

Chernoff information, 380, 384, 386, 388, 399

Chernoff-Stein lemma, 347, 376, 380, 383, 399

Chervonenkis, A.Y., 717

chessboard, 80, 97

χ2 distance, 665

χ2 statistic, 400

Chi-squared distribution, 661 Chiang, M.S., xxi, 610, 695 chicken, 301

Choi, B.S., 425, 694, 697 Chomsky, N., 694

Chou, P.A., 694 Chuang, I., 241, 710 Chung, K.L., 69, 694 Church’s thesis, 465 Cioffi, J.M., 611, 702 cipher, 170

substitution, 170 Clarke, B.S., 694 classroom, 561 Cleary, J.G., 691, 719 closed system, 11 closure, 363

Cocke, J., 690 cocktail party, 515

code, 10, 20, 61, 62, 98, 104 – 158, 172, 194 – 232, 264 – 281, 302 – 339, 357, 360, 428, 429, 434, 436, 438, 440, 443, 444, 456, 460 – 463, 492, 525 – 602

arithmetic, see arithmetic coding block, see block code

channel, see channel code

distributed source, see distributed source coding

error correcting, see error correcting code extension, 105, 126

Gaussian channel, 264 Hamming, see Hamming code Huffman, see Huffman code

instantaneous, 103, 106, 107, 110, 118, 124, 142, 143, 146, 152, see

also code, prefix minimum distance, 212 minimum weight, 212 Morse, see Morse code nonsingular, 105, 152 optimal, 124, 144 prefix, 106

random, 199, 201 self-punctuating, 105, 106 Shannon, see Shannon code

source, see source code

uniquely decodable, 105, 110, 116 – 118, 127, 131, 141 – 143, 147, 148, 150, 152, 157, 158

zero-error, 205

codebook, 204, 206, 212, 266, 268, 321, 322, 327, 328, 513 – 519, 530, 534, 565, 574, 580

codelength, 114, 115, 122 code points, 302 codeword, 109, 122, 131

optimal length, 113 Cohn, M., xxi

coin flips, 44, 103, 155

coin tosses, 37, 134, 137, 139, 225, 375 coin weighing, 45

coins, bent, 48

biased, 96, 375, 407 fair, 134

large deviations, 365 colored noise, 288 coloring,

random, 558 comma, 105, 468

common information, 563, 564, 567, 568

communication channel, 1, 7, 187, 223, 261, 560, 663

communication networks, 509 communication system, 8, 184, 192 communication theory, 1

compact disc, 3, 215 compact set, 432, 538 company mergers, 149 competitive optimality,

log-optimal portfolio, 627 Shannon code, 130

competitively optimal, 103, 613 composition class, 348 compression, see data compression

computable, 466, 479, 482, 484, 491 computable probability distribution, 482 computable statistical tests, 479 computation, 4, 5, 12, 68, 190, 438, 463,

466, 594

channel capacity, 191, 332 halting, see halting computation models of, 464

INDEX 731

physical limits, 56 rate distortion, 332

computer science, xvii, 1, 463, 483 computer source code, 360 computers, 4, 442, 464, 504 concatenation, 105, 116

concavity, 27, 31, 33, 222, 453, 474, 616, see also convexity

conditional entropy, 17, 38 – 40, 75, 77, 88, 89, 206, 417, 669, 670

conditional limit theorem, 366, 371, 375, 389, 398

conditional mutual information, 24, 589 conditional rate distortion,

convexity, 583, 585

conditional rate distortion function, 584, 585

conditional relative entropy, 25 conditional type, 342, 366 conditionally typical set, 327, 342

conditioning reduces entropy, 33, 39, 85, 311, 313, 318, 418, 578, 584, 682

constant rebalanced portfolio, 615, 630 constrained sequences, 94, 101 continuous alphabet, 261, 305 continuous channel, 263

continuous random variable, 21, 37, 243, 245, 248, 256, 301, 304, 338, 500, see also differential entropy, rate distortion theory

quantization, see quantization continuous time, 270 convergence of random variables,

convergence in mean square, 58 convergence in probability, 58 convergence with probability, 1, 58

converse,

broadcast channel, 599

discrete memoryless channel, 206 with feedback, 216

general multiterminal network, 589 multiple access channel, 538

rate distortion, 315 Slepian-Wolf coding, 555

732 INDEX

convex closure, 538 convex families, 655

convex hull, 526, 530, 532, 534 – 536, 538, 543, 544, 565, 591, 594, 595

convex set, 330, 534 convexification, 598

convexity, 26, 28, 32, 42, 432, 616, 657, see also concavity

capacity region, broadcast channel, 598

multiple access channel, 534 rate distortion function, 316 strict, 26

convolution, 270, 674 convolutional code, 215

interleaved, 215 cookie-cutting, 240 copper, 274 Coppersmith, D., 689

correlated random variables, coding, see Slepian-Wolf coding

correlation, 46, 252, 258, 294, 295, 593 Costa, M.H.M., 593, 694, 703

Costello, D.J., 708 counterfeit, 45

covariance matrix, 249, 277, 279, 280, 284, 292, 397, 416, 417, 681

Cover, T.M., xxi, 69, 158, 182, 240, 299, 425, 508, 575, 593, 609 – 611, 626, 656, 687, 690, 691, 694 – 699, 701, 712, 717

CPU, 465 Cramer,´ H., 696

Cramer´-Rao inequality, 392, 395, 396, 399 with bias, 402

crosstalk, 273 cryptography, 171

Csiszar,´ I., 55, 325, 332, 334, 335, 346, 347, 358, 408, 461, 610, 696, 697

Csiszar´-Tusnady´ algorithm, 335 cumulative distribution function, 127, 128,

130, 243, 437, 439 cut set, 512, 589, 597

D-adic distribution, 112

D-ary alphabet, 103, 109

Dantzig, G.B., 697

Darroch, J.N., 697

Daroczy,´ Z., 690

data compression, xv, xvii, xix, 1, 3, 5, 11, 103, 156, 163, 172, 173, 184, 218, 221, 301, 427, 442, 457, 549, 656, 686

data processing inequality, 35, 39, 44, 47, 371, 687

data transmission, 5, 686 Daubechies, I., 697

Davisson, L.D., 461, 697, 702, 703 de Bruijn’s identity, 672

decision function, 375

decision theory, see hypothesis testing decoder, 194

decoding, 194 robust, 296

decoding delay, 148

decoding function, 194, 219, 264, 552, 571, 583

degraded, 515, 516, 533, 565 – 570, 595, 599, 601, 604, 609

broadcast channel, see broadcast channel, degraded

physically degraded, 564

relay channel, see relay channel, degraded

stochastically degraded, 564 Dembo, A., xxiii, 687, 697, 698 demodulation, 3

Dempster, A.P., 698 density, 243

descriptive complexity, 463 determinant, 249, 279, 679, 681 determinant inequalities, xviii, 679 – 685

deterministic, 53, 173, 204, 511, 607, 609 deterministic decoding rule, 194 deterministic function, 339, 456

Devroye, L., 698

dice, 364, 375, 411, 412 dictionary, 442

differential entropy, xix, 243– 674 bound, 258

conditional, 249

Gaussian distribution, 244 relationship to discrete entropy, 247 table of, 661

Diggavi, S., xxi digital, 273, 274, 465 dimension, 211, 675

Dirichlet distribution, 436, 641 Dirichlet partition, 303

discrete, 184

discrete channel, 263, 264, 268 discrete entropy, 244, 259, 660 discrete memoryless channel, 183,

184 – 241, 280, 318, 344, 536 discrete memoryless source, 92

discrete random variable, 14, 49, 54, 134, 243, 245, 249, 251, 252, 258, 347, 371, 658

discrete time, 261 discrimination, 55

distance, 20, 301, 325, 332, 369, 432 between probability distributions, 13 Euclidean, 297, 332, 367

relative entropy, 356, 366 variational, 370

distinguishable inputs, 10, 222 distinguishable signals, 183

distortion, 10, 301 – 341, 580 – 582, 586, 587, see also rate distortion theory

asymmetric, 337

Hamming, see Hamming distortion squared error, see squared error

distortion

distortion function, 304, 307, 309, 312, 336, 340, 344, 345

distortion measure, 301, 302, 304, 305, 307, 314, 319, 321

bounded, 304

Hamming distortion, 304, Itakura-Saito, 305

squared error distortion, 305 distortion rate function, 306, 341 distortion typical, 319, 321, 322, 328 distributed source coding, 509, 511, 550,

556, 586, see also Slepian-Wolf coding

distribution, 427, 428, 456 two mass point, 28

divergence, 55 DiVincenzo, D.P., 691, 698

DMC, see discrete memoryless channel Dobrushin, R.L., 698, 711

dog, 93

Donoho, D.L., 698

doubling rate, xvii, 9, 11, 12, 160, 163 – 167, 175, 179, 180

doubly stochastic matrix, 88 DSL, 274

INDEX 733

duality, 610

data compression and data transmission, 184

data compression and gambling, 173 growth rate and entropy rate, 159, 613 multiple access channel and

Slepian-Wolf coding, 558

rate distortion and channel capacity, 311 source coding and generation of random

variables, 134 Duda, R.O., 698

Dueck, G., 609, 610, 690, 698 Durbin algorithm, 419

Dutch, 561, 562, 606 Dutch book, 164, 180 DVD, 3

dyadic, 129, 130, 132, 137 – 139 dyadic distribution, 151

ear, 219

Ebert, P.M., 299, 698 echoes, 273 Eckschlager, K., 698 economics, 4

edge process, 98 effectively computable, 465 efficient estimator, 396 efficient frontier, 614

Effros, M., 462, 694, 698, 702 Efron, B., 699

Eggleston, H.G., 538, 699 eigenvalue, 95, 279, 282, 315, 336 Einstein, A., xvii

Ekroot, L., xxiii

El Gamal, A., xxiii, 575, 609 – 611, 694 – 696, 699, 701, 702, 710

elephant, 301 Elias, P., 158, 699 Ellis, R.S., 699 EM algorithm, 335

empirical, 68, 381, 409, 470, 474, 542, 660 empirical distribution, 68, 168, 209, 347,

356, 366, 630 convergence, 68

empirical entropy, 195 empirical frequency, 474

encoder, 173, 231, 304, 321, 357, 360, 443, 458, 459, 549, 553, 557, 574, 585, 588

734 INDEX

encoding function, 193, 264, 305, 359, 571, 583, 599

encrypted text, 506

energy, 261, 265, 272, 273, 294, 424 England, 82

English, 104, 168 – 171, 174, 175, 182, 360, 470, 506

entropy rate, 159, 182 models of, 168

entanglement, 56

entropy, xvii, 3, 4, 13 – 56, 87, 659, 671, 686

average, 49

axiomatic definition, 14, 54 base of logarithm, 14, 15 bounds, 663

chain rule, 23 concavity, 33, 34

conditional, 16, 51, see conditional entropy

conditioning, 42 cross entropy, 55

differential, see differential entropy discrete, 14

encoded bits, 156 functions, 45 grouping, 50

independence bound, 31 infinite, 49

joint, 16, 47, see joint entropy mixing increase, 51

mixture, 46

and mutual information, 21 properties of, 42

relative, see relative entropy Renyi, 676

sum, 47 thermodynamics, 14

entropy and relative entropy, 12, 28 entropy power, xviii, 674, 675, 678, 679,

687

entropy power inequality, xx, 298, 657, 674 – 676, 678, 679, 687

entropy rate, 4, 74, 71 – 101, 114, 115, 134, 151, 156, 159, 163, 167, 168, 171, 175, 182, 221, 223, 259, 417, 419, 420, 423 – 425, 428 – 462, 613, 624, 645, 667, 669

differential, 416

English, 168, 170, 174, 175 Gaussian process, 416 Hidden Markov model, 86 Markov chain, 77

subsets, 667 envelopes, 182 Ephremides, A., 611, 699

Epimenides liar paradox, 483 equalization, 611

Equitz, W., xxiii, 699

erasure, 188, 226, 227, 232, 235, 527, 529, 594

erasure channel, 219, 235, 433

ergodic, 69, 96, 167, 168, 175, 297, 360, 443, 444, 455, 462, 557, 613, 626, 644, 646, 647, 651

ergodic process, xx, 11, 77, 168, 444, 446, 451, 453, 644

ergodic source, 428, 644 ergodic theorem, 644 ergodic theory, 11 Erkip, E., xxi, xxiii Erlang distribution, 661

error correcting code, 205 error detecting code, 211

error exponent, 4, 376, 380, 384, 385, 388, 399, 403

estimation, xviii, 255, 347, 392, 425, 508 spectrum, 415

estimator, 39, 40, 52, 255, 392, 393,

395 – 397, 401, 402, 407, 417, 500, 663 bias, 393

biased, 401

consistent in probability, 393 domination, 393

efficient, 396

unbiased, 392, 393, 395 – 397, 399, 401, 402, 407

Euclidean distance, 514 Euclidean geometry, 378 Euclidean space, 538 Euler’s constant, 153, 662 exchangeable stocks, 653

expectation, 14, 167, 281, 306, 321, 328, 393, 447, 479, 617, 645, 647, 669, 670

expected length, 104

exponential distribution, 256, 661 extension of channel, 193 extension of code, 105