Теория информации / Cover T.M., Thomas J.A. Elements of Information Theory. 2006., 748p

Shannon lower bound, 337

Shannon’s first theorem (source coding theorem), 115

Shannon’s second theorem (channel coding theorem), 189, 192

Shannon’s third theorem (rate distortion theorem), 307

Shannon, C.E., xv, xviii, 55, 69, 100, 157, 171, 174, 182, 205, 240, 270, 299, 345, 609, 656, 687, 699, 714, 715, see also Shannon code, Shannon-Fano-Elias code, Shannon-McMiIlan-Breiman theorem

Shannon-Fano code, 158, 491, see also Shannon code

Shannon-Fano-Elias code, 127, 130, 428 Shannon-McMillan-Breiman theorem, 69,

644 – 649

Shannon-Nyquist sampling theorem, 272 Sharpe, W.F., 614, 715 Sharpe-Markowitz theory, 614

Shields, P.C., 462, 715 Shimizu, M., xxiii

Shor, P.W., 241, 691, 693, 698 Shore, J.E., 715

short selling, 181

Shtarkov, Y.M., 631, 656, 719 Shtarkov, Y.V., 715

shuffle, 84, 89 Shwartz, A., 715 side information,

and source coding, 575

side information, xvii, 12, 159, 165, 166, 180, 255, 574, 576, 580 – 583, 596, 610, 623, 652

and doubling rate, 165, 622 Siegel, P.H., 704

σ algebra, 644

signal, 1, 171, 192, 199, 234, 258,

262 – 299, 513, 517, 519, 533, 544, 561, 607

Sigurjonsson, S., xxi Silicon Dreams, 170 silver iodide crystals, 293

simplex, 348, 378, 380, 385, 386, 391, 408, 617, 618

sinc function, 271 Sleator, D., 691

Slepian, D., 272, 299, 549, 609, 715

INDEX 745

Slepian-Wolf coding, 10, 549 – 560, 575, 581, 586, 592, 593, 595, 598, 603 – 605, 608 – 610

achievability, 551 converse, 555

duality with multiple access channels, 558

slice code, 122 slice questions, 121

sliding window Lempel-Ziv, 441 Sloane, N.J.A., 707, 708, 720 smallest probable set, 64 Smolin, J., 691, 698

SNR (Signal to Noise Ratio), 273, 514, 516 Solomon, G., 214

Solomonoff, R.J., 3, 4, 507, 716 source, 103, 337

binary, 307 Gaussian, 310

source channel coding theorem, 218, 223 source channel separation, 218, 318, 344,

592, 593

source code, 103, 123, 552, 631 source coding, 60, 134, 447, 473, 511

and channel capacity, 430

with side information, 575, 595 source coding theorem, 144, 158 space-time coding, 611

Spanish, 561, 562, 606

spectral representation theorem, 315 spectrum, 271, 279, 280, 315, 415, 417,

419, 421

spectrum estimation, 415

speech, 1, 87, 101, 171, 218, 305, 416 sphere, 265, 297, 324, 675

sphere covering, 324

sphere packing, 10, 324, 325 squared error, 302, 393, 423, 683 squared error distortion, 336

St. Petersburg paradox, 181, 182 Stam, A., 674, 687, 716

state diagram, 95

state transition, 73, 465

stationary, 4, 69, 114, 168, 220, 221, 279, 297, 415 – 417, 423, 428, 444, 446, 451, 453, 455, 458, 462, 613, 625, 626, 646, 647, 651, 659, 681

stationary distribution, 73, 77 – 79, 96 stationary ergodic processes, 69

746 INDEX

stationary ergodic source, 219 stationary market, 624 stationary process, 71, 142, 644 statistic, 36, 38, 400

Kolmogorov sufficient, 496 minimal sufficient, 38 sufficient, 38

statistical mechanics, 4, 6, 11, 55, 56, 425 statistics, xvi – xix, 1, 4, 12, 13, 20, 36 – 38,

169, 347, 375, 497, 499 Steane, A., 716

Stein’s lemma, 399 stereo, 604

Stirling’s approximation, 351, 353, 405, 411, 666

stochastic process, 71, 72, 74, 75, 77, 78, 87, 88, 91, 93, 94, 97, 98, 100, 114, 166, 219, 220, 223, 279, 415, 417, 420, 423 – 425, 455, 625, 626, 646

ergodic, see ergodic process function of, 93

Gaussian, 315

without entropy rate, 75

stock, 9, 613 – 615, 619, 624, 626, 627, 629 – 634, 636, 637, 639 – 641, 652, 653

stock market, xix, 4, 9, 159, 613, 614 – 617, 619 – 622, 627, 629 – 631, 634, 636, 649, 652, 653, 655, 656

Stone, C.J., 693 stopping time, 55

Storer, J.A., 441, 459, 716

strategy, 160, 163, 164, 166, 178, 391, 392, 487

investment, 620

strong converse, 208, 240 strongly jointly typical, 327, 328

strongly typical, 326, 327, 357, 579, 580 strongly typical set, 342, 357

Stuart, A., 705

Student’s t distribution, 662 subfair odds, 164 submatrix, 680

subset, xx, 8, 66, 71, 183, 192, 211, 222, 319, 347, 520, 644, 657, 668 – 670

subset inequalities, 668 – 671 subsets, 505

entropy rate, 667 subspace, 211

sufficient statistic, 13, 36, 37, 38, 44, 56, 209, 497 – 499

minimal, 38, 56 suffix code, 145 superfair odds, 164, 180 supermartingale, 625

superposition coding, 609

support set, 29, 243, 244, 249, 251, 252, 256, 409, 676 – 678

surface area, 247 Sutivong, A., xxi Sweetkind-Singer, J., xxi symbol, 103

symmetric channel, 187, 190 synchronization, 94

Szasz’s inequality, 680 Szego,¨ G., 703 Szpankowski, W., 716, 708

Szymanski, T.G., 441, 459, 716

Tanabe, M., 713

Tang, D.L., 716

Tarjan, R., 691

TDMA (Time Division Multiple Access), 547, 548

Telatar, I.E., 716 telegraph, 441

telephone, 261, 270, 273, 274 channel capacity, 273

Teletar, E., 611, 716 temperature, 409, 411

ternary, 119, 145, 157, 239, 439, 504, 527 ternary alphabet, 349

ternary channel, 239 ternary code, 145, 152, 157 text, 428

thermodynamics, 1, 4, see also second law of thermodynamics

Thitimajshima, P., 692

Thomas, J.A., xxi, 687, 694, 695, 698, 700, 716

Thomasian, A.J., 692 Tibshirani, R., 699 time symmetry, 100

timesharing, 527, 532 – 534, 538, 562, 598, 600

Tjalkens, T.J., 716, 719 Toeplitz matrix, 416, 681 Tornay, S.C., 716

trace, 279, 547

transition matrix, 77, 92, 98, 144, 190 doubly stochastic, 83, 88, 190

transmitter, 266, 294, 296, 299, 515, 517 – 519, 546, 573, 574, 588, 601, 611

Treasury bonds, 614 tree,

code, 107 Huffman, 124 random, 89

tree structured Lempel-Ziv, 441, 442 triangle inequality, 20, 369 triangular distribution, 662

trigram model, 171 Trofimov, V.K., 706 Trott, M., xxiii Tseng, C.W., xxiii Tsoucas, P., 700 Tsybakov, B.S., 716 Tunstall, B.P., 716 Tunstall coding, 460

turbo codes, 3, 205, 215 Turing, A., 465

Turing machine, xix, 465, 466 Tusnady,´ G., 332, 335, 346, 697 Tuttle, D.L., 182, 707

TV, 509, 560, 561 twin, 171

two envelope problem, 179 two level signalling, 262 two stage description, 496

two-way channel, 510, 519, 594, 602, 609 type, 342, 347, 348 – 350, 353 – 356, 358,

360, 361, 366, 367, 371, 373, 374, 378, 391, 408, 474, 490, 499, 570, 666

type class, 348 – 351, 353 – 356, 666 typewriter, 74, 192, 224, 235, 482

typical sequence, 11, 12, 57, 63, 245, 381, 522

typical set, 57, 59, 61, 62, 64, 68, 77, 196, 220, 227, 245, 247, 258, 319, 321, 356, 381, 382, 384, 524, 551

conditionally typical, 341 data compression, 60 distortion typical, 319 properties, 59, 64, 245 strongly typical, 326 volume, 245

INDEX 747

Ullman, J.D., 704

uncertainty, 5, 6, 11, 13, 15, 20, 22, 24, 31, 53, 83, 89, 170, 517, 518, 593

Ungerboeck, G., 716

uniform distribution, 5, 30, 43, 83, 88, 148, 163, 190, 195, 202, 204, 209, 210, 228, 268, 338, 375, 408, 411, 412, 434, 436, 437, 553, 662, 663

uniform fair odds, 163, 166, 176, 626 uniquely decodable code, see code,

uniquely decodable universal computer, 465, 501

universal data compression, 333, 457 universal gambling, 487, 488, 507 universal portfolios, 629 – 643, 651

finite horizon, 631 horizon free, 638

universal probability, 481, 487, 489 – 491, 502, 503, 507, 686

universal probability mass function, 481 universal source, 358

universal source code, 357, 360, 461 universal source coding, xv, 355, 427 – 462

error exponent, 400

universal Turing machine, 465, 480 Unix, 443

Urbanke, R., 702, 712

V.90, 273

V’yugin, V.V., 507, 718 Vajda, I., 716

Valiant, L.G., 717

Van Campenhout, J.M., 717

Van der Meulen, E., xxiii, 609 – 611, 699, 702, 717

Van Trees, H.L., 716 Vapnik, V.N., 717

variable-to-fixed length coding, 460 variance, xx, 36, 37, 64, 65, 255, 261, 265,

272, 292, 315, 325, 389, 393, 394, 396, 513, 516, 520, 544, 614, 655, 681, 685, see also covariance matrix

variational distance, 370 vector quantization, 303, 306 Venkata, R., xxi

Venkatesh, S.S., 707

Venn diagram, 23, 47, 50, 213

Verdu, S., 690, 698, 703, 704, 714, 717, 718

748 INDEX

Verdugo Lazo, A.C.G., 662, 718 video, 218

video compression, xv Vidyasagar, M., 718 Visweswariah, K., 698, 718 vitamins, xx

Vitanyi, P., 508, 707 Viterbi, A.J., 718 Vitter, J.S., 718 vocabulary, 561

volume, 67, 149, 244 – 247, 249, 265, 324, 675, 676, 679

Von Neumann, J., 11 Voronoi partition, 303

waiting time, 99 Wald, A., 718 Wallace, M.S., 697 Wallmeier, H.M., 692 Washington, 551 Watanabe, Y., xxiii

water-filling, 164, 177, 277, 279, 282, 289, 315

waveform, 270, 305, 519 weak, 342

weakly typical, see typical wealth, 175

wealth relative, 613, 619 weather, 470, 471, 550, 551 Weaver, W.W., 715

web search, xv

Wei, V.K.W., 718, 691 Weibull distribution, 662 Weinberger, M.J., 711, 718 Weiner, N., 718

Weiss, A., 715

Weiss, B., 462, 710, 715 Welch, T.A., 462, 718 Wheeler, D.J., 462, 693

white noise, 270, 272, 280, see also Gaussian channel

Whiting, P.A., 702 Wiener, N., 718 Wiesner, S.J., 691 Wilcox, H.J., 718

Willems, F.M.J., xxi, 461, 594, 609, 716, 718, 719

window, 442 wine, 153

wireless, xv, 215, 611 Witsenhausen, H.S., 719 Witten, I.H., 691, 719

Wolf, J.K., 549, 593, 609, 700, 704, 715 Wolfowitz, J., 240, 408, 719

Woodward, P.M., 719

Wootters, W.K., 691 World Series, 48 Wozencraft, J.M., 666, 719 Wright, E.V., 168

wrong distribution, 115

Wyner, A.D., xxiii, 299, 462, 581, 586, 610, 703, 719, 720

Wyner, A.J., 720

Yaglom, A.M., 702 Yamamoto, H., xxiii Yang, E.-H., 690, 720 Yao, A.C., 705

Yard, J., xxi

Yeung, R.W., xxi, xxiii, 610, 692, 720 Yockey, H.P., 720

Young’s inequality, 676, 677 Yu, Bin, 691

Yule-Walker equations, 418, 419

Zeevi, A., xxi Zeger, K., 708 Zeitouni, O., 698

zero-error, 205, 206, 210, 226, 301 zero-error capacity, 210, 226 zero-sum game, 131

Zhang, Z., 609, 690, 712, 720

Ziv, J., 428, 442, 462, 581, 586, 610, 703, 707, 711, 719 – 721, see

also Lempel-Ziv coding

Ziv’s inequality, 449, 450, 453, 455, 456

Zurek, W.H., 507, 721 Zvonkin, A.K., 507, 707