19. Message sources

message source is what generates the message. This is the spoken word, writing, newspapers, books, messages on radio, television, the results of the measurements presented in the form of a sequence of digits, etc. The message may come in the form of a sequence or any code characters.

We are interested in the source from a mathematical point of view, so that you can distinguish the sources of each other from what is generic entries.

From a mathematical point of view, a source of information refers to the set of possible messages with a given on this set of probability measure.

20. Messages sources various models: discrete, continuous

Distinguish between discrete and continuous sources. The difference between them is that in the case of discrete elements form a countable set, and in the continuous  uncountable number (continuum).

Discrete source is defined, if it lists all the possible messages and their probabilities are.

                 x₁,      x₂,      x₃,   ... ,  x_m                                                                                      

                 p(x₁), p(x₂), p(x₃), ... ,p(x_m),          ,

Then the entropy of the source, or the amount of information coming at an average of one message will be:

(1)

Source (1) is a model of the first, the very rough approximation. The model takes into account the statistics of the second approximation relationships between adjacent letters p p(x_j/x_i). The model takes into account the third approximation relationships between the three consecutive letters p(x_k/x_ix_j) etc.

21. Markov's source main characteristics

Markov chain of n-th order is a sequence of dependent trials, in which the conditional probability of a certain outcome in - th trial when we know the outcomes in previous n trials, independent of the earlier outcomes. In other words, if

.

In the Markov source of n-th order probability distribution of letters is not constant, but depends on what were the last letters n of the message. In other words, the last letters n identify the state of a source , in which the probability of selecting -s the first letter of the alphabet is .

The number of different possible sequences of n letters of the alphabet with the volume of care l is equal . Consequently, the number r of different states of the Markov source is finite and does not exceed .

22. Source’s information characteristics: entropy, redundancy

In the simplest case, a source of independent messages, in which the probability of choosing one or another element of the message does not depend on the previously selected items

,                                     (1.7а)

where   — the amount of source symbols;   — the probability of selection k-th element ( -th letter).

- при равновероятных и взаимонезависимых элементах сообщения:

- при неравновероятных и взаимонезависимых элементах сообщения:

- при неравновероятных и взаимозависимых элементах сообщения

A measure of redundancy shows how well (efficiently) with characters that source

The redundancy of the source - a consequence of the physical properties of the source.

The natural source of redundancy is eliminated, and if necessary, increase immunity when sending a message entered the so-called rational redundancy, allowing to detect and correct errors.

If the message would be passed through equally likely letters of the alphabet and each statistically independent, the entropy of such messages would be maximized. In fact, the real messages are constructed from the letters of the alphabet are not equally likely with the presence of the statistical relationships between the letters. Therefore, the entropy of real-Hp messages, is much smaller than the optimum messages - Ho. Suppose you want to send a message containing the amount of information equal to I.

Springs with the letter entropy equal to Hp have to spend a certain number n_р , that is . If the entropy of the source would be H0, we would have had to expend fewer letters to the transfer of the same amount of information I = n₀H₀   .

Таким образом, часть букв n_р-n_о являются как бы лишними, избыточными. Таким образом, мера удлинения реальных сообщений по сравнению с оптимально закодированными и представляет собой избыточность D.                                      

23	Information coding main objectives

1. Coding. Basic concepts and definitions

Consider the basic concepts associated with the encoding of information. To transfer the communication channel signals are converted into messages. Spacing with which messages are generated to form the primary alphabet, wherein each symbol is characterized by the probability of its occurrence in the message. Each message is uniquely corresponds to a signal representing a sequence of elementary discrete symbols called codewords. Encoding - a message conversion into a signal, i.e. convert messages into codewords. Code - a system of correspondences between the elements of messages and code combinations. The encoder - a device that performs encoding. The decoder - a device that performs the inverse operation, ie conversion in a message codeword. Alphabet - a set of possible code elements, ie elementary symbols (code symbols) X = {xi}, where i = 1, 2, ..., m. The number of code elements - m is called its base. For binary xi = {0, 1} and m = 2. A finite sequence of symbols of the alphabet is called a codeword (codeword). The number of elements in the codeword - n is called the valence (long combination). The number of different codewords (N = mn) or output of said code.

If N0 - number of source messages, the N i N0. Set of states of the code should cover a set of states of the object. Full uniform n - digit code base contains m N = mn code combinations. Such a code is called primitive.

2. Classification Codes

Codes can be classified according to various criteria:

1. The base (the number of characters in the alphabet): binary (binary m = 2) and binary (m № 2).

2. In length codewords (words):

Uniform - if all codewords have the same length;

irregular - if the length of the codeword is not constant.

3. Mode of transmission:

serial and parallel;

block - the data is first buffered and then transferred to the binary and continuous feed.

4. Immunity:

simple (primitive, full) - communicate using all possible codewords (no redundancy);

corrective (jam-resistant) - used to transmit messages, not all but only a portion of (resolved) codewords.

5. Depending on the purpose and use of conditionally include the following types of code:

Internal codes - codes is used inside the device. This machine codes as well as codes that are based on the use of positional number system (binary, decimal, binary, decimal, octal, hexadecimal, etc.). The most common code in a binary computer code that allows you to simply implement a hardware device for storing, processing and transmitting data in binary code. It provides high reliability and easy execution of operations on data in binary code. Binary data, combined in groups of 4 to form the hexadecimal code, which agrees well with the architecture of the computer, working with multiple data byte (8 bits).

24	Effective and noiseproof coding

The theory of error-correcting coding is based on the results of studies conducted by Shannon and he formulated as a theorem:

whenever performance information source smaller than the channel capacity, there is a coding method which allows to transmit all the information generated by the source communications with an arbitrarily small probability of error;

There is no encoding method allows to transmit information with arbitrarily small probability of error, if the performance of the source of messages more bandwidth.

Although the proof of this theorem, proposed by Shannon, subsequently subjected to a deeper and more rigorous mathematical representation, the idea of ​​it remained unchanged. We prove only the existence of the desired encoding method, which found the average probability of error over all the possible ways of coding and show that it can be made arbitrarily small. In this case, there is at least one way of coding, for which the probability of error is less than average.

The source generates data at Vx letters per second and the entropy of each letter in the middle of H (x), ie its performance:

H (x) = vx · H (x) [bit / sec].

In accordance with the theorem on the coding efficiency of the average number of characters per letter lsr ≥ H (x), ie in a first approximation we can get that

H (x) = vx · lsr [bit / sec].

Condition information pass through the channel H (x) ≤ C = Vk.

If interference bandwidth connection drops, ie in a binary symmetric channel, we have

Ck = Vk · [1 - H (Posh)].

Theorem on the antinoise coding requires that

Vx · lsr = H (x) ≤ Ck = Vk · [1 - H (Posh)].

where lsr - the average length of a codeword to write a single letter.

Compared with the effective code lsr increased to:

However, such a value lsr1 jam-resistant codes do not reach. This can be regarded as the theoretical limit.

Ensuring the transfer of information from a very small probability of error and quite possibly high efficiency encoding extremely long sequences of characters. In practice, the degree of reliability and efficiency is limited by two factors: the size and cost of encoding and decoding equipment and the time delay of the transmitted message. http://peredacha-informacii.ru/ currently use relatively simple coding techniques that do not implement the opportunities identified by the theory. However, the ever increasing requirements for transmission reliability and success in the technology of large-scale integrated circuits to facilitate the implementation of these goals more and more sophisticated equipment.

25	Shannon's main coding theorem

Shannon's theorem is one of the fundamental theorems of information about the transmission of signals through the communication channels in the presence of interference, resulting in distortion. Let must pass a sequence of characters that appear with certain probabilities, and there is some probability that the transmitted symbol in the transfer process will be distorted. The easiest way to reliably reconstruct the original sequence produced by, is that each transmitted symbol to repeat a large number (N) time. However, this will reduce the transmission rate of N times, ie, it will be close to zero. Sh t asserts that there exists, depending only on the probabilities considered a positive number v, such that for any number of small ε> 0, there are ways of transmitting at speeds v '(v' <v), arbitrarily close to v, enabling restore the original sequence with an error of less than ε. At the same time, when the transmission rate v ', more v, it is no longer possible. These methods use proper transmission "noise-proof" codes. The critical velocity v is determined from the ratio of Hv = C, where H - entropy source to the character C - channel capacity in binary units per second.

26	Effective codes: Shannon-Fano's code, Huffman's code, and their characteristics

Huffman coding

One of the first algorithms for efficient coding of information was offered by DA Huffman in 1952. The idea of ​​the algorithm is as follows: the probability of knowing characters in the message can describe a procedure for constructing a variable length code consisting of a number of bits. Characters are more likely to be put in line shorter codes. Huffman codes have the property of the prefix (ie, no codeword is a prefix of the other) that allow one to decode them.

Classic Huffman algorithm input is a table of frequencies of characters in the message. Further on the basis of this table is constructed Huffman coding tree (tree-H).

Characters in the input alphabet form a list of free nodes. Each sheet has a weight which can be either equal to the probability, or the number of occurrences of a compressible message.

Selected two free tree node with the lowest weights.

Created their parent with a weight equal to their total weight.

Parent added to the list of free nodes, and two of his offspring are removed from this list.

One arc extending from the parent is assigned a 1 bit, the other - bit 0.

Steps, starting with the second, repeated as long as the list of free nodes will be left only one free node. It will be considered as the root of the tree.

Let's say we have the following table of frequencies:

15 6 6 5

A B C D

This process can be represented as the construction of a tree whose root - a symbol of the sum of the probabilities of combined symbols, obtained by combining characters from the last step, his descendants n0 - the characters from the previous step, etc.

To determine the code for each of the characters included in the message, we have to go through the path from the root to the leaf corresponding to the current symbol, accumulating bits as you move along the branches of the tree (the first branch in the path corresponds to the least significant bit). The thus obtained sequence of bits of the character code is recorded in the reverse order.

For the table of characters Huffman codes are as follows.

A B C D

0 100 101 110 111

Since none of the codes received is not a prefix of another, they can be uniquely decoded by reading them from the stream. In addition, the most frequent symbol A coded message fewest bits, and the most rare symbol D - the largest.

Thus the total length of the message consisting of the characters listed in the table, will be 87 bits (an average of 2.2308 bits per character). When using the uniform coding of the total length of the message would have amounted to 117 bits (exactly 3 bits per symbol). Note that the entropy of the source, independently of the generating symbols to indicate the frequency is ~ 2.1858 bits per symbol, ie, redundancy built for such source code Huffman, understood as the difference between the average number of bits per symbol of entropy, is less than 0.05 bits per character.

Classic Huffman algorithm has some significant drawbacks. First, to restore the contents of the compressed message decoder needs to know the frequency table that was used by the encoder. The length of the compressed message is increased by the length of the frequency table to be sent ahead of the data, which can negate all the efforts to compress the message. Furthermore, the need for complete frequency statistics before actually encoding requires two passes through the post, one for model building communication (frequency table and H-tree), the other for the actual encoding. Second, the redundancy coding vanishes only in those cases where the probability of encoded symbols are inverse powers of 2. Third, for a source with entropy not exceeding 1 (for example, for a binary source), the direct application of the Huffman code is meaningless.

The algorithm for calculation codes Shannon - Fano

Code Shannon - Fano constructed using wood. The construction of the tree starts from the root. The whole set of encoded elements corresponds to the root of a tree (top tier). It is divided into two subsets, with approximately the same total probability. These two peaks correspond to a subset of the second level are connected to the root. Next, each of these subsets is divided into two subsets with roughly equal total probability. They correspond to the top of the third level. If the subset contains a single element, then it corresponds to the end-point code tree, a subset of the partition can not be. Similarly, we proceed as long until we get all the top end. Branch mark up code tree symbols 1 and 0 as in the case of Huffman code.

When building code Shannon - Fano partitioning of elements can be produced, in general, in several ways. Select partition level n may degrade in the following embodiments of the partition level (n + 1), and lead to non-optimal code in general. In other words, the optimal behavior at every step of the way does not guarantee the optimality of the entire set of actions. Therefore, the code Shannon - Fano is not optimal in a general sense, while giving optimum results in certain probability distributions. For the same probability distribution can be constructed, in general, more than one code Shannon - Fano, and they can give different results. If you build all possible codes Shannon - Fano for this probability distribution, among them will be the Huffman codes and all that is optimal codes. Example code tree.

The original characters:

A (incidence 50)

B (frequency of 39)

C (incidence 18)

D (incidence 49)

E (frequency of 35)

F (incidence 24)

The resulting code: A - 11, B - 101, C - 100, D - 00, E - 011, F - 010.

Coding Shannon - Fano is old enough compression method, and today it is of no particular interest in practice. In most cases, the length of the sequence, compressed according to the method is the sequence length compressed using Huffman coding. But for some sequences may form sub-optimal codes Shannon - Fano, so the more effective is the method of Huffman compression.