356 Chapter 7 Advanced Topics

while((c=getchar())!=’\n’)

a[i++]=c;

a[i]=’\n’; encode(a,array,ARRAY_SIZE); decode(array,s); printf(“%s”,s);

}

Listing 7-4: Numeric Decode Test

Coding the alpha data

For encoding and decoding the name data to be stored in our phone book, we will use a Huffman code. We saw the decoding of a Huffman code in Chapter 5 and the decoding approach used here will be almost the same as was used there. In the discussion in Chap­ ter 5 there was no encoding and that feature must be added here. To do justice to the encoding technique, it is necessary to try to build the code to encode the type of text that a phone book represents. There is no reason to suspect that a collection of English names will contain the same character frequency as standard English text. It is necessary to understand the frequency of occurrence of each letter in the text to be encoded. With this understanding, you can write a code that as­ signs few bits to frequently occurring letters and more bits to letters that occur less frequently.

The program below reads in data, counts the occurrences of let­ ters, both upper and lower case, in a document. The occurrence of letters is sorted in order of decreasing occurrence. These data are printed out. The program calculates the average theoretical entropy, bits per character, of each character in the document and displays this number. Also included in the calculations are the space, ‘ ’, char­ acter and the new line, ‘\n’, character. These characters cause the output to be distorted, so the character ‘>’ is used to indicate a space character and a ‘<’ indicates a new line. These data will then be used to create a Huffman code used to compress the data prior to storage in the internal EEPROM.

This particular program, along with many variations, has been an exercise used for years in classes. It does demonstrate some im­ portant considerations. The shell sort was used in Chapters 2 and 5 to sort data, and here we will use it again. In this case, the data to be

360 Chapter 7 Advanced Topics

The purpose of this code is to calculate frequency of occurrence of letters in a document and provide some guidance as to how well the compression approach developed works. This program was run with a ten-page instruction manual and then with a telephone book with 200 entries. The results of these two executions are shown below.

There are 11393 characters

The theoretical average bits per character is 3.797302

Output 7-1: Calculation of entropy for the document manual.doc

The outputs shown above follow very closely the expected occurrence of letters found in the typical technical text. The bits per character should be about 4.5, but this value is distorted because the space character is included in the count, and its very frequent occurrences distort the overall averages and hence the entropy per character found in the document.

Shown below in Output 2 is a repeat of the same calculation on the contents of a phone book. Note here that occurrences of the letters and other characters are quite different from those found above. Even though the phone book used to create the table below contained only about 200 entries, these data will be used to create a Huffman code to compress the data when storing names into the microcomputer EEPROM.

Coding the Alpha Data 361

There are 2488 characters

The theoretical average bits per character is 4.392597

Output 7-2: Calculation of Entropy for Phone Book

Next, a Huffman code will be created to encode the data from the phone book. A Huffman code is built into a complete binary tree. Such a tree always has two descendents from every node unless the node is a leaf node. As such, whenever a Huffman tree is created to encode n characters, there will be 2n–1 nodes in the tree. Figure 7.1 shows an instance of such a tree. This tree encodes the data shown in Output 2 above. As with most trees, analysis, or encoding, starts at the root node at the top of the page. Whenever you traverse to the left, a code value of zero is recorded. When traversing to the right, a code value of 1 is recorded. For example, the character R will be encoded as 0010, and the character M will be 111100. This table is constructed and filled to keep the most frequently occurring letters at the top of the tree and the least frequently occurring letters at the bottom. Therefore, the number of bits for each character is inversely proportional to its frequency of occurrence. This choice for the letter codes requires less than the number of bits one would expect when using the standard 8 bits per character.

362 Chapter 7 Advanced Topics

E

' ' ' \n '

A R O I N L

T H C M S D

B

K

G J Y U P V

F W Z

X Q

Figure 7-1: Huffman Tree for Encoding the Phone Book Data

We have seen above that the minimum number of bits per character for the telephone book is 4.39. We cannot expect to reach this level, but we should expect to be significantly fewer than 8 bits per character. The code corresponding to the tree in Figure 7-1 is shown in the following table:

a0001

b0000111

c000010

d111110

e01

f0000110110

g000011000

Coding the Alpha Data 363

h000001

i1100

j000011001

k1111111

l1110

m111100

n1101

o0011

p111111001

q11111101111

r0010

s111101

t000000

u111111000

v111111010

w0000110111

x11111101110

y000011010

z1111110110

Table 7-1: Huffman Code for CompressingTelephone Book Names

The encoding routine is shown in Listing 7-6. Contained in the listing of the encoding routine is a look-up table that contains all of the codes. In this table, the first two entries correspond to a space character and a new line character. The following entries correspond to the letters in the alphabet. In other words, the third entry corresponds to the letter A and the seventh entry corresponds to the letter E. Notice that this table is defined as external, but it is labeled static so that there is no linkage to the table outside of the file encode.c.

In operation, this function receives three parameters. The first is a pointer to an array that contains the data to be encoded. This array contains a zero terminated string. The second array of unsigned integers is named array. Its length is the third passed parameter length. Encoded data are all loaded into this array. All of the local variables used by encode are straightforward. The variable bitbase is an unsigned int with its most significant bit set to one and the remainder of its bits zero. When the function is executed, the array[] is first filled with zeros. The variable i is initialized

364 Chapter 7 Advanced Topics

to zero and bit is given the value bitbase. Then the code for each character is retrieved successively. If the character read is a space, the first entry in the table is used; if it is a new line character, the second entry in the table is used; and if it is any other letter, the letter is converted to an index into the alphabet and that particular code, offset by two, is used as the code string for the input character.

static char *code[]={ “100”,“101”,”0001",”0000111",”000010",”111110", ”01",“0000110110”,”000011000",”000001",”1100", ”000011001",“1111111”,”1110",”111100",”1101", ”0011",”111111001",“11111101111”,”0010",”111101", ”000000",”111111000",“111111010”,”0000110111", ”11111101110",”000011010",“1111110110” };

#include <ctype.h>

int encode(char *a,unsigned *array,int length)

{

unsigned i,bit,bitbase=~(~0u>>1); int c=1;

char *ptr,*pa;

pa=a;

for(i=0;i<length;i++)

array[i]=0; /* initialize the array */ i=0;

bit=bitbase;

while(c!=’\n’)

{

c=*pa++; /* assumes file is not empty */ if(isalpha(c=toupper(c))||c==’ ‘||c==’\n’)

{

if(c==’ ‘) ptr=code[0];

else if(c==’\n’) ptr=code[1];

else

ptr=code[c-’A’+2];

Coding the Alpha Data 365

while(*ptr!=’\0')

{

if(*ptr++==’1')

array[i]|=bit;

bit>>=1;

if(bit==0)

{

bit=bitbase;

i++;

}

}

}

}

return ++i; /* the length of the coded array */

}

Listing 7-6: Encode Function

The program then enters a while loop that examines the contents of the code received. If the leftmost entry is a character ‘1’ a value of bit is ORed into the location array[i]. In either case, *ptr==’1’ or *ptr==’0’, the value of bit is replaced by bit shifted right by 1. Whenever bit has been shifted until its value becomes zero, it indicates that the unsigned int value pointed to by ptr has been filled and bit is reinitialized to bitbase. Also at this time i, the index into array[], is incremented to get the next character to decode.

Decoding the alpha data

The above function encodes the alpha data entered in the array s[] into a Huffman code of the same data and returns the encoded data in the array array[]. Perhaps the easiest way to test the encode routine is to execute it in conjunction with its corresponding decode routine. The decode operation essentially recreates the tree shown in Figure 7-1. Rather than a two-dimensional rendition, it must be a single-dimension list. The list will have built-in mechanisms for traversing the tree from its root node to the encoded character based on the 1 and 0 patterns in the encoded data.

366 Chapter 7 Advanced Topics

Recall in Chapter 5 the decode scheme involved intermixing jump distances in with the decoded characters in a table. There, the decode operation started at the zero entry in the table. The code being decoded was examined a bit at a time. If the code bit was zero, the table index was incremented by one. If the code bit was one, the value in that table location would be added to the table index. Whenever the table index fell to a location that contained a character, that character would be output and the index would be returned to the value zero.

That approach is fine for relatively small alphabets, as used in Chapter 5. Here, we are using the full alphabet, which makes the creation of the table above extremely complicated. Another approach was used this time. We still have a table that contains jump instructions intermixed with characters to be output. The characters to be output are each ORed with the hex value 0x80. The printable characters here are all identified with the least significant 7 bits of the character. Therefore, a test for a character is to determine if the value found in the table has a value when ANDed with 0x80.

Each numeric entry in the node table is broken into two nibbles. The left 4 bits correspond to the jump when a code 0 is found and the right 4 bits correspond to the jump when a code 1 is found. In other words, the decode operation starts at the beginning of the node table. If the first bit of the encoded data is a 0, the value found in the most significant 4 bits of the data is added to the node table index and the decoding is continued from that point in the node table. If the encoded data is a 1 the contents of the least significant 4 bits is added to the node table index. Whenever the node table index is changed, the value of that location is tested to see if the most significant bit is turned on. If so, that bit is turned off and the result is saved in output array. Otherwise, the process is repeated from that location until an output character is found. At that time, the node table index is reset to zero and the process repeated until a new line character is detected. Then the null character is put on the end of the output data and control is returned to the calling program.

One little problem with this approach: The number that contains the jump data can never have its most significant bit turned on. Therefore, the maximum jump when the encoded bit is 0 is seven. This restriction did not cause any difficulty when writing this tree. In fact, most of the time the jump caused by a 0 bit was 1 or 2. This

the table was quite easy to construct.

static const char node[]={ 0x1d,0x21,’E’|0x80,0x41,0x21,’O’|0x80,’R’|0x80,0x21, ‘A’|0x80,0x1e,0x21,’H’|0x80,’T’|0x80,0x14,0x21,’\n’|0x80, ‘‘|0x80,0x14,0x21,’N’|0x80,’I’|0x80,0x1f,’L’|0x80,0x12, ‘C’|0x80,0x21,’B’|0x80,0x14,0x12,’G’|0x80,’J’|0x80,0x12, ‘Y’|0x80,0x12,’F’|0x80,’W’|0x80,0x14,0x12,’M’|0x80, ‘S’|0x80,0x12,’D’|0x80,0x15,0x15,0x12,’U’|0x80,’P’|0x80, ‘K’|0x80,0x12,’V’|0x80,0x12,’Z’|0x80,0x12,’X’|0x80, ‘Q’ |0x80

};

int decode(unsigned M[],char *s)

{

unsigned mask,maskdo = ~(~0u>>1);

}

if((mask>>=1)==0) /* if the mask is 0, turn on MSB */

{

mask = maskdo;

i++; /* and get the next byte from message */

}

}