BookBody

declaration sequence. This, and efficiency considerations lead to a second option: to write a special purpose parser for the grammar at hand. Many of these special purpose parsers have been written, and most of them use an implementation technique called recursive descent. We will assume that the reader has some programming experience, and knows about procedures and recursion. If not, this section can be skipped. It does not describe a different parsing method, but merely an implementation technique that is often used in hand-written parsers and also in some machine-generated parsers.

6.6.1 A naive approach

As a first approach, we regard a grammar rule as a procedure for recognizing its lefthand side. The rule

S -> aB | bA

is regarded as a procedure to recognize an S. This procedure then states something like the following:

S succeeds if

a succeeds and then B succeeds or else

b succeeds and then A succeeds

This does not differ much from the grammar rule, but it does not look like a piece of Pascal or C either. Like a cookbook recipe that usually does not tell us that we must peel the potatoes, let alone how to do that, the procedure is incomplete.

There are several bits of information that we must maintain when carrying out such a procedure. First, there is the notion of a “current position” in the rule. This current position indicates what must be tried next. When we implement rules as procedures, this current position is maintained automatically, by the program counter, which tells us where we are within a procedure. Next, there is the input sentence itself. When implementing a backtracking parser, we usually keep the input sentence in a global array, with one element for each symbol in the sentence. The array must be global, because it contains information that must be accessible equally easily from all procedures. Then, there is the notion of a current position in the input sentence. When the current position in the rule indicates a terminal symbol, and this symbol corresponds to the symbol at the current position in the input sentence, both current positions will be advanced one position. The current position in the input sentence is also global information. We will therefore maintain this position in a global variable, of a type that is suitable for indexing the array containing the input sentence. Also, when starting a rule we must remember the current position in the input sentence, because we need it for the “or else” clauses. These must all be started at the same position in the input sentence. For instance, starting with the rule for S of grammar 6.1, suppose that the a matches the symbol at the current position of the input sentence. The current position is advanced and then B is tried. For B, we have a rule similar to that of S. Now suppose that B fails. We then have to try the next choice for S, and backup the position in the input sentence to what it was when we started the rule for S. This is backtracking, just as we have seen it earlier.

All this tells us how to deal with one rule. However, usually we are dealing with

Now, we are at the end of a choice for D. This means that it succeeds, and we remove this entry from the list of active rules, after updating the current positions in the entry above. Next, it is C’s turn:

Now, the c succeeds, so the C succeeds, and then the S also succeeds.

Now, the # also succeeds, and thus S’ succeeds, resulting in:

The “Parse” part now represents a left-most derivation of the sentence:

S’ -> S# -> DC# -> abC# -> abc#.

This method is called recursive descent. Descent, because it operates top-down, and recursive, because each non-terminal is implemented as a procedure that can directly or indirectly (through other procedures) invoke itself. It should be stressed that “recursive descent” is merely an implementation issue, albeit an important one. It should also be stressed that the parser described above is a backtracking parser, independent of the implementation method used. Backtracking is a property of the parser, not of the implementation.

The backtracking method developed above is aesthetically pleasing, because we in fact use the grammar itself as a program (or we transform the grammar rules into procedures, which can be done mechanically). There is only one problem: the recursive descent method, as described above, does not always work! We already know that it does not work for left-recursive grammars, but the problem is worse than that. For

instance, aabc and abcc are sentences that are not recognized, but should be. Parsing of the aabc sentence gets stuck after the first a, and parsing of the abcc sentence gets stuck after the first c. Yet, aabc can be derived as follows:

S -> AB -> aAB -> aaB -> aabc,

and abcc can be derived with

S -> DC -> abC -> abcC -> abcc.

So, let us examine why our method fails. A little investigation shows that we never try the A->aA choice when parsing aabc, because the A->a choice succeeds. Such a problem arises whenever more than one right-hand side can succeed, and this is the case whenever a right-hand side can derive a prefix of a string derivable from another right-hand side of the same non-terminal. The method developed so far is too optimistic, in that it assumes that if a choice succeeds, it must be the right choice. It does not allow us to backtrack over such a choice, when it was the wrong one. This is a particularly serious problem if the grammar has ε-rules, because ε-rules always succeed. Another consequence of being unable to backup over a succeeding choice is that it does not allow us to get all parses when there is more than one (this is possible for ambiguous grammars). Improvement is certainly needed here. Our criterion for determining whether a choice is the right one clearly is wrong. Looking back at the backtracking parser of the beginning of this section, we see that that parser does not have this problem, because it does not consider choices independently of their context. One can only decide that a choice is the right one if taking it results in a successful parse; even if the choice ultimately succeeds, we have to try the other choices as well if we want all parses. In the next section, we will develop a recursive-descent parser that solves all the problems mentioned above. Meanwhile, the method above only works for grammars that are prefix-free. A non-terminal A is prefix-free if A →* x and A →* xy, where x and y are strings of terminal symbols, implies that y = ε. A grammar is called prefix-free if all its non-terminals are prefix-free.

6.6.2 Exhaustive backtracking recursive descent

In the previous section we saw that we have to be careful not to accept a choice too early; it can only be accepted when it leads to a successful parse. Now this demand is difficult to express in a recursive-descent parser; how do we obtain a procedure that tells us whether a choice leads to a successful parse? In principle, there are infinitely many of these procedures, depending on the sentential form (the prediction) that must derive the rest of the input. We cannot just write them all. However, at any point during the parsing process we are dealing with only one such sentential form: the current prediction, so we could try to build a parsing procedure for this sentential form dynamically, during parsing. Many programming languages offer a useful facility for this purpose: procedure parameters. One procedure can accept a procedure as parameter, and call it, or pass it on to another procedure, or whatever other things one does with procedures. Some languages (for instance Pascal) require these procedures to be named, that is, the actual parameter must be declared as a procedure; other languages, like Algol 68, allow a procedure body for an actual parameter.

Let us see how we can write a parsing procedure for a symbol X, given that it is

passed a procedure, which we will call tail, that parses the rest of the sentence (the part that follows the X). This is the approach taken for all non-terminals, and, for the time being, for terminals as well.

The parsing procedure for a terminal symbol a is easy: it matches the current input symbol with a; if it succeeds, it advances the input position, and calls the tail parameter; then, when tail returns, it restores the input position and returns.

Obviously, the parsing procedure for a non-terminal A is more complicated. It depends on the type of grammar rule we have for A. The simplest case is A →ε. This is implemented as a call to tail. The next simple case is A →X, where X is either a terminal or a non-terminal symbol. To deal with this case, we must remember that we assume that we have a parsing procedure for X, so the implementation of this case consists of a call to X, with the tail parameter. The next case is A →XY, with X and Y symbols. The procedure for X expects a procedure for “what comes after the X” as parameter. Here, this parameter procedure is built using the Y and the tail procedures: we create a new procedure out of these two. This, by itself, is a simple procedure: it calls Y, with tail as parameter. If we call this procedure Y_tail, we can implement A by calling X with Y_tail as parameter.† And finally, if the right-hand side contains more than two symbols, this technique has to be repeated: for a rule A →X 1 X 2 . . . Xn we create a procedure for X 2 . . . Xn and tail using a procedure for X 3 . . . Xn and tail, and so on. Finally, if we have a choice, that is, we have A →α | β, the parsing procedure for A has two parts: one part for α, followed by a call to tail, and another part for β, followed by a call to tail. We have already seen how to implement these parts. If we only want one parsing, all parsing procedures may be implemented as functions that return either false or true, reflecting whether they result in a successful parse; the part for β is then only started if the part for α, followed by tail, fails. If we want all parses, we have to try both choices.

Applying this technique to all grammar rules almost results in a parser. Only, we don’t have a starting point yet; this is easily obtained: we just call the procedure for the start-symbol, with the procedure for recognizing the end-marker as parameter. This end-marker procedure is probably a bit different from the others, because this is the procedure where we finally find out whether a parsing attempt succeeds.

Figure 6.12 presents a fully backtracking recursive-descent parser for the grammar of Figure 6.6, written in Pascal. The program has a mechanism to remember the rules used, so these can be printed for each successful parse. Figure 6.13 presents a sample session with this program.

{$C+: distinguish between upper and lower case } program parse(input, output);

{This is an exhaustive backtracking recursive-descent parser that will correctly parse according to the grammar

S -> D C | A B A -> a | a A

B -> b c | b B c

† For some programming languages this is difficult. The problem is that tail must be accessible from Y_tail. Therefore, Y_tail should be a local procedure within the procedure for A. But, some languages do not allow for local procedures (for instance C), and others do not allow local procedures to be passed as parameters (like Modula-2). Some extensive trickery is required for these languages, but this is beyond the scope of this book.

D -> a b | a D b C -> c | c C

It implements proper backtracking by only checking one symbol at a time and passing the rest of the alternative as a parameter for evaluation on a lower level. A more naive backtracking parser will not accept e.g. aabc.

}

{ administration of rules used }

procedure pushrule (s: str); begin rp := rp + 1; rules[rp] := s end; procedure poprule; begin rp := rp - 1 end;

begin if text[tp] = ’a’ then begin tp := tp + 1; tail; tp := tp - 1 end end;

procedure b(procedure tail); { recognize a ’b’ and call tail }

begin if text[tp] = ’b’ then begin tp := tp + 1; tail; tp := tp - 1 end end;

procedure c(procedure tail); { recognize a ’c’ and call tail }

begin if text[tp] = ’c’ then begin tp := tp + 1; tail; tp := tp - 1 end end;

function readline: boolean; begin

write(’> ’); length := 1; if not eof then

begin while not eoln do begin

read(text[length]); length := length + 1;

end;

readln; readline := true

end

else readline := false;

end;

procedure parser;

begin text[length] := ’#’; tp := 1; rp := 0; S(endmark) end;

begin while readline do parser end.

Figure 6.12 A parser for the grammar of Figure 6.6

6.7 DEFINITE CLAUSE GRAMMARS

In the previous sections, we have seen how to create parsers that retain much of the original structure of the grammar. The programming language Prolog allows us to take this even one step further. Prolog has its foundations in logic. The programmer declares some facts about objects and their relationships, and asks questions about these. The Prolog system uses a built-in search and backtrack mechanism to answer

>aabc Derivation:

S -> AB A -> aA A -> a B -> bc

>abcc Derivation:

S -> DC D -> ab C -> cC C -> c

>abc Derivation:

S -> DC D -> ab C -> c

Derivation: S -> AB A -> a B -> bc

Figure 6.13 A session with the program of Figure 6.12

the questions with “yes” or “no”. For instance, if we have told the Prolog system about the fact that a table and a chair are pieces of furniture, as follows:

furniture(table). furniture(chair).

and we then ask if a bread is a piece of furniture:

| ?- furniture(bread).

the answer will be “no”, but the answer to the question

| ?- furniture(table).

will, of course, be “yes”. We can also use variables, which can be either instantiated (have a value), or not. Variables start with a capital letter or an underscore (_). We can use them for instance as follows:

| ?- furniture(X).

This is asking for an instantiation of the variable X. The Prolog system will search for a possible instantiation and respond:

X = table

We can then either stop by typing a RETURN, or continue searching by typing a