Cooper K.Engineering a compiler

The Algorithm The algorithm for constructing the canonical collection of sets of lr(1) items uses closure and goto to derive the set S of all parser states reachable from s0.

CC0 ← closure([S →•S, EOF]) while(new sets are still being added)

for Sj S and x (T N T )

if Goto(Sj , x) is a new set, add it to S and record the transition

It begins by initializing S to contain s0. Next, it systematically extends S by looking for any transition from a state in S to a state outside S. It does this constructively, by building all of the possible new states and checking them for membership in S.

Like the other computations in the construction, this is a fixed-point computation. The canonical collection, S, is a subset of 2items. Each iteration of the

while loop is monotonic; it can only add sets to S. Since S can grow no bigger than 2items, the computation must halt. As in closure, a worklist implemen-

tation can avoid much of the work implied by the statement of the algorithm. Each trip through the while loop need only consider sets sj added during the previous iteration. To further speed the process, it can inspect the items in sj and only compute goto(sj , x) for symbols that appear immediately after the

• in some item in sj . Taken together, these improvements should significantly decrease the amount of work required by the computation.

For SN, the computation proceeds as follows:

CC0 is computed as closure ([Goal → • SheepNoise, EOF]):

[Goal → • SheepNoise, EOF] [SheepNoise → • SheepNoise baa, EOF] [SheepNoise → • SheepNoise baa, baa] [SheepNoise → • baa, EOF] [SheepNoise → • baa, baa]

The first iteration produces two sets:

goto(CC0,SheepNoise) is CC1: [Goal → SheepNoise •, EOF]

[SheepNoise → SheepNoise • baa, EOF] [SheepNoise → SheepNoise • baa, baa]

goto(CC0,baa) is CC2: [SheepNoise → baa •, EOF] [SheepNoise → baa •, baa]

The second iteration produces one more set:

goto(CC1,baa) is CC3: [SheepNoise → SheepNoise baa •, EOF] [SheepNoise → SheepNoise baa •, baa]

The final iteration produces no additional sets, so the computation terminates.

Filling in the Tables Given S, the canonical collection of sets of lr(1) items, the parser generator fills in the Action and Goto tables by iterating through S and examining the items in each set sj S. Each set sj becomes a parser state. Its items generate the non-empty elements of one column of Action; the corresponding transitions recorded during construction of S specify the nonempty elements of Goto. Three cases generate entries in the Action table:

si S

item i si

if i is [α → β • aγ, b] and goto(si , a) = sj , a T then set Action[i,a] to “shift j”

else if i is [α → β•, a]

then set Action[i,a] to “reduce α → β” else if i is [S → S•, EOF]

then set Action[i,EOF] to “accept”

n N T

If goto(si, A) = sj

then set Goto[i, A] to j

Figure 3.12: lr(1) table-filling algorithm

1.An item of the form [α→β •bγ, a] indicates that encountering the terminal symbol b would be a valid next step toward discovering the non-terminal α. Thus, it generates a shift item on b in the current state. The next state for the recognizer is the state generated by computing goto on the current state with the terminal b. Either β or γ can be .

2.An item of the form [α→β•, a] indicates that the parser has recognized a β, and if the lookahead is a, then the item is a handle. Thus, it generates a reduce item for the production α→β on a in the current state.

3.The item [S →S•, eof] is unique. It indicates the accepting state for the parser; the parser has recognized an input stream that reduces to the goal symbol. If the lookahead symbol is eof, then it satisfies the other acceptance criterion—it has consumed all the input. Thus, this item generates an accept action on eof in the current state.

The code in Figure 3.12 makes this concrete. For an lr(1) grammar, these items should uniquely define the non-error entries in the Action and Goto tables.

Notice that the table-filling algorithm only essentially ignores items where the • precedes a non-terminal symbol. Shift actions are generated when • precedes a terminal. Reduce and accept actions are generated when • is at the right end of the production. What if si contains an item [α→β • γδ, a], where γ N T ? While this item does not generate any table entries itself, the items that closure derives from [α→β •γδ, a] must include some of the form [ν→•b, c], with b T . Thus, one e ect of closure is to directly instantiate the First set of γ into si. It chases down through the grammar until it finds each member of First(γ) and puts the appropriate items into si to generate shift items for every x First(γ).

For our continuing example, the table-filling algorithm produces these two tables:

At this point, the tables can be used with the skeleton parser in Figure 3.8 to create an lr(1) parser for SN.

Errors in the Process If the grammar is not lr(1), the construction will attempt to multiply define one or more entries in the Action table. Two kinds of conflicts occur:

shift/reduce This conflict arises when some state si contains a pair of items [α→β • aγ, b] and [δ→ν•, a]. The first item implies a shift on a, while the second implies a reduction by δ→[]ν. Clearly, the parser cannot do both. In general, this conflict arises from an ambiguity like the if-then-else ambiguity. (See Section 3.2.3.)

reduce/reduce This conflict arises when some state si contains both [α→γ•, a] and [β→γ•, a]. The former implies that the parser should reduce γ to α, while the latter implies that it should reduce the same γ to β. In general, this conflict arises when the grammar contains two productions that have the same right-hand side, di erent left-hand sides, and allows them both to occur in the same place. (See Section 3.6.1.)

Typically, when a parser generator encounters one of these conflicts, it reports the error to the user and fails. The compiler-writer needs to resolve the ambiguity, as discussed elsewhere in this chapter, and try again.

The parser-generator can resolve a shift-reduce conflict in favor of shifting; this causes the parser to always favor the longer production over the shorter one. A better resolution, however, it to disambiguate the grammar.

Detailed Example To make this discussion more concrete, consider the classic expression grammar, augmented with a production Goal→Expr.

1.Goal → Expr

2. Expr → Expr + Term

3.| Expr − Term

6.| Term ÷ Factor

9.| Num

Notice the productions have been renumbered. (Production numbers show up in “reduce” entries in the Action table.)

The First sets for the augmented grammar are as follows:

The initial step in constructing the Canonical Collection of Sets of lr(1) Items forms an initial item, [Goal → • Expr,EOF] and takes its closure to produce the first set.

CC0: [Goal → • Expr,EOF], [Expr → • Expr + Term,{EOF,+,-}],

[Expr → • Expr - Term,{EOF,+,-}], [Expr → • Term,{EOF,+,-}], [Term → • Term × Factor,{EOF,+,-,×,÷}],

[Term → • Term ÷ Factor,{EOF,+,-,×,÷}],

[Term → • Factor,{EOF,+,-,×,÷}], [Factor → • ( Expr ),{EOF,+,-,×,÷}], [Factor → • Num,{EOF,+,-,×,÷}], [Factor → • Id,{EOF,+,-,×,÷}],

The first iteration computes goto on cc0 and each symbol in the grammar. It produces six new sets, designating them cc1 through cc6.

CC1: [Goal → Expr •,EOF], [Expr → Expr • + Term,{EOF,+,-}], [Expr → Expr • - Term,{EOF,+,-}],

CC2: [Expr → Term •,{EOF,+,-}], [Term → Term • × Factor,{EOF,+,-,×,÷}], [Term → Term • ÷ Factor,{EOF,+,-,×,÷}],

CC3: [Term → Factor •,{EOF,+,-,×,÷}],

CC4: [Expr → • Expr + Term,{+,-,)}], [Expr → • Expr - Term,{+,-,)}], [Expr → • Term,{+,-,)}], [Term → • Term × Factor,{+,-,×,÷,)}],

[Term → • Term ÷ Factor,{+,-,×,÷,)}], [Term → • Factor,{+,-,×,÷,)}], [Factor → • ( Expr ),{+,-,×,÷,)}], [Factor → ( • Expr ),{EOF,+,-,×,÷}], [Factor → • Num,{+,-,×,÷,)}], [Factor → • Id,{+,-,×,÷,)}],

CC5: [Factor → Num •,{EOF,+,-,×,÷}],

CC6: [Factor → Id •,{EOF,+,-,×,÷}],

Iteration two examines sets from cc1 through cc6. This produces new sets labelled cc7 through cc16.

CC7: [Expr → Expr + • Term,{EOF,+,-}],

[Term → • Term × Factor,{EOF,+,-,×,÷}],

[Term → • Term ÷ Factor,{EOF,+,-,×,÷}],

[Term → • Factor,{EOF,+,-,×,÷}], [Factor → • ( Expr ),{EOF,+,-,×,÷}], [Factor → • Num,{EOF,+,-,×,÷}], [Factor → • Id,{EOF,+,-,×,÷}],

CC8: [Expr → Expr - • Term,{EOF,+,-}],

[Term → • Term × Factor,{EOF,+,-,×,÷}],

[Term → • Term ÷ Factor,{EOF,+,-,×,÷}],

[Term → • Factor,{EOF,+,-,×,÷}], [Factor → • ( Expr ),{EOF,+,-,×,÷}], [Factor → • Num,{EOF,+,-,×,÷}], [Factor → • Id,{EOF,+,-,×,÷}],

CC9: [Term → Term × • Factor,{EOF,+,-,×,÷}],

[Factor → • ( Expr ),{EOF,+,-,×,÷}], [Factor → • Num,{EOF,+,-,×,÷}], [Factor → • Id,{EOF,+,-,×,÷}],

CC10: [Term → Term ÷ • Factor,{EOF,+,-,×,÷}],

[Factor → • ( Expr ),{EOF,+,-,×,÷}], [Factor → • Num,{EOF,+,-,×,÷}], [Factor → • Id,{EOF,+,-,×,÷}],

CC11: [Expr → Expr • + Term,{+,-,)}], [Expr → Expr • - Term,{+,-,)}], [Factor → ( Expr • ),{EOF,+,-,×,÷}],

CC12: [Expr → Term •,{+,-,)}], [Term → Term • × Factor,{+,-,×,÷,)}], [Term → Term • ÷ Factor,{+,-,×,÷,)}],

CC13: [Term → Factor •,{+,-,×,÷,)}],

CC14: [Expr → • Expr + Term,{+,-,)}], [Expr → • Expr - Term,{+,-,)}], [Expr → • Term,{+,-,)}], [Term → • Term × Factor,{+,-,×,÷,)}],

[Term → • Term ÷ Factor,{+,-,×,÷,)}], [Term → • Factor,{+,-,×,÷,)}], [Factor → • ( Expr ),{+,-,×,÷,)}], [Factor → ( • Expr ),{+,-,×,÷,)}], [Factor → • Num,{+,-,×,÷,)}], [Factor → • Id,{+,-,×,÷,)}],

CC15: [Factor → Num •,{+,-,×,÷,)}],

CC16: [Factor → Id •,{+,-,×,÷,)}],

Iteration three processes sets from cc7 through cc16. This adds sets cc16 through cc26 to the Canonical Collection.

CC17: [Expr → Expr + Term •,{EOF,+,-}],

[Term → Term • × Factor,{EOF,+,-,×,÷}],

[Term → Term • ÷ Factor,{EOF,+,-,×,÷}],

CC18: [Expr → Expr - Term •,{EOF,+,-}],

[Term → Term • × Factor,{EOF,+,-,×,÷}],

[Term → Term • ÷ Factor,{EOF,+,-,×,÷}],

CC19: [Term → Term × Factor •,{EOF,+,-,×,÷}],

CC20: [Term → Term ÷ Factor •,{EOF,+,-,×,÷}],

CC21: [Expr → Expr + • Term,{+,-,)}], [Term → • Term × Factor,{+,-,×,÷,)}], [Term → • Term ÷ Factor,{+,-,×,÷,)}], [Term → • Factor,{+,-,×,÷,)}], [Factor → • ( Expr ),{+,-,×,÷,)}], [Factor → • Num,{+,-,×,÷,)}],

[Factor → • Id,{+,-,×,÷,)}],

CC22: [Expr → Expr - • Term,{+,-,)}], [Term → • Term × Factor,{+,-,×,÷,)}], [Term → • Term ÷ Factor,{+,-,×,÷,)}], [Term → • Factor,{+,-,×,÷,)}], [Factor → • ( Expr ),{+,-,×,÷,)}], [Factor → • Num,{+,-,×,÷,)}], [Factor → • Id,{+,-,×,÷,)}],

CC23: [Factor → ( Expr ) •,{EOF,+,-,×,÷}],

CC24: [Term → Term × • Factor,{+,-,×,÷,)}],

[Factor → • ( Expr ),{+,-,×,÷,)}], [Factor → • Num,{+,-,×,÷,)}], [Factor → • Id,{+,-,×,÷,)}],

CC25: [Term → Term ÷ • Factor,{+,-,×,÷,)}],

[Factor → • ( Expr ),{+,-,×,÷,)}], [Factor → • Num,{+,-,×,÷,)}], [Factor → • Id,{+,-,×,÷,)}],

CC26: [Expr → Expr • + Term,{+,-,)}], [Expr → Expr • - Term,{+,-,)}], [Factor → ( Expr • ),{+,-,×,÷,)}],

Iteration four looks at sets cc17 through cc26. This adds five more sets to the Canonical Collection, labelled cc27 through cc31.

CC27: [Expr → Expr + Term •,{+,-,)}], [Term → Term • × Factor,{+,-,×,÷,)}], [Term → Term • ÷ Factor,{+,-,×,÷,)}],

CC28: [Expr → Expr - Term •,{+,-,)}], [Term → Term • × Factor,{+,-,×,÷,)}], [Term → Term • ÷ Factor,{+,-,×,÷,)}],

CC29: [Term → Term × Factor •,{+,-,×,÷,)}],

CC30: [Term → Term ÷ Factor •,{+,-,×,÷,)}],

CC31: [Factor → ( Expr ) •,{+,-,×,÷,)}],

Iteration five finds that every set it examines is already in the Canonical Collection, so the algorithm has reached its fixed point and it halts. Applying the table construction algorithm from Figure 3.12 produces the Action table shown in Figure 3.13 and the Goto table shown in Figure 3.14.

3.5.3Shrinking the Action and Goto Tables

As Figures 3.13 and 3.14 show, the lr(1) tables generated for relatively small grammars can be quite large. Many techniques exist for shrinking these tables. This section describes three approaches to reducing table size.

Combining Rows or Columns If the table generator can find two rows, or two columns, that are Identical, it can combine them. In Figure 3.13, the rows for states zero and seven through ten are identical, as are rows 4, 14, 21, 22, 24, and 25. The table generator can implement each of these sets once, and remap the states accordingly. This would remove five rows from the table, reducing its size by roughly fifteen percent. To use this table, the skeleton parser needs a mapping from a parser state to a row index in the Action table. The table generator can combine identical columns in the analogous way. A separate

Figure 3.14: Goto table for the classic expression grammar

inspection of the Goto table will yield a di erent set of state combinations—in particular, all of the rows containing only zeros should condense to a single row.

In some cases, the table generator can discover two rows or two columns that di er only in cases where one of the two has an “error” entry (denoted by a blank in our figures). In Figure 3.13, the column for EOF and for Num di er only where one or the other has a blank. Combining these columns produces a table that has the same behavior on correct inputs. The error behavior of the parser will change; several authors have published studies that show when such columns can be combined without adversely a ecting the parser’s ability to detect errors.

Combining rows and columns produces a direct reduction in table size. If this space reduction adds an extra indirection to every table access, the cost of those memory operations must trade o directly against the savings in memory. The table generator could also use other techniques for representing sparse matrices—again, the implementor must consider the tradeo of memory size against any increase in access costs.

Using Other Construction Algorithms Several other algorithms for constructing lr-style parsers exist. Among these techniques are the slr(1) construction, for simple lr(1), and the lalr(1) construction for lookahead lr(1). Both of these constructions produce smaller tables than the canonical lr(1) algorithm.

The slr(1) algorithm accepts a smaller class of grammars than the canonical lr(1) construction. These grammars are restricted so that the lookahead

symbols in the lr(1) items are not needed. The algorithm uses a set, called the follow set, to distinguish between cases where the parser should shift and those where it should reduce. (The follow set contains all of the terminals that can appear immediately after some non-terminal α in the input. The algorithm for constructing follow sets is similar to the one for building first sets.) In practice, this mechanism is powerful enough to resolve most grammars of practical interest. Because the algorithm uses lr(0 items, it constructs a smaller canonical collection and its table has correspondingly fewer rows.

The lalr(1) algorithm capitalizes on the observation that some items in the set representing a state are critical, and the remaining ones can be derived from the critical items. The table construction item only represents these critical items; again, this produces a smaller canonical collection. The table entries corresponding to non-critical items can be synthesized late in the process.

The lr(1) construction presented earlier in the chapter is the most general of these table construction algorithms. It produces the largest tables, but accepts the largest class of grammars. With appropriate table reduction techniques, the lr(1) tables can approximate the size of those produced by the more limited techniques. However, in a mildly counter-intuitive result, any language that has an lr(1) grammar also has an lalr(1) grammar and an slr(1) grammar. The grammars for these more restrictive forms will be shaped in a way that allows their respective construction algorithms to resolve the di erence between situations where the parser should shift and those where it should reduce.

Shrinking the Grammar In many cases, the compiler writer can recode the grammar to reduce the number of productions that it contains. This usually leads to smaller tables. For example, in the classic expression grammar, the distinction between a number and an identifier is irrelevant to the productions for Goal, Expr, Term, and Factor. Replacing the two productions Factor → Num and Factor → Id with a single production Factor → Val shrinks the grammar by a production. In the Action table, each terminal symbol has its own column. Folding Num and Id into a single symbol, Val, removes a column from the action table. To make this work, in practice, the scanner must return the same syntactic category, or token, for both numbers and identifiers.

Similar arguments can be made for combining × and ÷ into a single terminal MulDiv, and for combining + and − into a single terminal AddSub. Each of these replacements removes a terminal symbol and a production. This shrinks the size of the Canonical Collection of Sets of lr(1) Items, which removes rows from the table. It reduces the number of columns as well.

These three changes produce the following reduced expression grammar:

1.Goal → Expr