Cooper K.Engineering a compiler

Figure 2.4: Recognizer Tables for the Refined Register Specification

implement this tighter register specification, we need only to build a di erent set of tables for the skeleton recognizer.

2.6Non-deterministic Finite Automata

Recall from the definition of a regular expression that we designated the empty string, as a regular expression. What role does play in a recognizer? We can use transitions on to combine fas and form fas for more complex regular expressions. For example, assume that we have fas for the regular expressions m and n.

s0 m - s1 s0 n - s1

We can build an fa for mn by adding a transition on from the final state of fam to the initial state of fan, renumbering the states, and using Fn as the set of final states for the new fa.

s0 m - s1 - s2 n- s3

For this fa, the notion of acceptance changes slightly. In s1, the fa takes a transition on (or no input) to s2. Since is the empty string, it can occur between any two letters in the input stream without changing the word. This is a slight modification to the acceptance criterion, but seems intuitive.

By inspection, we can see that states s1 and s2 can be combined and the transition on eliminated.

s0 m - s1 n - s2

32 CHAPTER 2. LEXICAL ANALYSIS

si a- sj sk b- sl

nfa for a nfa for b

si a- sj - sk b- sl

nfa for ab

sp - si a- sj - sq

*

nfa for a

Figure 2.5: Components of Thompson’s Construction

of final states—every final state becomes a non-final state and every non-final state becomes a final state. The resulting recognizer fails on every word in

L(r) and succeeds on every word not in L(r). By definition, this automaton recognizes L(r) = {w Σ | w L(r)}.

Closure under complement is not strictly necessary for scanner construction. However, we can use this property to expand the range of input expressions allowed by a scanner generator. In some situations, complement provides a convenient notation for specifying a lexical pattern. Scanner generators often allow its use.

The argument for closure under intersection is somewhat more complex. It involves constructing an automaton whose state space contains the Cartesian product of the state spaces of the recognizers for r and s. The details are left as an exercise for the reader.

2.7From Regular Expression to Scanner

The goal of our work with fas is to automate the process of building an executable scanner from a collections of regular expressions. We will show how to accomplish this in two steps: using Thompson’s construction to build an nfa from a re [50] (Section 2.7.1) and using the subset construction to convert the nfa into a dfa (Section 2.7.2). The resulting dfa can be transformed into a minimal dfa—that is, one with the minimal number of states. That transformation is presented later, in Section 2.8.1.

In truth, we can construct a dfa directly from a re. Since the direct construction combines the two separate steps, it may be more e cient. However, understanding the direct method requires a thorough knowledge of the individual steps. Therefore, we first present the two-step construction; the direct method is presented later, along with other useful transformations on automata.

2.7.1Regular Expression to NFA

The first step in moving from a regular expression to an implemented scanner is deriving an nfa from the re. The construction follows a straightforward idea. It has a template for building the nfa that corresponds to a single letter re, and transformations on the nfas to represent the impact of the re operators, concatenation, alternation, and closure. Figure 2.5 shows the trivial nfa for the res a and b, as well as the transformations to form the res ab, a|b, and a .

The construction proceeds by building a trivial nfa, and applying the transformations to the collection of trivial nfas in the order of the relative precedence of the operators. For the regular expression a(b|c) , the construction would proceed by building nfas for a, b, and c. Next, it would build the nfa for b|c, then (b|c) , and, finally, for a(b|c) . Figure 2.6 shows this sequence of transformations.

Thompson’s construction relies on several properties of res. It relies on the obvious and direct correspondence between the re operators and the transformations on the nfas. It combines this with the closure properties on res for assurance that the transformations produce valid nfas. Finally, it uses -moves to connect the subexpressions; this permits the transformations to be simple templates. For example, the template for a looks somewhat contrived; it adds extra states to avoid introducing a cycle of -moves.

The nfas derived from Thompson’s construction have a number of useful properties.

1.Each has a single start state and a single final state. This simplifies the application of the transformations.

2.Any state that is the source or sink of an -move was, at some point in the process, the start state or final state of one of the nfas representing a partial solution.

3.A state has at most two entering and two exiting -moves, and at most one entering and one exiting move on a symbol in the alphabet.

s0 a- s1 s2 b- s3 s4 c- s5

nfas for a, b and c

Figure 2.6: Thompson’s construction for a(b|c)

1.S ← -closure(q0N )

2.while (S is still changing)

3.for each si S

4.for each character α Σ

Figure 2.7: The Subset Construction

These properties simplify an implementation of the construction. For example, instead of iterating over all the final states in the nfa for some arbitrary subexpression, the construction only needs to deal with a single final state.

Notice the large number of states in the nfa that Thompson’s construction built for a(b|c) . A human would likely produce a much simpler nfa, like the following:

a,b

?

s0 a- s1

We can automatically remove many of the -moves present in the nfa built by Thompson’s construction. This can radically shrink the size of the nfa. Since the subset construction can handle -moves in a natural way, we will defer an algorithm for eliminating -moves until Section 2.9.1.

2.7.2The Subset Construction

To construct a dfa from an nfa, we must build the dfa that simulates the behavior of the nfa on an arbitrary input stream. The process takes as input

an nfa N = (QN , Σ, δN , q0N , FN ) and produces a dfa D = (QD , Σ, δD, q0D , FD ). The key step in the process is deriving QD and δD from QN and δN (q0D and

FD will fall out of the process in a natural way.) Figure 2.7 shows an algorithm that does this; it is often called the subset construction.

The algorithm builds a set S whose elements are themselves sets of states in QN . Thus, each si S is itself a subset of QN . (We will denote the set of all subsets of QN as 2QN , called the powerset of QN .) Each si S represents a state in QD , so each state in QD represents a collection of states in QN (and, thus, is an element of 2QN ). To construct the initial state, s0 S, it puts q0N into s0 and then augments s0 with every state in QN that can be reached from q0N by following one or more -transitions.

The algorithm abstracts this notion of following -transitions into a function, called -closure For a state, qi, -closure(qi) is the set containing qi and any other states reachable from qi by taking only -moves. Thus, the first step is to construct s0 as -closure(q0N ).

S ← -closure( q0N )

while ( unmarked si S) mark si

for each character α Σ

t ← -closure(move(si ,α)) if t S then

add t to S as an unmarked state

T [si, α] ← t

Figure 2.8: A faster version of the Subset Construction

it can add at most | 2QN | elements to S; after that, it must halt. (It may, of course, halt much earlier.) Many fixed point computations have considerably tighter bounds, as we shall see.

E ciency The algorithm shown in Figure 2.7 is particularly ine cient. It recomputes the transitions for each state in S on each iteration of the while loop. These transitions cannot change; they are wholly determined by the structure of the input nfa. We can reformulate the algorithm to capitalize on this fact; Figure 2.8 shows one way to accomplish this.

The algorithm in Figure 2.8 adds a “mark” to each element of S. When sets are added to S, they are unmarked. When the body of the while loop processes a set si, it marks si. This lets the algorithm avoid processing each

si multiple times. It reduces the number of invocations of -closure(move(si ,α)) from O(| S |2 · | Σ |) to O(| S | · | Σ |). Recall that S can be no larger than 2QN .

Unfortunately, S can become rather large. The principal determinant of how much state expansion occurs is the degree of nondeterminism found in the input nfa. Recall, however, that the dfa makes exactly one transition per input character, independent of the size of QD . Thus, the use of non-determinism in specifying and building the nfa increases the space required to represent the corresponding dfa, but not the amount of time required for recognizing an input string.

Computing -closure as a Fixed Point To compute -closure(), we use one of two approaches: a straightforward, online algorithm that follows paths in the nfa’s transition graph, or an o ine algorithm that computes the -closure for each state in the nfa in a single fixed point computation.

for each state n N E(n) ←

while (some E(n) has changed) for each state n N

E(n) ← n,s, E(s)

monotonically since line five uses (not ∩). The algorithm halts when no E(n) changes in an iteration of the outer loop. When it halts, E(n) contains the names of all states in -closure(n).

We can obtain a tighter time bound by observing that | E(n) | can be no larger than the number of states involved in a path leaving n that is labeled entirely with ’s. Thus, the time required for a computation must be related to the number of nodes in that path. The largest E(n) set can have N nodes. Consider that longest path. The algorithm cannot halt until the name of the last node on the path reaches the first node on the path. In each iteration of the outer loop, the name of the last node must move one or more steps closer to the head of the path. Even with the worst ordering for that path, it must move along one edge in the path.

At the start of the iteration, nlast E(ni) for some ni. If it has not yet reached the head of the path, then there must be an edge ni, nj , in the path. That node will be visited in the loop at line six, so nlast will move from E(ni) to E(nj ). Fortuitous ordering can move it along more than one -transition in a single iteration of the loop at line six, but it must always move along at least one -transition, unless it is in the last iteration of the outer loop.

Thus, the algorithm requires at most one while loop iteration for each edge in the longest -path in the graph, plus an extra iteration to recognize that the E sets have stabilized. Each iteration visits N nodes and does E unions. Thus, its complexity is O(N (N + E)) or O(max(N 2, N E)). This is much better than O(2N ).

We can reformulate the algorithm to improve its specific behavior by using a worklist technique rather than a round-robin technique.

for each state n N E(n) ←

WorkList ← N

while (WorkList = ) remove ni from worklist

E(nj ) ← ni,nj , E(nj )

if E(nj ) changed then

WorkList ← WorkList {nk | nk , ni, δNF A }

This version only visits a node when the E set at one of its -successors has changed. Thus, it may perform fewer union operations than the round robin version. However, its asymptotic behavior is the same. The only way to improve its asymptotic behavior is to change the order in which nodes are removed from the worklist. This issue will be explored in some depth when we encounter data-flow analysis in Chapter 13.