- Table of Contents
- Previous Chapter
- Next Chapter

# Chapter 7: About Analyzing LALR(1) Grammars

This chapter describes the algorithm that is used by bisonc++. Generating parsers of course begins with a grammar to be analyzed. The analysis consists of these steps:

- First, a set of `FIRST` tokens is determined. The `FIRST` set of a nonterminal defines all terminal tokens that can be encountered when beginning to recognize that nonterminal.
- Having determined the `FIRST` set, the grammar itself is analyzed. Starting from the start rule all possible syntactically correct derivations of the grammar are determined.
- At each state actions are defined that are eventually used by the parser to determine what to do in a particular state. These actions are based on the next available token and may either involve a transition to another state (called a shift, since it involves processing a token by `shifting' over the next part of the input) or it may involve a reduction (a reduce action), in which the parser stack's size is somewhat reduced. In the latter case, if an action is associated with a particular rule, the action is executed as a side effect of the reduction.
- Sometimes the analysis takes the parser generator to a state in which a choice between a shift or a reduce action must be made. This is called a shift-reduce conflict. Sometimes the parser generator is able to solve these conflicts itself, by looking at the next available token (the look-ahead token). If the next token is not a possible continuation for either a shift or a reduce action that particular continuation can be discarded. Likewise, two reductions may be encountered in a state (a reduce-reduce conflict). Here the same reasoning can be applied, maybe resulting in discarding one of the possible reductions. In all other cases (i.e., if the look-ahead token is possible for both continuations) a true conflict is observed which somehow is solved by the grammar designer. If the designer doesn't select an particular action, the parser generator reports a conflict and selects a default action. By default a shift is used, rather than a reduce. With a reduce-reduce conflict the default action is to reduce by the rule listed first in the grammar.
- In cases where input is not structured according to the rules of the grammar, a syntactic error is observed. Error recovery may be attempted, to allow the parser to continue parsing. This is a desirable characteristic since it provides the user of a program with a full syntactic error report, rather than one error at a time.
- Following the analysis of the grammar, code is generated and the parsing algorithm (implemented in the parser's `parse()` function) processes input according to the tables generated by the parser generator.

All the above phases are illustrated and discussed in the next sections. Additional details of the parsing process can be found in various books about compiler construction, e.g., in Aho, Sethi and Ullman's (2003) book Compilers (Addison-Wesley).

In the sections below, the following grammar is used to illustrate the various phases:

        %token NR

        %left '+'

        %%

        start:
            start expr
        |
            // empty
        ;

        expr:
            NR
        |
            expr '+' expr
        ;

The grammar is interesting as it has a rule containing an empty alternative and because it harbors a shift-reduce conflict. The shift-reduce conflict is solved by explictly assigning a priority and association to the `'+'` token.

The analysis starts by defining an additional rule, which is recognized (reduced) at end of input. This rule and the rules specified in the grammar together define what is known as the augmented grammar. In the coming sections the symbol `$` is used to indicate `end of input'. From the above grammar the following augmented grammar is derived:

        1.  start:      start expr
        2.  start:      // empty
        3.  expr:       NR
        4.  expr:       expr '+' expr
        5.  start_$:    start   (input ends here)

bisonc++ itself produces an extensive analysis of any grammar it is offered when the option `--construction` is provided.

### 7.0.1: The FIRST Sets

The `FIRST` set defines all terminal tokens that can be encountered when beginning to recognize a grammatical symbol. For each grammatical symbol (terminal and nonterminal) a `FIRST` set can be determined as follows:

- The `FIRST` set of a terminal symbol is the symbol itself.
- The `FIRST` set of an empty alternative is the empty set. The empty set is indicated by ε and is considered an actual element of the `FIRST` set (So, a `FIRST` set could contain two elements: `'+'` and ε ).
- If X has a production rule `X: X1 X2 X3..., Xi, ...Xn`, then initialize `FIRST(X)` to empty (i.e., not even holding ε ). Then, for each Xi (1..n):
  - add `FIRST(Xi)` to `FIRST(X)`
  - stop when `FIRST(Xi)` does not contain ε

  If `FIRST(Xn)` does not contain ε remove ε from `FIRST(X)` (unless analyzing another production rule) ε is already part of `FIRST(X)`.

When starting this algorithm, only the nonterminals need to be considered. Also, required `FIRST` sets may not yet be available. Therefore the above algorithm iterates over all nonterminals until no changes were observed. In the algorithm `$` is not considered.

Applying the above algorithm to the rules of our grammar we get:

nonterminal | rule |  | FIRST set
`start_$` | `start` |  | not yet available
`start` | `start expr` |  | not yet available
`start` | `// empty` |  | ε
`expr` | `NR` |  | `NR`
`expr` | `expr '+' expr` |  | `NR`
changes in the next cycle:
`start` | `start expr` |  | `NR` ε
`start` | `// empty` |  | `NR` ε
changes in the next cycle:
`start_$` | `start` |  | `NR` ε
no further changes

### 7.0.2: The States

Having determined the `FIRST` set, bisonc++ determines the states of the grammar. The analysis starts at the augmented grammar rule and proceeds until all possible states have been determined. In this analysis the concept of the dot symbol is used. The dot shows the position we are at when analyzing production rules defined by a grammar. Using the provided example grammar the analysis proceeds as follows:

- State 0: `start_$ -> . start`
  At this point we haven't seen anything yet. The dot is before the grammar's start symbol. The above is called an item and the initial set of items of a state is called the set of kernel items. Except for the start rule, kernel items never have a dot before the very first symbol of a rule. In this particular state there's only one kernel item. Items are indexed, so this item receives index 0. Beyond, item indices are shown together with the items themselves. If a nonterminal follows immediately to the right of the dot, then all production rules of that nonterminal are added to the state as non-kernel items. Non-kernel items always have their dots at the very first position. Adding non-kernel items is a recursive process. If the rules of thus added items also show nonterminals to the right of the dot, then the production rules of those nonterminals are added too (unless they were already added).

  The above kernel item results in the addition of the following non-kernel items:

  - item 1: `start -> . start expr`
  - item 2: `start -> .`

  From each of the items new states may be derived. New states are reached when the symbol to the right of the dot has been recognized. In that case a transition (a goto) to the next state takes place, where the dot has moved one postition to the right, defining a kernel item of the new state. Once the dot has reached the end of the rule, a reduction may take place. Following a reduction a transition based on rule's LHS is performed. This procedure is discussed in more detail in section 7.0.5.

  Looking at the current state's items, two actions are possible:

  - On `start`, to a state in which `start` has been seen (state 1)
  - By default, a reduction by the rule `start -> .`

- State 1: kernel items:
  - item 0: `start_$ -> start .`
  - item 1: `start -> start . expr`
  Since `expr` is a nonterminal to the right of the dot, we add all `expr` rules as this state's non-kernel items:
  - item 2: `expr -> . NR`
  - item 3: `expr -> . expr '+' expr`
  This state becomes the accepting state: if EOF is reached in this state, the `start_$` rule has been recognized, and so the input was syntactically correct. But in this state transitions to other states are also possible:
  - On `expr` to state 2
  - On NR to state 3

- State 2: kernel items:
  - item 0: `start -> start expr .`
  - item 1: `expr -> expr . '+' expr`
  No nonterminal symbols appear to the right of the dots in these items, so no non-kernel items are added to this state. Transitions from this state are:
  - On `'+'` to state 4
  - Or reduce to `start` according to its first item (removing two elements from the parser's stack).

- State 3: kernel items:
  - item 0: `expr -> NR .`
  In this state only one action is possible: a reduction to `expr` (removing one element from the parser's stack).

- State 4: kernel item:
  - item 0: `expr -> expr '+' . expr`
  Since `expr` is a nonterminal to the right of the dot, we add all `expr` rules as this state's non-kernel items:
  - item 1: `expr -> . NR`
  - item 2: `expr -> . expr '+' expr`
  In this state the following transitions are possible:
  - On `expr` to state 5
  - On `NR` we reach the situation `expr -> NR .` which has already been encountered at state 3. That's OK, so on `NR` there is a transition to state 3.
  Note that in order to return to a previously defined state that state must have exactly the required kernel items. So, if state 3 would contain multiple kernel items, a new state would have been required, merely having the `expr -> NR .` kernel item.

- State 5: kernel items:
  - item 0: `expr -> expr '+' expr .`
  - item 1: `expr -> expr . '+' expr`
  In this state two actions are possible:
  - On `'+'` to state 4
  - Or reduce to `expr` according to its first item (removing three elements from the parser's stack).
  As explained in the next section this state's first action is never selected: in this state only the reduction is selected.

### 7.0.3: The Look-ahead Sets

In the previous section a grammar was discussed whose fifth state contained two items: one resulting in a shift-action, the other resulting in a reduce-action. This state contained these two items:

- item 0: `expr -> expr '+' expr .`
- item 1: `expr -> expr . '+' expr`

Although this state in theory defines two different actions, in practice only one is used. This is a direct consequence of the `%left '+'` specification, which is explained in this and the next section.

When analyzing a grammar all states that can be reached from the augmented start rule are determined. In the current grammar's fifth state bisonc++ must decide which action to take: should it shift on `'+'` or should it reduce according to the item ``expr -> expr '+' expr .`'? What choice will bisonc++ make?

Here the fact that bisonc++ implements a parser for a Look Ahead Left to Right (1) (LALR(1)) grammar becomes relevant. Bisonc++ computes look-ahead sets to determine which alternative to select when confronted with a choice. The look-ahead set can be used to favor one action over another when generating tables for the parsing function.

Sometimes the look-ahead sets allow bisonc++ simply to remove one action from the set of possible actions. When bisonc++ is called to process the example grammar while specifying the `--construction` option state five only shows the reduction and not the shifting action, as bisonc++ has removed that latter action from the action set. In state five the choice is between shifting a `'+'` token on the stack, or reducing the stack according to the rule

            expr -> expr '+' expr

Here, as we will shortly see, the `'+'` is also an element of the look-ahead set of the reducible item, creating a conflict: what to do on `'+'`?

In this case the grammar designer has provided bisonc++ with a way out: the `%left` directive tells bisonc++ to favor a reduction over a shift, and so it removed `expr -> expr . '+' expr` from its set of actions in state five.

#### 7.0.3.1: The look-ahead token

The bisonc++ parser does not always perform a reduction when a state is reached where an item has its dot position beyond the last element of its production rule. For most languages such a simple strategy is incorrect. Instead, when a reduction is possible, the parser sometimes `looks ahead' to the next token to decide what to do.

Whenever a token is read, it is not immediately shifted; first it becomes the look-ahead token, which is not yet shifted on the stack. This allows the parser to perform one or more reductions, with the look-ahead token still waiting to be processed. Only when all available reductions have been performed the look-ahead token is shifted on the stack. The phrase `all available reductions' does not necessarily mean all possible reductions. Depending on the look-ahead token, a shift rather than a reduce may be performed in states in which both actions are possible.

Here is a simple case where a look-ahead token is required. The production rules define expressions which may contain binary addition operators and postfix unary factorial operators (``!`'), as well as parentheses for grouping expressions:

        expr:
            term '+' expr
        |
            term
        ;

        term:
            '(' expr ')'
        |
            term '!'
        |
            NUMBER
        ;

Suppose that the tokens ``1 + 2`' have been read and shifted; what should be done? If the following token is `)', then the first three tokens must be reduced, forming an `expr`. This is the only valid course, because shifting the `)' would produce the sequence of symbols

        term 'CLOSEPAR'

which is not syntactically correct.

But if the next token is ``!`', then that token must be shifted so that ``2 !`' can be reduced to recognize a `term`. If in this case the parser would perform a reduction then ``1 + 2`' would become an `expr`. In that case the ``!`' can't be shifted because doing so would result in the sequence

        expr '!'

which is also syntactically incorrect.

#### 7.0.3.2: How look-ahead sets are determined

Once the items of all the grammar's states have been determined the LA sets for the states' items are computed. Starting from the LA set of the kernel item of state 0 (representing the augmented grammar's production rule `S_$: . S`, where `S` is the grammar's start rule) the LA sets of all items of all of the grammar's states are determined. By definition, the LA set of state 0's kernel item equals `$`, representing end-of-file.

Starting from the function `State::determineLAsets`, which is called for state 0, the LA sets of all items of all states are computed.

For each state, the LA sets of its items are computed first. Once they have been computed, the LA sets of items from where transitions to other states are possible are then propagated to the matching kernel items of those destination states. When the LA sets of kernel items of those destination states are enlarged then their state indices are added to a set `todo`. LA sets of the items of states whose indices are stored in the `todo` set are (re)computed (by calling `determineLAsets` for those states) until `todo` is empty, at which point all LA sets have been computed. Initially `todo` only contains 0, the index of the initial state, representing the augmented grammar's production rule.

To compute the LA sets of a state's items the LA set of each of its kernel items is distributed (by the member `State::distributeLAsetOf`) over the items which are implied by the item being considered. E.g., for item `X: a . Y z`, where `a` and `z` are any sequence of grammar symbols and `X` and `Y` are nonterminal symbols, all of `Y's` production rules are added as new items to the current state.

Then the member `distributeLAsetOfItem(idx)` matches the item's rule specification with the specification `a.Bc`, where `a` and `c` are (possibly empty) sequences of grammatical symbols, and `B` is a (possibly empty) nonterminal symbol appearing immediately to the right of the item's dot position. if `B` is empty then there are no additional production rules and `distributeLAsetOf` may return. Otherwise, the set `b = FIRST(c)` is computed. This set holds all symbols which may follow `B`. If `b` contains ε (i.e., the element representing the empty set), then the currently defined LA set of the item can also be observed. In that case ε is removed, and the currently defined LA set is added to `b`. Finally, the LA sets of all items representing a production rule for `B` are inspected: if `b` contains unique elements compared to the LA sets of those items, then the unique elements of `b` are added to the LA sets of those items. Finally, `distributeLAsetOfItem` is recursively called for those items whose LA sets were enlarged.

Once the LA sets of the items of a state have thus been computed, `inspectTransitions` is called to propagate the LA sets of items from where transitions to other states are possible to the affected (kernel) items of those other (destination) states. The member `inspectTransitions` inspects all `Next` objects of the current state's `d_nextVector`. Next objects provide

- the state index of a state to transfer to from the current state;
- a size_t vector of item transitions. Each element is the index of an item in the current state (the source-item), its index is the index of a (kernel) item of the state to transfer to (the destination index).

If the LA set of a destination item can be enlarged from the LA set of the source item then the LA sets of the destination state's items must be recomputed. This is realized by inserting the destation state's index into the `todo' set.

To illustrate an LA-set computation we will now compute the LA sets of (some of) the items of the states of the grammar introduced at the beginning of this chapter. Its augmented grammar consists of the following production rules:

        1.  start:      start expr
        2.  start:      // empty
        3.  expr:       NR
        4.  expr:       expr '+' expr
        5.  start_$:    start

When analyzing this grammar, we found the following five states, consisting of several items and transitions (kernel items are marked with K following their item indices). Next to the items, where applicable, the goto-table is shown: the state to go to when the mentioned grammatical symbol has been recognized:

                                                Goto table
                                                -----------
    State 0:                                    start
        0K:     start_$ ->  . start               1
        1:      start   ->  . start expr          1
        2:      start   ->  .

    State 1:                                    expr    NR
        0K:     start_$ ->  start  .
        1K:     start   ->  start  . expr         2
        2:      expr    ->  . NR                         3
        3:      expr    ->  . expr '+' expr

    State 2:                                     '+'
        0K:     start   -> start expr  .
        1K:     expr    -> expr  . '+' expr       4

    State 3:
        0K:     expr    -> NR  .

    State 4:                                    expr    NR
        0K:     expr    -> expr '+'  . expr       5
        1:      expr    -> . NR                          3
        2:      expr    -> . expr '+'  expr       5

    State 5:                                     '+'
        0K:     expr    -> expr '+' expr  .
        1K:     expr    -> expr . '+' expr        4

Item 0 of state 0 by definition has LA symbol $, and LA computation therefore always starts at item 0 of state 0. The interesting part of the LA set computation is encountered in the recursive member `distributeLAsets`:

    distributeLAsetsOfItem(0)
      start_$ -> . start:     LA: {$}, B: start, c: {}, so b: {$}

      items 1 and 2 refer to production rules of B (start) and are inspected:

      1: LA(1): {}: b contains unique elements. Therefore:
        LA(1) = {$}
        distributeLAsetsOfItem(1):
          start -> . start expr: LA: {$}, B: start, c: {expr}, so b: {NR}
          inspect items 1 and 2 as they refer to production rules of B (start):

          1: LA(1): {}: b contains unique elements. Therefore:
            LA(1) = {$,NR}
            distributeLAsetsOfItem(1)
              start -> . start expr: LA: {$,NR}, B: start, c: {expr}, so b: {NR}
              inspect items 1 and 2 as they refer to prod. rules of B (start):

              1: LA(1): {$,NR}, so b does not contain unique elements: done

              2: LA(2): {}, b contains unique elements
                LA(2) = {NR}
                distributeLAsetsOfItem(2)
                  start -> .: LA: {NR}, B: -, c: {}, so b: {NR}
                  inspect items 1 and 2 as they refer to prod. rules of B (start):

                  1: LA(1): {$,NR}, b does not contain unique elements: done

                  2: LA(2): {NR}, so b does not contain unique elements: done

          2: LA(2): {NR}, so b does not contain unique elements: done

      2: LA(2): {NR}: b contains unique elements. Therefore:
        LA(2) = {$,NR}
        distributeLAsetsOfItem(2)
          start -> .: LA: {$,NR}, B: -, c: {}
          B empty, so return.

So, item 0 has LA set `{$}`, items 1 and 2 have LA sets `{$,NR}`.

The next step involves propagating the LA sets to kernel items of the states to where transitions are possible:

- Item 0, state 0 transits to item 0 state 1. Item 0 of state 1's current LA set is empty, so it receives LA set `{$}`, and 1 (state 1's index) is inserted into the `todo` set.
- Item 1, state 0 transits to item 1 state 1. Item 1 of state 1's current LA set is empty, so it receives LA set `{$,NR}`, and 1 (state 1's index) is inserted into the `todo` set.

Following this LA set propagation the LA sets of all items of state 1 are computed, which in turn is followed by LA propagation to other states (states 2 and 3), etc. etc.

In this grammar there are no transitions to the current state (i.e., transitions from state x to state x). If such transitions are encountered then they can be ignored by `inspectTransitions` as the LA sets of the items of a state have already be computed by the time `inspectTransitions` is called.

### 7.0.4: The Final Transition Tables

#### 7.0.4.1: Preamble

The member function `parse()` is implemented using a finite-state machine. The values pushed on the parser stack are not simply token type codes; they represent the entire sequence of terminal and nonterminal symbols at or near the top of the stack. The current state collects all the information about previous input which is relevant to deciding what to do next.

Each time a look-ahead token is read, the current parser state together with the current (not yet processed) token are looked up in a table. This table entry can say Shift the token. This also specifies a new parser state, which is then pushed onto the top of the parser stack. Or it can say Reduce using rule number n. This means that a certain number of tokens or nonterminals are removed from the stack, and that the rule's nonterminal becomes the `next token' to be considered. That `next token' is then used in combination with the state then at the stack's top, to determine the next state to consider. This (next) state is then again pushed on the stack, and a new token is requested from the lexical scanner, and the process repeats itself.

There are two special situations the parsing algorithm must consider:

- First, the lexical scanner may reach end-of-input. If the current state on top of the parser's stack is the start-state, then the reduction (which is called for in this situation) is in fact the (successful) end of the parsing process, and `parse()` returns the value 0, indicating a successful parsing.
- There is one other alternative: the table can say that the token is erroneous in the current state. This causes error processing to begin (see chapter 8).

Once bisonc++ has successfully analyzed the grammar it generates the tables that are used by the parsing function to parse input according to the provided grammar. Each state results in a state transition table. For the example grammar used so far there are five states. Each table consists of rows having two elements. The meaning of the elements depends on their position in the table.

- For the first row,
  - the first element indicates the type of the state. The following types are recognized:

    NORMAL | Despite its name, it's not used
    ERR_ITEM | The state allows error recovery
    REQ_TOKEN | The state requires a token (which may already be available)
    ERR_REQ | combines ERR_ITEM and REQ_TOKEN
    DEF_RED | This state has a default reduction
    ERR_DEF | combines ERR_ITEM and DEF_RED
    REQ_DEF | combines REQ_TOKEN and DEF_RED
    ERR_REQ_DEF | combines ERR_ITEM, REQ_TOKEN and DEF_RED

  - the second element indicates the index of the table's last element.
- For the last row,
  - the first element stores the current token (it is not used when the option `--thread-safe` was specified)
  - the second element defines the action to perform. A positive value indicates a shift to the indicated state; a negative value a reduction according to the indicated rule number, disregarding its sign (note that it's rule number, rather than rule offset; zero indicates the input is accepted as correct according to the parser's grammar. )
  - For all intermediate remaining rows: the first element stores the value of a required token for the action specified in the second element, similar to the way an action is specified in the last row. Symbolic values (like `PARSE_ACCEPT` rather than 0) may be used as well.

Here are the tables defining the five states of the example grammar as they are generated by bisonc++ in the file containing the parsing function:

        SR_ s_0[] =
        {
            { { DEF_RED}, {  2} },
            { {     258}, {  1} }, // start
            { {       0}, { -2} },

        };

        SR_ s_1[] =
        {
            { { REQ_TOKEN}, {            4} },
            { {       259}, {            2} }, // expr
            { {       257}, {            3} }, // NR
            { {     EOF_}, { PARSE_ACCEPT} },
            { {         0}, {            0} },

        };

        SR_ s_2[] =
        {
            { { REQ_DEF}, {  2} },
            { {      43}, {  4} }, // '+'
            { {       0}, { -1} },

        };

        SR_ s_3[] =
        {
            { { DEF_RED}, {  1} },
            { {       0}, { -3} },

        };

        SR_ s_4[] =
        {
            { { REQ_TOKEN}, { 3} },
            { {       259}, { 5} }, // expr
            { {       257}, { 3} }, // NR
            { {         0}, { 0} },

        };

        SR_ s_5[] =
        {
            { { REQ_DEF}, {  1} },
            { {       0}, { -4} },

        };

### 7.0.5: Processing Input

Bisonc++ implements the parsing function in the member function `parse()`. This function obtains its tokens from the member `lex()` and processes all tokens until a syntactic error, a non-recoverable error, or the end of input is encountered.

The algorithm used by `parse()` is the same, irrespective of the used grammar. In fact, the `parse()` member's behavior is completely determined by the tables generated by bisonc++.

The parsing algorithm is known as the shift-reduce (S/R) algorithm, and it allows `parse()` to perform two actions while processing series of tokens:

- When a token is received in a state in which that token is required for a transition to another state (e.g., a `NR` token is observed in state 1 of the example's grammar) a transition to state 3 is performed.
- When a state is reached which calls for a (default) reduction (e.g., state 3 of the example's grammar) a reduction is performed.

The parsing function maintains two stacks, which are manipulated by the above two actions: a state stack and a value stack. These stacks are not accessible to the parser: they are private data structures defined in the parser's base class. The parsing member `parse()` may use the following member functions to manipulate these stacks:

- `push_(stateIdx)` pushes `stateIdx` on the state stack and pushes the current semantic value (i.e., `LTYPE_ d_val_`) on the value stack;
- `pop_(size_t count = 1)` removes `count` elements from the two stacks;
- `top_()` returns the state currently on top of the state stack;

Apart from the state- and semantic stacks, the S/R algorithm itself sometimes needs to push a token on a two-element stack. Rather than using a formal stack, two variables (`d_token_` and `d_nextToken_`) are used to implement this little token-stack. The member function `pushToken_()` pushes a new value on the token stack, the member `popToken_()` pops a previously pushed value from the token stack. At any time, `d_token_` contains the topmost element of the token stack.

The member `nextToken()` determines the next token to be processed. If the token stack contains a value it is returned. Otherwise, `lex()` is called to obtain the next token to be pushed on the token stack.

The member `lookup()` looks up the current token in the current state's `SR_` table. For this a simple linear search algorithm is used. If searching fails to find an action for the token an `UNEXPECTED_TOKEN_` exception is thrown, which starts the error recovery. If an action was found, it is returned.

Rules may have actions associated with them. These actions are executed when a grammatical rule has been completely recognized. This is always at the end of a rule: mid-rule actions are converted by bisonc++ into pseudo nonterminals, replacing mid-rule action blocks by these pseudo nonterminals. The pseudo nonterminals show up in the verbose grammar output as rules having LHSs starting with `#`. So, once a rule has been recognized its action (if defined) is executed. For this the member function `executeAction()` is available.

Finally, the token stack can be cleared using the member `clearin()`.

Now that the relevant support functions have been introduced, the S/R algorithm itself turns out to be a fairly simple algorithm. First, the parser's stack is initialized with state 0 and the token stack is cleared. Then, in a never ending loop:

- If a state needs a token (i.e., `REQ_TOKEN` has been specified for that state), `nextToken()` is called to obtain the next token;
- From the token and the current state `lookup()` determines the next action;
- If a shifting action was called for the next state is pushed on the stack and the token is popped off the token stack.
- If a reduction was called for that rule's action block is executed followed by a reduction of the production rule (performed by `reduce_()`): the semantic and state stacks are reduced by the number of elements found in that production rule, and the production rule's LHS is pushed on the token stack
- If the state/token combination indicates that the input is accepted (normally: when `EOF` is encountered in state 1) then the parsing function terminates, returning 0.

The following table shows the S/R algorithm in action when the example grammar is given the input `3 + 4 + 5`. The first column shows the (remaining) input, the second column the current token stack (with `-` indicating an empty token stack), the third column the state stack. The fourth column provides a short description. The leftmost elements of the stacks represent the tops of the stacks. The information shown below is also (in more elaborate form) shown when the `--debug` option is provided to Bisonc++ when generating the parsing function.

remaining input |  | token stack |  | state stack |  | description
`3 + 4 + 5` |  | `-` |  | `0` |  | initialization
`3 + 4 + 5` |  | `start` |  | `0` |  | reduction by rule 2
`3 + 4 + 5` |  | `-` |  | `1 0` |  | shift `start'
`+ 4 + 5` |  | `NR` |  | `1 0` |  | obtain NR token
`+ 4 + 5` |  | `-` |  | `3 1 0` |  | shift NR
`+ 4 + 5` |  | `expr` |  | `1 0` |  | reduction by rule 3
`+ 4 + 5` |  | `-` |  | `2 1 0` |  | shift `expr'
`4 + 5` |  | `+` |  | `2 1 0` |  | obtain `+' token
`4 + 5` |  | `-` |  | `4 2 1 0` |  | shift `+'
`+ 5` |  | `NR` |  | `4 2 1 0` |  | obtain NR token
`+ 5` |  | `-` |  | `3 4 2 1 0` |  | shift NR
`+ 5` |  | `expr` |  | `4 3 1 0` |  | reduction by rule 3
`+ 5` |  | `-` |  | `5 4 3 1 0` |  | shift `expr'
`5` |  | `+` |  | `5 4 3 1 0` |  | obtain `+' token
`5` |  | `expr +` |  | `1 0` |  | reduction by rule 4
`5` |  | `+` |  | `2 1 0` |  | shift `expr'
`5` |  | `-` |  | `4 2 1 0` |  | shift '+'
`` |  | `NR` |  | `4 2 1 0` |  | obtain NR token
`` |  | `-` |  | `3 4 2 1 0` |  | shift NR
`` |  | `expr` |  | `4 2 1 0` |  | reduction by rule 3
`` |  | `-` |  | `5 4 2 1 0` |  | shift `expr'
`` |  | `EOF` |  | `5 4 2 1 0` |  | obtain EOF
`` |  | `expr EOF` |  | `1 0` |  | reduction by rule 4
`` |  | `EOF` |  | `2 1 0` |  | shift `expr'
`` |  | `start EOF` |  | `2 1 0` |  | reduction by rule 1
`` |  | `EOF` |  | `1 0` |  | shift `start'
`` |  | `EOF` |  | `1 0` |  | ACCEPT

## 7.1: Shift/Reduce Conflicts

Suppose we are parsing a language which has `if` and `if-else` statements, with a pair of rules like this:

        if_stmt:
            IF '(' expr ')' stmt
        |
            IF '(' expr ')' stmt ELSE stmt
        ;

Here we assume that `IF` and `ELSE` are terminal symbols for specific keywords, and that `expr` and `stmnt` are defined nonterminals.

When the `ELSE` token is read and becomes the look-ahead token, the contents of the stack (assuming the input is valid) are just right for reduction by the first rule. But it is also legitimate to shift the `ELSE`, because that would lead to eventual reduction by the second rule.

This situation, where either a shift or a reduction would be valid, is called a `shift/reduce` conflict. Bisonc++ is designed to resolve these conflicts by implementing a shift, unless otherwise directed by operator precedence declarations. To see the reason for this, let's contrast it with the other alternative.

Since the parser prefers to shift the `ELSE`, the result is to attach the else-clause to the innermost if-statement, making these two inputs equivalent:

        if (x) if (y) then win(); else lose();

        if (x)
        {
            if (y) then win(); else lose();
        }

But if the parser would perform a reduction whenever possible rather than a shift, the result would be to attach the else-clause to the outermost if-statement, making these two inputs equivalent:

        if (x) if (y) then win(); else lose();

        if (x)
        {
            if (y) win();
        }
        else
            lose();

The conflict exists because the grammar as written is ambiguous: either parsing of the simple nested if-statement is legitimate. The established convention is that these ambiguities are resolved by attaching the else-clause to the innermost if-statement; this is what bisonc++ accomplishes by implementing a shift rather than a reduce. This particular ambiguity was first encountered in the specifications of Algol 60 and is called the dangling else ambiguity.

To avoid warnings from bisonc++ about predictable, legitimate shift/reduce conflicts, use the `%expect n` directive. There will be no warning as long as the number of shift/reduce conflicts is exactly `n`. See section 4.5.6.

The definition of `if_stmt` above is solely to blame for the conflict, but the plain `stmnt` rule, consisting of two recursive alternatives will of course never be able to match actual input, since there's no way for the grammar to eventually derive a sentence this way. Adding one non-recursive alternative is enough to convert the grammar into one that does derive sentences. Here is a complete bisonc++ input file that actually shows the conflict:

    %token IF ELSE VAR

    %%

    stmt:
        VAR ';'
    |
        IF '(' VAR ')' stmt
    |
        IF '(' VAR ')' stmt ELSE stmt
    ;

Looking again at the dangling else problem note that there are multiple ways to handle `stmnt` productions. Depending on the particular input that is provided it could either be reduced to a `stmt` or the parser could continue to consume input by processing an `ELSE` token, eventually resulting in the recognition of `IF '(' VAR ')' stmt ELSE stmt` as a `stmt`.

There is little we can do but resorting to `%expect` to handle the dangling else problem. The default handling is what most people intuitively expect and so in this case using `%expect 1` is an easy way to prevent bisonc++ from reporting a shift/reduce conflict. But shift/reduce conflicts are most often solved by specifying disambiguating rules specifying priorities or associations, usually in the context of arithmetic expressions, as discussed in the next sections.

However, shift-reduce conflicts can also be observed in grammars where a state contains items that could be reduced to a certain nonterminal and items in which a shift is possible in an item of a production rule of a completely different nonterminal. Here is an example of such a grammar:

        %token  ID
        %left  '-'
        %left  '*'
        %right UNARY

        %%

        expr:
            expr '-' term
        |
            term
        ;

        term:
            term '*' factor
        |
            factor
        ;

        factor:
           '-' expr %prec UNARY
        |
            ID
        ;

Why these grammars show shift reduce conflicts and how these are solved is discussed in the next section.

## 7.2: Operator Precedence

Shift/reduce conflicts are frequently encountered in grammars specifying rules of arithmetic expressions. Here shifting is not always the preferred resolution; the bisonc++ directives for operator precedence allow you to specify when to shift and when to reduce. How and when to do so is discussed next.

### 7.2.1: When Precedence is Needed

Consider the following ambiguous grammar fragment (ambiguous because the input ``1 - 2 * 3`' can be parsed in two different ways):

        expr:
            expr '-' expr
        |
            expr '*' expr
        |
            expr '<' expr
        |
            '(' expr ')'
        ...
        ;

Suppose the parser has seen the tokens ``1`', ``-'` and ``2`'; should it reduce them via the rule for the addition operator? It depends on the next token. Of course, if the next token is `)', we must reduce; shifting is invalid because no single rule can reduce the token sequence ``- 2` )' or anything starting with that. But if the next token is ``*`' or ``<`', we have a choice: either shifting or reduction would allow the parse to complete, but with different results.

To decide which one bisonc++ should do, we must consider the results. If the next operator token `op` is shifted, then it must be reduced first in order to permit another opportunity to reduce the sum. The result is (in effect) ``1 - (2 op 3)`'. On the other hand, if the subtraction is reduced before shifting `op`, the result is ``(1 - 2) op 3`'. Clearly, then, the choice of shift or reduce should depend on the relative precedence of the operators ``-`' and `op`: ``*`' should be shifted first, but not ``<`'.

What about input such as ``1 - 2 - 5`'; should this be ``(1 - 2) - 5`' or should it be ``1 - (2 - 5)`'? For most operators we prefer the former, which is called left association. The latter alternative, right association, is desirable for, e.g., assignment operators. The choice of left or right association is a matter of whether the parser chooses to shift or reduce when the stack contains ``1 - 2`' and the look-ahead token is ``-`': shifting results in right-associativity.

### 7.2.2: Specifying Operator Precedence

Bisonc++ allows you to specify these choices with the operator precedence directives `%left` and `%right`. Each such directive contains a list of tokens, which are operators whose precedence and associativity is being declared. The `%left` directive makes all those operators left-associative and the `%right` directive makes them right-associative. A third alternative is `%nonassoc`, which declares that it is a syntax error to find the same operator twice `in a row'. Actually, `%nonassoc` is not currently (0.98.004) punished that way by bisonc++. Instead, `%nonassoc` and `%left` are handled identically.

The relative precedence of different operators is controlled by the order in which they are declared. The first `%left` or `%right` directive in the file declares the operators whose precedence is lowest, the next such directive declares the operators whose precedence is a little higher, and so on.

### 7.2.3: Precedence Examples

In our example, we would want the following declarations:

        %left '<'
        %left '-'
        %left '*'

In a more complete example, which supports other operators as well, we would declare them in groups of equal precedence. For example, '`+`' is declared with '`-`':

        %left '<' '>' '=' NE LE GE
        %left '+' '-'
        %left '*' '/'

(Here `NE` and so on stand for the operators for `not equal' and so on. We assume that these tokens are more than one character long and therefore are represented by names, not character literals.)

### 7.2.4: How Precedence Works

The first effect of the precedence directives is to assign precedence levels to the terminal symbols declared. The second effect is to assign precedence levels to certain rules: each rule gets its precedence from the last terminal symbol mentioned in the components. (You can also specify explicitly the precedence of a rule. See section 7.3).

Finally, the resolution of conflicts works by comparing the precedence of the rule being considered with that of the look-ahead token. If the token's precedence is higher, the choice is to shift. If the rule's precedence is higher, the choice is to reduce. If they have equal precedence, the choice is made based on the associativity of that precedence level. The verbose output file made by ``-V`' (see section 9) shows how each conflict was resolved.

Not all rules and not all tokens have precedence. If either the rule or the look-ahead token has no precedence, then the default is to shift.

### 7.2.5: Rule precedence

Consider the following (somewhat peculiar) grammar:

        %token  ID
        %left  '-'
        %left  '*'
        %right UNARY

        %%

        expr:
            expr '-' term
        |
            term
        ;

        term:
            term '*' factor
        |
            factor
        ;

        factor:
           '-' expr %prec UNARY
        |
            ID
        ;

Even though operator precedence and association rules are used the grammar still displays a shift/reduce conflict. One of the grammar's states consists of the following two items:

        0: expr -> term  .
        1: term -> term  . '*' factor

and bisonc++ reduces to item 0, dropping item 1 rather than shifting a `'*'` and proceeding with item 0.

When considering states where shift/reduce conflicts are encountered the `shiftable' items of these states shift when encountering terminal tokens that are also in the follow sets of the reducible items of these states. In the above example item 1 shifts when `'*'` is encountered, but `'*'` is also an element of the set of look-ahead tokens of item 0. Bisonc++ must now decide what to do. In cases we've seen earlier bisonc++ could make the decision because the reducible item itself had a well known precedence. The precedence of a reducible item is defined as the precedence of the rule's LHS. Item 0 in the above example is an item of the rule `expr -> term`.

The precedence of a production rule is defined as follows:

- If `%prec` is used then the precedence of the production rule is equal to the precedence of the terminal that is specified with the `%prec` directive;
- If `%prec` is not used then the production rule's precedence is equal to the precedence of the first terminal token that is used in the production rule;
- In all other cases the production rule's precedence is set to the maximum possible precedence.

Since `expr -> term` does not contain a terminal token and does not use `%prec`, its precedence is the maximum possible precedence. Consequently in the above state the shift/reduce conflict is solved by reducing rather than shifting.

Some final remark as to why the above grammar is peculiar. It is peculiar as it combines precedence and association specifying directives with auxiliary nonterminals that may be useful conceptually (or when implementing an expression parser `by hand') but which are not required when defining grammars for bisonc++. The following grammar does not use `term` and `factor` but recognizes the same grammar as the above `peculiar' grammar without reporting any shift/reduce conflict:

        %token  ID
        %left  '-'
        %left  '*'
        %right UNARY

        %%

        expr:
            expr '-' expr
        |
            expr '*' expr
        |
           '-' expr %prec UNARY
        |
            ID
        ;

## 7.3: Context-Dependent Precedence

Often the precedence of an operator depends on the context. This sounds outlandish at first, but it is really very common. For example, a minus sign typically has a very high precedence as a unary operator, and a somewhat lower precedence (lower than multiplication) as a binary operator.

The bisonc++ precedence directives, %left, %right and %nonassoc, can only be used once for a given token; so a token has only one precedence declared in this way. For context-dependent precedence, you need to use an additional mechanism: the %prec modifier for rules.

The %prec modifier declares the precedence of a particular rule by specifying a terminal symbol whose precedence should be used for that rule. It's not necessary for that symbol to appear otherwise in the rule. The modifier's syntax is:

    %prec terminal-symbol

and it is written after the components of the rule. Its effect is to assign the rule the precedence of terminal-symbol, overriding the precedence that would be deduced for it in the ordinary way. The altered rule precedence then affects how conflicts involving that rule are resolved (see section Operator Precedence).

Here is how %prec solves the problem of unary minus. First, declare a precedence for a fictitious terminal symbol named UMINUS. There are no tokens of this type, but the symbol serves to stand for its precedence:

        ...
        %left '+' '-'
        %left '*'
        %left UMINUS

Now the precedence of UMINUS can be used in specific rules:

        exp:
            ...
            | exp '-' exp
            ...
            | '-' exp %prec UMINUS

## 7.4: Reduce/Reduce Conflicts

A reduce/reduce conflict occurs if there are two or more rules that apply to the same sequence of input. This usually indicates a serious error in the grammar.

For example, here is an erroneous attempt to define a sequence of zero or more words:

        %stype char *
        %token WORD

        %%

        sequence:
            // empty
            {
                cout << "empty sequence\n";
            }
        |
            maybeword
        |
            sequence WORD
            {
                cout << "added word " << $2 << endl;
            }
        ;

        maybeword:
            // empty
            {
                cout << "empty maybeword\n";
            }
        |
            WORD
            {
                cout << "single word " << $1 << endl;
            }
        ;

The error is an ambiguity: there is more than one way to parse a single word into a sequence. It could be reduced to a maybeword and then into a sequence via the second rule. Alternatively, nothing-at-all could be reduced into a sequence via the first rule, and this could be combined with the word using the third rule for sequence.

There is also more than one way to reduce nothing-at-all into a sequence. This can be done directly via the first rule, or indirectly via maybeword and then the second rule.

You might think that this is a distinction without a difference, because it does not change whether any particular input is valid or not. But it does affect which actions are run. One parsing order runs the second rule's action; the other runs the first rule's action and the third rule's action. In this example, the output of the program changes.

Bisonc++ resolves a reduce/reduce conflict by choosing to use the rule that appears first in the grammar, but it is very risky to rely on this. Every reduce/reduce conflict must be studied and usually eliminated. Here is the proper way to define sequence:

        sequence:
            // empty
            { printf ("empty sequence\n"); }
        | sequence word
            { printf ("added word %s\n", $2); }
        ;

Here is another common error that yields a reduce/reduce conflict:

        sequence:
            // empty
            | sequence words
            | sequence redirects
        ;

        words:
            // empty
            | words word
        ;

        redirects:
            // empty
            | redirects redirect
        ;

The intention here is to define a sequence containing either `word` or `redirect` nonterminals. The individual definitions of sequence, words and redirects are error-free, but the three together make a subtle ambiguity: even an empty input can be parsed in infinitely many ways!

Consider: nothing-at-all could be a words. Or it could be two words in a row, or three, or any number. It could equally well be a redirects, or two, or any number. Or it could be a words followed by three redirects and another words. And so on.

Here are two ways to correct these rules. First, to make it a single level of sequence:

        sequence:
            // empty
            | sequence word
            | sequence redirect
        ;

Second, to prevent either a words or a redirects from being empty:

        sequence:
            // empty
            | sequence words
            | sequence redirects
        ;

        words:
            word
            | words word
        ;

        redirects:
            redirect
            | redirects redirect
        ;

## 7.5: Mysterious Reduce/Reduce Conflicts

Sometimes reduce/reduce conflicts occur that are puzzling at first sight. Here is an example:

        %token ID

        %%
        def:
            param_spec return_spec ','
        ;

        param_spec:
            type
        |
            name_list ':' type
        ;

        return_spec:
            type
        |
            name ':' type
        ;

        type:
            ID
        ;

        name:
            ID
        ;

        name_list:
            name
        |
            name ',' name_list
        ;

It would seem that this grammar can be parsed with only a single look-ahead token: when a param_spec is being read, an `ID` is a `name` if a comma or colon follows, or a `type` if another `ID` follows. In other words, this grammar is LR(1).

However, bisonc++, like most parser generators, cannot actually handle all LR(1) grammars. In this grammar two contexts, one after an `ID` at the beginning of a `param_spec` and another one at the beginning of a `return_spec`, are similar enough for bisonc++ to assume that they are identical. They appear similar because the same set of rules would be active--the rule for reducing to a name and that for reducing to a type. Bisonc++ is unable to determine at that stage of processing that the rules would require different look-ahead tokens in the two contexts, so it makes a single parser state for them both. Combining the two contexts causes a conflict later. In parser terminology, this occurrence means that the grammar is not LALR(1).

In general, it is better to fix deficiencies than to document them. But this particular deficiency is intrinsically hard to fix; parser generators that can handle LR(1) grammars are hard to write and tend to produce parsers that are very large. In practice, bisonc++ is more useful the way it's currently operating.

When the problem arises, you can often fix it by identifying the two parser states that are being confused, and adding something to make them look distinct. In the above example, adding one rule to `return_spec` as follows makes the problem go away:

        %token BOGUS
        ...
        %%
        ...
        return_spec:
            type
        |
            name ':' type
        |
            ID BOGUS        // This rule is never used.
        ;

This corrects the problem because it introduces the possibility of an additional active rule in the context after the `ID` at the beginning of `return_spec`. This rule is not active in the corresponding context in a `param_spec`, so the two contexts receive distinct parser states. As long as the token `BOGUS` is never generated by the parser's member function `lex()`, the added rule cannot alter the way actual input is parsed.

In this particular example, there is another way to solve the problem: rewrite the rule for `return_spec` to use `ID` directly instead of via name. This also causes the two confusing contexts to have different sets of active rules, because the one for `return_spec` activates the altered rule for `return_spec` rather than the one for name.

        param_spec:
            type
        |
            name_list ':' type
        ;

        return_spec:
            type
        |
            ID ':' type
        ;

- Table of Contents
- Previous Chapter
- Next Chapter