Syntax and parser generators II-III
Lecture 4-5

Review
Implementing semantic rules
Visitor pattern
Example: Counting addition/subtraction operations
Visitor implementation pseudo-code
Defining traversal return type: generics
Implementing the Counter visitor
Program transformation with visitors
Implementing infix to postfix
Implementing an evaluator
Homework

Review

Questions about the last class?

Infix-to-postfix transformation

Grammar
Semantic rules

Tree traversals

Binary trees
Write preorder, postorder, and inorder traversals of the tree

What do you notice about {post,pre,in}fix notation and traversals?

Quiz

Write pseudo-code for a pre-order traversal of a binary tree.

Quiz Discussion

Implementing semantic rules

Parser creates trees based on grammar
Compiler traverses tree
- Applies semantic rules to tree

Visitor pattern

Visitors add methods to objects without modifying the class
We can abstract away new traversal into new visitor classes

Example implementation of double-dispatch Visitor in java

https://github.com/appleseedlab/superc/blob/master/src/xtc/tree/Visitor.java

Uses reflection to pick the right visitXXX function based on the object type of the child

Example: Counting addition/subtraction operations

Recall Calc grammar
Only want to count additions
- One visitor function per grammar construct
Visitor class handles dispatching to correct method

Visitor implementation pseudo-code

int visitMulDiv(n) {
  return visit(leftexpr) + visit(rightexpr);
}

int visitAddSub(n) {
  return visit(leftexpr) + visit(rightexpr) + 1;
}

int visitInt(n) {
  return 0;
}

int visitParens(n) {
  return visit(expr);
}

Defining traversal return type: generics

ANTLR's generate visitor base class:

public class CalcBaseVisitor<T>
  extends AbstractParseTreeVisitor<T>
  implements CalcVisitor<T> {
  // ...
}

Our implementation

public class Counter extends CalcBaseVisitor<Integer> {
  // ...
}

Implementing the Counter visitor

Generating the parser with a visitor

# add antlr to your classpath; your path to the runtime may vary
export CLASSPATH=/usr/share/java/antlr4-runtime.jar:$CLASSPATH
# this generates the parser, -visitor includes the visitor classes
antlr4 -visitor Calc.g4

Using ANTLR's visitor

Have the grammar file open as a reference
Use the CalcVisitor (or CalcBaseVisitor) as a guide
Visiting child nodes
- visit(ctx.expr())
Many child nodes of same type
- visit(ctx.expr(0)) + visit(ctx.expr(1))

Implementing the visitor

(Together in class)

public class Counter extends CalcBaseVisitor<Integer> {
  @Override public Integer visitParens(CalcParser.ParensContext ctx) {
    return visit(ctx.expr());
  }

  @Override public Integer visitMulDiv(CalcParser.MulDivContext ctx) {
    return visit(ctx.expr(0)) + visit(ctx.expr(1));
  }

  @Override public Integer visitAddSub(CalcParser.AddSubContext ctx) {
    return visit(ctx.expr(0)) + visit(ctx.expr(1)) + 1;
  }

  @Override public Integer visitInt(CalcParser.IntContext ctx) {
    return 0;
  }
}

Running the visitor

Boilerplate for ANTLR-generated parsers
Visiting the root of the node

Building

javac RunCounter.java; javac Counter.java

Running

echo "1+2*3" | java RunCounter
echo "((1))+2*3+3+5*(((7+2))*2)" | java RunCounter

import org.antlr.v4.runtime.*;
import org.antlr.v4.runtime.tree.*;
import java.io.PrintWriter;
import java.io.FileInputStream;
import java.io.InputStream;
import java.io.FileWriter;

public class RunCounter {
  public static void main(String[] args) throws Exception {
    // process input file
    String inputFile = null;
    if (args.length > 0) inputFile = args[0];
    CharStream input;
    if (inputFile != null) {
      input = CharStreams.fromFileName(inputFile);
    } else {
      input = CharStreams.fromStream(System.in);
    }

    // lexing and parsing
    CalcLexer lexer = new CalcLexer(input);
    CommonTokenStream tokens = new CommonTokenStream(lexer);
    CalcParser parser = new CalcParser(tokens);
    ParseTree tree = parser.expr();
    System.err.println(tree.toStringTree(parser));

    // visitor
    Counter counter = new Counter();
    Integer result = counter.visit(tree);
    System.out.println(result);
  }
}

Program transformation with visitors

Transformation source code

Input: program source code
Output: program source code
Simplest transformation: the identity transfom
- Rewrites to same program
Why identity?
- Starting point for transformations

Identity transform with ANTLR visitors

What is the return type?
Creating the base visitor
Using a StringBuilder for ease-of-use/performance

Getting the text of tokens in ANTLR

Getting the text of a token
- ctx.INT().getText()
- ctx.op.getText()

Creating the visitor

(Together in class)
Keep the grammar open as a reference
Start with the base visitor
Define the return type
Define the transformation for each construct
- Use ANTLR's accessors to get values

import java.lang.StringBuilder;

public class Identity extends CalcBaseVisitor<String> {
  @Override public String visitParens(CalcParser.ParensContext ctx) {
    StringBuilder sb = new StringBuilder();
    sb.append("(");
    sb.append(visit(ctx.expr()));
    sb.append(")");
    return sb.toString();
  }

  @Override public String visitMulDiv(CalcParser.MulDivContext ctx) {
    StringBuilder sb = new StringBuilder();
    sb.append(visit(ctx.expr(0)));
    sb.append(ctx.op.getText());
    sb.append(visit(ctx.expr(1)));
    return sb.toString();
  }

  @Override public String visitAddSub(CalcParser.AddSubContext ctx) {
    StringBuilder sb = new StringBuilder();
    sb.append(visit(ctx.expr(0)));
    sb.append(ctx.op.getText());
    sb.append(visit(ctx.expr(1)));
    return sb.toString();
  }

  @Override public String visitInt(CalcParser.IntContext ctx) {
    StringBuilder sb = new StringBuilder();
    sb.append(ctx.INT().getText());
    return sb.toString();
  }
}

import org.antlr.v4.runtime.*;
import org.antlr.v4.runtime.tree.*;
import java.io.PrintWriter;
import java.io.FileInputStream;
import java.io.InputStream;
import java.io.FileWriter;

public class RunIdentity {
  public static void main(String[] args) throws Exception {
    // process input file
    String inputFile = null;
    if (args.length > 0) inputFile = args[0];
    CharStream input;
    if (inputFile != null) {
      input = CharStreams.fromFileName(inputFile);
    } else {
      input = CharStreams.fromStream(System.in);
    }

    // lexing and parsing
    CalcLexer lexer = new CalcLexer(input);
    CommonTokenStream tokens = new CommonTokenStream(lexer);
    CalcParser parser = new CalcParser(tokens);
    ParseTree tree = parser.expr();
    System.err.println(tree.toStringTree(parser));

    // visitor
    Identity identity = new Identity();
    String result = identity.visit(tree);
    System.out.println(result);
  }
}

Compiling the transform

Building and running the identity transform

javac Identity.java; javac RunIdentity.java
echo "1*2+3" | java RunIdentity

Implementing infix to postfix

How can we modify the identity transform to make postfix?

Implementing an evaluator

How could we use the visitor to evaluate the expression?

ANTLR tip

Checking the op
```
ctx.op.getType() == CalcParser.ADD
```

Adding storage to the evaluator

How could we add variables to our calculator?

Homework

(1 week)

Create an infix to prefix translator based on the identity transform and its runner

Syntax and parser generators II-III Lecture 4-5

Table of Contents