Implementing External DSLs Using Scala Parser Combinators

Implementing External DSLs Using Scala Parser Combinators St. Louis Lambda Lounge Sept. 3, 2009 Tim Dalton Senior Software Engineer Object Computing Inc.

External vs Internal DSL
Internal DSLs are implemented using syntax of “host” programming language
Examples
Fluent APIs in Java
RSpec and ScalaSpec
Constrained by features of programming language
External DSLs syntax is only limited by capabilities of the parser

What is a Combinator ?
Combinators are functions that can be combined to perform more complex operations
Concept originates in Lambda Calculus
Mostly comes from the Haskell community
Haskell implementations use Monads
Scala implementation “almost Monadic”

Scala’s Parser Implementation
Context-free LL grammar
Left to right
Leftmost derivation
Recursive descent
Backtracking
There are ways to prevent backtracking
Advances planned for Scala 2.8
Support for Packrat parsing
Parser Expression Grammar
More predictive with less recursion and backtracking

Scala Combinator Parser Hierarchy
scala.util.parsing.combinator.Parsers
scala.util.parsing.combinator.syntactical.TokenParsers
scala.util.parsing.combinator.syntactical.StdTokenParsers
scala.util.parsing.combinator.RegexParsers
scala.util.parsing.combinator.JavaTokenParsers

A Simple Logo(-Like) Interpreter
Only a few commands:
Right Turn <angle-degrees>
Left Turn <angle-degrees>
Forward <number-of-pixels>
Repeat <nested sequence of other commands>

Grammar for Simple Logo
forward = (“FORWARD” | “FD”) positive-integer
right = (“RIGHT” | “RT) positive-integer
left = (“LEFT” | “LT”) positive-integer
repeat = “REPEAT” positive-integer “[“{statement}”]”
statement = right | left | forward | repeat
program = { statement }

Scala Code to Implement Parser
object LogoParser extends RegexParsers {
def positiveInteger = """d+"""r
def forward = ("FD"|"FORWARD")~positiveInteger
def right = ("RT"|"RIGHT")~positiveInteger
def left = ("LT"|"LEFT")~positiveInteger
def repeat = "REPEAT" ~ positiveInteger ~ "[" ~ rep(statement) ~ "]"
def statement:Parser[Any] = forward | right | left | repeat
def program = rep(statement)
}

Scala Code to Implement Parser
An internal DSL is used to implement an External One
Methods on preceding slide are referred to as parser generators
RegexParsers is subclass of Parsers trait that provides a generic parser combinator

A Closer Look
def positiveInteger = """d+"""r
The trailing “r” is a method call the converts the string to a Regex object
More verbose syntax:
"""d+""".r()
String does not have an r() method !!
Class RichString does, so an implicit conversion is done

Implicit Conversions
One of the more powerful / dangerous features of Scala is implicit conversions
RichString.r method signature
def r : Regex
scala.Predef implicit convertor
implicit def stringWrapper( x : java.lang.String) : RichString
The Scala compiler will look for implicit convertors in scope and insert them implicitly
“ With great power, comes great responsibility”

Back to the Parser
The “|” and “~” are methods of class Parsers.Parser[T] !!
RegexParser has implicit conversions:
implicit def literal(s : String) : Parser[String]
implicit def regex(r : Regex) : Parser[String]
Parser generator methods should return something that can be at least be converted to Parser[T]

Parser[T]’s and ParseResult[T]’s
Parsers.Parser[T]
Extends Reader => ParseResult[T]
This makes it a function object
ParserResult[T] Hierarchy:
Parsers.Success
Parsers.NoSuccess
Parsers.Failure
Parsers.Error
Invoking Parsers[T] function object return one of the above subclasses

Combining Parser[T]’s
Signature for Parser[T].| method:
def |[U >: T](q : => Parser[U]) : Parser[U]
Parser Combinator for alternative composition (OR)
Succeeds (returns Parsers.Success) if either “this” Parser[T] succeeds or “q” Parser[U] succeeds
Type U must be same or super-class of type T.

Combining Parser[T]’s
Signature of Parser[T].~ method:
def ~[U](p : => Parser[U]) : Parser[~[T, U]]
Parser Combinator for sequential composition
Succeeds only if “this” Parser succeeds and “q” Parser succeeds
Return an instance “~” that contain both results
Yes, “~” is also a class !
Like a Pair, but easier to pattern match on

Forward March
Back to the specification of forward:
For this combinator to succeed,
Either the Parser for literal “FD” or “FORWARD”
And the Parser for the positiveInt Regex
Both the literal strings and Regex result of positiveInt are implicitly converted to Parser[String]

Repetition
Next lines of note:
def repeat =
"REPEAT" ~ positiveInteger ~ "[" ~ rep(statement) ~ "]"
def statement:Parser[Any] =
forward | right | left | repeat
Type for either repeat or statement need to be explicitly specified due to recursion
The rep method specifies that Parser can be repeated

Repetition
Signature for Parsers.rep method:
def rep[T](p : => Parser[T]) : Parser[List[T]]
Parser Combinator for repetitions
Parses input until Parser, p, fails. Returns consecutive successful results as List.

Other Forms of Repetition
def repsep[T](p: => Parser[T], q: => Parser[Any]) : Parser[List[T]]
Specifies a Parser to be interleaved in the repetition
Example: repsep(term, ",")
def rep1[T](p: => Parser[T]): Parser[List[T]]
Parses non-empty repetitions
def repN[T](n : Int, p : => Parser[T]) : Parser[List[T]]
Parses a specified number of repetitions

Execution
Root Parser Generator:
def program = rep(statement)
To Execute the Parser
parseAll(program, "REPEAT 4 [FD 100 RT 90]")
Returns Parsers.Success[List[Parsers.~[…]]]
Remember ,Parsers.Success[T] is subclass of ParseResult[T]
toString:
[1.24] parsed: List(((((REPEAT~4)~[)~List((FD~100), (RT~90)))~]))
The “…” indicates many levels nested Parsers

Not-so-Happy Path
Example of failed Parsing:
parseAll(program, "REPEAT 4 [FD 100 RT 90 ) ")
Returns Parsers.Failure
Subclass of ParseResult[Nothing]
toString:
[1.23] failure: `]' expected but `)' found
REPEAT 4 [FD 100 RT 90)
^
Failure message not always so “precise”

Making Something Useful
Successful parse results need to transformed into something that can be evaluated
Enter the “eye brows” method of Parser[T]:
def ^^[U](f : (T) => U) : Parser[U]
Parser combinator for function application

Eye Brows Example
Example of “^^” method:
def positiveInteger = """d+""".r ^^
{ x:String => x.toInt }
Now positiveInteger generates Parser[Int] instead of Parser[String]
Transformer can be shortened to “{ _.toInt }”

Implementing Commands
For the statements we need a hierarchy of command classes:
sealed abstract class LogoCommand
case class Forward(x: Int) extends LogoCommand
case class Turn(x: Int) extends LogoCommand
case class Repeat(i: Int, e: List[LogoCommand]) extends LogoCommand

Transforming into Commands
The Forward command:
def forward = ("FD"|"FORWARD")~positiveInteger ^^
{ case _~value => Forward(value) }
A ~[String, Int] is being passed in the transformer
Pattern matching is to extract the Int, value and construct a Forward instance
Forward is a case class, so “new” not needed
Case constructs can be partial functions themselves. Longer form:
… ^^ { tilde => tilde match
{ case _~value => Forward(value) }}

Derivates of “~”
Two methods related to “~”:
def <~ [U](p: => Parser[U]): Parser[T]
Parser combinator for sequential composition which keeps only the left result
def ~> [U](p: => Parser[U]): Parser[U]
Parser combinator for sequential composition which keeps only the right result
Note, neither returns a “~” instance
The forward method can be simplified:
def forward = ("FD"|"FORWARD")~>positiveInteger ^^
{ Forward(_) }

Updated Parser
def positiveInteger = """d+""".r ^^ { _.toInt }
def forward = ("FD"|"FORWARD")~>positiveInteger ^^
{ Forward(_) }
def right = ("RT"|"RIGHT")~>positiveInteger ^^
{ x => Turn(-x) }
def left = ("LT"|"LEFT")~>positiveInteger ^^
{ Turn(_) }
def repeat = "REPEAT" ~> positiveInteger ~ "[" ~ rep(statement) <~ "]" ^^
{ case number~_~statements => Repeat(number, statements)}

Updated Parser Results
Executing the Parser now:
parseAll(program, "REPEAT 4 [FD 100 RT 90]")
Results:
[1.24] parsed: List(Repeat(4,List(Forward(100), Turn(-90))))
Returns Parsers.Success[List[Repeat]]
This can be evaluated !!

Evaluation
class LogoEvaluationState {
var x = 0
var y = 0
var heading = 0
}
implicit def dblToInt(d: Double):Int = if (d > 0) (d+0.5).toInt else (d-0.5).toInt
def parse(s: String) : List[LogoCommand] = LogoParser.parse(s).get
def evaluate(parseResult: LogoParser.ParseResult[List[LogoCommand]], g:Graphics2D) {
var state = new LogoEvaluationState
if (parseResult.successful) {
evaluate(parseResult.get, g, state)
}
// draw turtle
evaluate(parse("RT 90 FD 3 LT 110 FD 10 LT 140 FD 10 LT 110 FD 3"), g, state)
} // Continued...

Evaluation (More Functional)
private def evaluate(list: List[LogoCommand], g:Graphics2D, state:LogoEvaluationState) {
if (!list.isEmpty) {
val head :: tail = list
head match {
case Forward(distance) => {
val (nextX, nextY) =
(state.x + distance * sin(toRadians(state.heading)),
state.y + distance * cos(toRadians(state.heading)))
g.drawLine(state.x, state.y, nextX, nextY)
state.x = nextX
state.y = nextY
evaluate(tail, g, state)
}
case Turn(degrees) => {
state.heading += degrees
evaluate(tail, g, state)
}
case Repeat(0, _) => evaluate(tail, g, state)
case Repeat(count, statements) =>
evaluate(statements ::: Repeat(count-1, statements)::tail, g, state)
}
}
}

Evaluation (More Imperative)
def evaluate(list: List[LogoCommand], g:Graphics2D, state:LogoEvaluationState) { list.foreach(evaluate(_, g, state))
}
def evaluate(command:LogoCommand, g:Graphics2D, state:LogoEvaluationState) {
command match {
case Forward(distance) => {
val (nextX, nextY) = (state.x + distance * Math.sin(Math.toRadians(state.heading)),
state.y + distance * Math.cos(Math.toRadians(state.heading)))
g.drawLine(state.x, state.y, nextX, nextY)
state.x = nextX
state.y = nextY
}
case Turn(degrees) => state.heading += degrees
case Repeat(count, statements) => (0 to count).foreach { _ =>
evaluate(statements, g, state)
}
}
}

Demonstration

Implementing External DSLs Using Scala Parser Combinators

by Tim Dalton

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Implementing External DSLs Using Scala Parser Combinators Presentation Transcript