A dive into Graham Hutton's "Higher Order Functions for Parsing", which appeared in the Journal of Functional Programming in 19992. All the techniques have been worked through in Haskell.

1. Combinator
Parsing
By Swanand Pagnis

2. Higher-Order Functions for
Parsing
By Graham Hutton

3. • Abstract & Introduction
• Build a parser, one fn at a time
• Moving beyond toy parsers

4. Abstract

5. In combinator parsing, the text of parsers resembles BNF
notation. We present the basic method, and a number of
extensions. We address the special problems presented by
whitespace, and parsers with separate lexical and syntactic
phases. In particular, a combining form for handling the
“offside rule” is given. Other extensions to the basic
method include an “into” combining form with many useful
applications, and a simple means by which combinator
parsers can produce more informative error messages.

6. • Combinators that resemble BNF notation
• Whitespace handling through "Offside
Rule"
• "Into" combining form for advanced parsing
• Strategy for better error messages

7. Introduction

8. Primitive Parsers
• Take input
• Process one character
• Return results and unused input

9. Combinators
• Combine primitives
• Define building blocks
• Return results and unused input

10. Lexical analysis and syntax
• Combine the combinators
• Define lexical elements
• Return results and unused input

11. input: "from:swiggy to:me"
output: [("f", "rom:swiggy to:me")]

12. input: "42 !=> ans"
output: [("4", "2 !=> ans")]

13. rule: 'a' followed by 'b'
input: "abcdef"
output: [(('a','b'),"cdef")]

14. rule: 'a' followed by 'b'
input: "abcdef"
output: [(('a','b'),"cdef")]
Combinator

15. Language choice

16. Suggested:
Lazy Functional Languages

17. Miranda:
Author's choice

18. Haskell:
An obvious choice. 🤓

19. Racket:
Another obvious choice. 🤓

20. Ruby:
🍎 to 🍊 so $ for learning

21. OCaml:
Functional, but not lazy.

22. Haskell %

23. Simple when stick to fundamental FP
• Higher order functions
• Immutability
• Recursive problem solving
• Algebraic types

24. Let's build a parser,
one fn at a time

25. type Parser a b = [a] !-> [(b, [a])]

26. Types help with abstraction
• We'll be dealing with parsers and combinators
• Parsers are functions, they accept input and
return results
• Combinators accept parsers and return
parsers

27. A parser is a function that
accepts an input and returns
parsed results and the
unused input for each result

28. Parser is a function type that
accepts a list of type a
and returns all possible results
as a list of tuples of type (b, [a])

29. (Parser Char Number)
input: "42 it is!" !-- a is a [Char]
output: [(42, " it is!")] !-- b is a Number

30. type Parser a b = [a] !-> [(b, [a])]

31. Primitive Parsers

32. succeed !:: b !-> Parser a b
succeed v inp = [(v, inp)]

33. Always succeeds
Returns "v" for all inputs

34. failure !:: Parser a b
failure inp = []

35. Always fails
Returns "[]" for all inputs

36. satisfy !:: (a !-> Bool) !-> Parser a a
satisfy p [] = failure []
satisfy p (x:xs)
| p x = succeed x xs !-- if p(x) is true
| otherwise = failure []

37. satisfy !:: (a !-> Bool) !-> Parser a a
satisfy p [] = failure []
satisfy p (x:xs)
| p x = succeed x xs !-- if p(x) is true
| otherwise = failure []
Guard Clauses, if you want to Google

38. literal !:: Eq a !=> a !-> Parser a a
literal x = satisfy (!== x)

39. match_3 = (literal '3')
match_3 "345" !-- !=> [('3',"45")]
match_3 "456" !-- !=> []

40. succeed
failure
satisfy
literal

41. Combinators

42. match_3_or_4 = match_3 `alt` match_4
match_3_or_4 "345" !-- !=> [('3',"45")]
match_3_or_4 "456" !-- !=> [('4',"56")]

43. alt !:: Parser a b !-> Parser a b !-> Parser a b
(p1 `alt` p2) inp = p1 inp !++ p2 inp

44. (p1 `alt` p2) inp = p1 inp !++ p2 inp
List concatenation

45. (match_3 `and_then` match_4) "345"
# !=> [(('3','4'),"5")]

46. 🐉

47. and_then !:: Parser a b !-> Parser a c !-> Parser a (b, c)
(p1 `and_then` p2) inp =
[ ((v1, v2), out2) | (v1, out1) !<- p1 inp,
(v2, out2) !<- p2 out1 ]

48. and_then !:: Parser a b !-> Parser a c !-> Parser a (b, c)
(p1 `and_then` p2) inp =
[ ((v1, v2), out2) | (v1, out1) !<- p1 inp,
(v2, out2) !<- p2 out1 ]
List comprehensions

49. (v11, out11)
(v12, out12)
(v13, out13)
…
(v21, out21)
(v22, out22)
…
(v31, out31)
(v32, out32)
…
p1
p2

50. ((v11, v21), out21)
((v11, v22), out22)
…

51. (match_3 `and_then` match_4) "345"
# !=> [(('3','4'),"5")]

52. Manipulating values

53. match_3 = (literal '3')
match_3 "345" !-- !=> [('3',"45")]
match_3 "456" !-- !=> []

54. (number "42") "42 !=> answer"
# !=> [(42, " answer")]

55. (keyword "for") "for i in 1!..42"
# !=> [(:for, " i in 1!..42")]

56. using !:: Parser a b !-> (b !-> c) !-> Parser a c
(p `using` f) inp =
[(f v, out) | (v, out) !<- p inp ]

57. ((string "3") `using` float) "3"
# !=> [(3.0, "")]

58. Levelling up

59. many !:: Parser a b !-> Parser a [b]
many p =
((p `and_then` many p) `using` cons)
`alt` (succeed [])

60. 0 or many

61. (many (literal 'a')) "aab"
!=> [("aa","b"),("a","ab"),("","aab")]

62. (many (literal 'a')) "xyz"
!=> [("","xyz")]

63. some !:: Parser a b !-> Parser a [b]
some p =
((p `and_then` many p)
`using` cons)

64. 1 or many

65. (some (literal 'a')) "aab"
!=> [("aa","b"),("a","ab")]

66. (some (literal 'a')) "xyz"
!=> []

67. positive_integer = some (satisfy Data.Char.isDigit)
negative_integer = ((literal '-')
`and_then` positive_integer) `using` cons
positive_decimal = (positive_integer `and_then`
(((literal '.') `and_then` positive_integer)
`using` cons)) `using` join
negative_decimal = ((literal '-')
`and_then` positive_decimal) `using` cons

68. number !:: Parser Char [Char]
number = negative_decimal
`alt` positive_decimal
`alt` negative_integer
`alt` positive_integer

69. word !:: Parser Char [Char]
word = some (satisfy isLetter)

70. string !:: (Eq a) !=> [a] !-> Parser a [a]
string [] = succeed []
string (x:xs) =
(literal x `and_then` string xs)
`using` cons

71. (string "begin") "begin end"
# !=> [("begin"," end")]

72. xthen !:: Parser a b !-> Parser a c !-> Parser a c
p1 `xthen` p2 = (p1 `and_then` p2) `using` snd

73. thenx !:: Parser a b !-> Parser a c !-> Parser a b
p1 `thenx` p2 = (p1 `and_then` p2) `using` fst

74. ret !:: Parser a b !-> c !-> Parser a c
p `ret` v = p `using` (const v)

75. succeed, failure, satisfy, literal,
alt, and_then, using, string,
many, some, string, word, number,
xthen, thenx, ret

76. Expression Parser & Evaluator

77. data Expr = Const Double
| Expr `Add` Expr
| Expr `Sub` Expr
| Expr `Mul` Expr
| Expr `Div` Expr

78. (Const 3) `Mul`
((Const 6) `Add` (Const 1)))
# !=> "3*(6+1)"

79. parse "3*(6+1)"
# !=> (Const 3) `Mul`
((Const 6) `Add` (Const 1)))

80. (Const 3) Mul
((Const 6) `Add` (Const 1)))
# !=> 21

81. BNF Notation
expn !::= expn + expn
| expn − expn
| expn ∗ expn
| expn / expn
| digit+
| (expn)

82. Improving a little:
expn !::= term + term
| term − term | term
term !::= factor ∗ factor
| factor / factor | factor
factor !::= digit+ | (expn)

83. Parsers that resemble BNF

84. addition =
((term `and_then`
((literal '+') `xthen` term))
`using` plus)

85. subtraction =
((term `and_then`
((literal '-') `xthen` term))
`using` minus)

86. multiplication =
((factor `and_then`
((literal '*') `xthen` factor))
`using` times)

87. division =
((factor `and_then`
((literal '/') `xthen` factor))
`using` divide)

88. parenthesised_expression =
((nibble (literal '('))
`xthen` ((nibble expn)
`thenx`(nibble (literal ')'))))

89. value xs = Const (numval xs)
plus (x,y) = x `Add` y
minus (x,y) = x `Sub` y
times (x,y) = x `Mul` y
divide (x,y) = x `Div` y

90. expn = addition
`alt` subtraction
`alt` term

91. term = multiplication
`alt` division
`alt` factor

92. factor = (number `using` value)
`alt` parenthesised_expn

93. expn "12*(5+(7-2))"
# !=> [
(Const 12.0 `Mul`
(Const 5.0 `Add`
(Const 7.0 `Sub` Const 2.0)),""), … ]

94. value xs = Const (numval xs)
plus (x,y) = x `Add` y
minus (x,y) = x `Sub` y
times (x,y) = x `Mul` y
divide (x,y) = x `Div` y

95. value = numval
plus (x,y) = x + y
minus (x,y) = x - y
times (x,y) = x * y
divide (x,y) = x / y

96. expn "12*(5+(7-2))"
# !=> [(120.0,""),
(12.0,"*(5+(7-2))"),
(1.0,"2*(5+(7-2))")]

97. expn "(12+1)*(5+(7-2))"
# !=> [(130.0,""),
(13.0,"*(5+(7-2))")]

98. Moving beyond toy parsers

99. Whitespace? 🤔 (

100. white = (literal " ")
`alt` (literal "t")
`alt` (literal "n")

101. white = many (any literal " tn")

102. /s!*/

103. any p = foldr (alt.p) fail

104. any p [x1,x2,!!...,xn] =
(p x1) `alt` (p x2) `alt` !!...
`alt` (p xn)

105. white = many (any literal " tn")

106. nibble p =
white `xthen` (p `thenx` white)

107. The parser (nibble p) has the
same behaviour as parser p,
except that it eats up any white-
space in the input string before
or afterwards

108. (nibble (literal 'a')) " a "
# !=> [('a',""),('a'," "),('a'," ")]

109. symbol = nibble.string

110. symbol "$fold" " $fold "
# !=> [("$fold", ""), ("$fold", " ")]

111. The Offside Rule

112. w = x + y
where
x = 10
y = 15 - 5
z = w * 2

113. w = x + y
where
x = 10
y = 15 - 5
z = w * 2

114. When obeying the offside rule,
every token must lie
either directly below,
or to the right of its first token

115. i.e. A weak indentation policy

116. The Offside Combinator

117. type Pos a = (a, (Integer, Integer))

118. prelex "3 + n 2 * (4 + 5)"
# !=> [('3',(0,0)),
('+',(0,2)),
('2',(1,2)),
('*',(1,4)), … ]

119. satisfy !:: (a !-> Bool) !-> Parser a a
satisfy p [] = failure []
satisfy p (x:xs)
| p x = succeed x xs !-- if p(x) is true
| otherwise = failure []

120. satisfy !:: (a !-> Bool) !-> Parser (Pos a) a
satisfy p [] = failure []
satisfy p (x:xs)
| p a = succeed a xs !-- if p(a) is true
| otherwise = failure []
where (a, (r, c)) = x

121. satisfy !:: (a !-> Bool) !-> Parser (Pos a) a
satisfy p [] = failure []
satisfy p (x:xs)
| p a = succeed a xs !-- if p(a) is true
| otherwise = failure []
where (a, (r, c)) = x

122. offside !:: Parser (Pos a) b !-> Parser (Pos a) b
offside p inp = [(v, inpOFF) | (v, []) !<- (p inpON)]
where inpON = takeWhile (onside (head inp)) inp
inpOFF = drop (length inpON) inp
onside
(a, (r, c)) (b, (r', c')) =
r' !>= r !&& c' !>= c

123. offside !:: Parser (Pos a) b !-> Parser (Pos a) b

124. offside !:: Parser (Pos a) b !-> Parser (Pos a) b
offside p inp = [(v, inpOFF) | (v, []) !<- (p inpON)]

125. (nibble (literal 'a')) " a "
# !=> [('a',""),('a'," "),('a'," ")]

126. offside !:: Parser (Pos a) b !-> Parser (Pos a) b
offside p inp = [(v, inpOFF) | (v, []) !<- (p inpON)]

127. offside !:: Parser (Pos a) b !-> Parser (Pos a) b
offside p inp = [(v, inpOFF) | (v, []) !<- (p inpON)]
where inpON = takeWhile (onside (head inp)) inp

128. offside !:: Parser (Pos a) b !-> Parser (Pos a) b
offside p inp = [(v, inpOFF) | (v, []) !<- (p inpON)]
where inpON = takeWhile (onside (head inp)) inp
inpOFF = drop (length inpON) inp

129. offside !:: Parser (Pos a) b !-> Parser (Pos a) b
offside p inp = [(v, inpOFF) | (v, []) !<- (p inpON)]
where inpON = takeWhile (onside (head inp)) inp
inpOFF = drop (length inpON) inp
onside
(a, (r, c)) (b, (r', c')) =
r' !>= r !&& c' !>= c

130. (3 +
2 * (4 + 5))
+ (8 * 10)
(3 +
2 * (4 + 5))
+ (8 * 10)

131. (offside expn) (prelex inp_1)
# !=> [(21.0,[('+',(2,0)),('(',(2,2)),('8',(2,3)),('*',
(2,5)),('1',(2,7)),('0',(2,8)),(')',(2,9))])]
(offside expn) (prelex inp_2)
# !=> [(101.0,[])]

132. Quick recap before we 🛫

133. ∅
!|> succeed, fail
!|> satisfy, literal
!|> alt, and_then, using
!|> many, some
!|> string, thenx, xthen, return
!|> expression parser & evaluator
!|> any, nibble, symbol
!|> prelex, offside

134. Practical parsers

135. 🎯 Syntactical analysis
🎯 Lexical analysis
🎯 Parse trees

136. type Parser a b = [a] !-> [(b, [a])]
type Pos a = (a, (Integer, Integer))

137. data Tag = Ident | Number
| Symbol | Junk
deriving (Show, Eq)
type Token = (Tag, [Char])

138. (Symbol, "if")
(Number, "123")

139. Parse the string with parser p,
& apply token t to the result

140. (p `tok` t) inp =
[ (((t, xs), (r, c)), out)
| (xs, out) !<- p inp]
where (x, (r,c)) = head inp

141. (p `tok` t) inp =
[ ((<token>,<pos>),<unused input>)
| (xs, out) !<- p inp]
where (x, (r,c)) = head inp

142. (p `tok` t) inp =
[ (((t, xs), (r, c)), out)
| (xs, out) !<- p inp]
where (x, (r,c)) = head inp

143. ((string "where") `tok` Symbol) inp
# !=> ((Symbol,"where"), (r, c))

144. many ((p1 `tok` t1)
`alt` (p2 `tok` t2)
`alt` !!...
`alt` (pn `tok` tn))

145. [(p1, t1),
(p2, t2),
…,
(pn, tn)]

146. lex = many.(foldr op failure)
where (p, t) `op` xs
= (p `tok` t) `alt` xs

147. 🐉

148. lex = many.(foldr op failure)
where (p, t) `op` xs
= (p `tok` t) `alt` xs

149. # Rightmost computation
cn = (pn `tok` tn) `alt` failure

150. # Followed by
(pn-1 `tok` tn-1) `alt` cn

151. many ((p1 `tok` t1)
`alt` (p2 `tok` t2)
`alt` !!...
`alt` (pn `tok` tn))

152. lexer =
lex [
((some (any_of literal " nt")), Junk),
((string "where"), Symbol),
(word, Ident),
(number, Number),
((any_of string ["(", ")", "="]), Symbol)]

153. lexer =
lex [
((some (any_of literal " nt")), Junk),
((string "where"), Symbol),
(word, Ident),
(number, Number),
((any_of string ["(", ")", "="]), Symbol)]

154. lexer =
lex [
((some (any_of literal " nt")), Junk),
((string "where"), Symbol),
(word, Ident),
(number, Number),
((any_of string ["(", ")", "="]), Symbol)]

155. head (lexer (prelex "where x = 10"))
# !=> ([((Symbol,"where"),(0,0)),
((Ident,"x"),(0,6)),
((Symbol,"="),(0,8)),
((Number,"10"),(0,10))
],[])

156. (head.lexer.prelex) "where x = 10"
# !=> ([((Symbol,"where"),(0,0)),
((Ident,"x"),(0,6)),
((Symbol,"="),(0,8)),
((Number,"10"),(0,10))
],[])

157. (head.lexer.prelex) "where x = 10"
# !=> ([((Symbol,"where"),(0,0)),
((Ident,"x"),(0,6)),
((Symbol,"="),(0,8)),
((Number,"10"),(0,10))
],[])
Function composition

158. length ((lexer.prelex) "where x = 10")
# !=> 198

159. Conflicts? Ambiguity?

160. In this case, "where" is a source
of conflict.
It can be a symbol, or identifier.

161. lexer =
lex [
{- 1 -} ((some (any_of literal " nt")), Junk),
{- 2 -} ((string "where"), Symbol),
{- 3 -} (word, Ident),
{- 4 -} (number, Number),
{- 5 -} ((any_of string ["(",")","="]), Symbol)]

162. Higher priority,
higher precedence

163. Removing Junk

164. strip !:: [(Pos Token)] !-> [(Pos Token)]
strip = filter ((!!= Junk).fst.fst)

165. ((!!= Junk).fst.fst) ((Symbol,"where"),(0,0))
# !=> True
((!!= Junk).fst.fst) ((Junk,"where"),(0,0))
# !=> False

166. (fst.head.lexer.prelex) "where x = 10"
# !=> [((Symbol,"where"),(0,0)),
((Junk," "),(0,5)),
((Ident,"x"),(0,6)),
((Junk," "),(0,7)),
((Symbol,"="),(0,8)),
((Junk," "),(0,9)),
((Number,"10"),(0,10))]

167. (strip.fst.head.lexer.prelex) "where x = 10"
# !=> [((Symbol,"where"),(0,0)),
((Ident,"x"),(0,6)),
((Symbol,"="),(0,8)),
((Number,"10"),(0,10))]

168. Syntax Analysis

169. characters !|> lexical analysis !|> tokens

170. tokens !|> syntax analysis !|> parse trees

171. f x y = add a b
where
a = 25
b = sub x y
answer = mult (f 3 7) 5

172. f x y = add a b
where
a = 25
b = sub x y
answer = mult (f 3 7) 5
Script

173. f x y = add a b
where
a = 25
b = sub x y
answer = mult (f 3 7) 5
Definition

174. f x y = add a b
where
a = 25
b = sub x y
answer = mult (f 3 7) 5
Body

175. f x y = add a b
where
a = 25
b = sub x y
answer = mult (f 3 7) 5
Expression

176. f x y = add a b
where
a = 25
b = sub x y
answer = mult (f 3 7) 5
Definition

177. f x y = add a b
where
a = 25
b = sub x y
answer = mult (f 3 7) 5
Primitives

178. data Script = Script [Def]
data Def = Def Var [Var] Expn
data Expn = Var Var
| Num Double
| Expn `Apply` Expn
| Expn `Where` [Def]
type Var = [Char]

179. prog = (many defn) `using` Script

180. defn = (
(some (kind Ident))
`and_then` ((lit "=")
`xthen`
(offside body)))
`using` defnFN

181. body = (
expr `and_then`
(((lit "where")
`xthen` (some defn)) `opt` []))
`using` bodyFN

182. expr = (some prim)
`using` (foldl1 Apply)

183. prim =
((kind Ident) `using` Var)
`alt` ((kind Number) `using` numFN)
`alt` ((lit "(")
`xthen`
(expr `thenx` (lit ")")))

184. !-- only allow a kind of tag
kind !:: Tag !-> Parser (Pos Token) [Char]
kind t = (satisfy ((!== t).fst)) `using` snd
— only allow a given symbol
lit !:: [Char] !-> Parser (Pos Token) [Char]
lit xs = (literal (Symbol, xs)) `using` snd

185. prog = (many defn) `using` Script

186. defn = (
(some (kind Ident))
`and_then` ((lit "=")
`xthen`
(offside body)))
`using` defnFN

187. body = (
expr `and_then`
(((lit "where")
`xthen` (some defn)) `opt` []))
`using` bodyFN

188. expr = (some prim)
`using` (foldl1 Apply)

189. prim =
((kind Ident) `using` Var)
`alt` ((kind Number) `using` numFN)
`alt` ((lit "(")
`xthen`
(expr `thenx` (lit ")")))

190. data Script = Script [Def]
data Def = Def Var [Var] Expn
data Expn = Var Var
| Num Double
| Expn `Apply` Expn
| Expn `Where` [Def]
type Var = [Char]

191. Orange functions are for
transforming values.

192. Use data constructors to
generate parse trees

193. Use evaluation functions to
evaluate and generate a value

194. f x y = add a b
where
a = 25
b = sub x y
answer = mult (f 3 7) 5

195. Script [
Def "f" ["x","y"] (
((Var "add" `Apply` Var "a") `Apply` Var "b")
`Where` [
Def "a" [] (Num 25.0),
Def "b" [] ((Var "sub" `Apply` Var "x") `Apply` Var "y")]),
Def "answer" [] (
(Var "mult" `Apply` (
(Var "f" `Apply` Num 3.0) `Apply` Num 7.0)) `Apply` Num 5.0)]

196. Strategy for writing parsers

197. 1. Identify components
i.e. Lexical elements

198. lexer =
lex [
((some (any_of literal " nt")), Junk),
((string "where"), Symbol),
(word, Ident),
(number, Number),
((any_of string ["(", ")", "="]), Symbol)]

199. 2. Structure these elements
a.k.a. syntax

200. defn = ((some (kind Ident)) `and_then` ((lit "=")
`xthen` (offside body))) `using` defnFN
body = (expr `and_then` (((lit "where")
`xthen` (some defn)) `opt` [])) `using` bodyFN
expr = (some prim) `using` (foldl1 Apply)
prim = ((kind Ident) `using` Var) `alt`
((kind Number) `using` numFN) `alt`
((lit "(") `xthen` (expr `thenx` (lit ")")))

201. 3. BNF notation is very helpful

202. 4. TDD in the absence of types

203. Where to, next?

204. Monadic Parsers
Graham Hutton, Eric Meijer

205. Introduction to FP
Philip Wadler

206. The Dragon Book
If your interest is in compilers

207. Libraries?

208. Haskell: Parsec, MegaParsec. ✨
OCaml: Angstrom. ✨ 🚀
Ruby: rparsec, or roll you own
Elixir: Combine, ExParsec
Python: Parsec. ✨

209. Thank you!

210. Twitter: @_swanand
GitHub: @swanandp