Every program needs some input and some logic to translate such input into structured data in memory. Language parsers, in particular, are notoriously error-prone to write,so programmers often define a high-level grammar and use parsergenerators to make the parsing process easier. What does this entail for a security analyst? In this talk you will learn what to do when you encounter GNU bison-generated code in a binary, what kind of code it creates, and how to exploit its structure to analyze targets faster and improve your fuzzers in the process. You will also walk away with a free (as in freedom) tool to automate your reverse engineering efforts.

1. Reflex:
You give me a parser, I give you a token generator
@ CODE BLUE 2020
Paolo “babush” Montesel // rev.ng srls
Mario Polino, Ph.D // Politecnico di Milano
Prof. Stefano Zanero // Politecnico di Milano
paolo.montesel@gmail.com
@pmontesel
https://babush.me/

2. Who am I
● CTF w/ mHackeroni + Tower of Hanoi + NoPwnIntended
● Contractor @ rev.ng srls
○ Security Research Engineer
○ LLVM-based Dynamic Binary Translation
○ Machine Learning for Reverse Engineering
● Main Interests
○ Reverse engineering
○ Full system fuzzing
○ Machine Learning on code/binaries
○ Obfuscation
2

3. Overview
1. Introduction
○ What’s Reflex
○ How it was born
○ Why it’s useful
2. How it works
3

4. Overview
1. Introduction
○ What’s Reflex
○ How it was born
○ Why it’s useful
2. How it works
4

5. What’s Reflex?
5
“def”
“if”
“then”
“variable”
“switch”
...
<token>
tokensbinary

6. Scenario
● looking at a binary target
● custom language with poor or no docs
● few or no input samples available
● might not be able to instrument the target (e.g.: firmware)
6

7. Overview
1. Introduction
○ What’s Reflex
○ How it was born
○ Why it’s useful
2. How it works
7

8. Story time!
8

9. 9

10. 10

11. 11

12. 12

13. 13

14. 14

15. 15
./quit-phd.sh

16. 16

17. Overview
1. Introduction
○ What’s Reflex
○ How it was born
○ Why it’s useful
2. How it works
17

18. Analyzing parsers in binaries is time-consuming
● Auto-generated code
● Finite-state machine
● Table-based generators
○ Flex / bison
● CFG-based generators
18

19. How to get valid tokens out of binaries?
● Dynamic approaches
○ Assume target can be executed
○ Exploit coverage information
○ Fail with table-based parsers
● Require large corpus of valid inputs
● Make assumptions on program behavior (e.g.: early exit)
19

20. State of the art TL;DR
20“Parser-Directed Fuzzing”, Björn Mathis et al. (PLDI 2019)

21. State of the art TL;DR
21“Parser-Directed Fuzzing”, Björn Mathis et al. (PLDI 2019)
What about closed-source programs?

22. Popular open-source parser generators
22
bison / flex antlr
parseclemon

23. Popular open-source parser generators
23
bison / flex antlr
parseclemon

24. Where can you find bison in the wild?
24

25. Where can you find bison? (lulz version)
25https://lists.gnu.org/archive/html/help-bison/2005-11/msg00029.html

26. Overview
1. Introduction
○ What’s Reflex
○ How it was born
○ Why it’s useful
2. How it works
26

27. How it works
1. Some background
2. Identifying the information we need from the target
3. Recover the Finite State Machine
4. FSM => regular expression
5. Generate valid tokens
27

28. How it works
1. Some background
2. Identifying the information we need from the target
3. Recover the Finite State Machine
4. FSM => regular expression
5. Generate valid tokens
28

29. (simplified) Parser theory in ~2 minutes
29
developer grammar
parser
lexer

30. The lexer
30
“x = y * 2 + 3”
x = y * 2 + 3 tokens
text
lexer

31. The parser
31
x = y * 2 + 3
abstract
syntax
tree
(AST)
tokens
parser
x
=
+
y 2
* 3

32. How does flex work?
32
Rules:
enum {
B = 1,
DEFINE = 2,
A = 3,
}
1. a => A
2. def(ine)? => DEFINE
3. b => B
...
N. <regexp> => <action_id>

33. 33
See: “Compilers: Principles, Techniques, and Tools“ by Aho, Lam, Sethi, Ullman (AKA “The Dragon Book”)

34. How it works
1. Some background
2. Identifying the information we need from the target
3. Recover the Finite State Machine
4. FSM => regular expression
5. Generate valid tokens
34

35. What do we need from flex?
35
table offsets
+
element size
max state

36. 36
Steps needed
1. Find yylex() in the binary
2. Find tables using data-flow analysis
3. Infer their element size from memory accesses
4. Interpret the tables to recover the finite-state machine

37. 37
How to find the tables in a binary?
2
VAR
+
VAR
LOAD
2
+
VAR
LOAD

38. jge
38
Example: dataflow query
global_var1 global_var2
[ ]
>=
+
constant

39. How it works
1. Some background
2. Identifying the information we need from the target
3. Recover the Finite State Machine
4. FSM => regular expression
5. Generate valid tokens
39

40. 40
How to rewind time?

41. 41
Example: table decoding (simplified)
XX XX XX XX XX XX
XX XX XX XX XX XX
XX XX XX XX XX XX
XX XX XX XX XX XX
XX XX XX XX XX XX
XX XX XX XX XX XX
next state
inputchar
curr state
next = next_state[curr_state][input_char]

42. XX 0A XX XX XX XX
XX 08 XX XX XX XX
XX 0A XX XX XX XX
XX 09 XX XX XX XX
XX 0A XX XX XX XX
XX 02 XX XX XX XX
42
Example: table decoding (simplified)
curr state = 1
“a”
next = next_state[1][input_char]
“b”
“d”

43. 43
Real decoding

44. How it works
1. Some background
2. Identifying the information we need from the target
3. Recover the Finite State Machine
4. FSM => regular expression
5. Generate valid tokens
44

45. 45

46. 46

47. 47

48. 48
def(ine)?
See: “Introduction to Automata Theory, Languages, and Computation” by Hopcroft, Ullman
Reflex
simplify
Recovering the regular expressions

49. How it works
1. Some background
2. Identifying the information we need from the target
3. Recover the Finite State Machine
4. FSM => regular expression
5. Generate valid tokens
49

50. 50
Regex => Tokens

51. DEMO
51

52. 52
Takeaways
● Identify flex’s tables automatically ✅
● Extract the original FSM
● Recover the lexer patterns
● Generate valid tokens
● No need to run the target

53. 53
Takeaways
● Identify flex’s tables automatically ✅
● Extract the original FSM ✅
● Recover the lexer patterns
● Generate valid tokens
● No need to run the target

54. 54
Takeaways
● Identify flex’s tables automatically ✅
● Extract the original FSM ✅
● Recover the lexer patterns ✅
● Generate valid tokens
● No need to run the target

55. 55
Takeaways
● Identify flex’s tables automatically ✅
● Extract the original FSM ✅
● Recover the lexer patterns ✅
● Generate valid tokens ✅
● No need to run the target

56. 56
Takeaways
● Identify flex’s tables automatically ✅
● Extract the original FSM ✅
● Recover the lexer patterns ✅
● Generate valid tokens ✅
● No need to run the target ✅

57. 57
Limitations
● Ghidra-based dataflow analysis not reliable
● Works only for table-driven parsers
● Implementation limited to bison/flex
● Doesn’t (yet) handle flex => bison
○ No structured file generation

58. 58
Future work
● Find more bison-based targets (contact me)
● Reverse bison
● Improve yylex()/tables identification
○ rev.ng + LLVM IR + SCEV
● Look at other parser generators (e.g. antlr)
● Find bugs? (:

59. 59
https://github.com/thebabush/reflex

60. Thank you!
(^o^)/
質問がありますか？
60
paolo.montesel@gmail.com
@pmontesel
https://babush.me/

61. Credits
Icons
● flaticon.com
● public domain
61