Language-Based Testing and Debugging Andreas Zeller, CISPA Helmholtz Center for Information Security Joint work with Dominic Steinhöfel and the CISPA team If one knows the input language of a software system, one can use it for generating test inputs or for determining input patterns that cause specific behavior. But how does one specify an input language such that all properties of inputs are covered? In this talk, I introduce the ISLa specification language, combining grammars and constraints over nonterminals to capture even the most complex input languages. ISLa serves as a fuzzer, allowing to test programs with a large variety of inputs systematically, but also as a parser, decomposing inputs and outputs into individual elements. This opens up new directions for debugging, determining the exact conditions under which a program fails: "The program fails if the <check-temperature-option> is given and <temperature> < 0 holds." Includes live demos! Andreas Zeller is faculty at the CISPA Helmholtz Center for Information Security and professor for Software Engineering at Saarland University. His research on automated debugging, mining software archives, specification mining, and security testing has proven highly influential. Zeller is one of the few researchers to have received two ERC Advanced Grants, most recently for his S3 project. Zeller is an ACM Fellow and holds an ACM SIGSOFT Outstanding Research Award.

1. Andreas Zeller • DEBT Workshop • 2023-07-17
Joint work with Dominic Steinhöfel • Martin Eberlein • Lars Grunske
Language-Based
Testing and Debugging
Why and When Does My Program Fail?

2. Testing Challenges
program inputs
program outputs
Test
generation
problem
/ / / / / / / / / / / / / / / / /
/ / / / / / / / / / / / / / / / /
Oracle
problem
/ / / / / / / / / / / / / / / / /
/ / / / / / / / / / / / / / / / /

3. Testing and Debugging Challenges
program inputs
program outputs
Test
generation
problem
/ / / / / / / / / / / / / / / / /
/ / / / / / / / / / / / / / / / /
Oracle
problem
/ / / / / / / / / / / / / / / / /
/ / / / / / / / / / / / / / / / /

4. Testing and Debugging Challenges
Language
problem
}
Test
generation
problem
/ / / / / / / / / / / / / / / / /
/ / / / / / / / / / / / / / / / /
Oracle
problem
/ / / / / / / / / / / / / / / / /
/ / / / / / / / / / / / / / / / /
/ / / / / / / / / / / / / / / / /
/ / / / / / / / / / / / / / / / /

5. Testing the SSL/TLS Heartbeat Protocol
0x1 0x5 "Hello" ...
0x2 0x5 "Hello" ...
Client Server
payload must be identical

6. Testing the SSL/TLS Heartbeat Protocol
0x1 <length> <payload> <padding>
0x2 <length> <payload> <padding>
Client Server

7. Testing the SSL/TLS Heartbeat Protocol
0x2 <length> <payload> <padding>
<length> ::= <uint16>
<payload> ::= <byte>*
<padding> ::= <byte>*
<length> ::= <uint16>
length must match
...
0x1 <length> <payload> <padding>
<client-request> ::=
payload must
be identical
<server-response> ::=
Client Server

8. Specifying Languages
<length> ::= <uint16>
<payload> ::= <byte>*
<padding> ::= <byte>*
<exchange> ::= <client-request> <server-response>
uint16(<length>) = len(<payload>)
<client-request>.<payload> = <server-response>.<payload>
Syntax
I/O Grammar
Semantics
Constraints
0x1 <length> <payload> <padding>
<client-request> ::=
0x2 <length> <payload> <padding>
<server-response> ::=
length must match
payload must be identical
syntax alone
does not su
ffi
• A language to specify inputs (and outputs)
• A fuzzer to produce inputs that are valid syntactically and semantically
• A checker to parse, check, mutate, and repair inputs according to constraints
<exchange>

9. ISLa Language
Numbers uint16(<id>) = str.len(<name>) + str.len(<email>) + 2
Strings str.suffixof(<email>, '@cispa.de', True)
Based on SMT-LIB Theories
Booleans str.len(<email>) > 4096 and str.contains(<email>, '0', True)
Quanti
fi
ers exists <option> in <command_line>:
<option> = '--config' or <option> = '-C'
S-Expressions (>= (str.len <buffer> 4096))
• A language to specify inputs (and outputs)
• A fuzzer to produce inputs that are valid syntactically and semantically
• A checker to parse, check, mutate, and repair inputs according to constraints

10. ISLa Usage
Fuzzing $ isla solve grammar.bnf --constraint 'str.len(<payload>) > 65536'
Checking $ isla find customers.bnf --constraint '<company> = "CISPA"' *.csv
Command Line + Python API
Mutating $ isla mutate grammar.bnf spec.isla input.txt
Repairing $ isla repair grammar.bnf spec.isla input.txt
Python API from isla import ISLaSolver
• A language to specify inputs (and outputs)
• A fuzzer to produce inputs that are valid syntactically and semantically
• A checker to parse, check, mutate, and repair inputs according to constraints

11. Specifying Languages
<length> ::= <uint16>
<payload> ::= <byte>*
<padding> ::= <byte>*
<exchange> ::= <client-request> <server-response>
uint16(<length>) = len(<payload>)
<client-request>.<payload> = <server-response>.<payload>
Syntax
I/O Grammar
Semantics
Constraints
0x1 <length> <payload> <padding>
<client-request> ::=
0x2 <length> <payload> <padding>
<server-response> ::=
<exchange>

12. Producing Inputs
<length> ::= <uint16>
<payload> ::= <byte>*
<padding> ::= <byte>*
<exchange> ::= <client-request> <server-response>
uint16(<length>) = len(<payload>)
<client-request>.<payload> = <server-response>.<payload>
Syntax
I/O Grammar
Semantics
Constraints
0x1 <length> <payload> <padding>
<client-request> ::=
0x2 <length> <payload> <padding>
<server-response> ::=
<exchange>
<client-request> <server-response>

13. Producing Inputs
<length> ::= <uint16>
<payload> ::= <byte>*
<padding> ::= <byte>*
<exchange> ::= <client-request> <server-response>
uint16(<length>) = len(<payload>)
<client-request>.<payload> = <server-response>.<payload>
Syntax
I/O Grammar
Semantics
Constraints
0x1 <length> <payload> <padding>
<client-request> ::=
0x2 <length> <payload> <padding>
<server-response> ::=
<exchange> <client-request> <server-response>

14. <client-request>
Producing Inputs
<length> ::= <uint16>
<payload> ::= <byte>*
<padding> ::= <byte>*
<exchange> ::= <client-request> <server-response>
uint16(<length>) = len(<payload>)
<client-request>.<payload> = <server-response>.<payload>
Syntax
I/O Grammar
Semantics
Constraints
0x1 <length> <payload> <padding>
<client-request> ::=
0x2 <length> <payload> <padding>
<server-response> ::=
<exchange> <server-response>

15. <client-request> <server-response>
Producing Inputs
<length> ::= <uint16>
<payload> ::= <byte>*
<padding> ::= <byte>*
<exchange> ::= <client-request> <server-response>
uint16(<length>) = len(<payload>)
<client-request>.<payload> = <server-response>.<payload>
Syntax
I/O Grammar
Semantics
Constraints
0x1 <length> <payload> <padding>
<client-request> ::=
0x2 <length> <payload> <padding>
<server-response> ::=
<exchange>
<length> <payload> <padding>
0x1

16. <server-response>
Producing Inputs
<length> ::= <uint16>
<payload> ::= <byte>*
<padding> ::= <byte>*
<exchange> ::= <client-request> <server-response>
uint16(<length>) = len(<payload>)
<client-request>.<payload> = <server-response>.<payload>
Syntax
I/O Grammar
Semantics
Constraints
0x1 <length> <payload> <padding>
<client-request> ::=
0x2 <length> <payload> <padding>
<server-response> ::=
<exchange>
<length> <payload> <padding>
0x1
<client-request>

17. <server-response>
Producing Inputs
<length> ::= <uint16>
<payload> ::= <byte>*
<padding> ::= <byte>*
<exchange> ::= <client-request> <server-response>
uint16(<length>) = len(<payload>)
<client-request>.<payload> = <server-response>.<payload>
Syntax
I/O Grammar
Semantics
Constraints
0x1 <length> <payload> <padding>
<client-request> ::=
0x2 <length> <payload> <padding>
<server-response> ::=
<length> <payload> <padding>
0x1

18. Producing Inputs
<length> ::= <uint16>
<payload> ::= <byte>*
<padding> ::= <byte>*
<exchange> ::= <client-request> <server-response>
uint16(<length>) = len(<payload>)
<client-request>.<payload> = <server-response>.<payload>
Syntax
I/O Grammar
Semantics
Constraints
0x1 <length> <payload> <padding>
<client-request> ::=
0x2 <length> <payload> <padding>
<server-response> ::=
<server-response>
<length>
<payload> <padding>
0x1
<uint16>

19. Producing Inputs
<length> ::= <uint16>
<payload> ::= <byte>*
<padding> ::= <byte>*
<exchange> ::= <client-request> <server-response>
uint16(<length>) = len(<payload>)
<client-request>.<payload> = <server-response>.<payload>
Syntax
I/O Grammar
Semantics
Constraints
0x1 <length> <payload> <padding>
<client-request> ::=
0x2 <length> <payload> <padding>
<server-response> ::=
<server-response>
<length>
<payload> <padding>
0x1 <uint16>

20. 0x1 0x0005 <payload> <padding> <server-response>
Producing Inputs
<length> ::= <uint16>
<payload> ::= <byte>*
<padding> ::= <byte>*
<exchange> ::= <client-request> <server-response>
uint16(<length>) = len(<payload>)
<client-request>.<payload> = <server-response>.<payload>
Syntax
I/O Grammar
Semantics
Constraints
0x1 <length> <payload> <padding>
<client-request> ::=
0x2 <length> <payload> <padding>
<server-response> ::=

21. 0x1 0x0005 "hello" <padding> <server-response>
Producing Inputs
<length> ::= <uint16>
<payload> ::= <byte>*
<padding> ::= <byte>*
<exchange> ::= <client-request> <server-response>
uint16(<length>) = len(<payload>)
<client-request>.<payload> = <server-response>.<payload>
Syntax
I/O Grammar
Semantics
Constraints
0x1 <length> <payload> <padding>
<client-request> ::=
0x2 <length> <payload> <padding>
<server-response> ::=

22. 0x1 0x0005 "hello" 0x0 0x0... <server-response>
Producing Inputs
<length> ::= <uint16>
<payload> ::= <byte>*
<padding> ::= <byte>*
<exchange> ::= <client-request> <server-response>
uint16(<length>) = len(<payload>)
<client-request>.<payload> = <server-response>.<payload>
Syntax
I/O Grammar
Semantics
Constraints
0x1 <length> <payload> <padding>
<client-request> ::=
0x2 <length> <payload> <padding>
<server-response> ::=
complete and valid input

23. 0x1 0x0005 "hello" 0x0 0x0... <server-response>
Parsing Outputs
<length> ::= <uint16>
<payload> ::= <byte>*
<padding> ::= <byte>*
<exchange> ::= <client-request> <server-response>
uint16(<length>) = len(<payload>)
<client-request>.<payload> = <server-response>.<payload>
Syntax
I/O Grammar
Semantics
Constraints
0x1 <length> <payload> <padding>
<client-request> ::=
0x2 <length> <payload> <padding>
<server-response> ::=
0x0005 "hello"
0x2 0x0 0x0...

24. 0x1 0x0005 "hello" 0x0 0x0... 0x2 <length> <payload> <padding>
Parsing Outputs
<length> ::= <uint16>
<payload> ::= <byte>*
<padding> ::= <byte>*
<exchange> ::= <client-request> <server-response>
uint16(<length>) = len(<payload>)
<client-request>.<payload> = <server-response>.<payload>
Syntax
I/O Grammar
Semantics
Constraints
0x1 <length> <payload> <padding>
<client-request> ::=
0x2 <length> <payload> <padding>
<server-response> ::=
0x0005 "hello"
0x2 0x0 0x0...

25. 0x1 0x0005 "hello" 0x0 0x0... 0x2 <uint16> <payload> <padding>
Parsing Outputs
<length> ::= <uint16>
<payload> ::= <byte>*
<padding> ::= <byte>*
<exchange> ::= <client-request> <server-response>
uint16(<length>) = len(<payload>)
<client-request>.<payload> = <server-response>.<payload>
Syntax
I/O Grammar
Semantics
Constraints
0x1 <length> <payload> <padding>
<client-request> ::=
0x2 <length> <payload> <padding>
<server-response> ::=
0x0005 "hello"
0x2 0x0 0x0...

26. 0x1 0x0005 "hello" 0x0 0x0... 0x2 0x0005 "hello" 0x0 0x0...
Parsing Outputs
<length> ::= <uint16>
<payload> ::= <byte>*
<padding> ::= <byte>*
<exchange> ::= <client-request> <server-response>
uint16(<length>) = len(<payload>)
<client-request>.<payload> = <server-response>.<payload>
Syntax
I/O Grammar
Semantics
Constraints
0x1 <length> <payload> <padding>
<client-request> ::=
0x2 <length> <payload> <padding>
<server-response> ::=
Test
generation
problem
/ / / / / / / / / / / / / / / / /
/ / / / / / / / / / / / / / / / /
Oracle
problem
/ / / / / / / / / / / / / / / / /
/ / / / / / / / / / / / / / / / /

27. Test
generation
problem
/ / / / / / / / / / / / / / / / /
/ / / / / / / / / / / / / / / / /
Oracle
problem
/ / / / / / / / / / / / / / / / /
/ / / / / / / / / / / / / / / / /
Language Specifications
<length> ::= <uint16>
<payload> ::= <byte>*
<padding> ::= <byte>*
<exchange> ::= <client-request> <server-response>
uint16(<length>) = len(<payload>)
<client-request>.<payload> = <server-response>.<payload>
Syntax
I/O Grammar
Semantics
Constraints
Testing Mocking
0x1 <length> <payload> <padding>
<client-request> ::=
0x2 <length> <payload> <padding>
<server-response> ::=
Monitoring

29. Where does the Language Come From?
Language
problem
}
Test
generation
problem
/ / / / / / / / / / / / / / / / /
/ / / / / / / / / / / / / / / / /
Oracle
problem
/ / / / / / / / / / / / / / / / /
/ / / / / / / / / / / / / / / / /
/ / / / / / / / / / / / / / / / /
/ / / / / / / / / / / / / / / / /

30. 0x1 0x5 "Hello" ...
0x1 0x4 "Dear" ...
0x1 0x5 "World" ...
Synthesizing Language Specifications
<client-request> ::=
uint16(<length>) =
Learning Grammars
Learning Constraints
Learn language speci
fi
cations from existing interactions
Di
ff
erent I/O elements take
di
ff
erent paths in processing program
0x1 0x5 "Hello" ...
0x1 <length> <payload> <padding>
Synthesize predicates
that match all observed values
len(<payload>)
Interactions Learner Language Spec Test Generator

31. Exploring Program Behavior
Re
fi
ne language specs through generated tests
Gradually negate constraints
uint16(<length>) len(<payload>)
uint16(<length>) = len(<payload>) =>
<client-request>.<payload> = <server-response>.<payload>
Re
fi
ned language spec (and failure diagnosis)
≠
=
uint16(<length>) ≥ len(<payload>) =>
str.prefixof(<server-response>.<payload>, <client-request>.<payload>, True)
length no longer matches
Interactions Learner Language Spec Test Generator
length matches:
payload is identical
length is too high:
payload di
ff
"hello"
0x1 0x2000 ...
...
0x2 0x2000 "hello0secret0password0"

32. Features on Demand
"hello" ...
<length> <payload> <length> <payload>
<client-request> <server-response>
<exchange>
0x1 0x2000 ... 0x2 0x2000 "hello0secret0password0"
"37h%$"
0x1 0x5 ...
"0"
0x1 0x1 ...
"37h%$"
0x2 0x5 ...
"0"
0x2 0x1 ...
Learner
Test
Generator
Perfect conditions for automated learners
"${}"
0x1 0x3 ... "${}"
0x2 0x3 ...
""
0x1 0x0 ... ""
0x2 0x0 ...
"'; --"
0x1 0x5 ... "'; --"
0x2 0x5 ...

33. AVICENNA
Semantic Debugging
Input
Language Spec
Failure
Circumstances
<client-request> ::=
0x1 <length> <payload>
uint16(<length>)
> len(<payload>)
Failing Inputs
"hello"
0x1 0x2000

34. AVICENNA
Semantic Debugging
Input
Language Spec
Failure
Circumstances
<client-request> ::=
0x1 <length> <payload>
uint16(<length>)
> len(<payload>)
Failing Inputs
"hello"
0x1 0x2000
Determine relevant
input elements
Determine failure-
related properties
Generate new inputs
to re
fi
ne theory
Negate constraints
<length>
<payload>
<length> = 0x2000
<payload> = "hello"
<length> ≠ 0x2000
<payload> ≠ "hello"
"xyz"
0x1 0x0003

35. TABLE II
AVICENNA DIAGNOSES VS. HUMAN DIAGNOSES
Bug AVICENNA diagnosis (using ISLa syntax [13]) Expert Bug Description
HeartBleed str.to.int(<length>) > str.len(<payload>)
“Attackers can send Heartbeat requests with the
value of the length field greater than the actual
length of the payload” [37]
grep.7aa698d3
exists <utf8> in <lc_all>:
<utf8> = "UTF-8" and
exists <ignore_case> in <general_options>:
<ignore_case> = "-i"
“If grep conducts a case-insensitive search (-i) on
an input that contains multibyte characters and the
locale is UTF8, then grep prints a match of incorrect
length.” [36]
grep.5fa8c7c9
exists <patterns> in <command>:
<patterns> = "’’" and
exists <utf8> in <lc_all>:
inside(<utf8>, <lc_all>) and
exists <fixed_string> in <cmd_1>:
inside(<fixed_string>, <cmd_1>)
“Searching with grep -F for an empty string in a
multibyte locals [sic] would freeze grep.” [36]
(Note: hfixed stringi expands to -F and
--fixed-strings)
grep.c96b0f2c
exists <regex_> in <patterns>:
<regex_> = "ˆ$" and
(exists <ignore_case> in <matching_control>:
inside(<ignore_case>, <matching_control>) or
exists <line_no> in <output_line_prefix_control>:
inside(<line_no>, <output_line_prefix_control>))
“Options -i and -n will not work when applied to an
empty line” [36] [in a UTF-8 locale]
(Note: hignore casei expands to -i and
--ignore-case; hline noi expands to -n
and --line-number)
“Core dump with pattern ’(ˆ| )*( |$)’” [36]
Some Benchmark Results
Subject AVICENNA Diagnosis Expert Diagnosis (from DBGBench)

36. grep.c96b0f2c
exists <regex_> in <patterns>:
<regex_> = "ˆ$" and
(exists <ignore_case> in <matching_control>:
inside(<ignore_case>, <matching_control>) or
exists <line_no> in <output_line_prefix_control>:
inside(<line_no>, <output_line_prefix_control>))
“Options -i and -n will not work when applied to an
empty line” [36] [in a UTF-8 locale]
(Note: hignore casei expands to -i and
--ignore-case; hline noi expands to -n
and --line-number)
grep.3c3bdace
exists <extended_regex> in <matcher_selection>:
inside(<extended_regex>, <matcher_selection>) and
exists <repetition> in <patterns>:
<repetition> = "*"
“Core dump with pattern ’(ˆ| )*( |$)’” [36]
[and -E option]
(Note: hextended regexi expands to -E and
--extended-regexp)
grep.3220317a
exists <bracket_expr> in <first_expression>:
inside(<bracket_expr>, <first_expression>) and
exists <utf_characters> in <bracket_char>:
inside(<utf_characters>, <bracket_char>)
“Segmentation fault on multibyte character
classes” [36]
(Note: hbracket expri expands to [...] in
a regular expression; hutf charactersi occur
within hbracket chari, i.e. the characters within
hbracket expri)
find.07b941b1
exists <match_opts> in <find_expression>:
<match_opts> = "-regex " and
exists <character_expr_no_minus> in <first_expression>:
<character_expr_no_minus> = "."
“find segfaults when using -regex, for instance
./find -regex ’.*’” [36]
find.091557f6
exists <file_properties> elem in <find_expression>:
<file_properties> = "-type f" and
exists <directory_name> elem_0 in <starting_dir_list>:
inside(<directory_name>, <starting_dir_list>)
“assertion failure on symbolic link loop:
Let’s say we accidentally create a symlink
loop $ mkdir tmp; cd tmp and
$ ln -s a b; ln -s b a and use find
to find files and follow symlinks inside the tmp-
folder: ../find -L -type f” [36]
Some Benchmark Results
Subject AVICENNA Diagnosis Expert Diagnosis (from DBGBench)

37. find.091557f6
exists <file_properties> elem in <find_expression>:
<file_properties> = "-type f" and
exists <directory_name> elem_0 in <starting_dir_list>:
inside(<directory_name>, <starting_dir_list>)
“assertion failure on symbolic link loop:
Let’s say we accidentally create a symlink
loop $ mkdir tmp; cd tmp and
$ ln -s a b; ln -s b a and use find
to find files and follow symlinks inside the tmp-
folder: ../find -L -type f” [36]
find.dbcb10e9
exists <digit> in <last_modified>:
inside(<digit>, <last_modified>) and
exists <numeric_arg> in <find_command>:
inside(<numeric_arg>, <find_command>)
“-mtime produces segmentation fault, e.g., ./find
-mtime 2” [36]
(Note: hlast modifiedi expands to -mtime)
find.ff248a20
exists <ln_file> in <ln>:
<ln_file> = "al_ln -s . link" and
exists <find_expression_or_empty> in <command>:
<find_expression_or_empty> = " -follow"
“infinite loop with -follow; e.g.,
$ mkdir testingfindagain;
ln -s . testingfindagain/symlink;
./find testingfindagain -follow” [36]
For details on all DBGBench bugs (including expert bug descriptions), visit https://dbgbench.github.io/.
For the HeartBleed description, see https://www.synopsys.com/blogs/software-security/heartbleed-bug/.
Some Benchmark Results
Subject AVICENNA Diagnosis Expert Diagnosis (from DBGBench)
At FSE 2023

38. Demo

39. Testing and Debugging Challenges
Language
problem
}
Test
generation
problem
/ / / / / / / / / / / / / / / / /
/ / / / / / / / / / / / / / / / /
Oracle
problem
/ / / / / / / / / / / / / / / / /
/ / / / / / / / / / / / / / / / /
/ / / / / / / / / / / / / / / / /
/ / / / / / / / / / / / / / / / /

40. The Future of Software Development
Bot #2
Debugger
Bot #1
Tester
Ursula
UI Designer
Bob
Backend Developer
Frederica
Frontend Developer
Omar
Operator
Bot #3
Monitor

41. Andreas Zeller • APR Launch Event • National University of Singapore • 2022-11-11
Joint work with Dominic Steinhöfel • Martin Eberlein • Lars Grunske
Semantic Debugging
Why and When Does My Program Fail? TABLE II
AVICENNA DIAGNOSES VS. HUMAN DIAGNOSES
Bug AVICENNA diagnosis (using ISLa syntax [13]) Expert Bug Description
HeartBleed str.to.int(<length>) > str.len(<payload>)
“Attackers can send Heartbeat requests with the
value of the length field greater than the actual
length of the payload” [37]
grep.7aa698d3
exists <utf8> in <lc_all>:
<utf8> = "UTF-8" and
exists <ignore_case> in <general_options>:
<ignore_case> = "-i"
“If grep conducts a case-insensitive search (-i) on
an input that contains multibyte characters and the
locale is UTF8, then grep prints a match of incorrect
length.” [36]
grep.5fa8c7c9
exists <patterns> in <command>:
<patterns> = "’’" and
exists <utf8> in <lc_all>:
inside(<utf8>, <lc_all>) and
exists <fixed_string> in <cmd_1>:
inside(<fixed_string>, <cmd_1>)
“Searching with grep -F for an empty string in a
multibyte locals [sic] would freeze grep.” [36]
(Note: hfixed stringi expands to -F and
--fixed-strings)
grep.c96b0f2c
exists <regex_> in <patterns>:
<regex_> = "ˆ$" and
(exists <ignore_case> in <matching_control>:
inside(<ignore_case>, <matching_control>) or
exists <line_no> in <output_line_prefix_control>:
inside(<line_no>, <output_line_prefix_control>))
“Options -i and -n will not work when applied to an
empty line” [36] [in a UTF-8 locale]
(Note: hignore casei expands to -i and
--ignore-case; hline noi expands to -n
and --line-number)
“Core dump with pattern ’(ˆ| )*( |$)’” [36]
Some Benchmark Results
Subject AVICENNA Diagnosis Expert Diagnosis (from DBGBench)
Features on Demand
"hello" ...
<length> <payload> <length> <payload>
<client-request> <server-response>
<exchange>
0x1 0x2000 ... 0x2 0x2000 "hello0secret0password0"
"37h%$"
0x1 0x5 ...
"0"
0x1 0x1 ...
"37h%$"
0x2 0x5 ...
"0"
0x2 0x1 ...
Learner
Test
Generator
Perfect conditions for automated learners
"${}"
0x1 0x3 ... "${}"
0x2 0x3 ...
""
0x1 0x0 ... ""
0x2 0x0 ...
"'; --"
0x1 0x5 ... "'; --"
0x2 0x5 ...
Test
generation
problem
/ / / / / / / / / / / / / / / / /
/ / / / / / / / / / / / / / / / /
Oracle
problem
/ / / / / / / / / / / / / / / / /
/ / / / / / / / / / / / / / / / /
Testing with Language Specs
<length> ::= <uint16>
<payload> ::= <byte>*
<padding> ::= <byte>*
<exchange> ::= <client-request> <server-response>
uint16(<length>) = len(<payload>)
<client-request>.<payload> = <server-response>.<payload>
Syntax
I/O Grammar
Semantics
Constraints
Testing Mocking
0x1 <length> <payload> <padding>
<client-request> ::=
0x2 <length> <payload> <padding>
<server-response> ::=
Monitoring
AVICENNA
Learning Failure Circumstances
Input
Language Spec
Failure
Circumstances
<client-request> ::=
0x1 <length> <payload>
uint16(<length>)
> len(<payload>)
Failing Inputs
"hello"
0x1 0x2000
Determine relevant
input elements
Determine failure-
related properties
Generate new inputs
to refine theory
Negate constraints
<length>
<payload>
<length> = 0x2000
<payload> = "hello"
<length> ≠ 0x2000
<payload> ≠ "hello"
"xyz"
0x1 0x0003
@AndreasZeller

42. Backup Slides

43. The Fuzzing Book

44. The Debugging Book