Overview You’ve written your unit tests and your integration tests. And yet, you’re still finding bugs in your code. In this talk we’ll cover advanced types of testing that helps you write higher quality code with less bugs. Unit and integration tests are great first steps towards improving the quality of your python project. Ever wonder if there’s even more you can do? In this talk we’ll cover additional types of tests that can help improve the quality and robustness of your python projects: stateful property-based testing, generative fuzz testing, long term stability testing and advanced multithreaded testing. This talk is more than just theory. We’ll look at specific libraries and frameworks that help you write these advanced tests. I’ll also show you real world examples of bugs these tests have found from projects that I maintain.

1. Next Level Testing
Revisited
James Saryerwinnie
@jsaryer
PyGotham 2017

2. Next Level Testing
Revisited
James Saryerwinnie
@jsaryer
PyGotham 2017

3. Property based testing
* Overview
* State-based testing
Fuzz testing
* Overview
* Multi-argument fuzzing
Agenda

4. Property based testing
* Overview
* State-based testing
Fuzz testing
* Overview
* Multi-argument fuzzing
Agenda

5. abs.py
def test_abs_always_positive():
assert abs(1) == 1
def test_abs_negative():
assert abs(-1) == 1
def test_abs_zero_is_zero():
assert abs(0) == 0
Property example

6. def test_abs_always_positive():
assert abs(1) == 1
def test_abs_negative():
assert abs(-1) == 1
def test_abs_zero_is_zero():
assert abs(0) == 0
abs.pyProperty example
import random
def test_abs():
for _ in range(1000):
n = random.randint(
-sys.maxint,
sys.maxint)
assert abs(n) >= 0
For all integers i, abs(i) should always
be greater than or equal to 0.
Can we do better?

7. Property Based Testing
1. Write assertions about the properties of a function
2. Generate random input data that violates these assertions
3. Minimize the example to be as simple as possible

8. Hypothesis
• Integrates with unittest/pytest
• Powerful test data generation
• Generates minimal test cases on failure
• pip install hypothesis

9. Strategies
>>> import hypothesis.strategies as s
>>> test_data = s.integers()
>>> test_data.example()
-214460024625629886
>>> test_data.example()
288486085571772
>>> test_data.example()
-2199980509
>>> test_data.example()
-207377623894
>>> test_data.example()
4588868

10. abs.py
import random
def test_abs():
for _ in range(1000):
n = random.randint(
-sys.maxint,
sys.maxint)
assert abs(n) >= 0
Property example
from hypothesis import given
import hypothesis.strategies as s
@given(s.integers())
def test_abs(x):
assert abs(x) >= 0

11. abs.py
import random
def test_abs():
for _ in range(1000):
n = random.randint(
-sys.maxint,
sys.maxint)
assert abs(n) >= 0
Property example
from hypothesis import given
import hypothesis.strategies as s
@given(s.integers())
def test_abs(x):
assert abs(x) >= 0

12. abs.py
import random
def test_abs():
for _ in range(1000):
n = random.randint(
-sys.maxint,
sys.maxint)
assert abs(n) >= 0
Property example
from hypothesis import given
import hypothesis.strategies as s
@given(s.integers())
def test_abs(x):
assert abs(x) >= 0

13. abs.py
import random
def test_abs():
for _ in range(1000):
n = random.randint(
-sys.maxint,
sys.maxint)
assert abs(n) >= 0
Property example
from hypothesis import given
import hypothesis.strategies as s
@given(s.integers())
def test_abs(x):
assert abs(x) >= 0

14. abs.py
import random
def test_abs():
for _ in range(1000):
n = random.randint(
-sys.maxint,
sys.maxint)
assert abs(n) >= 0
Property example
from hypothesis import given
import hypothesis.strategies as s
@given(s.integers())
def test_abs(x):
assert abs(x) >= 0

15. State based testing
• Instead of a single value, produce a set of steps
• At each step, some type of internal state is updated
• An failed test gives you a set of instructions to create the failure

16. Chalice
• Serverless microframework for python
• Create AWS Lambda Functions and Amazon API Gateway APIs
• Multiple deployment backends

17. app.py
from chalice import Chalice
app = Chalice(app_name='myproject')
@app.lambda_function()
def index(event, context):
return {'hello': 'world'}
@app.schedule('rate(5 minutes)')
def cron(event):
return {}
Lambda functions

18. app.py
from chalice import Chalice
app = Chalice(app_name='myproject')
@app.lambda_function()
def index(event, context):
return {'hello': 'world'}
@app.schedule('rate(5 minutes)')
def cron(event):
return {}
Lambda functions
AWS Lambda
AWS Lambda Amazon CloudWatch Events

19. app.py
from chalice import Chalice
app = Chalice(app_name='myproject')
@app.lambda_function()
def index(event, context):
return {'hello': 'world'}
@app.schedule('rate(5 minutes)')
def cron(event):
return {}
Lambda functions
AWS Lambda
AWS Lambda Amazon CloudWatch Events
T1 - Create index/cron
functions

20. app.py
from chalice import Chalice
app = Chalice(app_name='myproject')
@app.lambda_function()
def index(event, context):
return {'hello': 'world'}
@app.schedule('rate(5 minutes)')
def cron(event):
return {}
Lambda functions
AWS Lambda
AWS Lambda Amazon CloudWatch Events
T1 - Create index/cron
functions
Create new function

21. app.py
from chalice import Chalice
app = Chalice(app_name='myproject')
@app.lambda_function()
def index(event, context):
return {'hello': 'world'}
@app.lambda_function()
def hello_pygotham(event, context):
return {'hello': 'pygotham'}
@app.schedule('rate(5 minutes)')
def cron(event):
return {}
Lambda functions
AWS Lambda
AWS Lambda Amazon CloudWatch Events

22. app.py
from chalice import Chalice
app = Chalice(app_name='myproject')
@app.lambda_function()
def index(event, context):
return {'hello': 'world'}
@app.lambda_function()
def hello_pygotham(event, context):
return {'hello': 'pygotham'}
@app.schedule('rate(5 minutes)')
def cron(event):
return {}
Lambda functions
AWS Lambda
AWS Lambda
AWS Lambda Amazon CloudWatch Events

23. app.py
from chalice import Chalice
app = Chalice(app_name='myproject')
@app.lambda_function()
def index(event, context):
return {'hello': 'world'}
@app.lambda_function()
def hello_pygotham(event, context):
return {'hello': 'pygotham'}
@app.schedule('rate(5 minutes)')
def cron(event):
return {}
Lambda functions
AWS Lambda
AWS Lambda
AWS Lambda Amazon CloudWatch Events
T1 - Create index/cron
functions
T2 - Create hello_pygotham
function

24. app.py
from chalice import Chalice
app = Chalice(app_name='myproject')
@app.lambda_function()
def index(event, context):
return {'hello': 'world'}
@app.lambda_function()
def hello_pygotham(event, context):
return {'hello': 'pygotham'}
@app.schedule('rate(5 minutes)')
def cron(event):
return {}
Lambda functions
AWS Lambda
AWS Lambda
AWS Lambda Amazon CloudWatch Events
T1 - Create index/cron
functions
T2 - Create hello_pygotham
function

25. app.py
from chalice import Chalice
app = Chalice(app_name='myproject')
@app.lambda_function()
def index(event, context):
return {'hello': 'world'}
@app.lambda_function()
def hello_pygotham(event, context):
return {'hello': 'pygotham'}
Lambda functions
AWS Lambda
AWS Lambda
T1 - Create index/cron
functions
T2 - Create hello_pygotham
function
T3 - Delete cron
function

26. app.py
from chalice import Chalice
app = Chalice(app_name='myproject')
@app.lambda_function()
def index(event, context):
return {'hello': 'world'}
@app.lambda_function()
def hello_pygotham(event, context):
return {'hello': 'pygotham'}
Lambda functions
AWS Lambda
AWS Lambda
T1 - Create index/cron
functions
T2 - Create hello_pygotham
function
T3 - Delete cron
function

27. Deployment Properties
• Every new function in python code should correspond to a create API call
• Deleting a function in python code deletes the remote resource (unreferenced resources)
• Existing resources should have update calls (if needed)

28. app.py
from chalice import Chalice
app = Chalice(app_name='myproject')
@app.lambda_function()
def index(event, context):
return {'hello': 'world'}
@app.lambda_function()
def hello_pygotham(event, context):
return {'hello': 'pygotham'}
@app.schedule('rate(5 minutes)')
def cron(event):
return {}
Deployment properties
AWS Lambda
AWS Lambda
AWS Lambda Amazon CloudWatch Events

29. app.py
from chalice import Chalice
app = Chalice(app_name='myproject')
@app.lambda_function()
def index(event, context):
return {'hello': 'world'}
@app.lambda_function()
def hello_pygotham(event, context):
return {'hello': 'pygotham'}
@app.schedule('rate(5 minutes)')
def cron(event):
return {}
Deployment properties
AWS Lambda
AWS Lambda
AWS Lambda Amazon CloudWatch Events
T1 - Delete hello_pygotham

30. app.py
from chalice import Chalice
app = Chalice(app_name='myproject')
@app.lambda_function()
def index(event, context):
return {'hello': 'world'}
@app.schedule('rate(5 minutes)')
def cron(event):
return {}
Deployment properties
AWS Lambda
AWS Lambda
AWS Lambda Amazon CloudWatch Events
T1 - Delete hello_pygotham

31. app.py
from chalice import Chalice
app = Chalice(app_name='myproject')
@app.lambda_function()
def index(event, context):
return {'hello': 'world'}
@app.schedule('rate(5 minutes)')
def cron(event):
return {}
Deployment properties
AWS Lambda
AWS Lambda
AWS Lambda Amazon CloudWatch Events
T1 - Delete hello_pygotham
unreferenced resource

32. app.py
from chalice import Chalice
app = Chalice(app_name='myproject')
@app.lambda_function()
def index(event, context):
return {'hello': 'world'}
@app.schedule('rate(5 minutes)')
def cron(event):
return {}
Deployment properties
AWS Lambda
AWS Lambda
AWS Lambda Amazon CloudWatch Events
T1 - Delete hello_pygotham
T2 - Create hello_pygotham

33. app.py
from chalice import Chalice
app = Chalice(app_name='myproject')
@app.lambda_function()
def index(event, context):
return {'hello': 'world'}
@app.lambda_function()
def hello_pygotham(event, context):
return {'hello': 'pygotham'}
@app.schedule('rate(5 minutes)')
def cron(event):
return {}
Deployment properties
AWS Lambda
AWS Lambda
AWS Lambda Amazon CloudWatch Events
T1 - Delete hello_pygotham
T2 - Create hello_pygotham

34. app.py
from chalice import Chalice
app = Chalice(app_name='myproject')
@app.lambda_function()
def index(event, context):
return {'hello': 'world'}
@app.lambda_function()
def hello_pygotham(event, context):
return {'hello': 'pygotham'}
@app.schedule('rate(5 minutes)')
def cron(event):
return {}
Deployment properties
AWS Lambda
AWS Lambda
AWS Lambda Amazon CloudWatch Events
T1 - Delete hello_pygotham
T2 - Create hello_pygotham
Already exists!

35. filename.py
T1 - Create function named index
T2 - Create function named cron
T3 - Create function named hello_pygotham
T4 - Delete function named hello_pygotham
T5 - Create function named hello_pygotham (error)
Example input

36. filename.py
T1 - Create function named index
T2 - Create function named cron
T3 - Create function named hello_pygotham
T4 - Delete function named hello_pygotham
T5 - Create function named hello_pygotham (error)
Example input
Steps
}

37. filename.py
T1 - Create function named index
T2 - Create function named cron
T3 - Create function named hello_pygotham
T4 - Delete function named hello_pygotham
T5 - Create function named hello_pygotham (error)
Example input
Steps
}
How do we do this in code?

38. test_deployer.py
from hypothesis.stateful import GenericStateMachine
class DeployerTracker(GenericStateMachine):
def __init__(self, chalice_app=None):
…
def steps(self):
…
def execute_step(self, step):
…
State based testing

39. test_deployer.py
from hypothesis.stateful import GenericStateMachine
class DeployerTracker(GenericStateMachine):
def __init__(self, chalice_app=None):
…
def steps(self):
…
def execute_step(self, step):
…
State based testing
1. Subclass from GenericStateMachine

40. test_deployer.py
from hypothesis.stateful import GenericStateMachine
class DeployerTracker(GenericStateMachine):
def __init__(self, chalice_app=None):
…
def steps(self):
…
def execute_step(self, step):
…
State based testing
1. Subclass from GenericStateMachine
2. Initialize starting state in __init__()

41. test_deployer.py
from hypothesis.stateful import GenericStateMachine
class DeployerTracker(GenericStateMachine):
def __init__(self, chalice_app=None):
…
def steps(self):
…
def execute_step(self, step):
…
State based testing
1. Subclass from GenericStateMachine
2. Initialize starting state in __init__()
3. steps(), valid actions based on current state

42. test_deployer.py
from hypothesis.stateful import GenericStateMachine
class DeployerTracker(GenericStateMachine):
def __init__(self, chalice_app=None):
…
def steps(self):
…
def execute_step(self, step):
…
State based testing
1. Subclass from GenericStateMachine
2. Initialize starting state in __init__()
3. steps(), valid actions based on current state
4. Execute randomly selected step

43. test_deployer.py
from hypothesis.stateful import GenericStateMachine
from hypothesis.strategies import tuples, sampled_from, just, text
class DeployerTracker(GenericStateMachine):
def __init__(self, chalice_app=None):
self.chalice_app = chalice_app
def steps(self):
add_lambda_function = tuples(
just(self._add_function),
text(alphabet=string.ascii_lowercase)
)
if not self.chalice_app.lambda_functions:
return add_lambda_function
else:
delete_lambda_function = tuples(
just(self._delete_function),
sampled_from(self.chalice_app.lambda_functions)
)
return (add_lambda_function | delete_lambda_function)
def execute_step(self, step):
…
State based testing

44. test_deployer.py
from hypothesis.stateful import GenericStateMachine
from hypothesis.strategies import tuples, sampled_from, just, text
class DeployerTracker(GenericStateMachine):
def __init__(self, chalice_app=None):
self.chalice_app = chalice_app
def steps(self):
add_lambda_function = tuples(
just(self._add_function),
text(alphabet=string.ascii_lowercase)
)
if not self.chalice_app.lambda_functions:
return add_lambda_function
else:
delete_lambda_function = tuples(
just(self._delete_function),
sampled_from(self.chalice_app.lambda_functions)
)
return (add_lambda_function | delete_lambda_function)
def execute_step(self, step):
…
State based testing

45. test_deployer.py
from hypothesis.stateful import GenericStateMachine
from hypothesis.strategies import tuples, sampled_from, just, text
class DeployerTracker(GenericStateMachine):
def __init__(self, chalice_app=None):
self.chalice_app = chalice_app
def steps(self):
add_lambda_function = tuples(
just(self._add_function),
text(alphabet=string.ascii_lowercase)
)
if not self.chalice_app.lambda_functions:
return add_lambda_function
else:
delete_lambda_function = tuples(
just(self._delete_function),
sampled_from(self.chalice_app.lambda_functions)
)
return (add_lambda_function | delete_lambda_function)
def execute_step(self, step):
…
State based testing

46. test_deployer.py
from hypothesis.stateful import GenericStateMachine
from hypothesis.strategies import tuples, sampled_from, just, text
class DeployerTracker(GenericStateMachine):
def __init__(self, chalice_app=None):
self.chalice_app = chalice_app
def steps(self):
add_lambda_function = tuples(
just(self._add_function),
text(alphabet=string.ascii_lowercase)
)
if not self.chalice_app.lambda_functions:
return add_lambda_function
else:
delete_lambda_function = tuples(
just(self._delete_function),
sampled_from(self.chalice_app.lambda_functions)
)
return (add_lambda_function | delete_lambda_function)
def execute_step(self, step):
…
State based testing
(self._add_function, 'random-name')

47. test_deployer.py
from hypothesis.stateful import GenericStateMachine
from hypothesis.strategies import tuples, sampled_from, just, text
class DeployerTracker(GenericStateMachine):
def __init__(self, chalice_app=None):
self.chalice_app = chalice_app
def steps(self):
add_lambda_function = tuples(
just(self._add_function),
text(alphabet=string.ascii_lowercase)
)
if not self.chalice_app.lambda_functions:
return add_lambda_function
else:
delete_lambda_function = tuples(
just(self._delete_function),
sampled_from(self.chalice_app.lambda_functions)
)
return (add_lambda_function | delete_lambda_function)
def execute_step(self, step):
…
State based testing
# type: List[LambdaFunction]

48. test_deployer.py
from hypothesis.stateful import GenericStateMachine
from hypothesis.strategies import tuples, sampled_from, just, text
class DeployerTracker(GenericStateMachine):
def __init__(self, chalice_app=None):
self.chalice_app = chalice_app
def steps(self):
add_lambda_function = tuples(
just(self._add_function),
text(alphabet=string.ascii_lowercase)
)
if not self.chalice_app.lambda_functions:
return add_lambda_function
else:
delete_lambda_function = tuples(
just(self._delete_function),
sampled_from(self.chalice_app.lambda_functions)
)
return (add_lambda_function | delete_lambda_function)
def execute_step(self, step):
…
State based testing

49. test_deployer.py
from hypothesis.stateful import GenericStateMachine
from hypothesis.strategies import tuples, sampled_from, just, text
class DeployerTracker(GenericStateMachine):
def __init__(self, chalice_app=None):
self.chalice_app = chalice_app
def steps(self):
add_lambda_function = tuples(
just(self._add_function),
text(alphabet=string.ascii_lowercase)
)
if not self.chalice_app.lambda_functions:
return add_lambda_function
else:
delete_lambda_function = tuples(
just(self._delete_function),
sampled_from(self.chalice_app.lambda_functions)
)
return (add_lambda_function | delete_lambda_function)
def execute_step(self, step):
…
State based testing
(self._delete_function, <LambdaFunction()>)

50. test_deployer.py
from hypothesis.stateful import GenericStateMachine
from hypothesis.strategies import tuples, sampled_from, just, text
class DeployerTracker(GenericStateMachine):
def __init__(self, chalice_app=None):
self.chalice_app = chalice_app
def steps(self):
add_lambda_function = tuples(
just(self._add_function),
text(alphabet=string.ascii_lowercase)
)
if not self.chalice_app.lambda_functions:
return add_lambda_function
else:
delete_lambda_function = tuples(
just(self._delete_function),
sampled_from(self.chalice_app.lambda_functions)
)
return (add_lambda_function | delete_lambda_function)
def execute_step(self, step):
…
State based testing

51. test_deployer.py
from hypothesis.stateful import GenericStateMachine
from hypothesis.strategies import tuples, sampled_from, just, text
class DeployerTracker(GenericStateMachine):
def __init__(self, chalice_app=None):
self.chalice_app = chalice_app
def steps(self):
add_lambda_function = tuples(
just(self._add_function),
text(alphabet=string.ascii_lowercase)
)
if not self.chalice_app.lambda_functions:
return add_lambda_function
else:
delete_lambda_function = tuples(
just(self._delete_function),
sampled_from(self.chalice_app.lambda_functions)
)
return (add_lambda_function | delete_lambda_function)
def execute_step(self, step):
…
State based testing

52. test_deployer.py
from hypothesis.stateful import GenericStateMachine
from hypothesis.strategies import tuples, sampled_from, just, text
class DeployerTracker(GenericStateMachine):
def execute_step(self, step):
action, arg = step
action(arg)
# After updating our state we can go ahead and go through our
# deploy process.
self._mock_deploy()
def _add_function(self, name):
function = LambdaFunction(func=lambda x, y: {}, name=name,
handler_string='app.%s' % name)
self.chalice_app.lambda_functions.append(function)
def _delete_function(self, function):
lambda_functions = self.chalice_app.lambda_functions
lambda_functions.remove(function)
State based testing

53. test_deployer.py
from hypothesis.stateful import GenericStateMachine
from hypothesis.strategies import tuples, sampled_from, just, text
class DeployerTracker(GenericStateMachine):
def execute_step(self, step):
action, arg = step
action(arg)
# After updating our state we can go ahead and go through our
# deploy process.
self._mock_deploy()
def _add_function(self, name):
function = LambdaFunction(func=lambda x, y: {}, name=name,
handler_string='app.%s' % name)
self.chalice_app.lambda_functions.append(function)
def _delete_function(self, function):
lambda_functions = self.chalice_app.lambda_functions
lambda_functions.remove(function)
State based testing

54. test_deployer.py
from hypothesis.stateful import GenericStateMachine
from hypothesis.strategies import tuples, sampled_from, just, text
class DeployerTracker(GenericStateMachine):
def execute_step(self, step):
action, arg = step
action(arg)
# After updating our state we can go ahead and go through our
# deploy process.
self._mock_deploy()
def _add_function(self, name):
function = LambdaFunction(func=lambda x, y: {}, name=name,
handler_string='app.%s' % name)
self.chalice_app.lambda_functions.append(function)
def _delete_function(self, function):
lambda_functions = self.chalice_app.lambda_functions
lambda_functions.remove(function)
State based testing
(self._add_function, 'random-name')

55. test_deployer.py
from hypothesis.stateful import GenericStateMachine
from hypothesis.strategies import tuples, sampled_from, just, text
class DeployerTracker(GenericStateMachine):
def execute_step(self, step):
action, arg = step
action(arg)
# After updating our state we can go ahead and go through our
# deploy process.
self._mock_deploy()
def _add_function(self, name):
function = LambdaFunction(func=lambda x, y: {}, name=name,
handler_string='app.%s' % name)
self.chalice_app.lambda_functions.append(function)
def _delete_function(self, function):
lambda_functions = self.chalice_app.lambda_functions
lambda_functions.remove(function)
State based testing

56. test_deployer.py
from hypothesis.stateful import GenericStateMachine
from hypothesis.strategies import tuples, sampled_from, just, text
class DeployerTracker(GenericStateMachine):
def execute_step(self, step):
action, arg = step
action(arg)
# After updating our state we can go ahead and go through our
# deploy process.
self._mock_deploy()
def _add_function(self, name):
function = LambdaFunction(func=lambda x, y: {}, name=name,
handler_string='app.%s' % name)
self.chalice_app.lambda_functions.append(function)
def _delete_function(self, function):
lambda_functions = self.chalice_app.lambda_functions
lambda_functions.remove(function)
State based testing

57. test_deployer.py
from hypothesis.stateful import GenericStateMachine
from hypothesis.strategies import tuples, sampled_from, just, text
class DeployerTracker(GenericStateMachine):
def execute_step(self, step):
action, arg = step
action(arg)
# After updating our state we can go ahead and go through our
# deploy process.
self._mock_deploy()
def _add_function(self, name):
function = LambdaFunction(func=lambda x, y: {}, name=name,
handler_string='app.%s' % name)
self.chalice_app.lambda_functions.append(function)
def _delete_function(self, function):
lambda_functions = self.chalice_app.lambda_functions
lambda_functions.remove(function)
State based testing

58. test_deployer.py
from hypothesis.stateful import GenericStateMachine
from hypothesis.strategies import tuples, sampled_from, just, text
class DeployerTracker(GenericStateMachine):
# Core business logic
# a) Do a real deploy to AWS, query for real services (integration test)
# b) Use mock/fake components and verify properties (unit tests)
def _mock_deploy(self):
…
State based testing

59. Summary
• State based tests assert properties after executing a sequence of steps
• Used as both unit and integration tests
• Verify correct state of the world after deploying multiple versions a chalice app

60. Property based testing
* Overview
* State-based testing
Fuzz testing
* Overview
* Multi-argument fuzzing
Agenda

61. fuzzing.py
while True:
fuzzing_input = generate_random_input()
try:
code_under_test(fuzzing_input)
except AllowedExceptions:
pass
except:
# An unexpected exception was raised.
report_fuzzing_failure(fuzzing_input)
Simplified fuzzing

62. AFL - American Fuzzy Lop
• Coverage guided genetic fuzzer
• Fast
• Simple to use

63. python-afl
• Create a python program that reads input from stdin
• Raise an exception on error cases (“crashes”)
• Create a set of sample input files, one sample input per file

64. bug.py
def test(x):
# Invoke functions/classes to
# test given random input of ‘x’.
…
def main():
test(sys.stdin.read())
import afl
while afl.loop():
main()
AFL Fuzzing Example

65. bug.py
def test(x):
# Invoke functions/classes to
# test given random input of ‘x’.
…
def main():
test(sys.stdin.read())
import afl
while afl.loop():
main()
AFL Fuzzing Example

66. bug.py
def test(x):
# Invoke functions/classes to
# test given random input of ‘x’.
…
def main():
test(sys.stdin.read())
import afl
while afl.loop():
main()
AFL Fuzzing Example

67. bug.py
def test(x):
# Invoke functions/classes to
# test given random input of ‘x’.
…
def main():
test(sys.stdin.read())
import afl
while afl.loop():
main()
AFL Fuzzing Example
Test code
AFL fuzz integration
1. Each iteration, new input is generated on stdin
2. Read from stdin and pass the new input
to the test function

68. $ cat corpus/a
a
$ py-afl-fuzz -o results/ -i corpus/ -- $(which python) bug.py
Running python-afl
Sample inputCrashes

69. $ tree results/
results/
├── crashes
│ ├── id:000000,sig:10,src:000005,op:arith8,pos:4,val:+23
│ └── README.txt
├── fuzz_bitmap
├── fuzzer_stats
├── hangs
├── plot_data
└── queue
├── id:000000,orig:a
├── id:000001,src:000000,op:havoc,rep:16,+cov
├── id:000002,src:000001,op:havoc,rep:16,+cov
├── id:000003,src:000002,op:arith8,pos:1,val:+19,+cov
├── id:000004,src:000003,op:arith8,pos:2,val:+5,+cov
└── id:000005,src:000004,op:arith8,pos:3,val:+5,+cov
AFL Fuzzing Example

70. $ tree results/
results/
├── crashes
│ ├── id:000000,sig:10,src:000005,op:arith8,pos:4,val:+23
│ └── README.txt
├── fuzz_bitmap
├── fuzzer_stats
├── hangs
├── plot_data
└── queue
├── id:000000,orig:a
├── id:000001,src:000000,op:havoc,rep:16,+cov
├── id:000002,src:000001,op:havoc,rep:16,+cov
├── id:000003,src:000002,op:arith8,pos:1,val:+19,+cov
├── id:000004,src:000003,op:arith8,pos:2,val:+5,+cov
└── id:000005,src:000004,op:arith8,pos:3,val:+5,+cov
AFL Fuzzing Example
Value that caused the crash

71. def code_path(a, b, c):
power = 0
if a:
power += 1
if b:
power += 1
if c:
power += 1
return [0] * (4096 ** power)
Path coverage

72. def code_path(a, b, c):
power = 0
if a:
power += 1
if b:
power += 1
if c:
power += 1
return [0] * (4096 ** power)
Path coverage
True,False,False
True,False,False

73. def code_path(a, b, c):
power = 0
if a:
power += 1
if b:
power += 1
if c:
power += 1
return [0] * (4096 ** power)
Path coverage
True,False,False False,True,False
False,True,False

74. def code_path(a, b, c):
power = 0
if a:
power += 1
if b:
power += 1
if c:
power += 1
return [0] * (4096 ** power)
Path coverage
True,False,False False,True,False False,False,True Total
False,False,True

75. def code_path(a, b, c):
power = 0
if a:
power += 1
if b:
power += 1
if c:
power += 1
return [0] * (4096 ** power)
Path coverage
True,False,False False,True,False False,False,True Total
100% line coverage
100% branch coverage

76. def code_path(a, b, c):
power = 0
if a:
power += 1
if b:
power += 1
if c:
power += 1
return [0] * (4096 ** power)
Path coverage
True,True,True
Each branch is taken in a single invocation

77. def code_path(a, b, c):
power = 0
if a:
power += 1
if b:
power += 1
if c:
power += 1
return [0] * (4096 ** power)
Path coverage
True,True,True
Each branch is taken in a single invocation
4096 ^ 3 = 68719476736 elements
68719476736 * 8 bytes/element = 512 GB

78. def code_path(a, b, c):
power = 0
if a:
power += 1
if b:
power += 1
if c:
power += 1
return [0] * (4096 ** power)
Path coverage
True,True,True
Each branch is taken in a single invocation
4096 ^ 3 = 68719476736 elements
68719476736 * 8 bytes/element = 512 GB

79. A Query Language for JSON
import jmespath
# .search(expression, input_data)
jmespath.search(‘a.b', {'a': {'b': {'c': 'd'}}}) # {'c': 'd'}
$ aws ec2 describe-instances --query 'Reservations[].Instances[].[InstanceId, State.Name]'
- name: "Display all cluster names"
debug: var=item
with_items: "{{domain_definition|json_query('domain.cluster[*].name')}}"
AWS CLI
Ansible
Python API

80. >>> import jmespath
>>> # jmespath.search(expression: str, data: Any)
>>> jmespath.search('a.b', {'a': {'b': {'c': {'d': [0, 1, 2]}}}})
JMESPath search() usage

81. >>> import jmespath
>>> # jmespath.search(expression: str, data: Any)
>>> jmespath.search('a.b', {'a': {'b': {'c': {'d': [0, 1, 2]}}}})
JMESPath search() usage

82. >>> import jmespath
>>> # jmespath.search(expression: str, data: Any)
>>> jmespath.search('a.b', {'a': {'b': {'c': {'d': [0, 1, 2]}}}})
JMESPath search() usage

83. >>> import jmespath
>>> # jmespath.search(expression: str, data: Any)
>>> jmespath.search('a.b', {'a': {'b': {'c': {'d': [0, 1, 2]}}}})
{'c': {'d': [0, 1, 2]}}
JMESPath search() usage

84. >>> import jmespath
>>> # jmespath.search(expression: str, data: Any)
>>> jmespath.search('a.b', {'a': {'b': {'c': {'d': [0, 1, 2]}}}})
{'c': {'d': [0, 1, 2]}}
>>> jmespath.search('a.b.c.d', {'a': {'b': {'c': {'d': [0, 1, 2]}}}})
[0, 1, 2]
>>> jmespath.search('a.b.c.d[0]', {'a': {'b': {'c': {'d': [0, 1, 2]}}}})
0
JMESPath search() usage

85. >>> jmespath.compile('a.b.c.d[0]')
JMESPath compilation

86. >>> jmespath.compile('a.b.c.d[0]')
JMESPath compilation
Generated by AFL fuzz
(arbitrary byte sequence)

87. >>> jmespath.compile(‘a.b.c.d[0]')
>>> # jmespath.search(expression: str, data: Any)
>>> jmespath.search('a.b', {'a': {'b': {'c': {'d': [0, 1, 2]}}}})
JMESPath compilation

88. >>> jmespath.compile(‘a.b.c.d[0]')
>>> # jmespath.search(expression: str, data: Any)
>>> jmespath.search('a.b', {'a': {'b': {'c': {'d': [0, 1, 2]}}}})
JMESPath compilation
How do we generate multiple arguments with different types?

89. Multiple Input Parameters
• Random data is provided through sys.stdin
• Coverage based mutations
• Starting corpus of input comes from a set of files
• Create a pseudo-file format representing arguments
• afl-fuzz magically figures out what we mean?

90. >>> # jmespath.search(expression: str, data: Any)
>>> jmespath.search('a.b', {'a': {'b': {'c': {'d': [0, 1, 2]}}}})
JMESPath compilation
input.txt
a.b
----DELIMITER----
{"a": {"b": {"c": {"d": [0, 1, 2]}}}}

91. Starting Corpus
`2.0ý`
----DELIMITER----
{"": null, "uf1bcnu001du91feu0010uff1fn000u0207u55bfu0156r!": ""}
input.txt
foo.bar
----DELIMITER----
{"": null, "a": [1,2,3]}
input.txt
foo.bar[0].baz
----DELIMITER----
{"b": true, "a": [1,2,3]}
input.txt
.
.
.

92. fuzz.py
import jmespath
from jmespath import exceptions
def main():
try:
newval = sys.stdin.read()
expression, partition, data = newval.partition('n----DELIMITER----n')
if not expression or not data:
return
if not partition:
return
try:
parsed = json.loads(data)
except Exception as e:
return
result = jmespath.search(expression, parsed)
except exceptions.JMESPathError as e:
# JMESPathError is allowed.
return 0
return 0
import afl
while afl.loop():
main()
Fuzzing

93. fuzz.py
import jmespath
from jmespath import exceptions
def main():
try:
newval = sys.stdin.read()
expression, partition, data = newval.partition('n----DELIMITER----n')
if not expression or not data:
return
if not partition:
return
try:
parsed = json.loads(data)
except Exception as e:
return
result = jmespath.search(expression, parsed)
except exceptions.JMESPathError as e:
# JMESPathError is allowed.
return 0
return 0
import afl
while afl.loop():
main()
Fuzzing
input.txt
a.b
----DELIMITER----
{"a": {"b": {"c": {"d": [0, 1, 2]}}}}

94. fuzz.py
import jmespath
from jmespath import exceptions
def main():
try:
newval = sys.stdin.read()
expression, partition, data = newval.partition('n----DELIMITER----n')
if not expression or not data:
return
if not partition:
return
try:
parsed = json.loads(data)
except Exception as e:
return
result = jmespath.search(expression, parsed)
except exceptions.JMESPathError as e:
# JMESPathError is allowed.
return 0
return 0
import afl
while afl.loop():
main()
Fuzzing
input.txt
a.b
----DELIMITER----
{"a": {"b": {"c": {"d": [0, 1, 2]}}}}

95. fuzz.py
import jmespath
from jmespath import exceptions
def main():
try:
newval = sys.stdin.read()
expression, partition, data = newval.partition('n----DELIMITER----n')
if not expression or not data:
return
if not partition:
return
try:
parsed = json.loads(data)
except Exception as e:
return
result = jmespath.search(expression, parsed)
except exceptions.JMESPathError as e:
# JMESPathError is allowed.
return 0
return 0
import afl
while afl.loop():
main()
Fuzzing
input.txt
a.b
----DELIMITER----
{"a": {"b": {"c": {"d": [0, 1, 2]}}}}

96. fuzz.py
import jmespath
from jmespath import exceptions
def main():
try:
newval = sys.stdin.read()
expression, partition, data = newval.partition('n----DELIMITER----n')
if not expression or not data:
return
if not partition:
return
try:
parsed = json.loads(data)
except Exception as e:
return
result = jmespath.search(expression, parsed)
except exceptions.JMESPathError as e:
# JMESPathError is allowed.
return 0
return 0
import afl
while afl.loop():
main()
Fuzzing
input.txt
a.b
----DELIMITER----
{"a": {"b": {"c": {"d": [0, 1, 2]}}}}

97. bad-delimiter - 0 ( 0.00%)
bad-json - 23398576 (47.30%)
jp-error - 6575956 (13.29%)
no-errors - 337233 ( 0.68%)
Does it work?

98. Summary
• python-afl intelligently figures out meaningful input data
• Treat multi-param input as pseudo file format
• Possible to use property-based testing approach

99. Property based testing
* Overview
* State-based testing
Fuzz testing
* Overview
* Multi-argument fuzzing
Agenda

100. Thanks!
• American Fuzzy Lop - http://lcamtuf.coredump.cx/afl/
• Hypothesis - http://hypothesis.works/
• JMESPath - http://jmespath.org/
• Chalice - http://github.com/aws/chalice
• Next Level Testing (PyCon 2017) - https://www.youtube.com/watch?v=jmsk1QZQEvQ
• @jsaryer