Benchy is a lightweight Python framework for performing benchmarks on code. It allows generating performance and memory usage graphs to compare different code implementations. Benchmarks can be written as objects and executed via a BenchmarkRunner to obtain results. Results are stored in a SQLite database and full reports can be generated in reStructuredText format. The framework aims to provide an easy way to integrate benchmarks into the development workflow.

1. Benchy
Lightweight performing benchmark framework for
Python scripts
Marcel Caraciolo
@marcelcaraciolo
Developer, Cientist, contributor to the Crab recsys project,
works with Python for 6 years, interested at mobile,
education, machine learning and dataaaaa!
Recife, Brazil - http://aimotion.blogspot.com

2. About me
Co-founder of Crab - Python recsys library
Cientist Chief at Atepassar, e-learning social network
Co-Founder and Instructor of PyCursos, teaching Python on-line
Co-Founder of Pingmind, on-line infrastructure for MOOC’s
Interested at Python, mobile, e-learning and machine learning!

3. Why do we test ?

4. Freedom from fear

5. Testing for
performance

6. What made my
code slower ?

8. me

9. Solutions ?
In
[1]:
def
f(x):
...:
return
x*x
...:
In
[2]:
%timeit
for
x
in
range
(100):
f(x)
100000
loops,
best
of
3:
20.3
us
per
loop

10. Stop. Help is near
Performance benchmarks to compare several python code
alternatives
Generates graphs using matplotlib
Memory consumption, Performance timing available
https://github.com/python-recsys/benchy

11. Performance
benchmarks

12. Writing benchmarks
$
easy_install
-­‐U
benchy
#
pip
install
-­‐U
benchy

13. Writing benchmarks
from
benchy.api
import
Benchmark
common_setup
=
""
statement
=
"lst
=
['i'
for
x
in
range(100000)]"
benchmark1
=
Benchmark(statement,
common_setup,
name=
"range")
statement
=
"lst
=
['i'
for
x
in
xrange(100000)]"
benchmark2
=
Benchmark(statement,
common_setup,
name=
"xrange")
statement
=
"lst
=
['i']
*
100000"
benchmark3
=
Benchmark(statement,
common_setup,
name=
"range")

14. Use them in your
workflow
[1]:
print
benchmark1.run()
{'memory':
{'repeat':
3,
'success':
True,
'units':
'MB',
'usage':
2.97265625},
'runtime':
{'loops':
100,
'repeat':
3,
'success':
True,
'timing':
7.5653696060180664,
'units':
'ms'}}
Same code as %timeit
and %memit

15. Beautiful reports
rst_text
=
benchmark1.to_rst(results)

16. Benchmark suite
from
benchy.api
import
BenchmarkSuite
suite
=
BenchmarkSuite()
suite.append(benchmark1)
suite.append(benchmark2)
suite.append(benchmark3)

17. Run the benchmarks
from
benchy.api
import
BenchmarkRunner
runner
=
BenchmarkRunner(benchmarks=suite,
tmp_dir='.',
name=
'List
Allocation
Benchmark')
n_benchs,
results
=
runner.run()

18. Who is the faster ?
{Benchmark('list
with
"*"'):
{'runtime':
{'timing':
0.47582697868347168,
'repeat':
3,
'success':
True,
'loops':
1000,
'timeBaselines':
1.0,
'units':
'ms'},
'memory':
{'usage':
0.3828125,
'units':
'MB',
'repeat':
3,
'success':
True}},
Benchmark('list
with
xrange'):
{'runtime':
{'timing':
5.623779296875,
'repeat':
3,
'success':
True,
'loops':
100,
'timeBaselines':
11.818958463504936,
'units':
'ms'},
'memory':
{'usage':
0.71484375,
'units':
'MB',
'repeat':
3,
'success':
True}},
Benchmark('list
with
range'):
{
'runtime':
{'timing':
6.5933513641357422,
'repeat':
3,
'success':
True,
'loops':
100,
'timeBaselines':
13.856615239384636,
'units':
'ms'},
'memory':
{'usage':
2.2109375,
'units':
'MB',
'repeat':
3,
'success':
True}}}

19. Plot relative
fig
=
runner.plot_relative(results,
horizontal=True)
plt.savefig('%s_r.png'
%
runner.name,
bbox_inches='tight')

20. Plot absolute
runner.plot_absolute(results,
horizontal=False)
plt.savefig('%s.png'
%
runner.name)
#
bbox_inches='tight')

21. Full report
rst_text
=
runner.to_rst(results,
runner.name
+
'png',
runner.name
+
'_r.png')
with
open('teste.rst',
'w')
as
f:
f.write(rst_text)

22. Full report

23. Full report

24. Why ?
Benchmark pairwise functions at Crab recsys library
http://aimotion.blogspot.com.br/2013/03/performing-runtime-benchmarks-with.html

25. Get involved
Create the benchmarks as TestCases
Check automatically for benchmark files and run like %nose.test()
More setup and teardown control
Group benchmarks at the same graph

26. Improvements
Added Database Handler
Added Git Support
Added New Runner
Run benchmarks

27. db.py
import
sqlite3
class
BenchmarkDb(object):
"""
Persistence
handler
for
bechmark
results
"""
def
_create_tables(self):
self._cursor.execute("drop
table
if
exists
benchmarksuites")
self._cursor.execute("drop
table
if
exists
benchmarks")
self._cursor.execute("drop
table
if
exists
results")
...
self._cursor.execute('CREATE
TABLE
benchmarks(checksum
text
PRIMARY
KEY,
name
text,
description
text,
suite_id
integer,
FOREIGN
KEY(suite_id)
REFERENCES
benchmarksuites(id))')
self._cursor.execute('CREATE
TABLE
results(id
integer
PRIMARY
KEY
AUTOINCREMENT,
checksum
text,
timestamp
timestamp,
ncalls
text,
timing
float,
traceback
text,
FOREIGN
KEY(checksum)
REFERENCES
benchmarks(checksum))')
self._con.commit()
def
write_benchmark(self,
bm,
suite=None):
if
suite
is
not
None:
self._cursor.execute('SELECT
id
FROM
benchmarksuites
where
name
=
"%s"'
%
suite.name)
row
=
self._cursor.fetchone()
else:
row
=
None
if
row
==
None:
self._cursor.execute('INSERT
INTO
benchmarks
VALUES
(?,
?,
?,
?)',
(bm.checksum,
bm.name,
bm.description,
None))
else:
self._cursor.execute('INSERT
INTO
benchmarks
VALUES
(?,
?,
?,
?)',
(bm.checksum,
bm.name,
bm.description,
row[0]))

28. Improvements
Added Database Handler
Added Git Support
Added New Runner
Run benchmarks

29. Git Repo
class
GitRepository(Repository):
"""
Read
some
basic
statistics
about
a
git
repository
"""
def
__init__(self,
repo_path):
self.repo_path
=
repo_path
self.git
=
_git_command(self.repo_path)
(self.shas,
self.messages,
self.timestamps,
self.authors)
=
self._parse_commit_log()
[('d87fdf2', datetime.datetime(2013, 3, 22, 16, 55, 38)), ('a90a449', datetime.datetime(2013, 3, 22, 16, 54, 36)),
('fe66a86', datetime.datetime(2013, 3, 22, 16, 51, 2)), ('bea6b21', datetime.datetime(2013, 3, 22, 13, 14, 22)),
('bde5e63', datetime.datetime(2013, 3, 22, 5, 2, 56)), ('89634f6', datetime.datetime(2013, 3, 20, 4, 16, 19))]

30. Git Repo
class
BenchmarkRepository(object):
"""
Manage
an
isolated
copy
of
a
repository
for
benchmarking
"""
...
def
_copy_repo(self):
if
os.path.exists(self.target_dir):
print
'Deleting
%s
first'
%
self.target_dir
#
response
=
raw_input('%s
exists,
delete?
y/n'
%
self.target_dir)
#
if
response
==
'n':
#
raise
Exception('foo')
cmd
=
'rm
-­‐rf
%s'
%
self.target_dir
print
cmd
os.system(cmd)
self._clone(self.target_dir_tmp,
self.target_dir)
self._prep()
self._copy_benchmark_scripts_and_deps()
def
_clone(self,
source,
target):
cmd
=
'git
clone
%s
%s'
%
(source,
target)
print
cmd
os.system(cmd)
def
_copy_benchmark_scripts_and_deps(self):
pth,
_
=
os.path.split(os.path.abspath(__file__))
deps
=
[os.path.join(pth,
'run_benchmarks.py')]
if
self.dependencies
is
not
None:
deps.extend(self.dependencies)
for
dep
in
deps:
cmd
=
'cp
%s
%s'
%
(dep,
self.target_dir)
print
cmd
proc
=
subprocess.Popen(cmd,
shell=True)
proc.wait()

31. Improvements
Added Database Handler
Added Git Support
Added New Runner
Run benchmarks

32. New Runner
class
BenchmarkGitRunner(BenchmarkRunner):
...
def
_register_benchmarks(self):
ex_benchmarks
=
self.db.get_benchmarks()
db_checksums
=
set(ex_benchmarks.index)
for
bm
in
self.benchmarks:
if
bm.checksum
in
db_checksums:
self.db.update_name(bm)
else:
print
'Writing
new
benchmark
%s,
%s'
%
(bm.name,
bm.checksum)
self.db.write_benchmark(bm)

33. New runner
class
BenchmarkGitRunner(BenchmarkRunner):
...
def
_run_revision(self,
rev):
need_to_run
=
self._get_benchmarks_for_rev(rev)
if
not
need_to_run:
print
'No
benchmarks
need
running
at
%s'
%
rev
return
0,
{}
print
'Running
%d
benchmarks
for
revision
%s'
%
(len(need_to_run),
rev)
for
bm
in
need_to_run:
print
bm.name
self.bench_repo.switch_to_revision(rev)
pickle_path
=
os.path.join(self.tmp_dir,
'benchmarks.pickle')
results_path
=
os.path.join(self.tmp_dir,
'results.pickle')
if
os.path.exists(results_path):
os.remove(results_path)
pickle.dump(need_to_run,
open(pickle_path,
'w'))
#
run
the
process
cmd
=
'python
%s/run_benchmarks.py
%s
%s'
%
(pickle_path,
results_path)
print
cmd
proc
=
subprocess.Popen(cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
shell=True,
cwd=self.tmp_dir)
stdout,
stderr
=
proc.communicate()

34. New runner
class
BenchmarkGitRunner(BenchmarkRunner):
...
def
_run_revision(self,
rev):
need_to_run
=
self._get_benchmarks_for_rev(rev)
if
not
need_to_run:
print
'No
benchmarks
need
running
at
%s'
%
rev
return
0,
{}
print
'Running
%d
benchmarks
for
revision
%s'
%
(len(need_to_run),
rev)
for
bm
in
need_to_run:
print
bm.name
self.bench_repo.switch_to_revision(rev)
#
run
the
process
cmd
=
'python
%s/run_benchmarks.py
%s
%s'
%
(pickle_path,
results_path)
print
cmd
proc
=
subprocess.Popen(cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
shell=True,
cwd=self.tmp_dir)
stdout,
stderr
=
proc.communicate()
if
stderr:
if
("object
has
no
attribute"
in
stderr
or
'ImportError'
in
stderr):
print
stderr
print
'HARD
CLEANING!'
self.bench_repo.hard_clean()
print
stderr
if
not
os.path.exists(results_path):
print
'Failed
for
revision
%s'
%
rev
return
len(need_to_run),
{}
results
=
pickle.load(open(results_path,
'r'))

35. Improvements
Added Database Handler
Added Git Support
Added New Runner
Run benchmarks

36. Running
from
benchmark
import
Benchmark,
BenchmarkRepository,
BenchmarkGitRunner
try:
REPO_PATH
=
config.get('setup',
'repo_path')
REPO_URL
=
config.get('setup',
'repo_url')
DB_PATH
=
config.get('setup',
'db_path')
TMP_DIR
=
config.get('setup',
'tmp_dir')
except:
REPO_PATH
=
os.path.abspath(os.path.join(os.path.dirname(__file__),
"../"))
REPO_URL
=
'git@github.com:python-­‐recsys/crab.git'
DB_PATH
=
os.path.join(REPO_PATH,
'suite/benchmarks.db')
TMP_DIR
=
os.path.join(HOME,
'tmp/base_benchy/')
PREPARE
=
"""
python
setup.py
clean
"""
BUILD
=
"""
python
setup.py
build_ext
-­‐-­‐inplace
"""
repo
=
BenchmarkRepository(REPO_PATH,
REPO_URL,
DB_PATH,
TMP_DIR)

37. Running
common_setup
=
"""
import
numpy
from
crab.metrics
import
cosine_distances
X
=
numpy.random.uniform(1,5,(1000,))
"""
bench
=
Benchmark(statement,
setup_bk1,
name="Crab
Cosine")
suite
=
BenchmarkSuite()
suite.append(bench)
statement
=
"cosine_distances(X,
X)"
runner
=
BenchmarkGitRunner(suite,
'.',
'Absolute
timing
in
ms')
n_benchs,
results
=
runner.run()
runner.plot_history(results)
plt.show()

38. Improvements
Historical commits from version control now
benchmarked

39. Working now:
Module detection
by_module
=
{}
benchmarks
=
[]
modules
=
['metrics',
'recommenders',
'similarities']
for
modname
in
modules:
ref
=
__import__(modname)
by_module[modname]
=
[v
for
v
in
ref.__dict__.values()
if
isinstance(v,
Benchmark)]
benchmarks.extend(by_module[modname])
for
bm
in
benchmarks:
assert(bm.name
is
not
None)

40. https://github.com/python-recsys/benchy
Forks and pull requests are welcomed!

43. Benchy
Lightweight performing benchmark framework for
Python scripts
Marcel Caraciolo
@marcelcaraciolo
Developer, Cientist, contributor to the Crab recsys project,
works with Python for 6 years, interested at mobile,
education, machine learning and dataaaaa!
Recife, Brazil - http://aimotion.blogspot.com