Gen AI in Business - Global Trends Report 2024.pdf
Benchy, python framework for performance benchmarking of Python Scripts
1. Benchy
Lightweight performing benchmark framework for
Python scripts
Marcel Caraciolo
@marcelcaraciolo
Developer, Cientist, contributor to the Crab recsys project,
works with Python for 6 years, interested at mobile,
education, machine learning and dataaaaa!
Recife, Brazil - http://aimotion.blogspot.com
2. About me
Co-founder of Crab - Python recsys library
Cientist Chief at Atepassar, e-learning social network
Co-Founder and Instructor of PyCursos, teaching Python on-line
Co-Founder of Pingmind, on-line infrastructure for MOOC’s
Interested at Python, mobile, e-learning and machine learning!
9. Solutions ?
In
[1]:
def
f(x):
...:
return
x*x
...:
In
[2]:
%timeit
for
x
in
range
(100):
f(x)
100000
loops,
best
of
3:
20.3
us
per
loop
10. Stop. Help is near
Performance benchmarks to compare several python code
alternatives
Generates graphs using matplotlib
Memory consumption, Performance timing available
https://github.com/python-recsys/benchy
13. Writing benchmarks
from
benchy.api
import
Benchmark
common_setup
=
""
statement
=
"lst
=
['i'
for
x
in
range(100000)]"
benchmark1
=
Benchmark(statement,
common_setup,
name=
"range")
statement
=
"lst
=
['i'
for
x
in
xrange(100000)]"
benchmark2
=
Benchmark(statement,
common_setup,
name=
"xrange")
statement
=
"lst
=
['i']
*
100000"
benchmark3
=
Benchmark(statement,
common_setup,
name=
"range")
14. Use them in your
workflow
[1]:
print
benchmark1.run()
{'memory':
{'repeat':
3,
'success':
True,
'units':
'MB',
'usage':
2.97265625},
'runtime':
{'loops':
100,
'repeat':
3,
'success':
True,
'timing':
7.5653696060180664,
'units':
'ms'}}
Same code as %timeit
and %memit
16. Benchmark suite
from
benchy.api
import
BenchmarkSuite
suite
=
BenchmarkSuite()
suite.append(benchmark1)
suite.append(benchmark2)
suite.append(benchmark3)
17. Run the benchmarks
from
benchy.api
import
BenchmarkRunner
runner
=
BenchmarkRunner(benchmarks=suite,
tmp_dir='.',
name=
'List
Allocation
Benchmark')
n_benchs,
results
=
runner.run()
25. Get involved
Create the benchmarks as TestCases
Check automatically for benchmark files and run like %nose.test()
More setup and teardown control
Group benchmarks at the same graph
32. New Runner
class
BenchmarkGitRunner(BenchmarkRunner):
...
def
_register_benchmarks(self):
ex_benchmarks
=
self.db.get_benchmarks()
db_checksums
=
set(ex_benchmarks.index)
for
bm
in
self.benchmarks:
if
bm.checksum
in
db_checksums:
self.db.update_name(bm)
else:
print
'Writing
new
benchmark
%s,
%s'
%
(bm.name,
bm.checksum)
self.db.write_benchmark(bm)
33. New runner
class
BenchmarkGitRunner(BenchmarkRunner):
...
def
_run_revision(self,
rev):
need_to_run
=
self._get_benchmarks_for_rev(rev)
if
not
need_to_run:
print
'No
benchmarks
need
running
at
%s'
%
rev
return
0,
{}
print
'Running
%d
benchmarks
for
revision
%s'
%
(len(need_to_run),
rev)
for
bm
in
need_to_run:
print
bm.name
self.bench_repo.switch_to_revision(rev)
pickle_path
=
os.path.join(self.tmp_dir,
'benchmarks.pickle')
results_path
=
os.path.join(self.tmp_dir,
'results.pickle')
if
os.path.exists(results_path):
os.remove(results_path)
pickle.dump(need_to_run,
open(pickle_path,
'w'))
#
run
the
process
cmd
=
'python
%s/run_benchmarks.py
%s
%s'
%
(pickle_path,
results_path)
print
cmd
proc
=
subprocess.Popen(cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
shell=True,
cwd=self.tmp_dir)
stdout,
stderr
=
proc.communicate()
34. New runner
class
BenchmarkGitRunner(BenchmarkRunner):
...
def
_run_revision(self,
rev):
need_to_run
=
self._get_benchmarks_for_rev(rev)
if
not
need_to_run:
print
'No
benchmarks
need
running
at
%s'
%
rev
return
0,
{}
print
'Running
%d
benchmarks
for
revision
%s'
%
(len(need_to_run),
rev)
for
bm
in
need_to_run:
print
bm.name
self.bench_repo.switch_to_revision(rev)
#
run
the
process
cmd
=
'python
%s/run_benchmarks.py
%s
%s'
%
(pickle_path,
results_path)
print
cmd
proc
=
subprocess.Popen(cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
shell=True,
cwd=self.tmp_dir)
stdout,
stderr
=
proc.communicate()
if
stderr:
if
("object
has
no
attribute"
in
stderr
or
'ImportError'
in
stderr):
print
stderr
print
'HARD
CLEANING!'
self.bench_repo.hard_clean()
print
stderr
if
not
os.path.exists(results_path):
print
'Failed
for
revision
%s'
%
rev
return
len(need_to_run),
{}
results
=
pickle.load(open(results_path,
'r'))
39. Working now:
Module detection
by_module
=
{}
benchmarks
=
[]
modules
=
['metrics',
'recommenders',
'similarities']
for
modname
in
modules:
ref
=
__import__(modname)
by_module[modname]
=
[v
for
v
in
ref.__dict__.values()
if
isinstance(v,
Benchmark)]
benchmarks.extend(by_module[modname])
for
bm
in
benchmarks:
assert(bm.name
is
not
None)
43. Benchy
Lightweight performing benchmark framework for
Python scripts
Marcel Caraciolo
@marcelcaraciolo
Developer, Cientist, contributor to the Crab recsys project,
works with Python for 6 years, interested at mobile,
education, machine learning and dataaaaa!
Recife, Brazil - http://aimotion.blogspot.com