qiBuild, the meta build framework

● Lots of C++ (1,6 billions SLOC)
○ Desktop tools to program robots (Choregraphe), on Linux, Mac and
Windows
○ Robotic services running on robots (NAOqi) to help write applications
● Some Python:
○ Python bindings for our C++ middleware SDK
○ Language of choice for scripting and testing
● Lots of git:
○ > 100 git repositories
○ 10 development branches
○ Long running maintenance branches
Writing software at Aldebaran

● Started as a rewrite of Aldebaran’s release scripts in Python
● Morphed into a set of various command line tools for developers managing sources, tests
and C++ cross-platform build, all wrapped in a single package
● Added more features to build documentation and work with Python projects
● Started in 2010, open-sourced since 2011
● Active usage outside Aldebaran since recently
● Now a big project:
○ 4872 commits
○ 26960 Python SLOC
○ 879 tests, 86% line coverage
What is qiBuild ?

Two git projects:
● One containing the world library
● One containing
○ the hello library, that
■ depends on world
■ is wrapped with swig (helloswig)
○ A Python module wrapping helloswig called pyhello
■ with tests using py.test
How would you get the sources, compile everything and run
the tests?
Time for a demo

● Packaging
● tests:
○ Using py.test fixtures for fun and profit
○ Continuous integration
● Porting to Python3
Topics for this talk

● Write a README.rst and add in MANIFEST.in so that you
have a beautiful Pypi page
● Please use setuptools.find_packages() instead of hard-
coding your modules
● Let it generate the scripts
entry_points = {
"console_scripts" : [
"qidoc = qisys.main:main",
"qibuild = qisys.main:main"
● Then do magic with sys.argv[0]
Packaging qiBuild

● Used every day by > 50 developers internally
● Also used for continuous integration and release scripts
● Not much room for error or breaking retro-compatibility
Testing qiBuild

● Before:
○ non-PEP8, lots of assertStuff()
○ Had to use inheritance to share setUp() / tearDown()
○ no discovery
○ unittest sucks
● After:
○ PEP8, using plain assert
○ Easy sharing of fixtures
○ discovery and lots of handy plugins
■ xdist, cache, xml, coverage
○ py.test rocks
From unittest to py.test

● Making sure tests run in isolation:
○ Using py.test fixtures
● Writing integration tests:
○ Example: test_qibuild_configure.py
■ Again, everything in a tmpdir
py.test fixtures

● At Aldebaran:
○ Jenkins plugged to gerrit (code review)
○ Each change is automatically checked for pylint errors
and tests
● For external contributors:
○ travis
● Note: using invoke for driving CI (better than tox …)
○ tasks.py for qibuild
Continuous integration

Continuous integration (2)

● 2 day’s work
○ mandatory : have a good coverage
○ run 2to3
○ fix all the tests for Python3
○ fix all the tests for Python2, using six
○ make sure Python3 tests still pass
○ push everything to a branch
● How to make sure the branch stays alive ?
○ jenkins job to automatically rebase and run the tests for
Python2 and Python3
Porting to Python3

Development done on github:
https://github.com/aldebaran/qibuild
Join the discussion !
qibuild-dev@aldebaran.com
#qi on Freenode
We are hiring :)
https://www.aldebaran.com/fr/jobs/nos-offres
Contact me:
dmerejkowsky@aldebaran.com
See the demo again:
https://asciinema.org/a/35360
The discussion continues ...