• the most important thing for compiled software
• set up a build server
• with jenkins (or another type of software)
• backup your built objects
• copying them to s3 is not a horrible idea
• regardless of source control system or branching strategy,
ensure you can always rebuild any version of your
• git plugin didn’t work on windows
• branch building is painful, jenkins API can
• getting windows build agent working is
an operation that changes the apparent root directory
for the current running process [...].
A program that is run in such a modified environment
cannot name (and therefore normally not access) files
outside the designated directory tree.
• Create a base image on disk
• Clone base image
• Boot the cloned image
• Do the build and copy built object out.
• Delete cloned image when done
• Base image is still pristine and can be reused.
Create builds in cleanroom
• Avoid contaminating builds with artifacts from previous
• chroots help
• use mock or pbuilder for RPMs and DEBs
• KVM, EC2, or equivalent for everything else
• Always create pristine base images and do builds in a
• Use SSDs
• git-buildpackage can’t set changelog distribution field
• signing key setup is really painful (especially for
• deb naming scheme for packages is quite painful
• all tools are undocumented and very hard to actually
• recent versions of KVM provided by ubuntu fail to
boot VMs sometimes
two types of
• calls to functions in library are resolved at compile
• code from the library is copied into the resulting
• calls to functions are resolved at
• code for a library lives in it’s own
• figure out which libraries your app needs
• pick a supported major release, if possible
• build and package this library
• link it statically against your binary during build
• you now have fewer stones to turn over when
• you will need to track your upstream deps
• you will probably want to package your upstream
• you can then point your chroots and build envs at
private deb, rpm, etc repos in your internal
• merging upstream changes in becomes its own
Use the build system
Determine which file
to build at compile
• break up ifdef soup into separate files
• use the build system to compile the right file at
• this seems obvious but many C libraries and
programs are full of crazy ifdef soup.
• very easy to fall down a rabbit hole breaking things
• can make the build process more complicated and
harder to debug
• DEB and RPM can both output debug packages
that contain debug symbols.
• output these packages.
• store these and make backups.
• (or just don’t strip your binary)
Use googlecoredumper to catch
• you can use google-coredumper
segfaults, bus errors, and other bad things.
• you can output a coredump when this
• you can use this coredump and your debug
symbols to figure out what is going on.
• Have a backup plan
• Capture debug symbols during your automated
• Store them somewhere safe (and make
• Capture coredumps (if possible).
• Use coredumps and debug symbols to figure
out what happened.
• can significantly increase complexity
• google coredumper can’t help if your kernel is
• some linux distributions don’t allow ptrace
• google coredumper only supports linux
• Is the binary actually statically linked?
• Does it get copied to the right path?
• Are the right config files autogenerated?
• Does the version string the program outputs match
the package version?
Testing test every
• It will be impossible to build and
change on every supported platform.
• Use your build server to do this for you.
• Test things like:
• installing/uninstalling the object
• object is actually statically linked
• correctness testing (RSpec can be useful)
testing too much
• Fine line between not enough and
• Windows can be painful, but cygwin can help with
• Easy to forget about testing the actual package
• Can be difficult to get working with branch builds