Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Joe Damato


Published on

HighLoad++ 2014

Published in: Technology
  • Be the first to comment

Joe Damato

  1. 1. scaling compiled applications Joe Damato @joedamato
  2. 2. about me • Joe Damato • San Francisco, CA • i like: • systems programming • debugging • performance analysis
  3. 3. @joedamato
  4. 4.
  5. 5. sassanosystems.c om
  6. 6.
  7. 7. Repeatable builds • the most important thing for compiled software • set up a build server • with jenkins (or another type of software) • backup your built objects • copying them to s3 is not a horrible idea • regardless of source control system or branching strategy, ensure you can always rebuild any version of your software
  8. 8. Jenkins problems (maybe • git plugin didn’t work on windows fixed?) • branch building is painful, jenkins API can help • getting windows build agent working is painful
  9. 9. почему? why?
  10. 10. tools for... • RPMs • DEBs • Everything else • windows installers (MSIs) • other linux/unix like OS’s • etc
  11. 11. chroot ??
  12. 12. chroot: • • • an operation that changes the apparent root directory for the current running process [...]. A program that is run in such a modified environment cannot name (and therefore normally not access) files outside the designated directory tree. (from wikipedia)
  13. 13. RPM mock
  14. 14. DEB pbuilder
  15. 15. Everything else • KVM • Amazon EC2 • other virtualization
  16. 16. KVM EC2
  17. 17. KVM • Create a base image on disk • Clone base image • Boot the cloned image • Do the build and copy built object out. • Delete cloned image when done • Base image is still pristine and can be reused.
  18. 18. Create builds in cleanroom • Avoid contaminating builds with artifacts from previous builds. • chroots help • use mock or pbuilder for RPMs and DEBs • KVM, EC2, or equivalent for everything else • Always create pristine base images and do builds in a copy. • Use SSDs
  19. 19. Tool problems • git-buildpackage can’t set changelog distribution field • signing key setup is really painful (especially for RPMs) • deb naming scheme for packages is quite painful • all tools are undocumented and very hard to actually use • recent versions of KVM provided by ubuntu fail to boot VMs sometimes
  20. 20. two types of linking.... • • dynamic linking static linking
  21. 21. static linking • calls to functions in library are resolved at compile time. • code from the library is copied into the resulting binary.
  22. 22. dynamic linking • calls to functions are resolved at runtime. • code for a library lives in it’s own object: • • libblah.dll
  23. 23.
  24. 24. почему? why?
  25. 25. static linking • figure out which libraries your app needs • pick a supported major release, if possible • build and package this library • link it statically against your binary during build • you now have fewer stones to turn over when debugging
  26. 26. static linking • you will need to track your upstream deps • you will probably want to package your upstream deps • you can then point your chroots and build envs at private deb, rpm, etc repos in your internal infrastructure • merging upstream changes in becomes its own project
  27. 27. почему? why?
  28. 28. Use files • src/ •redhat5/ •some_internal_thing.c •ubuntu1204/ •some_internal_thing.c •debian6/ •some_internal_thing.c
  29. 29. Use the build system Determine which file to build at compile time.
  30. 30. Use modules • break up ifdef soup into separate files • use the build system to compile the right file at build time • this seems obvious but many C libraries and programs are full of crazy ifdef soup.
  31. 31. Use modules • very easy to fall down a rabbit hole breaking things apart • can make the build process more complicated and harder to debug
  32. 32. почему? why?
  33. 33. Capture debug symbols • DEB and RPM can both output debug packages that contain debug symbols. • output these packages. • store these and make backups. • (or just don’t strip your binary)
  34. 34. Use googlecoredumper to catch • you can use google-coredumper segfaults, bus errors, and other bad things. • you can output a coredump when this happens. • you can use this coredump and your debug symbols to figure out what is going on.
  35. 35. Plan for failure • Have a backup plan • Capture debug symbols during your automated build process. • Store them somewhere safe (and make backups). • Capture coredumps (if possible). • Use coredumps and debug symbols to figure out what happened.
  36. 36. Plan for failure • can significantly increase complexity • google coredumper can’t help if your kernel is buggy • some linux distributions don’t allow ptrace • google coredumper only supports linux
  37. 37.
  38. 38. Check things like... • Is the binary actually statically linked? • Does it get copied to the right path? • Are the right config files autogenerated? • Does the version string the program outputs match the package version? • ....
  39. 39. You also need... correctness testing.
  40. 40. RSpec is useful for this.
  41. 41. Automated Testing test every • It will be impossible to build and change on every supported platform. • Use your build server to do this for you. • Test things like: • installing/uninstalling the object • object is actually statically linked • correctness testing (RSpec can be useful)
  42. 42. Automated testing too much • Fine line between not enough and • Windows can be painful, but cygwin can help with scripting • Easy to forget about testing the actual package install/removal • Can be difficult to get working with branch builds
  43. 43. To summarize...
  44. 44. sassanosystems.c om @joedamato
  45. 45. спасиб о