Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove

7,277 views

Published on

Deployment issues surrounding the use of virtualenv within Debian packages at Spotify

Published in: Technology

Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove

  1. 1. 1 Spotify Tech Web/iOS/Android/Native clients Service oriented architecture > 50 services in Java and Python Primarily using Cassandra for persistent storage 7,000 Servers in production ~500 in Engineers worldwide
  2. 2. 2 What have we been doing? Lost in the forest of dependency building
  3. 3. 3 Deploy via Debian Packages Use Debian Stable repositories Package/Backport needed libraries as needed All Dependencies defined in Debian package Deployed via Puppet/Aptitude
  4. 4. 4 Mirroring   P a c k a g e   A p p s   P y P i  
  5. 5. 5 What’s wrong with that? Spending too much time managing dependencies and not enough building cool shit
  6. 6. 6 Using Debian Stable does not included latest and greatest
  7. 7. 7 Requires “Backporting” for newer libraries
  8. 8. 8 Does not used normal python packaging
  9. 9. 9 Limited use on developer systems (or bad/duplicated information)
  10. 10. 10 Full rollback non-existent
  11. 11. 11 What you test is NOT what you deploy
  12. 12. 12 Dependency lock in due to sharing a common python environment
  13. 13. 13 There MUST be a better way Avoid duplication of effort
  14. 14. 14 There MUST be a better way Increase stability and repeatability
  15. 15. 15 There MUST be a better way Offer easy rollback
  16. 16. 16 There MUST be a better way Scale
  17. 17. 17 What about now? Deploying Python @ Scale
  18. 18. 18 Enter Dh-viurtualenv
  19. 19. 19 Dh == Debian Helper •  •  •  •  •  Registers itself as a Debian Helper Bundles the requirements in requirements.txt into virtual environment Deploys virtualenv into /usr/share/python/<package-name> Still requires native dependencies to be installed via packaging Just include the following in the debian/rules: %:                                dh  $@  -­‐-­‐with  python-­‐virtualenv
  20. 20. 20 Mirroring   V i r t u a l e n v   Pip   A p p s  
  21. 21. 21 IT Works!
  22. 22. 22 Example: Using Sentry Define Dependencies in requirements.txt sentry[postgres]==6.2.0   eventlet==0.13.0   hiredis==0.1.1   django-­‐auth-­‐ldap==1.1.4
  23. 23. 23 Example: Using Sentry Define Debian dependencies in control file Build-Depends: python (>= 2.6.6-3~), debhelper (>= 8), dh-virtualenv, python-dev, libpq-dev, libldap2-dev, libsasl2-dev, Standards-Version: 3.9.3 X-Python-Version: >= 2.6
  24. 24. 24 Example: Using Sentry Use dh_virtualenv in Debian rules file %: dh $@ --with python-virtualenv override_dh_virtualenv: dh_virtualenv --index-url='http://localhost/simple'
  25. 25. 25 Have done this with RPMs as well (@ Amplify) Within RPM build virtaulenv and mark it as relocatable: /usr/local/bin/virtualenv --no-site-packages build_venv source build_venv/bin/activate env CFLAGS="$RPM_OPT_FLAGS" easy_install -Z . deactivate /usr/local/bin/virtualenv --relocatable build_venv
  26. 26. 26 Benefits No longer stuck on Debian stable
  27. 27. 27 Benefits No more backporting of new libraries or versions
  28. 28. 28 Benefits Code isolation/use own dependencies
  29. 29. 29 Benefits Single package rollback option
  30. 30. 30 Benefits Development and Deployed systems using same cross-platform setup
  31. 31. 31 What’s Next?
  32. 32. 32 Not Sure…
  33. 33. 33 Dh_virtualenv now available (ironically) in Debian Unstable repositories
  34. 34. 34 Still requires managing native dependencies locally
  35. 35. 35 But experimenting with containers!
  36. 36. 36 Stay Tuned!
  37. 37. 37 Thanks to Jyrki Pulliainen and Jim Whitehead @ Spotify Stockholm! Whitepaper: http://labs.spotify.com/2013/10/10/packagingin-your-packaging-dh-virtualenv/ Code: https://github.com/spotify/dh-virtualenv
  38. 38. 38 Questions? Chris Angove, Chapter Lead - Backend Email: cangove@spotify.com Blog: http://labs.spotify.com/ Listen: https://play.spotify.com/
  39. 39. Want to join the band? Check out spotify.com/jobs or @Spotifyjobs for more information. January 24, 2014

×