Your SlideShare is downloading. ×
0
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Deployment Tools and Techniques at Spotify: Virtualenv in debian by Chris Angove

5,649

Published on

Deployment issues surrounding the use of virtualenv within Debian packages at Spotify

Deployment issues surrounding the use of virtualenv within Debian packages at Spotify

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
5,649
On Slideshare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
4
Comments
0
Likes
5
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. 1 Spotify Tech Web/iOS/Android/Native clients Service oriented architecture > 50 services in Java and Python Primarily using Cassandra for persistent storage 7,000 Servers in production ~500 in Engineers worldwide
  • 2. 2 What have we been doing? Lost in the forest of dependency building
  • 3. 3 Deploy via Debian Packages Use Debian Stable repositories Package/Backport needed libraries as needed All Dependencies defined in Debian package Deployed via Puppet/Aptitude
  • 4. 4 Mirroring   P a c k a g e   A p p s   P y P i  
  • 5. 5 What’s wrong with that? Spending too much time managing dependencies and not enough building cool shit
  • 6. 6 Using Debian Stable does not included latest and greatest
  • 7. 7 Requires “Backporting” for newer libraries
  • 8. 8 Does not used normal python packaging
  • 9. 9 Limited use on developer systems (or bad/duplicated information)
  • 10. 10 Full rollback non-existent
  • 11. 11 What you test is NOT what you deploy
  • 12. 12 Dependency lock in due to sharing a common python environment
  • 13. 13 There MUST be a better way Avoid duplication of effort
  • 14. 14 There MUST be a better way Increase stability and repeatability
  • 15. 15 There MUST be a better way Offer easy rollback
  • 16. 16 There MUST be a better way Scale
  • 17. 17 What about now? Deploying Python @ Scale
  • 18. 18 Enter Dh-viurtualenv
  • 19. 19 Dh == Debian Helper •  •  •  •  •  Registers itself as a Debian Helper Bundles the requirements in requirements.txt into virtual environment Deploys virtualenv into /usr/share/python/<package-name> Still requires native dependencies to be installed via packaging Just include the following in the debian/rules: %:                                dh  $@  -­‐-­‐with  python-­‐virtualenv
  • 20. 20 Mirroring   V i r t u a l e n v   Pip   A p p s  
  • 21. 21 IT Works!
  • 22. 22 Example: Using Sentry Define Dependencies in requirements.txt sentry[postgres]==6.2.0   eventlet==0.13.0   hiredis==0.1.1   django-­‐auth-­‐ldap==1.1.4
  • 23. 23 Example: Using Sentry Define Debian dependencies in control file Build-Depends: python (>= 2.6.6-3~), debhelper (>= 8), dh-virtualenv, python-dev, libpq-dev, libldap2-dev, libsasl2-dev, Standards-Version: 3.9.3 X-Python-Version: >= 2.6
  • 24. 24 Example: Using Sentry Use dh_virtualenv in Debian rules file %: dh $@ --with python-virtualenv override_dh_virtualenv: dh_virtualenv --index-url='http://localhost/simple'
  • 25. 25 Have done this with RPMs as well (@ Amplify) Within RPM build virtaulenv and mark it as relocatable: /usr/local/bin/virtualenv --no-site-packages build_venv source build_venv/bin/activate env CFLAGS="$RPM_OPT_FLAGS" easy_install -Z . deactivate /usr/local/bin/virtualenv --relocatable build_venv
  • 26. 26 Benefits No longer stuck on Debian stable
  • 27. 27 Benefits No more backporting of new libraries or versions
  • 28. 28 Benefits Code isolation/use own dependencies
  • 29. 29 Benefits Single package rollback option
  • 30. 30 Benefits Development and Deployed systems using same cross-platform setup
  • 31. 31 What’s Next?
  • 32. 32 Not Sure…
  • 33. 33 Dh_virtualenv now available (ironically) in Debian Unstable repositories
  • 34. 34 Still requires managing native dependencies locally
  • 35. 35 But experimenting with containers!
  • 36. 36 Stay Tuned!
  • 37. 37 Thanks to Jyrki Pulliainen and Jim Whitehead @ Spotify Stockholm! Whitepaper: http://labs.spotify.com/2013/10/10/packagingin-your-packaging-dh-virtualenv/ Code: https://github.com/spotify/dh-virtualenv
  • 38. 38 Questions? Chris Angove, Chapter Lead - Backend Email: cangove@spotify.com Blog: http://labs.spotify.com/ Listen: https://play.spotify.com/
  • 39. Want to join the band? Check out spotify.com/jobs or @Spotifyjobs for more information. January 24, 2014

×