Continuous Deployment at Disqus (Pylons Minicon)

DISQUS Continuous Deployment Everything David Cramer @zeegFriday, April 29, 2011

Disclaimer: We don’t use PylonsFriday, April 29, 2011

Continuous Deployment Shipping new code as soon as it’s ready (It’s really just super awesome buildbots)Friday, April 29, 2011

Workﬂow 1. Developer commits code 2. CI server runs tests automatically 2.1.Build passes, code deploys 2.2.Build fails, block deploy 3. Developer tests feature on productionFriday, April 29, 2011

Pros Cons • Develop features • Culture Shock incrementally • Stability depends on • Release frequently test coverage • Less QA! (maybe) • Initial time investment We mostly just care about iteration and stabilityFriday, April 29, 2011

Painless DevelopmentFriday, April 29, 2011

Development • Production > Staging > CI > Dev • Automate testing of complicated processes and architecture • Simple is arguably better than complete • Especially for local development • python setup.py {develop,test} • Use Puppet/Chef or build a simple bootstrap.{py,sh}Friday, April 29, 2011

Production Staging • PostgreSQL • PostgreSQL • Memcache • Memcache • Redis • Redis • Solr • Solr • Apache • Apache • Nginx • Nginx • RabbitMQ • RabbitMQ CI Server Macbook • Memcache • PostgreSQL • PostgreSQL • Apache • Redis • Memcache • Solr • Redis • Apache • Solr • Nginx • Nginx • RabbitMQ • RabbitMQFriday, April 29, 2011

Bootstrapping Local • Simplify local setup • git clone dcramer@disqus:disqus.git • python setup.py develop • python manage.py runserver • Need to test dependancies? • virtualbox + vagrant upFriday, April 29, 2011

“Under Construction” • Iterate quickly by hiding features • Early adopters are free QA from gargoyle import gargoyle def my_view(request): if gargoyle.is_active(awesome, request): return new happy version :D else: return old sad version :(Friday, April 29, 2011

Gargoyle Deploy features to portions of a user base at a time to ensure smooth, measurable releases Being users of our product, we actively use early versions of features before public releaseFriday, April 29, 2011

Integration (or as we like to call it)Friday, April 29, 2011

Integration is Required Deploy only when things wont breakFriday, April 29, 2011

Reporting is CriticalFriday, April 29, 2011

CI Requirements ✓ Developers must know when they’ve broken something ✓ IRC, Email, IM ✓ Support proper reporting ✓ XUnit, Pylint, Coverage.py ✓ Painless setup ✓ apt-get install jenkinsFriday, April 29, 2011

Shortcomings • False positives lower awareness • Reporting isnt accurate • Services fail • Bad Tests • Not enough code coverage • Regressions on untested code • Test suite takes too long • Integration tests vs Unit testsFriday, April 29, 2011

Fixing False Positives • Re-run tests several times on a failure • Report continually failing tests • Fix continually failing tests • Rely less on 3rd parties • Mock/DingusFriday, April 29, 2011

Maintaining Coverage • Raise awareness with reporting • Fail/alert when coverage drops on a build • Commit tests with code • Drive it into your cultureFriday, April 29, 2011

Speeding Up Tests • Write true unit tests • vs slower integration tests • Mock 3rd party APIs • Distributed and parallel testing • http://github.com/disqus/muleFriday, April 29, 2011

Mule • Unstable (we’re working on it) • Mostly Django right now • Generic interfaces for unittest2 • Works with multi-processing and Celery • More complex than normal Celery usage • Full XUnit integration • Simple workﬂow • mule test --runner="python manage.py mule --worker $TEST"Friday, April 29, 2011

Deploy (ﬁnally)Friday, April 29, 2011

How DISQUS Does It • Incremental deploy with Fabric • Drop server from pool • Pull in requirements on each server • Isolated virtualenv’s built on each server • Push server back onlineFriday, April 29, 2011

Challenges • PyPi works on server A, but not B • Scale, or lack of • CPU cost per server • Schema changes, data model changes • Backwards compatibilityFriday, April 29, 2011

PyPi is Down • http://github.com/disqus/chishopFriday, April 29, 2011

Help, we have 100 servers! • Incremental (ours) vs Fanout • Push vs Pull • Twitter uses BitTorrent • Isolation vs Packaging (Complexity)Friday, April 29, 2011

SQL Schema Changes 1. Add column (NULLable) 2. Add app code to ﬁll column 3. Deploy 4. Backﬁll column 5. Add app code to read column 6. Deploy Cached Data Changes • Have a global version number • Have a data model cache version • maybe md5(cls.__dict__)?Friday, April 29, 2011

DISQUS Questions? psst, we’re hiring jobs@disqus.comFriday, April 29, 2011

References • Mule (distributed test runner) http://github.com/disqus/mule • Gargoyle (feature switches) https://github.com/disqus/gargoyle • Jenkins CI http://jenkins-ci.org/ code.disqus.comFriday, April 29, 2011

http://join5works.com	175
http://lanyrd.com	26
http://www.python.rk.edu.pl	18
http://quantagente.com.br	9
http://5works.co	6
http://www.utpl.edu.ec	4
http://pesquisacommetodo.blogspot.com	3
http://www.slideshare.net	2
http://www.mongodb.org	2
http://www.istikbal.gr	2
http://brandsuperhero.com	2
http://www.petrobras.com.br	2
http://ganocafeobrhayan.blogspot.com	1
http://www.emark.teicrete.gr	1
http://duckduckgo.com	1
http://www.geekagenda.it	1
http://python.rk.edu.pl	1
http://www.ronaldvandenhoff.nl	1
http://localhost	1
http://amoursaint.blogspot.com	1
http://m10.indicthreads.com	1
http://www.2g-cenergy.com	1
http://www.phila.gov	1
http://piscolabis.2creativo.net	1
http://www.onlydoo.com	1
http://pmsapp.mercadolibre.com.mx	1
http://www.etourisme-ardesi.fr	1
http://elhumordellicenciado-tomo1.blogspot.com	1
http://luzverdadera.blogspot.com	1

Continuous Deployment at Disqus (Pylons Minicon)

by zeeg on Apr 29, 2011

Accessibility

Categories

Upload Details

Usage Rights

29 Embeds 268

Statistics

Continuous Deployment at Disqus (Pylons Minicon) Presentation Transcript