SlideShare a Scribd company logo
DISQUS
                         Practicing Continuous Deployment



                                    David Cramer
                                       @zeeg




Saturday, March 10, 12
Shipping new code as soon
                 as it’s ready




Saturday, March 10, 12
Continuous Deployment




                         # Update the site every 5 minutes
                         */5 * * * * cd /www/example.com 
                                       && git pull 
                                       && service apache restart




Saturday, March 10, 12
Saturday, March 10, 12
When it’s ready




Saturday, March 10, 12
When is it ready?




                -        Reviewed by peers


                -        Passes automated tests


                -        Some level of QA




Saturday, March 10, 12
Focus on Stability and Iteration




Saturday, March 10, 12
Workflow


                          Review                  Commit




                         Integration             Failed Build




                          Deploy                 Reporting




                                                  Rollback




Saturday, March 10, 12
The Good
      -     Develop features
            incrementally
      -     Release frequently
      -     Smaller doses of QA
                                      The Bad
                                  -   Culture Shock
                                  -   Stability depends on
                                      test coverage
                                  -   Initial time investment

Saturday, March 10, 12
Keep Development Simple




Saturday, March 10, 12
Development




                -        Automate testing of complicated
                         processes and architecture
                -        Simple can be better than complete
                         -   Especially for local development
                -        python setup.py {develop,test}
                -        Puppet, Chef, Buildout, Fabric, etc.




Saturday, March 10, 12
Production                                 Staging
                                PostgreSQL                              PostgreSQL
                                Memcache                                Memcache
                                Redis                                   Redis
                                Solr                                    Solr
                                Apache                                  Apache
                                Nginx                                   Nginx
                                RabbitMQ                                RabbitMQ
                         (and 100 other painful-to-configure services)



                                CI Server                               Macbook
                                PostgreSQL                              PostgreSQL
                                Memcache                                Apache
                                Redis                                   Memcache
                                Solr                                    Redis
                                Apache                                  Solr
                                Nginx                                   Nginx
                                RabbitMQ                                RabbitMQ


Saturday, March 10, 12
Bootstrapping Local



                -        Simplify local setup
                         -   git clone dcramer@disqus:disqus.git
                         -   make
                         -   python manage.py runserver


                -        Need to test dependancies?
                         -   virtualbox + vagrant up



Saturday, March 10, 12
Progressive Rollout

                         We actively use early versions of features
                                   before public release




Saturday, March 10, 12
Deploy features to portions of a user base at a
                          time to ensure smooth, measurable releases




                                  https://github.com/disqus/gargoyle


Saturday, March 10, 12
•        Iterate quickly by hiding features
                •        Early adopters are free QA



                         from gargoyle import gargoyle

                         def my_view(request):
                             if gargoyle.is_active('awesome', request):
                                 return 'new happy version :D'
                             else:
                                 return 'old sad version :('




Saturday, March 10, 12
SWITCHES = {
                             # enable my_feature for 50%
                             'my_feature': range(0, 50),
                         }

                         def is_active(switch):
                             try:
                                  pct_range = SWITCHES[switch]
                             except KeyError:
                                  return False

                             ip_hash = sum([int(x) for x
                                            in ip_address.split('.')])

                             return (ip_hash % 100 in pct_range)




Saturday, March 10, 12
Review ALL the Commits




                            phabricator.org


Saturday, March 10, 12
Saturday, March 10, 12
Saturday, March 10, 12
Saturday, March 10, 12
Integration
                         (or as we like to call it)




Saturday, March 10, 12
Saturday, March 10, 12
Integration Requirements




                -        Developers must know when they’ve
                         broken something
                         -   IRC, Email, IM
                -        Support proper reporting
                         -   XUnit, Pylint, Coverage.py
                -        Painless setup
                         -   apt-get install jenkins *

                             https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins+on+Ubuntu


Saturday, March 10, 12
Shortcomings


                -        False positives
                         -   Reporting isn't accurate
                         -   Services fail
                         -   Bad Tests
                -        Test coverage
                         -   Regressions on untested code
                -        Feedback delay
                         -   Integration tests vs Unit tests


Saturday, March 10, 12
Fixing False Positives




                -        Re-run tests several times on a failure


                -        Report continually failing tests


                -        Replace external service tests with a
                         functional test suite




Saturday, March 10, 12
Maintaining Coverage




                -        Raise awareness with reporting
                         -   Fail/alert when coverage drops on a build
                -        Commit tests with code
                         -   Coverage against commit diff for
                             untested regressions
                -        Utilize code review




Saturday, March 10, 12
Speeding Up Tests




                -        Write true unit tests
                         -   vs slower integration tests
                -        Mock external services
                -        Distributed and parallel testing
                         -   Matrix builds




Saturday, March 10, 12
Reporting




Saturday, March 10, 12
<You> Why is mongodb-1 down?




             <Ops> It’s down? Must have crashed again

Saturday, March 10, 12
Meaningful Metrics




                -        Rate of traffic (not just hits!)
                         -   Business vs system
                -        Response time (database, web)
                -        Exceptions
                -        Social media
                         -   Twitter




Saturday, March 10, 12
Graphite




                         (Trafficgraphite.wikidot.com
                                  across a cluster of servers)


Saturday, March 10, 12
Sentry




                         sentry.readthedocs.org


Saturday, March 10, 12
Wrap Up




Saturday, March 10, 12
Getting Started




                -        Package your app
                -        Value code review
                -        Ease deployment; fast rollbacks
                -        Setup automated tests
                -        Gather some easy metrics




Saturday, March 10, 12
Going Further




                -        Build an immune system
                         -   Automate deploys, rollbacks (maybe)
                -        Adjust to your culture
                         -   There is no “right way”
                -        SOA == great success




Saturday, March 10, 12
DISQUS
                          Questions?




                          psst, we’re hiring
                          disqus.com/jobs

Saturday, March 10, 12
References



                -        Gargoyle (feature switches)
                         https://github.com/disqus/gargoyle

                -        Sentry (log aggregation)
                         https://github.com/dcramer/sentry

                -        Jenkins CI (continuous integration)
                         http://jenkins-ci.org/

                -        Phabricator (code reviews, bug tracking)
                         https://phabricator.org

                -        Graphite (metrics)
                         http://graphite.wikidot.com/




                                              code.disqus.com
Saturday, March 10, 12

More Related Content

What's hot

Groovy concurrency
Groovy concurrencyGroovy concurrency
Groovy concurrency
Alex Miller
 
Memcached Study
Memcached StudyMemcached Study
Memcached Study
nam kwangjin
 
Introduction to .Net Driver
Introduction to .Net DriverIntroduction to .Net Driver
Introduction to .Net Driver
DataStax Academy
 
Introduction to memcached
Introduction to memcachedIntroduction to memcached
Introduction to memcached
Jurriaan Persyn
 
Async and Non-blocking IO w/ JRuby
Async and Non-blocking IO w/ JRubyAsync and Non-blocking IO w/ JRuby
Async and Non-blocking IO w/ JRuby
Joe Kutner
 
Caching In The Cloud
Caching In The CloudCaching In The Cloud
Caching In The Cloud
Alex Miller
 
Distributed system coordination by zookeeper and introduction to kazoo python...
Distributed system coordination by zookeeper and introduction to kazoo python...Distributed system coordination by zookeeper and introduction to kazoo python...
Distributed system coordination by zookeeper and introduction to kazoo python...
Jimmy Lai
 
Spark / Mesos Cluster Optimization
Spark / Mesos Cluster OptimizationSpark / Mesos Cluster Optimization
Spark / Mesos Cluster Optimization
ebiznext
 
Hazelcast
HazelcastHazelcast
Hazelcast
oztalip
 
[245] presto 내부구조 파헤치기
[245] presto 내부구조 파헤치기[245] presto 내부구조 파헤치기
[245] presto 내부구조 파헤치기
NAVER D2
 
Above the clouds: introducing Akka
Above the clouds: introducing AkkaAbove the clouds: introducing Akka
Above the clouds: introducing Akka
nartamonov
 
JahiaOne - Performance Tuning
JahiaOne - Performance TuningJahiaOne - Performance Tuning
JahiaOne - Performance Tuning
Jahia Solutions Group
 
캐시 분산처리 인프라
캐시 분산처리 인프라캐시 분산처리 인프라
캐시 분산처리 인프라
Park Chunduck
 
Hazelcast and MongoDB at Cloud CMS
Hazelcast and MongoDB at Cloud CMSHazelcast and MongoDB at Cloud CMS
Hazelcast and MongoDB at Cloud CMS
uzquiano
 
Managing Postgres with Ansible
Managing Postgres with AnsibleManaging Postgres with Ansible
Managing Postgres with Ansible
Gulcin Yildirim Jelinek
 
Caching reboot: javax.cache & Ehcache 3
Caching reboot: javax.cache & Ehcache 3Caching reboot: javax.cache & Ehcache 3
Caching reboot: javax.cache & Ehcache 3
Louis Jacomet
 
Scaling symfony apps
Scaling symfony appsScaling symfony apps
Scaling symfony apps
Matteo Moretti
 
Ehcache 3 @ BruJUG
Ehcache 3 @ BruJUGEhcache 3 @ BruJUG
Ehcache 3 @ BruJUG
Louis Jacomet
 
Winter is coming? Not if ZooKeeper is there!
Winter is coming? Not if ZooKeeper is there!Winter is coming? Not if ZooKeeper is there!
Winter is coming? Not if ZooKeeper is there!
Joydeep Banik Roy
 
Hazelcast Essentials
Hazelcast EssentialsHazelcast Essentials
Hazelcast Essentials
Rahul Gupta
 

What's hot (20)

Groovy concurrency
Groovy concurrencyGroovy concurrency
Groovy concurrency
 
Memcached Study
Memcached StudyMemcached Study
Memcached Study
 
Introduction to .Net Driver
Introduction to .Net DriverIntroduction to .Net Driver
Introduction to .Net Driver
 
Introduction to memcached
Introduction to memcachedIntroduction to memcached
Introduction to memcached
 
Async and Non-blocking IO w/ JRuby
Async and Non-blocking IO w/ JRubyAsync and Non-blocking IO w/ JRuby
Async and Non-blocking IO w/ JRuby
 
Caching In The Cloud
Caching In The CloudCaching In The Cloud
Caching In The Cloud
 
Distributed system coordination by zookeeper and introduction to kazoo python...
Distributed system coordination by zookeeper and introduction to kazoo python...Distributed system coordination by zookeeper and introduction to kazoo python...
Distributed system coordination by zookeeper and introduction to kazoo python...
 
Spark / Mesos Cluster Optimization
Spark / Mesos Cluster OptimizationSpark / Mesos Cluster Optimization
Spark / Mesos Cluster Optimization
 
Hazelcast
HazelcastHazelcast
Hazelcast
 
[245] presto 내부구조 파헤치기
[245] presto 내부구조 파헤치기[245] presto 내부구조 파헤치기
[245] presto 내부구조 파헤치기
 
Above the clouds: introducing Akka
Above the clouds: introducing AkkaAbove the clouds: introducing Akka
Above the clouds: introducing Akka
 
JahiaOne - Performance Tuning
JahiaOne - Performance TuningJahiaOne - Performance Tuning
JahiaOne - Performance Tuning
 
캐시 분산처리 인프라
캐시 분산처리 인프라캐시 분산처리 인프라
캐시 분산처리 인프라
 
Hazelcast and MongoDB at Cloud CMS
Hazelcast and MongoDB at Cloud CMSHazelcast and MongoDB at Cloud CMS
Hazelcast and MongoDB at Cloud CMS
 
Managing Postgres with Ansible
Managing Postgres with AnsibleManaging Postgres with Ansible
Managing Postgres with Ansible
 
Caching reboot: javax.cache & Ehcache 3
Caching reboot: javax.cache & Ehcache 3Caching reboot: javax.cache & Ehcache 3
Caching reboot: javax.cache & Ehcache 3
 
Scaling symfony apps
Scaling symfony appsScaling symfony apps
Scaling symfony apps
 
Ehcache 3 @ BruJUG
Ehcache 3 @ BruJUGEhcache 3 @ BruJUG
Ehcache 3 @ BruJUG
 
Winter is coming? Not if ZooKeeper is there!
Winter is coming? Not if ZooKeeper is there!Winter is coming? Not if ZooKeeper is there!
Winter is coming? Not if ZooKeeper is there!
 
Hazelcast Essentials
Hazelcast EssentialsHazelcast Essentials
Hazelcast Essentials
 

Viewers also liked

Seguridad de la Informacion
Seguridad de la InformacionSeguridad de la Informacion
Seguridad de la Informacion
NFAG DTLF
 
18 TIPS TO-BE FOUNDERS
18 TIPS TO-BE FOUNDERS18 TIPS TO-BE FOUNDERS
18 TIPS TO-BE FOUNDERSAndre Marquet
 
אילוצים וריסונים במלחמה מקומית - מלחמת לבנון השניה
אילוצים וריסונים במלחמה מקומית - מלחמת לבנון השניהאילוצים וריסונים במלחמה מקומית - מלחמת לבנון השניה
אילוצים וריסונים במלחמה מקומית - מלחמת לבנון השניהIshai Aloni
 
شهادة الماجستر
شهادة الماجسترشهادة الماجستر
شهادة الماجسترSaleem Abudayeh
 
Through the Lens of an iPhone: Charleston, SC
Through the Lens of an iPhone: Charleston, SCThrough the Lens of an iPhone: Charleston, SC
Through the Lens of an iPhone: Charleston, SC
Paul Brown
 
6 Healthy Mother's Day Gifts
6 Healthy Mother's Day Gifts6 Healthy Mother's Day Gifts
6 Healthy Mother's Day Gifts
Eason Chan
 
Mas loco que una cabra
Mas loco que una cabraMas loco que una cabra
Mas loco que una cabra
Emilio Gil (unjubilado)
 
Fitriana bakar tugas 1 so ii
Fitriana bakar tugas 1 so iiFitriana bakar tugas 1 so ii
Fitriana bakar tugas 1 so ii
fhitCharis
 
Mobile application development |#Mobileapplicationdevelopment
Mobile application development |#MobileapplicationdevelopmentMobile application development |#Mobileapplicationdevelopment
Mobile application development |#Mobileapplicationdevelopment
Mobile App Developers India
 
หน่วยที่ 9 ระบบเครือข่ายอินเทอร์เน็ต
หน่วยที่ 9 ระบบเครือข่ายอินเทอร์เน็ตหน่วยที่ 9 ระบบเครือข่ายอินเทอร์เน็ต
หน่วยที่ 9 ระบบเครือข่ายอินเทอร์เน็ต
Ta Khanittha
 
How to Find a Good Work-Life Balance
How to Find a Good Work-Life Balance How to Find a Good Work-Life Balance
How to Find a Good Work-Life Balance
Wording Well
 
Local Government Balances Security, Flexibility and Productivity with BlackBe...
Local Government Balances Security, Flexibility and Productivity with BlackBe...Local Government Balances Security, Flexibility and Productivity with BlackBe...
Local Government Balances Security, Flexibility and Productivity with BlackBe...
BlackBerry
 
Perunapuu esittely 4
Perunapuu esittely 4Perunapuu esittely 4
Perunapuu esittely 4
Miikka Leinonen
 
Towards a Grand Unified Theory of Systems Engineering (GUTSE)
Towards a Grand Unified Theory of Systems Engineering (GUTSE)Towards a Grand Unified Theory of Systems Engineering (GUTSE)
Towards a Grand Unified Theory of Systems Engineering (GUTSE)
Joseph KAsser
 
Social Media & Networking - The Evolving Workforce
Social Media & Networking - The Evolving WorkforceSocial Media & Networking - The Evolving Workforce
Social Media & Networking - The Evolving WorkforceTodd Wheatland
 

Viewers also liked (19)

SFR Certification
SFR CertificationSFR Certification
SFR Certification
 
Seguridad de la Informacion
Seguridad de la InformacionSeguridad de la Informacion
Seguridad de la Informacion
 
18 TIPS TO-BE FOUNDERS
18 TIPS TO-BE FOUNDERS18 TIPS TO-BE FOUNDERS
18 TIPS TO-BE FOUNDERS
 
אילוצים וריסונים במלחמה מקומית - מלחמת לבנון השניה
אילוצים וריסונים במלחמה מקומית - מלחמת לבנון השניהאילוצים וריסונים במלחמה מקומית - מלחמת לבנון השניה
אילוצים וריסונים במלחמה מקומית - מלחמת לבנון השניה
 
شهادة الماجستر
شهادة الماجسترشهادة الماجستر
شهادة الماجستر
 
Things I Carry
Things I CarryThings I Carry
Things I Carry
 
Frraactioon
FrraactioonFrraactioon
Frraactioon
 
Research Design
Research DesignResearch Design
Research Design
 
Through the Lens of an iPhone: Charleston, SC
Through the Lens of an iPhone: Charleston, SCThrough the Lens of an iPhone: Charleston, SC
Through the Lens of an iPhone: Charleston, SC
 
6 Healthy Mother's Day Gifts
6 Healthy Mother's Day Gifts6 Healthy Mother's Day Gifts
6 Healthy Mother's Day Gifts
 
Mas loco que una cabra
Mas loco que una cabraMas loco que una cabra
Mas loco que una cabra
 
Fitriana bakar tugas 1 so ii
Fitriana bakar tugas 1 so iiFitriana bakar tugas 1 so ii
Fitriana bakar tugas 1 so ii
 
Mobile application development |#Mobileapplicationdevelopment
Mobile application development |#MobileapplicationdevelopmentMobile application development |#Mobileapplicationdevelopment
Mobile application development |#Mobileapplicationdevelopment
 
หน่วยที่ 9 ระบบเครือข่ายอินเทอร์เน็ต
หน่วยที่ 9 ระบบเครือข่ายอินเทอร์เน็ตหน่วยที่ 9 ระบบเครือข่ายอินเทอร์เน็ต
หน่วยที่ 9 ระบบเครือข่ายอินเทอร์เน็ต
 
How to Find a Good Work-Life Balance
How to Find a Good Work-Life Balance How to Find a Good Work-Life Balance
How to Find a Good Work-Life Balance
 
Local Government Balances Security, Flexibility and Productivity with BlackBe...
Local Government Balances Security, Flexibility and Productivity with BlackBe...Local Government Balances Security, Flexibility and Productivity with BlackBe...
Local Government Balances Security, Flexibility and Productivity with BlackBe...
 
Perunapuu esittely 4
Perunapuu esittely 4Perunapuu esittely 4
Perunapuu esittely 4
 
Towards a Grand Unified Theory of Systems Engineering (GUTSE)
Towards a Grand Unified Theory of Systems Engineering (GUTSE)Towards a Grand Unified Theory of Systems Engineering (GUTSE)
Towards a Grand Unified Theory of Systems Engineering (GUTSE)
 
Social Media & Networking - The Evolving Workforce
Social Media & Networking - The Evolving WorkforceSocial Media & Networking - The Evolving Workforce
Social Media & Networking - The Evolving Workforce
 

Similar to Practicing Continuous Deployment

Inside the Atlassian OnDemand Private Cloud
Inside the Atlassian OnDemand Private CloudInside the Atlassian OnDemand Private Cloud
Inside the Atlassian OnDemand Private Cloud
Atlassian
 
Improving Your Heroku App Performance with Asset CDN and Unicorn
Improving Your Heroku App Performance with Asset CDN and UnicornImproving Your Heroku App Performance with Asset CDN and Unicorn
Improving Your Heroku App Performance with Asset CDN and UnicornSimon Bagreev
 
Dev Ops 101
Dev Ops 101Dev Ops 101
Dev Ops 101
Thoughtworks
 
Rabbitmq Boot System
Rabbitmq Boot SystemRabbitmq Boot System
Rabbitmq Boot SystemAlvaro Videla
 
Cloud meets Fog & Puppet A Story of Version Controlled Infrastructure
Cloud meets Fog & Puppet A Story of Version Controlled InfrastructureCloud meets Fog & Puppet A Story of Version Controlled Infrastructure
Cloud meets Fog & Puppet A Story of Version Controlled Infrastructure
Habeeb Rahman
 
Continuous Integration Testing in Django
Continuous Integration Testing in DjangoContinuous Integration Testing in Django
Continuous Integration Testing in Django
Kevin Harvey
 
Building and running cloud native cassandra
Building and running cloud native cassandraBuilding and running cloud native cassandra
Building and running cloud native cassandra
Vinay Kumar Chella
 
Mcollective introduction
Mcollective introductionMcollective introduction
Mcollective introduction
Javier Turégano Molina
 
Testing Cassandra Guarantees under Diverse Failure Modes with Jepsen
Testing Cassandra Guarantees under Diverse Failure Modes with JepsenTesting Cassandra Guarantees under Diverse Failure Modes with Jepsen
Testing Cassandra Guarantees under Diverse Failure Modes with Jepsen
jkni
 
DataStax: Testing Cassandra Guarantees Under Diverse Failure Modes With Jepsen
DataStax: Testing Cassandra Guarantees Under Diverse Failure Modes With JepsenDataStax: Testing Cassandra Guarantees Under Diverse Failure Modes With Jepsen
DataStax: Testing Cassandra Guarantees Under Diverse Failure Modes With Jepsen
DataStax Academy
 
Corwin on Containers
Corwin on ContainersCorwin on Containers
Corwin on Containers
Corwin Brown
 
Configuration Management and Salt
Configuration Management and SaltConfiguration Management and Salt
Configuration Management and Salt
55020
 
Antons Kranga Building Agile Infrastructures
Antons Kranga   Building Agile InfrastructuresAntons Kranga   Building Agile Infrastructures
Antons Kranga Building Agile Infrastructures
Antons Kranga
 
Islands: Puppet at Bulletproof Networks
Islands: Puppet at Bulletproof NetworksIslands: Puppet at Bulletproof Networks
Islands: Puppet at Bulletproof Networks
Lindsay Holmwood
 
Eclipse con 2012 - Frictionless operations with Puppet - Luke Kanies
Eclipse con 2012 - Frictionless operations with Puppet - Luke KaniesEclipse con 2012 - Frictionless operations with Puppet - Luke Kanies
Eclipse con 2012 - Frictionless operations with Puppet - Luke KaniesPuppet
 
Writing & Sharing Great Modules on the Puppet Forge
Writing & Sharing Great Modules on the Puppet ForgeWriting & Sharing Great Modules on the Puppet Forge
Writing & Sharing Great Modules on the Puppet Forge
Puppet
 
Puppet Conf 2012 - Managing Network Devices with Puppet
Puppet Conf 2012 - Managing Network Devices with PuppetPuppet Conf 2012 - Managing Network Devices with Puppet
Puppet Conf 2012 - Managing Network Devices with PuppetNan Liu
 
Scalability at GROU.PS
Scalability at GROU.PSScalability at GROU.PS
Scalability at GROU.PS
esokullu
 
DrupalCampLA 2011: Drupal backend-performance
DrupalCampLA 2011: Drupal backend-performanceDrupalCampLA 2011: Drupal backend-performance
DrupalCampLA 2011: Drupal backend-performance
Ashok Modi
 
(ARC402) Deployment Automation: From Developers' Keyboards to End Users' Scre...
(ARC402) Deployment Automation: From Developers' Keyboards to End Users' Scre...(ARC402) Deployment Automation: From Developers' Keyboards to End Users' Scre...
(ARC402) Deployment Automation: From Developers' Keyboards to End Users' Scre...
Amazon Web Services
 

Similar to Practicing Continuous Deployment (20)

Inside the Atlassian OnDemand Private Cloud
Inside the Atlassian OnDemand Private CloudInside the Atlassian OnDemand Private Cloud
Inside the Atlassian OnDemand Private Cloud
 
Improving Your Heroku App Performance with Asset CDN and Unicorn
Improving Your Heroku App Performance with Asset CDN and UnicornImproving Your Heroku App Performance with Asset CDN and Unicorn
Improving Your Heroku App Performance with Asset CDN and Unicorn
 
Dev Ops 101
Dev Ops 101Dev Ops 101
Dev Ops 101
 
Rabbitmq Boot System
Rabbitmq Boot SystemRabbitmq Boot System
Rabbitmq Boot System
 
Cloud meets Fog & Puppet A Story of Version Controlled Infrastructure
Cloud meets Fog & Puppet A Story of Version Controlled InfrastructureCloud meets Fog & Puppet A Story of Version Controlled Infrastructure
Cloud meets Fog & Puppet A Story of Version Controlled Infrastructure
 
Continuous Integration Testing in Django
Continuous Integration Testing in DjangoContinuous Integration Testing in Django
Continuous Integration Testing in Django
 
Building and running cloud native cassandra
Building and running cloud native cassandraBuilding and running cloud native cassandra
Building and running cloud native cassandra
 
Mcollective introduction
Mcollective introductionMcollective introduction
Mcollective introduction
 
Testing Cassandra Guarantees under Diverse Failure Modes with Jepsen
Testing Cassandra Guarantees under Diverse Failure Modes with JepsenTesting Cassandra Guarantees under Diverse Failure Modes with Jepsen
Testing Cassandra Guarantees under Diverse Failure Modes with Jepsen
 
DataStax: Testing Cassandra Guarantees Under Diverse Failure Modes With Jepsen
DataStax: Testing Cassandra Guarantees Under Diverse Failure Modes With JepsenDataStax: Testing Cassandra Guarantees Under Diverse Failure Modes With Jepsen
DataStax: Testing Cassandra Guarantees Under Diverse Failure Modes With Jepsen
 
Corwin on Containers
Corwin on ContainersCorwin on Containers
Corwin on Containers
 
Configuration Management and Salt
Configuration Management and SaltConfiguration Management and Salt
Configuration Management and Salt
 
Antons Kranga Building Agile Infrastructures
Antons Kranga   Building Agile InfrastructuresAntons Kranga   Building Agile Infrastructures
Antons Kranga Building Agile Infrastructures
 
Islands: Puppet at Bulletproof Networks
Islands: Puppet at Bulletproof NetworksIslands: Puppet at Bulletproof Networks
Islands: Puppet at Bulletproof Networks
 
Eclipse con 2012 - Frictionless operations with Puppet - Luke Kanies
Eclipse con 2012 - Frictionless operations with Puppet - Luke KaniesEclipse con 2012 - Frictionless operations with Puppet - Luke Kanies
Eclipse con 2012 - Frictionless operations with Puppet - Luke Kanies
 
Writing & Sharing Great Modules on the Puppet Forge
Writing & Sharing Great Modules on the Puppet ForgeWriting & Sharing Great Modules on the Puppet Forge
Writing & Sharing Great Modules on the Puppet Forge
 
Puppet Conf 2012 - Managing Network Devices with Puppet
Puppet Conf 2012 - Managing Network Devices with PuppetPuppet Conf 2012 - Managing Network Devices with Puppet
Puppet Conf 2012 - Managing Network Devices with Puppet
 
Scalability at GROU.PS
Scalability at GROU.PSScalability at GROU.PS
Scalability at GROU.PS
 
DrupalCampLA 2011: Drupal backend-performance
DrupalCampLA 2011: Drupal backend-performanceDrupalCampLA 2011: Drupal backend-performance
DrupalCampLA 2011: Drupal backend-performance
 
(ARC402) Deployment Automation: From Developers' Keyboards to End Users' Scre...
(ARC402) Deployment Automation: From Developers' Keyboards to End Users' Scre...(ARC402) Deployment Automation: From Developers' Keyboards to End Users' Scre...
(ARC402) Deployment Automation: From Developers' Keyboards to End Users' Scre...
 

More from zeeg

Tools for Development and Debugging in Python
Tools for Development and Debugging in PythonTools for Development and Debugging in Python
Tools for Development and Debugging in Pythonzeeg
 
Pitfalls of Continuous Deployment
Pitfalls of Continuous DeploymentPitfalls of Continuous Deployment
Pitfalls of Continuous Deployment
zeeg
 
Building Scalable Web Apps
Building Scalable Web AppsBuilding Scalable Web Apps
Building Scalable Web Apps
zeeg
 
Continuous Deployment at Disqus (Pylons Minicon)
Continuous Deployment at Disqus (Pylons Minicon)Continuous Deployment at Disqus (Pylons Minicon)
Continuous Deployment at Disqus (Pylons Minicon)zeeg
 
PyCon 2011 Scaling Disqus
PyCon 2011 Scaling DisqusPyCon 2011 Scaling Disqus
PyCon 2011 Scaling Disqus
zeeg
 
Sentry (SF Python, Feb)
Sentry (SF Python, Feb)Sentry (SF Python, Feb)
Sentry (SF Python, Feb)
zeeg
 
Db tips & tricks django meetup
Db tips & tricks django meetupDb tips & tricks django meetup
Db tips & tricks django meetupzeeg
 

More from zeeg (7)

Tools for Development and Debugging in Python
Tools for Development and Debugging in PythonTools for Development and Debugging in Python
Tools for Development and Debugging in Python
 
Pitfalls of Continuous Deployment
Pitfalls of Continuous DeploymentPitfalls of Continuous Deployment
Pitfalls of Continuous Deployment
 
Building Scalable Web Apps
Building Scalable Web AppsBuilding Scalable Web Apps
Building Scalable Web Apps
 
Continuous Deployment at Disqus (Pylons Minicon)
Continuous Deployment at Disqus (Pylons Minicon)Continuous Deployment at Disqus (Pylons Minicon)
Continuous Deployment at Disqus (Pylons Minicon)
 
PyCon 2011 Scaling Disqus
PyCon 2011 Scaling DisqusPyCon 2011 Scaling Disqus
PyCon 2011 Scaling Disqus
 
Sentry (SF Python, Feb)
Sentry (SF Python, Feb)Sentry (SF Python, Feb)
Sentry (SF Python, Feb)
 
Db tips & tricks django meetup
Db tips & tricks django meetupDb tips & tricks django meetup
Db tips & tricks django meetup
 

Recently uploaded

Assuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyesAssuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyes
ThousandEyes
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Inflectra
 
Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........
Alison B. Lowndes
 
Knowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and backKnowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and back
Elena Simperl
 
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdfFIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance
 
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
Product School
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Product School
 
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
BookNet Canada
 
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Albert Hoitingh
 
Accelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish CachingAccelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish Caching
Thijs Feryn
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
DianaGray10
 
Elevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object CalisthenicsElevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object Calisthenics
Dorra BARTAGUIZ
 
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
Product School
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
RTTS
 
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
KatiaHIMEUR1
 
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdfFIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
Safe Software
 
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdfSmart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
91mobiles
 
Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR EventsMonitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
Ana-Maria Mihalceanu
 

Recently uploaded (20)

Assuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyesAssuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyes
 
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdfFIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
 
Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........
 
Knowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and backKnowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and back
 
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdfFIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdf
 
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...
 
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...Transcript: Selling digital books in 2024: Insights from industry leaders - T...
Transcript: Selling digital books in 2024: Insights from industry leaders - T...
 
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
 
Accelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish CachingAccelerate your Kubernetes clusters with Varnish Caching
Accelerate your Kubernetes clusters with Varnish Caching
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
 
Elevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object CalisthenicsElevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object Calisthenics
 
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
 
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
 
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdfFIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
 
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdfSmart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
 
Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR EventsMonitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
 

Practicing Continuous Deployment

  • 1. DISQUS Practicing Continuous Deployment David Cramer @zeeg Saturday, March 10, 12
  • 2. Shipping new code as soon as it’s ready Saturday, March 10, 12
  • 3. Continuous Deployment # Update the site every 5 minutes */5 * * * * cd /www/example.com && git pull && service apache restart Saturday, March 10, 12
  • 6. When is it ready? - Reviewed by peers - Passes automated tests - Some level of QA Saturday, March 10, 12
  • 7. Focus on Stability and Iteration Saturday, March 10, 12
  • 8. Workflow Review Commit Integration Failed Build Deploy Reporting Rollback Saturday, March 10, 12
  • 9. The Good - Develop features incrementally - Release frequently - Smaller doses of QA The Bad - Culture Shock - Stability depends on test coverage - Initial time investment Saturday, March 10, 12
  • 11. Development - Automate testing of complicated processes and architecture - Simple can be better than complete - Especially for local development - python setup.py {develop,test} - Puppet, Chef, Buildout, Fabric, etc. Saturday, March 10, 12
  • 12. Production Staging PostgreSQL PostgreSQL Memcache Memcache Redis Redis Solr Solr Apache Apache Nginx Nginx RabbitMQ RabbitMQ (and 100 other painful-to-configure services) CI Server Macbook PostgreSQL PostgreSQL Memcache Apache Redis Memcache Solr Redis Apache Solr Nginx Nginx RabbitMQ RabbitMQ Saturday, March 10, 12
  • 13. Bootstrapping Local - Simplify local setup - git clone dcramer@disqus:disqus.git - make - python manage.py runserver - Need to test dependancies? - virtualbox + vagrant up Saturday, March 10, 12
  • 14. Progressive Rollout We actively use early versions of features before public release Saturday, March 10, 12
  • 15. Deploy features to portions of a user base at a time to ensure smooth, measurable releases https://github.com/disqus/gargoyle Saturday, March 10, 12
  • 16. Iterate quickly by hiding features • Early adopters are free QA from gargoyle import gargoyle def my_view(request): if gargoyle.is_active('awesome', request): return 'new happy version :D' else: return 'old sad version :(' Saturday, March 10, 12
  • 17. SWITCHES = { # enable my_feature for 50% 'my_feature': range(0, 50), } def is_active(switch): try: pct_range = SWITCHES[switch] except KeyError: return False ip_hash = sum([int(x) for x in ip_address.split('.')]) return (ip_hash % 100 in pct_range) Saturday, March 10, 12
  • 18. Review ALL the Commits phabricator.org Saturday, March 10, 12
  • 22. Integration (or as we like to call it) Saturday, March 10, 12
  • 24. Integration Requirements - Developers must know when they’ve broken something - IRC, Email, IM - Support proper reporting - XUnit, Pylint, Coverage.py - Painless setup - apt-get install jenkins * https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins+on+Ubuntu Saturday, March 10, 12
  • 25. Shortcomings - False positives - Reporting isn't accurate - Services fail - Bad Tests - Test coverage - Regressions on untested code - Feedback delay - Integration tests vs Unit tests Saturday, March 10, 12
  • 26. Fixing False Positives - Re-run tests several times on a failure - Report continually failing tests - Replace external service tests with a functional test suite Saturday, March 10, 12
  • 27. Maintaining Coverage - Raise awareness with reporting - Fail/alert when coverage drops on a build - Commit tests with code - Coverage against commit diff for untested regressions - Utilize code review Saturday, March 10, 12
  • 28. Speeding Up Tests - Write true unit tests - vs slower integration tests - Mock external services - Distributed and parallel testing - Matrix builds Saturday, March 10, 12
  • 30. <You> Why is mongodb-1 down? <Ops> It’s down? Must have crashed again Saturday, March 10, 12
  • 31. Meaningful Metrics - Rate of traffic (not just hits!) - Business vs system - Response time (database, web) - Exceptions - Social media - Twitter Saturday, March 10, 12
  • 32. Graphite (Trafficgraphite.wikidot.com across a cluster of servers) Saturday, March 10, 12
  • 33. Sentry sentry.readthedocs.org Saturday, March 10, 12
  • 35. Getting Started - Package your app - Value code review - Ease deployment; fast rollbacks - Setup automated tests - Gather some easy metrics Saturday, March 10, 12
  • 36. Going Further - Build an immune system - Automate deploys, rollbacks (maybe) - Adjust to your culture - There is no “right way” - SOA == great success Saturday, March 10, 12
  • 37. DISQUS Questions? psst, we’re hiring disqus.com/jobs Saturday, March 10, 12
  • 38. References - Gargoyle (feature switches) https://github.com/disqus/gargoyle - Sentry (log aggregation) https://github.com/dcramer/sentry - Jenkins CI (continuous integration) http://jenkins-ci.org/ - Phabricator (code reviews, bug tracking) https://phabricator.org - Graphite (metrics) http://graphite.wikidot.com/ code.disqus.com Saturday, March 10, 12