SlideShare a Scribd company logo
1 of 50
Download to read offline
DISQUS
                           Continuous Deployment Everything



                                      David Cramer
                                         @zeeg




Wednesday, June 22, 2011
Continuous Deployment


          Shipping new code as soon
                 as it’s ready

                      (It’s really just super awesome buildbots)




Wednesday, June 22, 2011
Workflow


                           Commit (master)




                             Integration             Failed Build




                               Deploy                Reporting




                                                      Rollback




Wednesday, June 22, 2011
Pros                           Cons


              •     Develop features           •   Culture Shock
                    incrementally              •   Stability depends on
              •     Release frequently             test coverage
              •     Smaller doses of QA        •   Initial time
                                                   investment




                       We mostly just care about iteration and stability

Wednesday, June 22, 2011
Painless Development




Wednesday, June 22, 2011
Development



               •     Production > Staging > CI > Dev
                     •     Automate testing of complicated
                           processes and architecture
               •     Simple > complete
                     •     Especially for local development
               •     python setup.py {develop,test}
               •     Puppet, Chef, simple bootstrap.{py,sh}



Wednesday, June 22, 2011
Production            Staging
                           •    PostgreSQL   •   PostgreSQL
                           •    Memcache     •   Memcache
                           •    Redis        •   Redis
                           •    Solr         •   Solr
                           •    Apache       •   Apache
                           •    Nginx        •   Nginx
                           •    RabbitMQ     •   RabbitMQ


                               CI Server         Macbook

                           •    Memcache     •   PostgreSQL
                           •    PostgreSQL   •   Apache
                           •    Redis        •   Memcache
                           •    Solr         •   Redis
                           •    Apache       •   Solr
                           •    Nginx        •   Nginx
                           •    RabbitMQ     •   RabbitMQ


Wednesday, June 22, 2011
Bootstrapping Local



               •     Simplify local setup
                     •     git clone dcramer@disqus:disqus.git
                     •     ./bootstrap.sh
                     •     python manage.py runserver


               •     Need to test dependancies?
                     •     virtualbox + vagrant up



Wednesday, June 22, 2011
“Under Construction”



               •     Iterate quickly by hiding features
               •     Early adopters are free QA



                     from gargoyle import gargoyle

                     def my_view(request):
                         if gargoyle.is_active('awesome', request):
                             return 'new happy version :D'
                         else:
                             return 'old sad version :('




Wednesday, June 22, 2011
Gargoyle

                           Deploy features to portions of a user base at a
                            time to ensure smooth, measurable releases




                            Being users of our product, we actively use
                           early versions of features before public release

Wednesday, June 22, 2011
Conditions in Gargoyle


                    from gargoyle import gargoyle
                    from gargoyle.conditions import ModelConditionSet,
                                                    Percent, String

                    class UserConditionSet(ModelConditionSet):
                        # percent implicitly maps to ``id``
                        percent = Percent()
                        username = String()

                           def can_execute(self, instance):
                               return isinstance(instance, User)

                    # register with our main gargoyle instance
                    gargoyle.register(UserConditionSet(User))




Wednesday, June 22, 2011
Without Gargoyle


                    SWITCHES = {
                        # enable my_feature for 50%
                        'my_feature': range(0, 50),
                    }

                    def is_active(switch):
                        try:
                             pct_range = SWITCHES[switch]
                        except KeyError:
                             return False

                           ip_hash = sum([int(x) for x
                                          in ip_address.split('.')])

                           return (ip_hash % 100 in pct_range)


                                    If you use Django, use Gargoyle


Wednesday, June 22, 2011
Integration
                           (or as we like to call it)




Wednesday, June 22, 2011
Integration is Required




                           Deploy only when things wont break

Wednesday, June 22, 2011
Setup a Jenkins Build




Wednesday, June 22, 2011
Reporting is Critical




Wednesday, June 22, 2011
CI Requirements



               •     Developers must know when they’ve
                     broken something
                     •     IRC, Email, IM
               •     Support proper reporting
                     •     XUnit, Pylint, Coverage.py
               •     Painless setup
                     •     apt-get install jenkins *

                           https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins+on+Ubuntu


Wednesday, June 22, 2011
Shortcomings

               •     False positives lower awareness
                     •     Reporting isn't accurate
                     •     Services fail
                     •     Bad Tests
               •     Not enough code coverage
                     •     Regressions on untested code
               •     Test suite takes too long
                     •     Integration tests vs Unit tests
                     •     SOA, distribution

Wednesday, June 22, 2011
Fixing False Positives




               •     Re-run tests several times on a failure
               •     Report continually failing tests
                     •     Fix continually failing tests
               •     Rely less on 3rd parties
                     •     Mock/Dingus




Wednesday, June 22, 2011
Maintaining Coverage




               •     Raise awareness with reporting
                     •     Fail/alert when coverage drops on a build
               •     Commit tests with code
                     •     Coverage against commit di    for
                           untested regressions
               •     Drive it into your culture




Wednesday, June 22, 2011
Speeding Up Tests




               •     Write true unit tests
                     •     vs slower integration tests
               •     Mock 3rd party APIs
               •     Distributed and parallel testing
                     •     http://github.com/disqus/mule




Wednesday, June 22, 2011
Mule



               •     Unstable, will change a lot
               •     Mostly Django right now
                     •     Generic interfaces for unittest2
               •     Works with multi-processing and Celery
               •     Full XUnit integration
               •     Simple workflow
                     •     mule test --runner="python manage.py
                           mule --worker $TEST"



Wednesday, June 22, 2011
Deploy (finally)




Wednesday, June 22, 2011
How DISQUS Does It




               •     Incremental deploy with Fabric
               •     Drop server from pool
               •     Pull in requirements on each server
                     •     Isolated virtualenv’s built on each server
               •     Push server back online




Wednesday, June 22, 2011
How You Can Do It

                    # fabfile.py
                    from fabric.api import *

                    def deploy(revision):
                        # update sources, virtualenv, requirements
                        # ...

                           # copy ``current`` to ``previous``
                           run('cp -R %(path)s/current %(path)s/previous' % dict(
                               path=env.path,
                               revision=revision,
                           ))

                           # symlink ``revision`` to ``current``
                           run('ln -fs %(path)s/%(revision)s %(path)s/current' % dict(
                               path=env.path,
                               revision=revision,
                           ))

                           # restart apache
                           run('touch %(path)s/current/django.wsgi')



Wednesday, June 22, 2011
How YOU Can Do It (cont.)




                    # fabfile.py
                    from fabric.api import *

                    def rollback(revision=None):
                        # move ``previous`` to ``current``
                        run('mv %(path)s/previous %(path)s/current' % dict(
                            path=env.path,
                            revision=revision,
                        ))

                           # restart apache
                           run('touch %(path)s/current/django.wsgi')




Wednesday, June 22, 2011
Challenges




               •     PyPi works on server A, but not B
               •     Scale
               •     CPU cost per server
               •     Schema changes, data model changes
               •     Backwards compatibility




Wednesday, June 22, 2011
PyPi is Down




               •     http://github.com/disqus/chishop




Wednesday, June 22, 2011
Help, we have 100 servers!




               •     Incremental (ours) vs Fanout
               •     Push vs Pull
                     •     Twitter uses BitTorrent
               •     Isolation vs Packaging (Complexity)




Wednesday, June 22, 2011
SQL Schema Changes




               1. Add column (NULLable)
               2. Add app code to fill column
               3.Deploy
               4.Backfill column
               5. Add app code to read column
               6.Deploy




Wednesday, June 22, 2011
Updating Caches




               •     Have a global version number
                     •     CACHE_PREFIX = 9000
               •     Have a data model cache version
                     •     sha1(cls.__dict__)
               •     Use multiple caches




Wednesday, June 22, 2011
Reporting




Wednesday, June 22, 2011
It’s Important!




Wednesday, June 22, 2011
<You> Why is mongodb-1 down?

         <Ops> It’s down? Must have crashed again




Wednesday, June 22, 2011
Meaningful Metrics




               •     Rate of tra c (not just hits!)
                     •     Business vs system
               •     Response time (database, web)
               •     Exceptions
               •     Social media
                     •     Twitter




Wednesday, June 22, 2011
Standard Tools



                                                       Nagios

                           Graphite




Wednesday, June 22, 2011
Using Graphite


                    # statsd.py
                    # requires python-statsd

                    from pystatsd import Client
                    import socket

                    def with_suffix(key):
                        hostname = socket.gethostname().split('.')[0]
                        return '%s.%s' % (key, hostname)

                    client = Client(host=STATSD_HOST, port=STATSD_PORT)

                    # statsd.incr('key1', 'key2')
                    def incr(*keys):
                        keys = [with_suffix(k) for k in keys]:
                        client.increment(*keys):




Wednesday, June 22, 2011
Using Graphite (cont.)




                           (Tra c across a cluster of servers)


Wednesday, June 22, 2011
Logging



                           •   Realtime
                           •   Aggregates
                           •   History
                           •   Notifications
                           •   Scalable
                           •   Available
                           •   Metadata



Wednesday, June 22, 2011
Logging: Syslog


                           ✓   Realtime
                           x   Aggregates
                           ✓   History
                           x   Notifications
                           ✓   Scalable
                           ✓   Available
                           x   Metadata




Wednesday, June 22, 2011
Logging: Email Collection


                               ✓   Realtime
                               x   Aggregates
                               ✓   History
                               x   Notifications
                               x   Scalable
                               ✓   Available
                               ✓   Metadata


                           (Django provides this out of the box)


Wednesday, June 22, 2011
Logging: Sentry


                                 ✓   Realtime
                                 ✓   Aggregates
                                 ✓   History
                                 ✓   Notifications
                                 ✓   Scalable
                                 ✓   Available
                                 ✓   Metadata


                           http://github.com/dcramer/django-sentry


Wednesday, June 22, 2011
Setting up Sentry (1.x)



                    # setup your server first
                    $ pip install django-sentry
                    $ sentry start

                    # configure your Python (Django in our case) client
                    INSTALLED_APPS = (
                        # ...
                        'sentry.client',
                    )

                    # point the client to the servers
                    SENTRY_REMOTE_URL = ['http://sentry/store/']

                    # visit http://sentry in the browser




Wednesday, June 22, 2011
Setting up Sentry (cont.)


                    # ~/.sentry/sentry.conf.py

                    # use a better database
                    DATABASES = {
                        'default': {
                            'ENGINE': 'postgresql_psycopg2',
                            'NAME': 'sentry',
                            'USER': 'postgres',
                        }
                    }

                    # bind to all interfaces
                    SENTRY_WEB_HOST = '0.0.0.0'

                    # change data paths
                    SENTRY_WEB_LOG_FILE = '/var/log/sentry.log'
                    SENTRY_WEB_PID_FILE = '/var/run/sentry.pid'


Wednesday, June 22, 2011
Sentry (demo time)




Wednesday, June 22, 2011
Wrap Up




Wednesday, June 22, 2011
Getting Started




               •     Package your app
               •     Ease deployment; fast rollbacks
               •     Setup automated tests
               •     Gather some easy metrics




Wednesday, June 22, 2011
Going Further




               •     Build an immune system
                     •     Automate deploys, rollbacks (maybe)
               •     Adjust to your culture
                     •     CD doesn’t “just work”
               •     SOA == great success




Wednesday, June 22, 2011
DISQUS
                             Questions?




                             psst, we’re hiring
                            jobs@disqus.com

Wednesday, June 22, 2011
References



               •     Gargoyle (feature switches)
                     https://github.com/disqus/gargoyle
               •     Sentry (log aggregation)
                     https://github.com/dcramer/django-sentry (1.x)
                     https://github.com/dcramer/sentry (2.x)
               •     Jenkins CI
                     http://jenkins-ci.org/
               •     Mule (distributed test runner)
                     https://github.com/disqus/mule




                                              code.disqus.com
Wednesday, June 22, 2011

More Related Content

What's hot

Infrastructure Deployment with Docker & Ansible
Infrastructure Deployment with Docker & AnsibleInfrastructure Deployment with Docker & Ansible
Infrastructure Deployment with Docker & AnsibleRobert Reiz
 
Jenkins vs GitLab CI
Jenkins vs GitLab CIJenkins vs GitLab CI
Jenkins vs GitLab CICEE-SEC(R)
 
GITS Class #16: CI/CD (Continuous Integration & Continuous Deployment) with G...
GITS Class #16: CI/CD (Continuous Integration & Continuous Deployment) with G...GITS Class #16: CI/CD (Continuous Integration & Continuous Deployment) with G...
GITS Class #16: CI/CD (Continuous Integration & Continuous Deployment) with G...GITS Indonesia
 
Write microservice in golang
Write microservice in golangWrite microservice in golang
Write microservice in golangBo-Yi Wu
 
Monitoring with Graylog - a modern approach to monitoring?
Monitoring with Graylog - a modern approach to monitoring?Monitoring with Graylog - a modern approach to monitoring?
Monitoring with Graylog - a modern approach to monitoring?inovex GmbH
 
Get started with gitops and flux
Get started with gitops and fluxGet started with gitops and flux
Get started with gitops and fluxLibbySchulze1
 
Introducing GitLab (June 2018)
Introducing GitLab (June 2018)Introducing GitLab (June 2018)
Introducing GitLab (June 2018)Noa Harel
 
What's New for GitLab CI/CD February 2020
What's New for GitLab CI/CD February 2020What's New for GitLab CI/CD February 2020
What's New for GitLab CI/CD February 2020Noa Harel
 
Practical tools for Web Accessibility testing
Practical tools for Web Accessibility testingPractical tools for Web Accessibility testing
Practical tools for Web Accessibility testingToufic Sbeiti
 
CI and CD with Jenkins
CI and CD with JenkinsCI and CD with Jenkins
CI and CD with JenkinsMartin Málek
 
Artifacts management with DevOps
Artifacts management with DevOpsArtifacts management with DevOps
Artifacts management with DevOpsChen-Tien Tsai
 
Introduction to Gitlab
Introduction to GitlabIntroduction to Gitlab
Introduction to GitlabJulien Pivotto
 
Sonarqube
SonarqubeSonarqube
SonarqubeKalkey
 
Devops Porto - CI/CD at Gitlab
Devops Porto - CI/CD at GitlabDevops Porto - CI/CD at Gitlab
Devops Porto - CI/CD at GitlabFilipa Lacerda
 
Continuous Integration/Deployment with Gitlab CI
Continuous Integration/Deployment with Gitlab CIContinuous Integration/Deployment with Gitlab CI
Continuous Integration/Deployment with Gitlab CIDavid Hahn
 
CI/CD Overview
CI/CD OverviewCI/CD Overview
CI/CD OverviewAn Nguyen
 
CI/CD Tools Universe: The Ultimate List
CI/CD Tools Universe: The Ultimate ListCI/CD Tools Universe: The Ultimate List
CI/CD Tools Universe: The Ultimate ListPlutora
 

What's hot (20)

Infrastructure Deployment with Docker & Ansible
Infrastructure Deployment with Docker & AnsibleInfrastructure Deployment with Docker & Ansible
Infrastructure Deployment with Docker & Ansible
 
Jenkins vs GitLab CI
Jenkins vs GitLab CIJenkins vs GitLab CI
Jenkins vs GitLab CI
 
GITS Class #16: CI/CD (Continuous Integration & Continuous Deployment) with G...
GITS Class #16: CI/CD (Continuous Integration & Continuous Deployment) with G...GITS Class #16: CI/CD (Continuous Integration & Continuous Deployment) with G...
GITS Class #16: CI/CD (Continuous Integration & Continuous Deployment) with G...
 
Write microservice in golang
Write microservice in golangWrite microservice in golang
Write microservice in golang
 
Monitoring with Graylog - a modern approach to monitoring?
Monitoring with Graylog - a modern approach to monitoring?Monitoring with Graylog - a modern approach to monitoring?
Monitoring with Graylog - a modern approach to monitoring?
 
Get started with gitops and flux
Get started with gitops and fluxGet started with gitops and flux
Get started with gitops and flux
 
Introducing GitLab (June 2018)
Introducing GitLab (June 2018)Introducing GitLab (June 2018)
Introducing GitLab (June 2018)
 
What's New for GitLab CI/CD February 2020
What's New for GitLab CI/CD February 2020What's New for GitLab CI/CD February 2020
What's New for GitLab CI/CD February 2020
 
GitLab.pptx
GitLab.pptxGitLab.pptx
GitLab.pptx
 
DevOps Presentation.pptx
DevOps Presentation.pptxDevOps Presentation.pptx
DevOps Presentation.pptx
 
The Journey to DevSecOps
The Journey to DevSecOpsThe Journey to DevSecOps
The Journey to DevSecOps
 
Practical tools for Web Accessibility testing
Practical tools for Web Accessibility testingPractical tools for Web Accessibility testing
Practical tools for Web Accessibility testing
 
CI and CD with Jenkins
CI and CD with JenkinsCI and CD with Jenkins
CI and CD with Jenkins
 
Artifacts management with DevOps
Artifacts management with DevOpsArtifacts management with DevOps
Artifacts management with DevOps
 
Introduction to Gitlab
Introduction to GitlabIntroduction to Gitlab
Introduction to Gitlab
 
Sonarqube
SonarqubeSonarqube
Sonarqube
 
Devops Porto - CI/CD at Gitlab
Devops Porto - CI/CD at GitlabDevops Porto - CI/CD at Gitlab
Devops Porto - CI/CD at Gitlab
 
Continuous Integration/Deployment with Gitlab CI
Continuous Integration/Deployment with Gitlab CIContinuous Integration/Deployment with Gitlab CI
Continuous Integration/Deployment with Gitlab CI
 
CI/CD Overview
CI/CD OverviewCI/CD Overview
CI/CD Overview
 
CI/CD Tools Universe: The Ultimate List
CI/CD Tools Universe: The Ultimate ListCI/CD Tools Universe: The Ultimate List
CI/CD Tools Universe: The Ultimate List
 

Viewers also liked

Failing Continuous Delivery, JDays, 2015
Failing Continuous Delivery, JDays, 2015Failing Continuous Delivery, JDays, 2015
Failing Continuous Delivery, JDays, 2015Daniel Sawano
 
Code, ci, infrastructure - the gophers way
Code, ci, infrastructure - the gophers wayCode, ci, infrastructure - the gophers way
Code, ci, infrastructure - the gophers wayAlex Baitov
 
Agile Design - Chicago IXDA Presentation
Agile Design - Chicago IXDA PresentationAgile Design - Chicago IXDA Presentation
Agile Design - Chicago IXDA PresentationAlice Toth
 
The Hard Problems of Continuous Deployment
The Hard Problems of Continuous DeploymentThe Hard Problems of Continuous Deployment
The Hard Problems of Continuous DeploymentTimothy Fitz
 
Testing, CI and CD in the real world
Testing, CI and CD in the real worldTesting, CI and CD in the real world
Testing, CI and CD in the real worldRoc Boronat
 
FALCON's Tilt Tray Sorter: A new age in Packet Sorting
FALCON's Tilt Tray Sorter: A new age in Packet SortingFALCON's Tilt Tray Sorter: A new age in Packet Sorting
FALCON's Tilt Tray Sorter: A new age in Packet SortingFalcon Autotech
 
Cloud Application Development Lifecycle
Cloud Application Development LifecycleCloud Application Development Lifecycle
Cloud Application Development LifecycleSuhas Kelkar
 
Lviv PMDay: Дов Німрац Як зробити процес Continuous Integration ефективним
Lviv PMDay: Дов Німрац Як зробити процес Continuous Integration ефективнимLviv PMDay: Дов Німрац Як зробити процес Continuous Integration ефективним
Lviv PMDay: Дов Німрац Як зробити процес Continuous Integration ефективнимLviv Startup Club
 
Training for Third Sector Partners
Training for Third Sector PartnersTraining for Third Sector Partners
Training for Third Sector PartnersPaul McElvaney
 
Loraine Slinn at LP2010
Loraine Slinn at LP2010Loraine Slinn at LP2010
Loraine Slinn at LP2010Paul McElvaney
 
Stunning Photos
Stunning PhotosStunning Photos
Stunning PhotosJennAlm
 
ステルスマーケティングとニュースリリース
ステルスマーケティングとニュースリリースステルスマーケティングとニュースリリース
ステルスマーケティングとニュースリリースMinako Kambara
 
Niver Bah - 22.06.07
Niver Bah - 22.06.07Niver Bah - 22.06.07
Niver Bah - 22.06.07Jubrac Jacui
 
香港六合彩
香港六合彩香港六合彩
香港六合彩wejia
 
Learning Pool Webinar: Brand new new authoring tool templates
Learning Pool Webinar: Brand new new authoring tool templatesLearning Pool Webinar: Brand new new authoring tool templates
Learning Pool Webinar: Brand new new authoring tool templatesPaul McElvaney
 
Webanalytics2.0 sem jvol2
Webanalytics2.0 sem jvol2Webanalytics2.0 sem jvol2
Webanalytics2.0 sem jvol2Sonika Mishra
 
Pondres Social Marketing event 26 oktober
Pondres Social Marketing event 26 oktoberPondres Social Marketing event 26 oktober
Pondres Social Marketing event 26 oktoberSjef Kerkhofs
 

Viewers also liked (20)

Failing Continuous Delivery, JDays, 2015
Failing Continuous Delivery, JDays, 2015Failing Continuous Delivery, JDays, 2015
Failing Continuous Delivery, JDays, 2015
 
Code, ci, infrastructure - the gophers way
Code, ci, infrastructure - the gophers wayCode, ci, infrastructure - the gophers way
Code, ci, infrastructure - the gophers way
 
Agile Design - Chicago IXDA Presentation
Agile Design - Chicago IXDA PresentationAgile Design - Chicago IXDA Presentation
Agile Design - Chicago IXDA Presentation
 
The Hard Problems of Continuous Deployment
The Hard Problems of Continuous DeploymentThe Hard Problems of Continuous Deployment
The Hard Problems of Continuous Deployment
 
Testing, CI and CD in the real world
Testing, CI and CD in the real worldTesting, CI and CD in the real world
Testing, CI and CD in the real world
 
FALCON's Tilt Tray Sorter: A new age in Packet Sorting
FALCON's Tilt Tray Sorter: A new age in Packet SortingFALCON's Tilt Tray Sorter: A new age in Packet Sorting
FALCON's Tilt Tray Sorter: A new age in Packet Sorting
 
Cloud Application Development Lifecycle
Cloud Application Development LifecycleCloud Application Development Lifecycle
Cloud Application Development Lifecycle
 
Lviv PMDay: Дов Німрац Як зробити процес Continuous Integration ефективним
Lviv PMDay: Дов Німрац Як зробити процес Continuous Integration ефективнимLviv PMDay: Дов Німрац Як зробити процес Continuous Integration ефективним
Lviv PMDay: Дов Німрац Як зробити процес Continuous Integration ефективним
 
Training for Third Sector Partners
Training for Third Sector PartnersTraining for Third Sector Partners
Training for Third Sector Partners
 
Loraine Slinn at LP2010
Loraine Slinn at LP2010Loraine Slinn at LP2010
Loraine Slinn at LP2010
 
Stunning Photos
Stunning PhotosStunning Photos
Stunning Photos
 
ステルスマーケティングとニュースリリース
ステルスマーケティングとニュースリリースステルスマーケティングとニュースリリース
ステルスマーケティングとニュースリリース
 
Ana Flickr 101
Ana Flickr 101Ana Flickr 101
Ana Flickr 101
 
Niver Bah - 22.06.07
Niver Bah - 22.06.07Niver Bah - 22.06.07
Niver Bah - 22.06.07
 
香港六合彩
香港六合彩香港六合彩
香港六合彩
 
01.2008 AcampãO
01.2008   AcampãO01.2008   AcampãO
01.2008 AcampãO
 
Learning Pool Webinar: Brand new new authoring tool templates
Learning Pool Webinar: Brand new new authoring tool templatesLearning Pool Webinar: Brand new new authoring tool templates
Learning Pool Webinar: Brand new new authoring tool templates
 
Webanalytics2.0 sem jvol2
Webanalytics2.0 sem jvol2Webanalytics2.0 sem jvol2
Webanalytics2.0 sem jvol2
 
Pondres Social Marketing event 26 oktober
Pondres Social Marketing event 26 oktoberPondres Social Marketing event 26 oktober
Pondres Social Marketing event 26 oktober
 
Para Que Serve O Galego
Para Que Serve O GalegoPara Que Serve O Galego
Para Que Serve O Galego
 

Similar to Pitfalls of Continuous Deployment

Continuous Deployment at Disqus (Pylons Minicon)
Continuous Deployment at Disqus (Pylons Minicon)Continuous Deployment at Disqus (Pylons Minicon)
Continuous Deployment at Disqus (Pylons Minicon)zeeg
 
Android 1.5 to 3.0: a compatibility journey
Android 1.5 to 3.0: a compatibility journeyAndroid 1.5 to 3.0: a compatibility journey
Android 1.5 to 3.0: a compatibility journeyEmanuele Di Saverio
 
Donating a mature project to Eclipse
Donating a mature project to EclipseDonating a mature project to Eclipse
Donating a mature project to Eclipseglynnormington
 
Mozilla: Continuous Deploment on SUMO
Mozilla: Continuous Deploment on SUMOMozilla: Continuous Deploment on SUMO
Mozilla: Continuous Deploment on SUMOMatt Brandt
 
Continuous delivery with Jenkins Enterprise and Deployit
Continuous delivery with Jenkins Enterprise and DeployitContinuous delivery with Jenkins Enterprise and Deployit
Continuous delivery with Jenkins Enterprise and DeployitXebiaLabs
 
Building Scalable Web Apps
Building Scalable Web AppsBuilding Scalable Web Apps
Building Scalable Web Appszeeg
 
Continuous Delivery Overview
Continuous Delivery OverviewContinuous Delivery Overview
Continuous Delivery OverviewWill Iverson
 
State of jQuery June 2013 - Portland
State of jQuery June 2013 - PortlandState of jQuery June 2013 - Portland
State of jQuery June 2013 - Portlanddmethvin
 
Continuous integration for open source distros v 3.0
Continuous integration for open source distros v 3.0Continuous integration for open source distros v 3.0
Continuous integration for open source distros v 3.0Sriram Narayanan
 
Introduction to jenkins
Introduction to jenkinsIntroduction to jenkins
Introduction to jenkinsAbe Diaz
 
Continuous Delivery Using Jenkins
Continuous Delivery Using JenkinsContinuous Delivery Using Jenkins
Continuous Delivery Using JenkinsCliffano Subagio
 
Jython 2.7 and techniques for integrating with Java - Frank Wierzbicki
Jython 2.7 and techniques for integrating with Java - Frank WierzbickiJython 2.7 and techniques for integrating with Java - Frank Wierzbicki
Jython 2.7 and techniques for integrating with Java - Frank Wierzbickifwierzbicki
 
Test parallelization using Jenkins
Test parallelization using JenkinsTest parallelization using Jenkins
Test parallelization using JenkinsRogue Wave Software
 
Running productioninstance 1-localcopy
Running productioninstance 1-localcopyRunning productioninstance 1-localcopy
Running productioninstance 1-localcopyCloudBees
 
DevOps Fest 2020. Kohsuke Kawaguchi. GitOps, Jenkins X & the Future of CI/CD
DevOps Fest 2020. Kohsuke Kawaguchi. GitOps, Jenkins X & the Future of CI/CDDevOps Fest 2020. Kohsuke Kawaguchi. GitOps, Jenkins X & the Future of CI/CD
DevOps Fest 2020. Kohsuke Kawaguchi. GitOps, Jenkins X & the Future of CI/CDDevOps_Fest
 
Quality Assurance in a DevOps World
Quality Assurance in a DevOps WorldQuality Assurance in a DevOps World
Quality Assurance in a DevOps WorldKit Plummer
 
Php com con-2011
Php com con-2011Php com con-2011
Php com con-2011LB Denker
 
Jumping from Continuous Integration to Continuous Delivery with Jenkins Enter...
Jumping from Continuous Integration to Continuous Delivery with Jenkins Enter...Jumping from Continuous Integration to Continuous Delivery with Jenkins Enter...
Jumping from Continuous Integration to Continuous Delivery with Jenkins Enter...CloudBees
 

Similar to Pitfalls of Continuous Deployment (20)

Continuous Deployment at Disqus (Pylons Minicon)
Continuous Deployment at Disqus (Pylons Minicon)Continuous Deployment at Disqus (Pylons Minicon)
Continuous Deployment at Disqus (Pylons Minicon)
 
Hudson
HudsonHudson
Hudson
 
Android 1.5 to 3.0: a compatibility journey
Android 1.5 to 3.0: a compatibility journeyAndroid 1.5 to 3.0: a compatibility journey
Android 1.5 to 3.0: a compatibility journey
 
Donating a mature project to Eclipse
Donating a mature project to EclipseDonating a mature project to Eclipse
Donating a mature project to Eclipse
 
Mozilla: Continuous Deploment on SUMO
Mozilla: Continuous Deploment on SUMOMozilla: Continuous Deploment on SUMO
Mozilla: Continuous Deploment on SUMO
 
Continuous delivery with Jenkins Enterprise and Deployit
Continuous delivery with Jenkins Enterprise and DeployitContinuous delivery with Jenkins Enterprise and Deployit
Continuous delivery with Jenkins Enterprise and Deployit
 
Building Scalable Web Apps
Building Scalable Web AppsBuilding Scalable Web Apps
Building Scalable Web Apps
 
Continuous Delivery Overview
Continuous Delivery OverviewContinuous Delivery Overview
Continuous Delivery Overview
 
State of jQuery June 2013 - Portland
State of jQuery June 2013 - PortlandState of jQuery June 2013 - Portland
State of jQuery June 2013 - Portland
 
Continuous integration for open source distros v 3.0
Continuous integration for open source distros v 3.0Continuous integration for open source distros v 3.0
Continuous integration for open source distros v 3.0
 
Introduction to jenkins
Introduction to jenkinsIntroduction to jenkins
Introduction to jenkins
 
Continuous Delivery Using Jenkins
Continuous Delivery Using JenkinsContinuous Delivery Using Jenkins
Continuous Delivery Using Jenkins
 
Jython 2.7 and techniques for integrating with Java - Frank Wierzbicki
Jython 2.7 and techniques for integrating with Java - Frank WierzbickiJython 2.7 and techniques for integrating with Java - Frank Wierzbicki
Jython 2.7 and techniques for integrating with Java - Frank Wierzbicki
 
Test parallelization using Jenkins
Test parallelization using JenkinsTest parallelization using Jenkins
Test parallelization using Jenkins
 
Running productioninstance 1-localcopy
Running productioninstance 1-localcopyRunning productioninstance 1-localcopy
Running productioninstance 1-localcopy
 
DevOps Fest 2020. Kohsuke Kawaguchi. GitOps, Jenkins X & the Future of CI/CD
DevOps Fest 2020. Kohsuke Kawaguchi. GitOps, Jenkins X & the Future of CI/CDDevOps Fest 2020. Kohsuke Kawaguchi. GitOps, Jenkins X & the Future of CI/CD
DevOps Fest 2020. Kohsuke Kawaguchi. GitOps, Jenkins X & the Future of CI/CD
 
Quality Assurance in a DevOps World
Quality Assurance in a DevOps WorldQuality Assurance in a DevOps World
Quality Assurance in a DevOps World
 
Php com con-2011
Php com con-2011Php com con-2011
Php com con-2011
 
Building XWiki
Building XWikiBuilding XWiki
Building XWiki
 
Jumping from Continuous Integration to Continuous Delivery with Jenkins Enter...
Jumping from Continuous Integration to Continuous Delivery with Jenkins Enter...Jumping from Continuous Integration to Continuous Delivery with Jenkins Enter...
Jumping from Continuous Integration to Continuous Delivery with Jenkins Enter...
 

More from zeeg

Practicing Continuous Deployment
Practicing Continuous DeploymentPracticing Continuous Deployment
Practicing Continuous Deploymentzeeg
 
Tools for Development and Debugging in Python
Tools for Development and Debugging in PythonTools for Development and Debugging in Python
Tools for Development and Debugging in Pythonzeeg
 
PyCon 2011 Scaling Disqus
PyCon 2011 Scaling DisqusPyCon 2011 Scaling Disqus
PyCon 2011 Scaling Disquszeeg
 
Sentry (SF Python, Feb)
Sentry (SF Python, Feb)Sentry (SF Python, Feb)
Sentry (SF Python, Feb)zeeg
 
DjangoCon 2010 Scaling Disqus
DjangoCon 2010 Scaling DisqusDjangoCon 2010 Scaling Disqus
DjangoCon 2010 Scaling Disquszeeg
 
Db tips & tricks django meetup
Db tips & tricks django meetupDb tips & tricks django meetup
Db tips & tricks django meetupzeeg
 

More from zeeg (6)

Practicing Continuous Deployment
Practicing Continuous DeploymentPracticing Continuous Deployment
Practicing Continuous Deployment
 
Tools for Development and Debugging in Python
Tools for Development and Debugging in PythonTools for Development and Debugging in Python
Tools for Development and Debugging in Python
 
PyCon 2011 Scaling Disqus
PyCon 2011 Scaling DisqusPyCon 2011 Scaling Disqus
PyCon 2011 Scaling Disqus
 
Sentry (SF Python, Feb)
Sentry (SF Python, Feb)Sentry (SF Python, Feb)
Sentry (SF Python, Feb)
 
DjangoCon 2010 Scaling Disqus
DjangoCon 2010 Scaling DisqusDjangoCon 2010 Scaling Disqus
DjangoCon 2010 Scaling Disqus
 
Db tips & tricks django meetup
Db tips & tricks django meetupDb tips & tricks django meetup
Db tips & tricks django meetup
 

Recently uploaded

QCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesQCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesBernd Ruecker
 
Generative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptxGenerative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptxfnnc6jmgwh
 
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...panagenda
 
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical InfrastructureVarsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructureitnewsafrica
 
All These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFAll These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFMichael Gough
 
Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#Karmanjay Verma
 
WomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneWomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneUiPathCommunity
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Farhan Tariq
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch TuesdayIvanti
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI AgeCprime
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra
 
Accelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with PlatformlessAccelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with PlatformlessWSO2
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integrationmarketing932765
 
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesAssure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesThousandEyes
 
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...BookNet Canada
 
Digital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentDigital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentMahmoud Rabie
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024TopCSSGallery
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Nikki Chapple
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfNeo4j
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Hiroshi SHIBATA
 

Recently uploaded (20)

QCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesQCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architectures
 
Generative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptxGenerative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptx
 
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
 
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical InfrastructureVarsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
 
All These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFAll These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDF
 
Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#
 
WomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneWomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyone
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch Tuesday
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI Age
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
 
Accelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with PlatformlessAccelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with Platformless
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
 
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesAssure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
 
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
 
Digital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentDigital Tools & AI in Career Development
Digital Tools & AI in Career Development
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdf
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024
 

Pitfalls of Continuous Deployment

  • 1. DISQUS Continuous Deployment Everything David Cramer @zeeg Wednesday, June 22, 2011
  • 2. Continuous Deployment Shipping new code as soon as it’s ready (It’s really just super awesome buildbots) Wednesday, June 22, 2011
  • 3. Workflow Commit (master) Integration Failed Build Deploy Reporting Rollback Wednesday, June 22, 2011
  • 4. Pros Cons • Develop features • Culture Shock incrementally • Stability depends on • Release frequently test coverage • Smaller doses of QA • Initial time investment We mostly just care about iteration and stability Wednesday, June 22, 2011
  • 6. Development • Production > Staging > CI > Dev • Automate testing of complicated processes and architecture • Simple > complete • Especially for local development • python setup.py {develop,test} • Puppet, Chef, simple bootstrap.{py,sh} Wednesday, June 22, 2011
  • 7. Production Staging • PostgreSQL • PostgreSQL • Memcache • Memcache • Redis • Redis • Solr • Solr • Apache • Apache • Nginx • Nginx • RabbitMQ • RabbitMQ CI Server Macbook • Memcache • PostgreSQL • PostgreSQL • Apache • Redis • Memcache • Solr • Redis • Apache • Solr • Nginx • Nginx • RabbitMQ • RabbitMQ Wednesday, June 22, 2011
  • 8. Bootstrapping Local • Simplify local setup • git clone dcramer@disqus:disqus.git • ./bootstrap.sh • python manage.py runserver • Need to test dependancies? • virtualbox + vagrant up Wednesday, June 22, 2011
  • 9. “Under Construction” • Iterate quickly by hiding features • Early adopters are free QA from gargoyle import gargoyle def my_view(request): if gargoyle.is_active('awesome', request): return 'new happy version :D' else: return 'old sad version :(' Wednesday, June 22, 2011
  • 10. Gargoyle Deploy features to portions of a user base at a time to ensure smooth, measurable releases Being users of our product, we actively use early versions of features before public release Wednesday, June 22, 2011
  • 11. Conditions in Gargoyle from gargoyle import gargoyle from gargoyle.conditions import ModelConditionSet, Percent, String class UserConditionSet(ModelConditionSet): # percent implicitly maps to ``id`` percent = Percent() username = String() def can_execute(self, instance): return isinstance(instance, User) # register with our main gargoyle instance gargoyle.register(UserConditionSet(User)) Wednesday, June 22, 2011
  • 12. Without Gargoyle SWITCHES = { # enable my_feature for 50% 'my_feature': range(0, 50), } def is_active(switch): try: pct_range = SWITCHES[switch] except KeyError: return False ip_hash = sum([int(x) for x in ip_address.split('.')]) return (ip_hash % 100 in pct_range) If you use Django, use Gargoyle Wednesday, June 22, 2011
  • 13. Integration (or as we like to call it) Wednesday, June 22, 2011
  • 14. Integration is Required Deploy only when things wont break Wednesday, June 22, 2011
  • 15. Setup a Jenkins Build Wednesday, June 22, 2011
  • 17. CI Requirements • Developers must know when they’ve broken something • IRC, Email, IM • Support proper reporting • XUnit, Pylint, Coverage.py • Painless setup • apt-get install jenkins * https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins+on+Ubuntu Wednesday, June 22, 2011
  • 18. Shortcomings • False positives lower awareness • Reporting isn't accurate • Services fail • Bad Tests • Not enough code coverage • Regressions on untested code • Test suite takes too long • Integration tests vs Unit tests • SOA, distribution Wednesday, June 22, 2011
  • 19. Fixing False Positives • Re-run tests several times on a failure • Report continually failing tests • Fix continually failing tests • Rely less on 3rd parties • Mock/Dingus Wednesday, June 22, 2011
  • 20. Maintaining Coverage • Raise awareness with reporting • Fail/alert when coverage drops on a build • Commit tests with code • Coverage against commit di for untested regressions • Drive it into your culture Wednesday, June 22, 2011
  • 21. Speeding Up Tests • Write true unit tests • vs slower integration tests • Mock 3rd party APIs • Distributed and parallel testing • http://github.com/disqus/mule Wednesday, June 22, 2011
  • 22. Mule • Unstable, will change a lot • Mostly Django right now • Generic interfaces for unittest2 • Works with multi-processing and Celery • Full XUnit integration • Simple workflow • mule test --runner="python manage.py mule --worker $TEST" Wednesday, June 22, 2011
  • 24. How DISQUS Does It • Incremental deploy with Fabric • Drop server from pool • Pull in requirements on each server • Isolated virtualenv’s built on each server • Push server back online Wednesday, June 22, 2011
  • 25. How You Can Do It # fabfile.py from fabric.api import * def deploy(revision): # update sources, virtualenv, requirements # ... # copy ``current`` to ``previous`` run('cp -R %(path)s/current %(path)s/previous' % dict( path=env.path, revision=revision, )) # symlink ``revision`` to ``current`` run('ln -fs %(path)s/%(revision)s %(path)s/current' % dict( path=env.path, revision=revision, )) # restart apache run('touch %(path)s/current/django.wsgi') Wednesday, June 22, 2011
  • 26. How YOU Can Do It (cont.) # fabfile.py from fabric.api import * def rollback(revision=None): # move ``previous`` to ``current`` run('mv %(path)s/previous %(path)s/current' % dict( path=env.path, revision=revision, )) # restart apache run('touch %(path)s/current/django.wsgi') Wednesday, June 22, 2011
  • 27. Challenges • PyPi works on server A, but not B • Scale • CPU cost per server • Schema changes, data model changes • Backwards compatibility Wednesday, June 22, 2011
  • 28. PyPi is Down • http://github.com/disqus/chishop Wednesday, June 22, 2011
  • 29. Help, we have 100 servers! • Incremental (ours) vs Fanout • Push vs Pull • Twitter uses BitTorrent • Isolation vs Packaging (Complexity) Wednesday, June 22, 2011
  • 30. SQL Schema Changes 1. Add column (NULLable) 2. Add app code to fill column 3.Deploy 4.Backfill column 5. Add app code to read column 6.Deploy Wednesday, June 22, 2011
  • 31. Updating Caches • Have a global version number • CACHE_PREFIX = 9000 • Have a data model cache version • sha1(cls.__dict__) • Use multiple caches Wednesday, June 22, 2011
  • 34. <You> Why is mongodb-1 down? <Ops> It’s down? Must have crashed again Wednesday, June 22, 2011
  • 35. Meaningful Metrics • Rate of tra c (not just hits!) • Business vs system • Response time (database, web) • Exceptions • Social media • Twitter Wednesday, June 22, 2011
  • 36. Standard Tools Nagios Graphite Wednesday, June 22, 2011
  • 37. Using Graphite # statsd.py # requires python-statsd from pystatsd import Client import socket def with_suffix(key): hostname = socket.gethostname().split('.')[0] return '%s.%s' % (key, hostname) client = Client(host=STATSD_HOST, port=STATSD_PORT) # statsd.incr('key1', 'key2') def incr(*keys): keys = [with_suffix(k) for k in keys]: client.increment(*keys): Wednesday, June 22, 2011
  • 38. Using Graphite (cont.) (Tra c across a cluster of servers) Wednesday, June 22, 2011
  • 39. Logging • Realtime • Aggregates • History • Notifications • Scalable • Available • Metadata Wednesday, June 22, 2011
  • 40. Logging: Syslog ✓ Realtime x Aggregates ✓ History x Notifications ✓ Scalable ✓ Available x Metadata Wednesday, June 22, 2011
  • 41. Logging: Email Collection ✓ Realtime x Aggregates ✓ History x Notifications x Scalable ✓ Available ✓ Metadata (Django provides this out of the box) Wednesday, June 22, 2011
  • 42. Logging: Sentry ✓ Realtime ✓ Aggregates ✓ History ✓ Notifications ✓ Scalable ✓ Available ✓ Metadata http://github.com/dcramer/django-sentry Wednesday, June 22, 2011
  • 43. Setting up Sentry (1.x) # setup your server first $ pip install django-sentry $ sentry start # configure your Python (Django in our case) client INSTALLED_APPS = ( # ... 'sentry.client', ) # point the client to the servers SENTRY_REMOTE_URL = ['http://sentry/store/'] # visit http://sentry in the browser Wednesday, June 22, 2011
  • 44. Setting up Sentry (cont.) # ~/.sentry/sentry.conf.py # use a better database DATABASES = { 'default': { 'ENGINE': 'postgresql_psycopg2', 'NAME': 'sentry', 'USER': 'postgres', } } # bind to all interfaces SENTRY_WEB_HOST = '0.0.0.0' # change data paths SENTRY_WEB_LOG_FILE = '/var/log/sentry.log' SENTRY_WEB_PID_FILE = '/var/run/sentry.pid' Wednesday, June 22, 2011
  • 47. Getting Started • Package your app • Ease deployment; fast rollbacks • Setup automated tests • Gather some easy metrics Wednesday, June 22, 2011
  • 48. Going Further • Build an immune system • Automate deploys, rollbacks (maybe) • Adjust to your culture • CD doesn’t “just work” • SOA == great success Wednesday, June 22, 2011
  • 49. DISQUS Questions? psst, we’re hiring jobs@disqus.com Wednesday, June 22, 2011
  • 50. References • Gargoyle (feature switches) https://github.com/disqus/gargoyle • Sentry (log aggregation) https://github.com/dcramer/django-sentry (1.x) https://github.com/dcramer/sentry (2.x) • Jenkins CI http://jenkins-ci.org/ • Mule (distributed test runner) https://github.com/disqus/mule code.disqus.com Wednesday, June 22, 2011