DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)


Published on

Leo Zhandovsky (@leozh, formerly DNC, now AWS Solutions Architect) and I gave a brief talk about DevOps at the DNC and OFA campaign for DevOps Days in New York City.
Video (Starts ~1hr55min) -

Published in: Technology

DevOps at Obama for America(2012) and the DNC (DevOps Days NYC Jan 2013)

  1. 1. DevOps @DevOps @DNC & OFADNC & OFA
  2. 2. John Schneider Leo Zhadanovsky@jdotp @leozhFormerly Obama for America Director of Systems EngineeringCurrently DevOps / Internet @Jedi @ Mozilla Foundation Democratic National Committee
  3. 3. DevOps @ DNC - 2009 - 2011Year 1● Cleanup● Virtualization● Documentation● Version Control● Making everything HA● Replacing all our network gear● Automation
  4. 4. Labs Development Team● Created first versions of Call Tool, Voter Registration Site, Canvassing iPhone App
  5. 5. Labs Development TeamStart of DevOps Practices at the DNC● Agile Development● Usage of EC2● Continuous Integration● Deploys using Capistrano● Scaling up using custom AMIs● Monitoring using Nagios, Cacti, NewRelic, Graphite, Pingdom, StatsD
  6. 6. Analytics Team● Lessons learned from 2010 used in 2012● Big Data● Hardware vs AWS● Limitations of AWS at the time● What would we do differently now?● Developed software for data analysts, modelers and field staff
  7. 7. Cloud (AWS)● Unpredictable traffic patterns: No problem! (We had no clue those birther mugs would sell so well.)● Makes entire org far more agile● Apples to apples testing envs are a snap● Cloud rules changes the way we manage systems
  8. 8. Dev + Ops = <3 The New Mentality of Devops SOA / 12 Factor● Allow for easy migration, deployment● All components independently scaled without retooling or re-architecting● Critical for building scalable, high-availability apps in the cloud.● Goes well with Route53.
  9. 9. Dev + Ops = <3 The New Mentality of Devops Agile and Continuous● Tool agnostic (we used Pivatol, Basecamp Classic, Basecamp Next, spreadsheets, github issues, zendesk tickets, etc)● Small and fast releases. (we pushed ~40 new versions per day in the last weekend)● SHIPIT! Build good enough tools and iterate.
  10. 10. Dev + Ops = <3 The New Mentality of Devops Instrument Everything (and trust the data)● CPU, traffic, response time, etc● User/System events● Deployment-related change metrics● Logs are data too!
  11. 11. Dev + Ops = <3 The New Mentality of Devops Visible Ops (aka, how to turn PMs, Devs, Management into Ops partners)● They all get Nagios, NewRelic, and Pagerduty accounts and training● Activity feed of commits, deploys, alerts, changes.● Cross-pollinate team stands, retros● Give everyone contextual data
  12. 12. Who doesnt love color charts?
  13. 13. Load Testing with NewRelic Deploys
  14. 14. Dev + Ops = <3 The New Mentality of Devops Fail Early, Fail Fast, Fail Well● It is not if, but when.● Learn how to detect & mitigate, and then iterate.● Reduce the time to detect, the time to resolve.● Preventing all failure is not the DevOps role, making failure hurt less or not hurt at all is DevOps role
  15. 15. Yo, mind if we bring the donationsdb down during the VP debate?
  16. 16. Oh Hai East Virginia Datacenter!
  17. 17. Other AWS Tips-Use EBS-backed instances, AMI-baseddeploys, and provisioned IOPS.-Use SQS.-Multi-AZ, Multi-Region early-Autoscale from any metric, push your ownmetrics to Cloudwatch even-Setup ifttt or zapier alerts on AWS statuspages-Check out billing broken out by tag, andTrustedAdvisor
  18. 18. S3 microsites are awesome
  19. 19. Other Things That HelpedNew Relic (We love you New Relic)Asgard (Thanks Netflix)StatsD/Graphite and CubismLogglyBeesWithMachineGunsAkamai DSA and GTMBotoPalominoDB is awesome!Lots of caffeineLastly....culture, culture, culture.
  20. 20. Election Day