The Road to the White House with Puppet & AWS

  • 5,268 views
Uploaded on

Learn how the Obama campaign leveraged Amazon Web Services (AWS) and Puppet to rapidly scale their infrastructure up for the needs of the election in a sustainable manner. Using the automation that …

Learn how the Obama campaign leveraged Amazon Web Services (AWS) and Puppet to rapidly scale their infrastructure up for the needs of the election in a sustainable manner. Using the automation that AWS and Puppet enabled -- the Obama campaign build a significant AWS infrastructure (http://awsofa.info) while having a lean DevOps team, tight deadlines and applications that needed to be highly available. Learn about using bootstrapping puppet on Amazon EC2 instances with CloudInit, using it with autoscaling groups and secure handling of credentials in manifests. Find out how to scale puppet masters and take advantage of Amazon S3 backed RPM/Debian repos with them.

Leo Zhadanovsky
Senior Solutions Architect, Amazon Web Services
Leo Zhadanovsky is a Senior Solutions Architect at Amazon Web Services. He helps customers best leverage AWS services, in order to help them succeed in building highly-available, scalable and elastic architectures for their business needs. He was previously the Director of Systems Engineering at the Democratic National Committee. From 2009 to early 2013, he ran the DNC's physical server and cloud footprint and supported infrastructure which was in use by the Obama campaign, state and local Democratic parties. In 2010, the DNC successfully ran and deployed many applications, such as a Call Tool and Voter Registration website, that were written in Ruby and ran on AWS. In 2012, the DNC supported the Obama campaign with various backend APIs, web sites, voter file databases and a large data warehouse.

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
5,268
On Slideshare
0
From Embeds
0
Number of Embeds
7

Actions

Shares
Downloads
67
Comments
0
Likes
9

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • Not your normal technology professionals
  • Not your normal office environment
  • A few friends in high places
  • Cloud computing is a better way to run your business. The cloud helps companies of all sizesbecome moreagile. Instead of running your applications yourself you can run them on the cloud where IT infrastructure is offered as a service like a utility. With the cloud, your company saves money: there are no up-front capital expenses as you don’t have to buy hardware for your projects. The massive scale and fast pace of innovation of the cloud drive the costs down for you. In the cloud, you pay only for what you use just like electricity.The cloud can also help your company save time and improve agility – it’s faster to get started: you can build new environments in minutes as you don’t need to wait for new servers to arrive. The elastic nature of the cloud makes it easy to scale up and down as needed. At the end of the day you have more resources left for innovation which allows you to focus on projects that can really impact your businesses like building and deploying more applications. “With the high growth nature of our business, we were looking for a cloud solution to enable us to scale fast. Think twice before buying your next server. Cloud computing is the way forward.” - Sami Lababidi, CTO, Playfish

Transcript

  • 1. The Road to the White House with Puppet & AWS Leo Zhadanovsky – Solutions Architect – leo@amazon.com @leozh
  • 2. What am I talking about today? What was OFA Tech? • Who did it? • What did they build? How did they do that? • Technologies and Tradeoffs • Services vs. Software How did they leverage puppet? What did they learn from building something so big?
  • 3. Who Am I? I work for AWS I worked for the DNC 2009-2012 I was embedded at OFA AWS does not endorse political candidates I love Star Trek (TNG is the best)
  • 4. So here’s the Idea ~30th biggest E-commerce operation, globally ~200 distinct new applications, many mobile Hundreds of new, untested analytical approaches Processing hundreds of TB of data on thousands of servers Spikes of hundreds of thousands of concurrent users FUN FUN FUN
  • 5. a few constraints… ~30th biggest E-commerce operation, globally ~200 distinct applications, many mobile Hundreds of new, untested analytical approaches Processing hundreds of TB of data on thousands of servers Spikes of hundreds of thousands of concurrent users Critically compressed budget Less than a year to execute Volunteer and near-volunteer development team Core systems will be used for a single critical day Constitutionally-mandated completion date NOT NOT
  • 6. Built by guys and gals like these: Obama For America
  • 7. Business as usual.. …for a technology startup
  • 8. Election Day – OFA Headquarters
  • 9. So they built it all, and it worked
  • 10. Typical Charts
  • 11. How?
  • 12. The old approach, even from Amazon 
  • 13. The old approach.. Might have some problems..
  • 14. No Up-Front Capital Expense Pay Only for What You Use Self-Service Infrastructure Easily Scale Up and Down Improve Agility & Time-to-Market Low Cost Cloud Computing Benefits Deploy
  • 15. OFA’s Infrastructure awsofa.info
  • 16. Web-Scale Applications
  • 17. 500k+ IOPS DB Systems
  • 18. Services API
  • 19. Ingredients Ubuntu nginx boundary Unity jQuery SQLServer hbase NewRelic EC2 node.js Cybersource hive ElasticSearch Ruby Twilio EE S3 ELB boto Magento PHP EMR SES Route53 SimpleDB Campfire nagios Paypal CentOS CloudSearch levelDB mongoDB python securitygroups Usahidhi PostgresSQL Github apache bootstrap SNS cloudformation Jekyll RoR EBS FPS VPC Mashery Vertica RDS Optimizely MySQL puppet tsunamiUDP R asgard cloudwatch ElastiCache cloudopt SQS cloudinit DirectConnect BSD rsync STS Objective-C DynamoDB
  • 20. Data Stores Ubuntu nginx boundary Unity jQuery SQLServer hbase NewRelic EC2 node.js Cybersource hive ElasticSearch Ruby Twilio EE S3 ELB boto Magento PHP EMR SES Route53 SimpleDB Campfire nagios Paypal CentOS CloudSearch levelDB mongoDB python securitygroups Usahidhi PostgresSQL Github apache bootstrap SNS cloudformation Jekyll RoR EBS FPS VPC Mashery Vertica RDS Optimizely MySQL puppet tsunamiUDP R asgard cloudwatch ElastiCache cloudopt SQS cloudinit DirectConnect BSD rsync STS Objective-C DynamoDB
  • 21. Development Frameworks Ubuntu nginx boundary Unity jQuery SQLServer hbase NewRelic EC2 node.js Cybersource hive ElasticSearch Ruby Twilio EE S3 ELB boto Magento PHP EMR SES Route53 SimpleDB Campfire nagios Paypal CentOS CloudSearch levelDB mongoDB python securitygroups Usahidhi PostgresSQL Github apache bootstrap SNS cloudformation Jekyll RoR EBS FPS VPC Mashery Vertica RDS Optimizely MySQL puppet tsunamiUDP R asgard cloudwatch ElastiCache cloudopt SQS cloudinit DirectConnect BSD rsync STS Objective-C DynamoDB
  • 22. Infrastructure, Configuration Management & Monitoring Ubuntu nginx boundary Unity jQuery SQLServer hbase NewRelic EC2 node.js Cybersource hive ElasticSearch Ruby Twilio EE S3 ELB boto Magento PHP EMR SES Route53 SimpleDB Campfire nagios Paypal CentOS CloudSearch levelDB mongoDB python securitygroups Usahidhi PostgresSQL Github apache bootstrap SNS cloudformation Jekyll RoR EBS FPS VPC Mashery Vertica RDS Optimizely MySQL puppet tsunamiUDP R asgard cloudwatch ElastiCache cloudopt SQS cloudinit DirectConnect BSD rsync STS Objective-C DynamoDB
  • 23. Configuration Management: Puppet In mid-2011, we look at options for configuration management and chose Puppet We needed to make it scale, and to get it to work with state- less, horizontally scalable infrastructure How did we do this?
  • 24. Bootstrapping Puppet with CloudInit CloudInit is built into Ubuntu and Amazon Linux • Allows you to pass bootstrap parameters in Amazon EC2 user-data field, in YAML format
  • 25. Bootstrapping Puppet with CloudInit Don’t store creds in puppet manifests, store them in private Amazon S3 buckets Either pass Amazon S3 creds through CloudInit: Even better – avoid this by using AWS Identity and Access Management (IAM) roles and the version of s3cmd in github
  • 26. Bootstrapping Puppet with CloudInit Built-in puppet support Use certname with %i for instance id to name the node Puppetmaster must have auto sign turned on • Use security groups and/or NACLs for network-level security In nodes.pp, use regex to match node names
  • 27. Puppet Tips Use a base class to define your standard install
  • 28. Use runstages Don’t store credentials in puppet, store them in private Amazon S3 buckets • Use AWS IAM to secure the credentials bucket/folders within that bucket Puppet Tips
  • 29. Puppet Tips Use puppet only for configuration files and what makes your apps unique For undifferentiated parts of apps, use Amazon S3 backed RPM/Debian repositories • Can be either public or private repos, depending on your needs • Amazon S3 Private RPM Repos: http://git.io/YAcsbg • Amazon S3 Private Debian Repos: http://git.io/ecCjWQ
  • 30. Puppet Tips By using packages for applications deploys, you can set ensure => latest, and just bump the package in the repo to update Log everything with rsyslog/graylog/loggly/NewRelic/splunk
  • 31. Scaling the Puppet Masters Use an Auto Scaling group for puppet masters • Min size => 2, use multiple Availability Zones Either have them build themselves off of existing puppet masters in the group or off packages storied in Amazon S3 and bootstrapped through user-data Auto-sign must be on
  • 32. Sites Communications Ad Targeting Ops Tools Analytics Apps Micro-targeting Micro-listening Reporting Registrations Volunteer Coordination Etc, etc, etc.
  • 33. Technology Choice Polyglot Development Cloud Hosting Diverse, App-centered Databases SOA, queue-based system integrations Expected Tradeoff More Complex Ops Less Infra Control, performance More Complex Ops, Fragility, Data Corruption Dev Complexity, slower system performance
  • 34. Technology Choice Polyglot Development Cloud Hosting Diverse, App- centered Databases SOA, queue-based system integrations Expected Tradeoff More Complex Ops Less Infra Control, performance More Complex Ops, Fragility, Data Corruption Dev Complexity, slower system performance Upside Build as little as possible, rev-1 faster, reuse dev skills Scale, Speed, Cost Heterogeneous Resilience, right tools for the job Scalability, serviceability, operational flexibility, and substantially faster in aggregate
  • 35. $5.2B retail business 7,800 employees A whole lot of servers 2003
  • 36. 2012 Every day, AWS adds enough server capacity to power this $5B enterprise $5.2B retail business 7,800 employees A whole lot of servers 2003
  • 37. 2012 Amazon Simple Queuing Service (SQS) Thousands of customers A whole lot of servers Over 5 Billion Queued Events 2006-8
  • 38. 2012 OFA Produced 8.4 Billion Amazon SQS Queued Events Amazon Simple Queuing Service (SQS) Thousands of customers A whole lot of servers Over 5 Billion Queued Events 2006-8
  • 39. 2012 OFA Produced 8.4 Billion Amazon SQS Queued Events Just the last month of the campaign 2006-8 Amazon Simple Queuing Service (SQS) Thousands of customers A whole lot of servers Over 5 Billion Queued Events
  • 40. No time to waste
  • 41. This applies to lots of services! Elastic Load Balancing Amazon ElastiCache Amazon RDS Amazon CloudSearch Amazon Route53 Amazon S3 Amazon CloudFront Amazon DynamoDB You can mostly do these on your own… But do you have extra: focus, expertise, time, research, money, risk-tolerance, staff, dedication to innovate, operations coverage, scalability in design...
  • 42. Looks pretty simple. Inserts 7.5m records in Amazon DynamoDB, in 8 minutes
  • 43. One thing that is difficult to prepare for…
  • 44. No pressure…
  • 45. They had this built for the previous 3 months, all on the East Coast.
  • 46. They had this built for the previous 3 months, all on the East Coast. We built this part in 9 hours to be safe. AWS + Puppet + Netflix Asgard + CloudOpt + DevOps = Cross-Continent Fault- Tolerance On-Demand
  • 47. Replication across the continent.. http://tsunami-udp.sourceforge.net/ 478.18 Mbps cross-continental data transit rate for a single cc2.8xlarge instance 1.72 Tb an hour 27 Tb of data to move 3.92 Hours required to move the data across the continent with four cc2.8xlarge instances
  • 48. So what did they learn? HA in Depth: Amazon S3 static pages, de-coupled UI, jekyll/hyde Game Day: Practice failures so you know what to do. ( http://www.awsgameday.com ) Loose-Coupling: Ops easy, scale easy, test easy, fix easy… Fail-Forward: features, quality, and focus are all critical. Cloud works.
  • 49. We showed it to the world at re: Invent 2012
  • 50. together with the OFA DevOps crew
  • 51. We presented in Tokyo…
  • 52. Born from the Campaign
  • 53. What will you do next?
  • 54. Maybe look at some of their Ruby code?
  • 55. Register Now! reinvent.awsevents.com $200 Off Discount Code: Zoltan2013 Gain New Skills & Knowledge Choose from 175+ technical sessions, training bootcamps, hands-on labs, and hackathons. Dive Deeper into AWS Dive deep into foundational AWS services and learn about the latest services and features. Get Your Questions Answered Get your technical questions answered by AWS architects, engineers, and product leads. Learn Best Practices Discover best practices, tips and tricks, and lessons learned from expert customers.
  • 56. Thank you! Questions? • Come talk to an AWS Solutions Architect at Table 22 Contact me! • @leozh • leo@amazon.com