Obama For America on AWS

Younjin Jeong
Solutions Architect
What am I talking about today?
What was OFA? Why is this relevant?
• Who did it?
• What did they build?

How did they do t...
Full Disclosure
I work for AWS
AWS does not endorse
political candidates
Yes, I talk too much
So here’s the Idea
~30th biggest E-commerce operation, globally
~200 distinct new applications, many mobile
Hundreds of ne...
a few constraints…
~30th biggest E-commerce operation, globally
~200 distinct applications, many mobile
Hundreds of new, u...
CHALLENGE ACCEPTED !
Built by guys and gals like these: Obama For America
Business as usual..

…for a technology startup
Election Day – OFA Headquarters
So they built it all, and it worked
Typical Charts
How?
The old approach, even from Amazon 
The old approach.. Might have some problems..
Cloud Computing Benefits
No Up-Front
Capital Expense

Low Cost

Pay Only for
What You Use

Self-Service
Infrastructure

Ea...
OFA’s Infrastructure

awsofa.info
Web-Scale Applications
500k+ IOPS DB Systems
Services API
Ingredients
Ubuntu nginx boundary Unity jQuery SQLServer hbase
NewRelic EC2 node.js Cybersource hive ElasticSearch
Ruby Tw...
Data Stores
Ubuntu nginx boundary Unity jQuery SQLServer hbase
NewRelic EC2 node.js Cybersource hive ElasticSearch
Ruby Tw...
Development Frameworks
Ubuntu nginx boundary Unity jQuery SQLServer hbase
NewRelic EC2 node.js Cybersource hive ElasticSea...
Sites

Communications
Ad Targeting
Ops Tools
Analytics
Apps

Micro-targeting
Micro-listening
Reporting
Registrations
Volun...
Technology Choice
Polyglot Development
Cloud Hosting

Expected Tradeoff
More Complex Ops

Diverse, App-centered
Databases
...
Technology Choice
Polyglot
Development
Cloud Hosting
Diverse, Appcentered Databases
SOA, queue-based
system integrations

...
No time to waste
This applies to lots of services!
ELB
ElastiCache
RDS
CloudSearch
Route53
S3
CloudFront
DynamoDB

You can mostly do
these ...
Looks pretty simple.

Inserts 7.5m records in DynamoDB, in 8 minutes
One thing that is difficult to prepare for…
No pressure…
They had this built for the previous 3
months, all on the East Coast.
They had this built for the previous 3
months, all on the East Coast.

We built this
part in 9 hours
to be safe.

AWS +
Pu...
Replication across the continent..

http://tsunami-udp.sourceforge.net/
So what did they learn?
Game Day: Practice failures so you know what to do.
Loose-Coupling: Ops easy, scale easy, test eas...
What will you do next?
Maybe look at some of their Ruby code?

https://github.com/democrats/voter-registration
AMAZON REDSHIFT
AMAZON REDSHIFT
Redshift runs on HS type instances

HS1.8XL: 128 Go RAM, 16 Coeurs, 16 To de contenu compressé, 2 Go/sec e...
Extra Large Node
(HS1.XL)

Single node
Cluster 2-32 Nodes (4 To – 64 To)

Eight Extra Large Node (HS1.8XL)
Cluster 2-100 N...
JDBC/ODBC

10 GigE
(HPC)

Ingestion
Backup
Restoration

Amazon DynamoDB
AMAZON EC2

AMAZON
DYNAMODB

AMAZON RDS

AMAZON ELASTIC
MAPREDUCE

AMAZON
REDSHIFT

AMAZON S3

AWS STORAGE
GATEWAY

DATA C...
Thank you!

Younjin Jeong - AWS
younjin@amazon.com
[판교에서 만나는 아마존웹서비스] Obama for America를 통해서 본 AWS에서의 데이터 분석
[판교에서 만나는 아마존웹서비스] Obama for America를 통해서 본 AWS에서의 데이터 분석
[판교에서 만나는 아마존웹서비스] Obama for America를 통해서 본 AWS에서의 데이터 분석
[판교에서 만나는 아마존웹서비스] Obama for America를 통해서 본 AWS에서의 데이터 분석
[판교에서 만나는 아마존웹서비스] Obama for America를 통해서 본 AWS에서의 데이터 분석
[판교에서 만나는 아마존웹서비스] Obama for America를 통해서 본 AWS에서의 데이터 분석
[판교에서 만나는 아마존웹서비스] Obama for America를 통해서 본 AWS에서의 데이터 분석
[판교에서 만나는 아마존웹서비스] Obama for America를 통해서 본 AWS에서의 데이터 분석
[판교에서 만나는 아마존웹서비스] Obama for America를 통해서 본 AWS에서의 데이터 분석
[판교에서 만나는 아마존웹서비스] Obama for America를 통해서 본 AWS에서의 데이터 분석
[판교에서 만나는 아마존웹서비스] Obama for America를 통해서 본 AWS에서의 데이터 분석
Upcoming SlideShare
Loading in...5
×

[판교에서 만나는 아마존웹서비스] Obama for America를 통해서 본 AWS에서의 데이터 분석

709

Published on

Obama for America를 통해서 본 AWS에서의 데이터 분석 (정윤진 책임, Solutions Architect)

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
709
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
25
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Transcript of "[판교에서 만나는 아마존웹서비스] Obama for America를 통해서 본 AWS에서의 데이터 분석 "

  1. 1. Obama For America on AWS Younjin Jeong Solutions Architect
  2. 2. What am I talking about today? What was OFA? Why is this relevant? • Who did it? • What did they build? How did they do that? • Technologies and Tradeoffs • Services vs. Software What did they learn from building something so big?
  3. 3. Full Disclosure I work for AWS AWS does not endorse political candidates Yes, I talk too much
  4. 4. So here’s the Idea ~30th biggest E-commerce operation, globally ~200 distinct new applications, many mobile Hundreds of new, untested analytical approaches Processing hundreds of TB of data on thousands of servers Spikes of hundreds of thousands of concurrent users FUN FUN FUN
  5. 5. a few constraints… ~30th biggest E-commerce operation, globally ~200 distinct applications, many mobile Hundreds of new, untested analytical approaches Processing hundreds of TB of data on thousands of servers Spikes of hundreds of thousands of concurrent users Critically compressed budget Less than a year to execute Volunteer and near-volunteer development team Core systems will be used for a single critical day Constitutionally-mandated completion date NOT NOT
  6. 6. CHALLENGE ACCEPTED !
  7. 7. Built by guys and gals like these: Obama For America
  8. 8. Business as usual.. …for a technology startup
  9. 9. Election Day – OFA Headquarters
  10. 10. So they built it all, and it worked
  11. 11. Typical Charts
  12. 12. How?
  13. 13. The old approach, even from Amazon 
  14. 14. The old approach.. Might have some problems..
  15. 15. Cloud Computing Benefits No Up-Front Capital Expense Low Cost Pay Only for What You Use Self-Service Infrastructure Easily Scale Up and Down Improve Agility & Time-to-Market Deploy
  16. 16. OFA’s Infrastructure awsofa.info
  17. 17. Web-Scale Applications
  18. 18. 500k+ IOPS DB Systems
  19. 19. Services API
  20. 20. Ingredients Ubuntu nginx boundary Unity jQuery SQLServer hbase NewRelic EC2 node.js Cybersource hive ElasticSearch Ruby Twilio EE S3 ELB boto Magento PHP EMR SES Route53 SimpleDB Campfire nagios Paypal CentOS CloudSearch levelDB mongoDB python securitygroups Usahidhi PostgresSQL Github apache bootstrap SNS cloudformation Jekyll RoR EBS FPS VPC Mashery Vertica RDS Optimizely MySQL puppet tsunamiUDP R asgard cloudwatch ElastiCache cloudopt SQS cloudinit DirectConnect BSD rsync STS Objective-C dynamoDB
  21. 21. Data Stores Ubuntu nginx boundary Unity jQuery SQLServer hbase NewRelic EC2 node.js Cybersource hive ElasticSearch Ruby Twilio EE S3 ELB boto Magento PHP EMR SES Route53 SimpleDB Campfire nagios Paypal CentOS CloudSearch levelDB mongoDB python securitygroups Usahidhi PostgresSQL Github apache bootstrap SNS cloudformation Jekyll RoR EBS FPS VPC Mashery Vertica RDS Optimizely MySQL puppet tsunamiUDP R asgard cloudwatch ElastiCache cloudopt SQS cloudinit DirectConnect BSD rsync STS Objective-C dynamoDB
  22. 22. Development Frameworks Ubuntu nginx boundary Unity jQuery SQLServer hbase NewRelic EC2 node.js Cybersource hive ElasticSearch Ruby Twilio EE S3 ELB boto Magento PHP EMR SES Route53 SimpleDB Campfire nagios Paypal CentOS CloudSearch levelDB mongoDB python securitygroups Usahidhi PostgresSQL Github apache bootstrap SNS cloudformation Jekyll RoR EBS FPS VPC Mashery Vertica RDS Optimizely MySQL puppet tsunamiUDP R asgard cloudwatch ElastiCache cloudopt SQS cloudinit DirectConnect BSD rsync STS Objective-C dynamoDB
  23. 23. Sites Communications Ad Targeting Ops Tools Analytics Apps Micro-targeting Micro-listening Reporting Registrations Volunteer Coordination Etc, etc, etc.
  24. 24. Technology Choice Polyglot Development Cloud Hosting Expected Tradeoff More Complex Ops Diverse, App-centered Databases Less Infra Control, performance More Complex Ops, Fragility, Data Corruption SOA, queue-based system integrations Dev Complexity, slower system performance
  25. 25. Technology Choice Polyglot Development Cloud Hosting Diverse, Appcentered Databases SOA, queue-based system integrations Expected Tradeoff More Complex Ops Upside Build as little as possible, rev-1 faster, reuse dev skills Less Infra Control, performance More Complex Ops, Fragility, Data Corruption Scale, Speed, Cost Dev Complexity, slower system performance Scalability, serviceability, operational flexibility, and substantially faster in aggregate Heterogeneous Resilience, right tools for the job
  26. 26. No time to waste
  27. 27. This applies to lots of services! ELB ElastiCache RDS CloudSearch Route53 S3 CloudFront DynamoDB You can mostly do these on your own… But do you have extra: focus, expertise, time, research, money, risk-tolerance, staff, dedication to innovate, operations coverage, scalability in design...
  28. 28. Looks pretty simple. Inserts 7.5m records in DynamoDB, in 8 minutes
  29. 29. One thing that is difficult to prepare for…
  30. 30. No pressure…
  31. 31. They had this built for the previous 3 months, all on the East Coast.
  32. 32. They had this built for the previous 3 months, all on the East Coast. We built this part in 9 hours to be safe. AWS + Puppet + Netflix Asgard + CloudOpt + DevOps = Cross-Continent FaultTolerance On-Demand
  33. 33. Replication across the continent.. http://tsunami-udp.sourceforge.net/
  34. 34. So what did they learn? Game Day: Practice failures so you know what to do. Loose-Coupling: Ops easy, scale easy, test easy, fix easy… Fail-Forward: features, quality, and focus are all critical. HA in Depth: S3 static pages, de-coupled UI, jekyll/hyde Cloud works.
  35. 35. What will you do next?
  36. 36. Maybe look at some of their Ruby code? https://github.com/democrats/voter-registration
  37. 37. AMAZON REDSHIFT
  38. 38. AMAZON REDSHIFT Redshift runs on HS type instances HS1.8XL: 128 Go RAM, 16 Coeurs, 16 To de contenu compressé, 2 Go/sec en lecture HS1.XL: 16 Go RAM, 2 Coeurs, 2 To de contenu compressé
  39. 39. Extra Large Node (HS1.XL) Single node Cluster 2-32 Nodes (4 To – 64 To) Eight Extra Large Node (HS1.8XL) Cluster 2-100 Nodes (32 To – 1.6 Po)
  40. 40. JDBC/ODBC 10 GigE (HPC) Ingestion Backup Restoration Amazon DynamoDB
  41. 41. AMAZON EC2 AMAZON DYNAMODB AMAZON RDS AMAZON ELASTIC MAPREDUCE AMAZON REDSHIFT AMAZON S3 AWS STORAGE GATEWAY DATA CENTER
  42. 42. Thank you! Younjin Jeong - AWS younjin@amazon.com
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×