Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
November 14, 2014 | Las Vegas, NV 
Valentino Volonghi, CTO, AdRoll 
Siva Raghupathy, Principal Solutions Architect, AWS
60 billion requests/day
We 
Must 
Stay 
Up 
1% downtime 
= 
>$1M
No 
Infinitely 
Deep 
Pockets
100ms MAX Latency
Paris-New York: ~6000km 
Speed of Light in fiber: 200,000 km/s 
RTT latency without hops and copper: 60ms 
Paris-New York:...
a 
a 
a 
r 
•Memcache 
aOpensource 
aMature 
aBlazinglyfast 
rNostrong guarantees 
•Redis 
aOpensource 
rStoragescale 
rNo...
http://techblog.netflix.com/2014/07/revisiting-1-million-writes-per-second.html 
Cassandra 
DynamoDB 
Delta 
10/90mix, $/m...
Instance 
Nginx 
ErlangBidder 
Black Box 
Supervisor 
ASG 
Amazon 
Kinesis 
Amazon S3 
DynamoDB 
Elastic Load Balancing 
I...
Amazon S3 
Amazon 
Kinesis 
DynamoDB 
DynamoDB 
DynamoDB
HashKey 
Value 
Key 
{‘segments’: set([‘123’, ‘345’]), 
‘timestamp’:‘1234556’, 
‘leads’: ‘to-write-contention’} 
HashKey 
...
ArrayListListWriteRequestArrayListWriteRequestKeyAtomicIntegerAtomicIntegerSetWriteRequestCollectionsHashSetWriteRequestSt...
Not for me!
Data Collection 
Data Storage 
Global 
Distribution 
Bid Storage 
Bidding
Data Collection 
Data Storage 
Global 
Distribution 
Bid Storage 
Bidding
Data Collection = Batch Layer 
Bidding = Speed Layer 
Data Collection 
Data Storage 
Global 
Distribution 
Bid Storage 
Bi...
US East region 
Availability Zone 
Availability Zone 
Elastic Load Balancing 
instances 
instances 
Auto Scaling group 
Am...
US East region 
Availability Zone Availability Zone 
Elastic Load 
Balancing 
instances instances 
Auto Scaling group 
Ama...
Data Collection Bidding 
US East region 
Availability Zone Availability Zone 
Elastic Load Balancing 
instances instances ...
Data Collection 
•Amazon EC2, Elastic Load Balancing, Auto Scaling 
Store 
•Amazon S3 + Amazon Kinesis 
Global Distributio...
http://bit.ly/awsevals
(ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014
(ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014
(ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014
(ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014
(ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014
(ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014
(ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014
(ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014
(ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014
(ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014
(ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014
(ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014
(ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014
(ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014
(ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014
(ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014
(ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014
(ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014
(ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014
(ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014
(ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014
(ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014
(ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014
Upcoming SlideShare
Loading in …5
×

(ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014

11,036 views

Published on

With Amazon Web Services it's possible to serve the needs of modern high performance advertising without breaking the bank. This session covers how AdRoll processes more than 60 billion requests per day in less than 100 milliseconds each using Amazon DynamoDB, Auto Scaling, and Elastic Load Balancing. This process generates more than 2 GB of data every single second, which will be processed and turned into useful models over the following hour. We discuss designing systems that can answer billions of latency-sensitive global requests every day and look into some tricks to pare down the costs.

Published in: Technology

(ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014

  1. 1. November 14, 2014 | Las Vegas, NV Valentino Volonghi, CTO, AdRoll Siva Raghupathy, Principal Solutions Architect, AWS
  2. 2. 60 billion requests/day
  3. 3. We Must Stay Up 1% downtime = >$1M
  4. 4. No Infinitely Deep Pockets
  5. 5. 100ms MAX Latency
  6. 6. Paris-New York: ~6000km Speed of Light in fiber: 200,000 km/s RTT latency without hops and copper: 60ms Paris-New York: ~6000km Speed of Light in fiber: 200,000 km/s RTT latency without hops and copper: 60ms 6000 km 60 ms c-RTT
  7. 7. a a a r •Memcache aOpensource aMature aBlazinglyfast rNostrong guarantees •Redis aOpensource rStoragescale rNotreally distributed rOperationallyintense. •Hbase(we still use this) aOpensource aMaturingquickly aGreatscale rReallyhard to operate
  8. 8. http://techblog.netflix.com/2014/07/revisiting-1-million-writes-per-second.html Cassandra DynamoDB Delta 10/90mix, $/month $287,064 $131,040 219% 50/50 mix, $/month $287,064 $280,800 ~0% 10/90, 3-yrreserved $27,075.6 ($904kupfront) $15,736 ($504k upfront) 180% •10 people Cassandra ops team: $150k/month (fully loaded) •0 DynamoDBops team: $0
  9. 9. Instance Nginx ErlangBidder Black Box Supervisor ASG Amazon Kinesis Amazon S3 DynamoDB Elastic Load Balancing Instance Nginx ErlangBidder Black Box Supervisor ASG Elastic Load Balancing
  10. 10. Amazon S3 Amazon Kinesis DynamoDB DynamoDB DynamoDB
  11. 11. HashKey Value Key {‘segments’: set([‘123’, ‘345’]), ‘timestamp’:‘1234556’, ‘leads’: ‘to-write-contention’} HashKey RangeKey Value (just a single timestamp) Key Segment 1234554343254 Key Segment1 1231231433235
  12. 12. ArrayListListWriteRequestArrayListWriteRequestKeyAtomicIntegerAtomicIntegerSetWriteRequestCollectionsHashSetWriteRequestStringAmazonDynamoDB RunnableRunnablerun ListWriteRequest BatchWriteItemResultBatchWriteItemRequestImmutableMapListWriteRequest Map
  13. 13. Not for me!
  14. 14. Data Collection Data Storage Global Distribution Bid Storage Bidding
  15. 15. Data Collection Data Storage Global Distribution Bid Storage Bidding
  16. 16. Data Collection = Batch Layer Bidding = Speed Layer Data Collection Data Storage Global Distribution Bid Storage Bidding
  17. 17. US East region Availability Zone Availability Zone Elastic Load Balancing instances instances Auto Scaling group Amazon S3 Amazon Kinesis
  18. 18. US East region Availability Zone Availability Zone Elastic Load Balancing instances instances Auto Scaling group Amazon S3 Amazon Kinesis Apache Storm DynamoDB US West region EU West region DynamoDB DynamoDB
  19. 19. Data Collection Bidding US East region Availability Zone Availability Zone Elastic Load Balancing instances instances Auto Scaling group Amazon S3 Amazon Kinesis Apache Storm DynamoDB Availability Zone Availability Zone Auto Scaling group Elastic Load Balancing
  20. 20. Data Collection •Amazon EC2, Elastic Load Balancing, Auto Scaling Store •Amazon S3 + Amazon Kinesis Global Distribution •Apache Storm on Amazon EC2 Bid Store •DynamoDB Bidding •Amazon EC2, Elastic Load Balancing, Auto Scaling Data CollectionBiddingAd Network 2Ad Network 1Auto Scaling GroupAuto Scaling GroupAuto Scaling GroupAuto Scaling GroupAuto Scaling GroupAuto Scaling GroupAuto Scaling GroupAuto Scaling GroupAuto Scaling GroupApache Stormv2V3V3v1v2V3V3v1V2V3V3V1Auto Scaling GroupV3V4Elastic Load BalancingElastic Load BalancingElastic Load BalancingElastic Load BalancingDynamoDBWriteReadReadReadReadReadReadWriteWritesWriteWriteReadV3` Elastic Load BalancingElastic Load BalancingElastic Load BalancingElastic Load BalancingElastic Load BalancingElastic Load BalancingDynamoDBData CollectionBiddingDynamoDBWriteReadReadWriteWriteWriteAmazon S3Amazon Kinesis
  21. 21. http://bit.ly/awsevals

×