Speed and Reliability at Any Scale –
Combining SQS and DB Services
Jonathan Desrocher, Amazon Web Services
Colin Vipurs, S...
AWS Messaging = Amazon SQS + Amazon SNS
Amazon SQS Core Features

•
•

New and improved

Payload size of up to 256KB
Message batching for higher throughput and re...
SQS Core mechanics
Basic Message Lifecycle

Writer
Basic Message Lifecycle

Writer
Basic Message Lifecycle

Writer
Basic Message Lifecycle

Reader A

Reader B
Basic Message Lifecycle

Reader A

Reader B

ReceiveMessage

A

B
Basic Message Lifecycle

Reader A

Reader B

A

B
Basic Message Lifecycle

B

Reader B
Basic Message Lifecycle

Reader B
Basic Message Lifecycle

B

Reader B
Basic Message Lifecycle

Reader B
Basic Message Lifecycle

Reader B
That covers reliability.
Now let’s go for the scale!
Bulk Transactional Reads

Reader A
Bulk Transactional Reads

A

RAM: 10 Msgs

ReceiveMessage

A

A

A

A

Reader A

A
A

A

A

A
Bulk Transactional Reads

A

ReceiveMessage

A

A

A

A

Reader A

A
A

A

A

A

A

A

A

A

A

A

A

A

A

A

RAM: 20 Msg...
Bulk Transactional Reads

A

ReceiveMessage

A

A

A

A

Reader A

A
A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A
...
Bulk Transactional Reads

Reader A

DeleteMessage
Bulk Transactional Reads

Reader A
Let’s take it to real life!
Scalability example: market trade volume by half hour
350,000
300,000
250,000
200,000
150,000

100,000
50,000
0
Scalability example: market trade volume by half hour
350,000
300,000
250,000
200,000

85%

150,000

100,000
50,000
0

15%
Design pattern #1:
Batch processing
Batch Processing
• Use SQS as a scalable and resilient short-term storage
solution.
• Simply configure the appropriate ret...
Batch Processing
• When appropriate, launch a fleet of Amazon EC2
workers and process the messages en masse.

ReceiveMessa...
Design pattern #2:
IAM Roles for Amazon EC2
Using IAM Roles for Amazon EC2
• Create an IAM role with the
appropriate permissions to
Amazon SQS.
• Launch EC2 instances...
Using IAM Roles for Amazon EC2
• Use the AWS SDK on the
Instance
• No need to type credentials
•
•
•

Not in code
Not in a...
Design pattern #3:
Using SQS to durably batch writes
Using SQS to durably batch writes
• The application:
–
–
–
–

An AWS Elastic Beanstalk application.
Clients upload data to...
Using SQS to durably batch writes
• The challenge:
– We have an external constraint that requires us to batch the upload i...
Using SQS to durably batch writes
• Enter SQS:
– Persist individual client PUTs as SQS messages.
– Have an Amazon EC2 work...
Using SQS to durably batch writes
• Also to consider:
– Some data stores are optimized for read workloads
– Buffering the ...
Design pattern #4:
Discarding stale messages
Discarding stale messages
• Controlled via the MessageRetentionPeriod property.
• Useful when there is no business value f...
Design pattern #5:
Simple Notification Service Fan-out
Simple Notification Service Fan-out
•

Atomically distribute a message to
multiple subscribers over different
transport me...
Simple Notification Service Fan-out
• Perform different operations on the same data
– Split different facets of the payloa...
Simple Notification Service Fan-out
• Deliver the same data to different environments

Production
ReceiveMessage

S3 PUT

...
Simple Notification Service Fan-out
• Distribute the same data to a multiple external environments:
– Push data to differe...
Simple Notification Service Fan-out
• Each recipient can have its own preferred transport protocol:
– SQS for guaranteed d...
Design pattern #6:
Send messages from the browser
Send messages from the browser
• Make direct calls to AWS services such as SQS
and DynamoDB directly from the user’s brows...
Send messages from the browser
• Back to our sample architecture:
– Browser authenticates against Elastic Beanstalk applic...
Colin Vipurs
Shazam Entertainment Ltd.
375 MILLION
USERS

75 MILLION

MONTHLY ACTIVE USERS

10 MILLION
NEW USERS
PER MONTH
2004

2005

2006

2007

2008

2009

2010

2011

2012

2013
Amazon SQS for surge protection

SQS
shield
c3po

single event

message

update request

queue worker

user data

batch

data req.

Facebook Realtime Updates

user da...
Facebook Realtime Updates
Queue Worker Anatomy
operation
read msg

versioned msg

data object

target system
pollers

unmarshallers

handlers
SQS for SLAs

data
ingester
SQS for SLAs
high priority
batch
data
ingester

std priority

queue worker
low priority
SQS as DynamoDB Buffer
data
API

DynamoDb
Time

Traffic Volume
Traffic Volume

large volume
event

Time
SQS as DynamoDB Buffer
data
API

DynamoDb

throughtput exceeded

data

data
data message
queue worker
SQS as DynamoDB Buffer
data
API

client fails
fast

DynamoDb

data

queue worker

client
retries
SQS as DynamoDB Buffer

public interface Writer <T> {
void write(T t);
}
SNS/SQS Datastore Segregation
SNS/SQS Datastore Segregation
Reporting
S3
tag data
API

API
SNS/SQS Datastore Segregation
Reporting
S3
tag data
API

API

Dev Hacking
SQS for Shazam is…
•
•
•
•

Protection from the outside world
Short term, unbounded persistence
Cost effective elastic cap...
Thank you Colin!
Design patterns recap
1.
2.
3.
4.
5.
6.

Batch processing
IAM Roles for EC2
Using SQS to durably batch writes
Discard stal...
Additional messaging resources
• Application Services Booth
• re:Invent sessions:
– ARC301 Controlling the Flood: Massive ...
Next stop:
ARC301 - Controlling the Flood!
Right here in this room.
Please give us your feedback on this
presentation

SVC206
As a thank you, we will select prize
winners daily for completed...
Speed and Reliability at Any Scale: Amazon SQS and Database Services (SVC206) | AWS re:Invent 2013
Speed and Reliability at Any Scale: Amazon SQS and Database Services (SVC206) | AWS re:Invent 2013
Speed and Reliability at Any Scale: Amazon SQS and Database Services (SVC206) | AWS re:Invent 2013
Upcoming SlideShare
Loading in...5
×

Speed and Reliability at Any Scale: Amazon SQS and Database Services (SVC206) | AWS re:Invent 2013

3,654

Published on

Amazon Simple Queue Service (Amazon SQS) makes it easy and inexpensive to enhance the scalability and reliability of your cloud application. In this session, we demonstrate design patterns for using Amazon SQS in conjunction with Amazon Simple Storage Service (Amazon S3), Amazon DynamoDB, Amazon Elastic MapReduce, Amazon Relational Database Service, and Amazon Redshift. Shazam will share their experience of combining Amazon SQS with Amazon DynamoDB to support a Super Bowl advertising campaign.

Published in: Technology, Business
0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,654
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
72
Comments
0
Likes
7
Embeds 0
No embeds

No notes for slide

Speed and Reliability at Any Scale: Amazon SQS and Database Services (SVC206) | AWS re:Invent 2013

  1. 1. Speed and Reliability at Any Scale – Combining SQS and DB Services Jonathan Desrocher, Amazon Web Services Colin Vipurs, Shazam Entertainment Ltd. November 14, 2013 © 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
  2. 2. AWS Messaging = Amazon SQS + Amazon SNS
  3. 3. Amazon SQS Core Features • • New and improved Payload size of up to 256KB Message batching for higher throughput and reduced costs • Supports long polling for reduced costs and latency • Cross-origin resource sharing support
  4. 4. SQS Core mechanics
  5. 5. Basic Message Lifecycle Writer
  6. 6. Basic Message Lifecycle Writer
  7. 7. Basic Message Lifecycle Writer
  8. 8. Basic Message Lifecycle Reader A Reader B
  9. 9. Basic Message Lifecycle Reader A Reader B ReceiveMessage A B
  10. 10. Basic Message Lifecycle Reader A Reader B A B
  11. 11. Basic Message Lifecycle B Reader B
  12. 12. Basic Message Lifecycle Reader B
  13. 13. Basic Message Lifecycle B Reader B
  14. 14. Basic Message Lifecycle Reader B
  15. 15. Basic Message Lifecycle Reader B
  16. 16. That covers reliability. Now let’s go for the scale!
  17. 17. Bulk Transactional Reads Reader A
  18. 18. Bulk Transactional Reads A RAM: 10 Msgs ReceiveMessage A A A A Reader A A A A A A
  19. 19. Bulk Transactional Reads A ReceiveMessage A A A A Reader A A A A A A A A A A A A A A A A RAM: 20 Msgs
  20. 20. Bulk Transactional Reads A ReceiveMessage A A A A Reader A A A A A A A A A A A A A A A A A A A A A A A A A A RAM: 30 Msgs
  21. 21. Bulk Transactional Reads Reader A DeleteMessage
  22. 22. Bulk Transactional Reads Reader A
  23. 23. Let’s take it to real life!
  24. 24. Scalability example: market trade volume by half hour 350,000 300,000 250,000 200,000 150,000 100,000 50,000 0
  25. 25. Scalability example: market trade volume by half hour 350,000 300,000 250,000 200,000 85% 150,000 100,000 50,000 0 15%
  26. 26. Design pattern #1: Batch processing
  27. 27. Batch Processing • Use SQS as a scalable and resilient short-term storage solution. • Simply configure the appropriate retention period and send away! SendMessage HTTP PUT Elastic Beanstalk Application
  28. 28. Batch Processing • When appropriate, launch a fleet of Amazon EC2 workers and process the messages en masse. ReceiveMessage
  29. 29. Design pattern #2: IAM Roles for Amazon EC2
  30. 30. Using IAM Roles for Amazon EC2 • Create an IAM role with the appropriate permissions to Amazon SQS. • Launch EC2 instances with this role. • Done! – Audit logs will correlate the EC2 instance ID to the SQS API calls. – IAM will automatically rotate the credentials on our behalf. { "Statement": [ { "Sid": "Stmt1384277213171", "Action": [ "sqs:ChangeMessageVisibility", "sqs:DeleteMessage", "sqs:GetQueueAttributes", "sqs:GetQueueUrl", "sqs:ListQueues", "sqs:ReceiveMessage" ], "Effect": "Allow", "Resource": "arn:aws:sqs:us-east1:455320512810:Sensor_Ingestion" } ] }
  31. 31. Using IAM Roles for Amazon EC2 • Use the AWS SDK on the Instance • No need to type credentials • • • Not in code Not in a configuration file Not via the console either require ‘rubygems’ require ‘aws-sdk’ sqs = AWS::SQS.new() myqueue = sqs.queues.named("Sensor_Ingestion") myqueue.poll do |msg| # Do something with the message end
  32. 32. Design pattern #3: Using SQS to durably batch writes
  33. 33. Using SQS to durably batch writes • The application: – – – – An AWS Elastic Beanstalk application. Clients upload data to the application through HTTP PUTs. Each upload is 100KB in size. Amazon S3 will be used as the permanent data store. S3 PUT HTTP PUT Elastic Beanstalk Application S3 Bucket
  34. 34. Using SQS to durably batch writes • The challenge: – We have an external constraint that requires us to batch the upload into Amazon S3. For example: • Amazon EMR best practices call for Amazon S3 object size of >10MB. • Hourly Amazon Redshift batch inserts. EMR Cluster S3 PUT HTTP PUT Elastic Beanstalk Application S3 Bucket Redshift Database
  35. 35. Using SQS to durably batch writes • Enter SQS: – Persist individual client PUTs as SQS messages. – Have an Amazon EC2 worker role that performs the following logic: • Receive SQS message and add to an in-memory buffer. • Once buffer is full, upload to Amazon S3. • Upon acknowledgement from S3, delete SQS messages from queue. SendMessage HTTP PUT Elastic Beanstalk Application ReceiveMessage EMR Cluster S3 PUT S3 Bucket Redshift Database
  36. 36. Using SQS to durably batch writes • Also to consider: – Some data stores are optimized for read workloads – Buffering the writes with Simple Queue Service will ensure both speed and reliability of data ingestion. SendMessage HTTP PUT Elastic Beanstalk Application ReceiveMessage BatchWriteItem RDS Database
  37. 37. Design pattern #4: Discarding stale messages
  38. 38. Discarding stale messages • Controlled via the MessageRetentionPeriod property. • Useful when there is no business value for data older than X minutes. – “Transactions that don’t complete within 5 minutes are abandoned, enabling client-side failure handling”. SendMessage HTTP PUT Elastic Beanstalk Application ReceiveMessage [Stale Messages] EMR Cluster S3 PUT S3 Bucket Redshift Database
  39. 39. Design pattern #5: Simple Notification Service Fan-out
  40. 40. Simple Notification Service Fan-out • Atomically distribute a message to multiple subscribers over different transport methods – – – – • SQS queues HTTP/S endpoints SMS Email Also used to abstract different mobile device push providers (MBL308) – Apple Push Notification Service – Google Cloud Messaging for Android – Amazon Device Messaging Publish
  41. 41. Simple Notification Service Fan-out • Perform different operations on the same data – Split different facets of the payload into different systems. – Duplicate the data into short-term and longterm storage systems. ReceiveMessage S3 PUT S3 Bucket EMR Cluster S3 Bucket Redshift Database SNS Publish HTTP PUT Elastic Beanstalk Application ReceiveMessage S3 PUT
  42. 42. Simple Notification Service Fan-out • Deliver the same data to different environments Production ReceiveMessage S3 PUT S3 Bucket DynamoDB Table S3 Bucket DynamoDB Table SNS Publish HTTP PUT Test Elastic Beanstalk Application ReceiveMessage S3 PUT
  43. 43. Simple Notification Service Fan-out • Distribute the same data to a multiple external environments: – Push data to different locations worldwide. – Seamlessly synchronize AWS and on-premises environments. – Pro tip: MessageID field is consistent across locations. us-east-1 eu-west-1 SNS Publish HTTP PUT Elastic Beanstalk Application ap-northeast-1 On-premises Data Center
  44. 44. Simple Notification Service Fan-out • Each recipient can have its own preferred transport protocol: – SQS for guaranteed delivery – Email for human-friendly delivery – HTTP/S for real-time push Partner/Process A Partner/Process B SNS Publish HTTP PUT Elastic Beanstalk Application Partner/Process C Partner/Process D
  45. 45. Design pattern #6: Send messages from the browser
  46. 46. Send messages from the browser • Make direct calls to AWS services such as SQS and DynamoDB directly from the user’s browser. • Authentication is based on STS tokens. • Supports S3, SQS, SNS and DynamoDB.
  47. 47. Send messages from the browser • Back to our sample architecture: – Browser authenticates against Elastic Beanstalk application – Response includes location of SQS Queue and STS Token for direct authentication. GetToken Elastic Beanstalk Application SendMessage ReceiveMessage EMR Cluster S3 PUT S3 Bucket Redshift Database
  48. 48. Colin Vipurs Shazam Entertainment Ltd.
  49. 49. 375 MILLION USERS 75 MILLION MONTHLY ACTIVE USERS 10 MILLION NEW USERS PER MONTH
  50. 50. 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013
  51. 51. Amazon SQS for surge protection SQS shield
  52. 52. c3po single event message update request queue worker user data batch data req. Facebook Realtime Updates user data user updater DynamoDB
  53. 53. Facebook Realtime Updates
  54. 54. Queue Worker Anatomy operation read msg versioned msg data object target system pollers unmarshallers handlers
  55. 55. SQS for SLAs data ingester
  56. 56. SQS for SLAs high priority batch data ingester std priority queue worker low priority
  57. 57. SQS as DynamoDB Buffer data API DynamoDb
  58. 58. Time Traffic Volume
  59. 59. Traffic Volume large volume event Time
  60. 60. SQS as DynamoDB Buffer data API DynamoDb throughtput exceeded data data data message queue worker
  61. 61. SQS as DynamoDB Buffer data API client fails fast DynamoDb data queue worker client retries
  62. 62. SQS as DynamoDB Buffer public interface Writer <T> { void write(T t); }
  63. 63. SNS/SQS Datastore Segregation
  64. 64. SNS/SQS Datastore Segregation Reporting S3 tag data API API
  65. 65. SNS/SQS Datastore Segregation Reporting S3 tag data API API Dev Hacking
  66. 66. SQS for Shazam is… • • • • Protection from the outside world Short term, unbounded persistence Cost effective elastic capacity Scalable data segregation
  67. 67. Thank you Colin!
  68. 68. Design patterns recap 1. 2. 3. 4. 5. 6. Batch processing IAM Roles for EC2 Using SQS to durably batch writes Discard stale messages Simple Notification Service Fan-out Send messages from the browser
  69. 69. Additional messaging resources • Application Services Booth • re:Invent sessions: – ARC301 Controlling the Flood: Massive Message Processing with AWS SQS and DynamoDB – MBL308 Engage Your Customers with Amazon SNS Mobile Push • AWS Support and Discussion Forums • AWS Architecture Center: http://aws.amazon.com/architecture • Documentation: http://aws.amazon.com/documentation/sqs
  70. 70. Next stop: ARC301 - Controlling the Flood! Right here in this room.
  71. 71. Please give us your feedback on this presentation SVC206 As a thank you, we will select prize winners daily for completed surveys!
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×