4. Summary
A team inside Comcast open sourced a
compatible version of Amazon SNS and
SQS on top of Cassandra & Redis
5. Who are we?
• Comcast Silicon Valley
– Consumer-facing internet products for Comcast
– Web, Mobile, TV
• Silicon Valley
– Product Development & Innovation Center
6. CMB
CMB – Message Bus
CQS – Queue Service
CNS – Notification Service
15. Requirements
• Compatible with AWS
• Support Active-Active Multi-Data-Center
• Horizontal Scalability
• Guaranteed Delivery
• Very Low Latency
16. NOT Required
1. Order is NOT guaranteed
2. Duplicate messages CAN happen
2. Duplicate messages CAN happen
17. First, the Queue
CMB – Message Bus
CQS – Queue Service
CNS – Notification Service
18. What is Simple Queue Service?
(from a Popular Cloud Service Provider)
“Simple Queue Service offers a reliable, highly
scalable, hosted queue for storing messages as
they travel between computers”
26. CQS Notes
• Minimal duplication and good ordering
– When things are happy..
• Cassandra tombstones degrade message
performance
– Normally reads => Redis
28. Why use Cassandra?
• very low latency
• very high write throughput
• extremely high availability (across multi-dc)
• Near linear horizontal scalability
30. Next, Notifications
CMB – Message Bus
CQS – Queue Service
CNS – Notification Service
31. What is Simple Notification Service?
(from a Popular Cloud Services Provider … in
Seattle… )
“Simple Notification Service is a web service that
makes it easy to set up, operate, and send
notifications from the cloud”
42. Next steps
• Does this fit into OpenStack?
• Get Feedback from the community (you!)
• Questions?
• ryan@sv.comcast.com
• @ryanking
• PS - We’re hiring @ Comcast SV!
Editor's Notes
This talk is for anyone who is considering whether OpenStack needs Queuing and Notification services as part of it, or anyone who is building applications on top of OpenStack and is finding these missing (esp as compared to AWS)
Comcast is a member of the OpenStack Community.Comcast also is a strategic collaborator with Cisco on OpenStack.This collaboration with Cisco is focused on getting Comcast's own applications to run on its own private OpenStack infrastructure.The intent is to share the results of this effort with the OpenStack community as open source.We are not disclosing which applications will run in the Comcast OpenStack environment at this time. However, we expect a wide variety of Comcast applications to be targeted.The expected result will be contributions to the OpenStack open source community that will enable OpenStack to become solid, scalable, and reliable infrastructure for service providers like Comcast.
Receive message marks as invisible
Payload Cache, key/value pairs (on miss pre-load next 10k items)
(10 queues and 2 producers/ 2 consumersper queue)
CNS API Server Receives Publish() requestsValidates the request (authenticated and correctly formatted) Sends the Message to CQS for handling by the Endpoint Publish Job Producers