Successfully reported this slideshow.
Your SlideShare is downloading. ×

How to choose the right messaging service for your workload

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 113 Ad

How to choose the right messaging service for your workload

Download to read offline

At the heart of every event-driven architecture is a conduit for messages to flow through. AWS offers many services that can act as such conduit - EventBridge, SNS, SQS, Kinesis, DynamoDB streams, MSK, IOT Core and Amazon MQ just to name a few! These services have different characteristics and trade-offs around performance, scalability and cost. Picking the right service for your workload is not always easy. In this talk, let’s talk about how to pick the right messaging service to use in your event-driven architecture and play the game of trade-offs to your advantage.

At the heart of every event-driven architecture is a conduit for messages to flow through. AWS offers many services that can act as such conduit - EventBridge, SNS, SQS, Kinesis, DynamoDB streams, MSK, IOT Core and Amazon MQ just to name a few! These services have different characteristics and trade-offs around performance, scalability and cost. Picking the right service for your workload is not always easy. In this talk, let’s talk about how to pick the right messaging service to use in your event-driven architecture and play the game of trade-offs to your advantage.

Advertisement
Advertisement

More Related Content

More from Yan Cui (20)

Advertisement

Recently uploaded (20)

How to choose the right messaging service for your workload

  1. 1. How to choose the right messaging service for your workload
  2. 2. O(n) NP P NP-Hard NP-Complete
  3. 3. Yan Cui http://theburningmonk.com @theburningmonk AWS user since 2010
  4. 4. Yan Cui http://theburningmonk.com @theburningmonk Developer Advocate @
  5. 5. Yan Cui http://theburningmonk.com @theburningmonk Independent Consultant advise training delivery
  6. 6. AWS services are the new data structures. Understanding their trade-offs is the new big O notation.
  7. 7. Event-Driven Architecture
  8. 8. Event-Driven Architecture
  9. 9. event stuff happens
  10. 10. ? event stuff happens
  11. 11. EventBridge
  12. 12. publisher publisher publisher consumer consumer consumer
  13. 13. service service service service service service service service
  14. 14. service service service service service service service service
  15. 15. event
  16. 16. event FIFO Archiving Replays Schema Discovery
  17. 17. event FIFO Archiving Replays Schema Discovery Scalability Cost Performance Debugging Error Handling
  18. 18. Kinesis EventBridge SNS SQS DynamoDB Stream IOT Core …
  19. 19. picking the right AWS service is a valuable skill
  20. 20. Kinesis EventBridge SNS SQS DynamoDB Stream IOT Core …
  21. 21. Scaling
  22. 22. What are the scaling constraint for the messaging service?
  23. 23. Google “{service name} quotas”
  24. 24. How does Lambda’s concurrency scale with throughput?
  25. 25. Concurrency Msgs/s
  26. 26. Concurrency Msgs/s SNS EventBridge
  27. 27. https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html
  28. 28. Concurrency Msgs/s SNS EventBridge SQS
  29. 29. Concurrency Msgs/s SNS EventBridge SQS Kinesis
  30. 30. https://amzn.to/2RudmGV
  31. 31. Concurrency Msgs/s SNS EventBridge SQS Kinesis
  32. 32. more concurrency is not always better…
  33. 33. if you want… maximum throughput precise control over throughput SNS EventBridge Kinesis Provisioned
  34. 34. if you want… maximum throughput precise control over throughput SNS EventBridge Kinesis Provisioned
  35. 35. Downstream System SNS Lambda use Reserved Concurrency to limit concurrency
  36. 36. Reserved Concurrency are taken out of the available regional Lambda concurrency
  37. 37. managing Reserved Concurrency for many functions is dif fi cult and error prone, easy to create more problems than it solves
  38. 38. Costs
  39. 39. always factor scale into the equation
  40. 40. $10.836 1 msg/s for a month, 1KB per msg 1 x 60s x 60m x 24hr x 30days @ $0.014 per mil + 24hrs x 30days @ $0.015 per shard per hr $2.592 SNS SQS EventBridge Kinesis 1 x 60s x 60m x 24hr x 30days @ $1.00 per mil 1 x 60s x 60m x 24hr x 30days @ $0.40 per mil 1 x 60s x 60m x 24hr x 30days @ $0.50 per mil $1.037 $1.296
  41. 41. Kinesis on-demand mode pricing
  42. 42. $10.836 1 msg/s for a month, 1KB per msg 1 x 60s x 60m x 24hr x 30days @ $0.014 per mil + 24hrs x 30days @ $0.015 per shard per hr $2.592 SNS SQS EventBridge Kinesis Provisioned 1 x 60s x 60m x 24hr x 30days @ $1.00 per mil 1 x 60s x 60m x 24hr x 30days @ $0.40 per mil 1 x 60s x 60m x 24hr x 30days @ $0.50 per mil $1.037 $1.296 Kinesis On-Demand 1kb x 60s x 60m x 24hr x 30days @ $0.08 per GB ingested + 24hrs x 30days @ $0.04 per stream per hr $28.998
  43. 43. $47.088 1,000 msg/s for a month, 1KB per msg 1000 x 60s x 60m x 24hr x 30days @ $0.014 per mil + 24hrs x 30days @ $0.015 per shard per hr $2592.00 SNS SQS EventBridge Kinesis Provisioned 1000 x 60s x 60m x 24hr x 30days @ $1.00 per mil 1000 x 60s x 60m x 24hr x 30days @ $0.40 per mil 1000 x 60s x 60m x 24hr x 30days @ $0.50 per mil $1036.80 $1296.00 Kinesis On-Demand 1000kb x 60s x 60m x 24hr x 30days @ $0.08 per GB ingested + 24hrs x 30days @ $0.04 per stream per hr $226.55
  44. 44. services that charge by uptime are order(s) of magnitude cheaper when running at scale
  45. 45. National broadcaster of Finland
  46. 46. National broadcaster of Finland 500+ millions events per day, peaks at 600K+ messages/min
  47. 47. National broadcaster of Finland 500+ millions events per day, peaks at 600K+ messages/min Anahit Pogosova
  48. 48. National broadcaster of Finland 500+ millions events per day, peaks at 600K+ messages/min Anahit Pogosova decisions driven by cost ef fi ciency
  49. 49. National broadcaster of Finland 500+ millions events per day, peaks at 600K+ messages/min Anahit Pogosova
  50. 50. National broadcaster of Finland 500+ millions events per day, peaks at 600K+ messages/min Anahit Pogosova fallback in case Kinesis is down
  51. 51. Error Handling
  52. 52. 0, 1, 2
  53. 53. DLQs only capture the failed event
  54. 54. DLQ
  55. 55. SNS, SQS, Lambda, EventBridge
  56. 56. prefer Lambda destination over DLQs (you can use both together, but there’s no clear reason for doing so)
  57. 57. Kinesis EventBridge SNS SQS DynamoDB Stream
  58. 58. DLQ
  59. 59. Observability
  60. 60. A measure of how well the internal state of an application can be inferred from its external outputs
  61. 61. X-Ray
  62. 62. X-Ray doesn’t trace through many popular messaging services.
  63. 63. X-Ray doesn’t trace through many popular messaging services. Need separate solution (e.g. correlation IDs) for Lambda logs.
  64. 64. all the indirect invocations are accounted for
  65. 65. all the Lambda logs from the transaction in chronological order
  66. 66. Step 1. Step 2.
  67. 67. No manual instrument. Better support for messaging services. Supports containers and Lambda.
  68. 68. “which messaging service should I use?”
  69. 69. task vs event
  70. 70. Task Event “Something happened” “Go do a thing”
  71. 71. Task Event “Something happened” “Go do a thing” Intended for a target receiver.
  72. 72. Task Event “Something happened” “Go do a thing” Intended for a target receiver. Often expects an answer.
  73. 73. Task Event “Something happened” “Go do a thing” Intended for a target receiver. Publishers are obvious to subscribers. Often expects an answer.
  74. 74. EventBridge SNS FIFO Do Stuff!
  75. 75. EventBridge SNS FIFO Do Stuff!
  76. 76. EventBridge SNS FIFO Do Stuff! order is lost
  77. 77. EventBridge SNS FIFO Do Stuff! order is lost
  78. 78. EventBridge SNS FIFO SQS FIFO Do Stuff (in order)! Other subscribers (not my problem!)
  79. 79. Kinesis Lambda
  80. 80. EventBridge Kinesis Lambda Other subscribers (not my problem!) send domain events
  81. 81. service service service service service service service
  82. 82. service service service
  83. 83. service service service
  84. 84. https://theburningmonk.com/hire-me Advise Training Delivery “Fundamentally, Yan has improved our team by increasing our ability to derive value from AWS and Lambda in particular.” Nick Blair Tech Lead
  85. 85. @theburningmonk theburningmonk.com github.com/theburningmonk

×