How to choose the right messaging service for your workload
Report
Share
•0 likes•65 views
How to choose the right messaging service for your workload
•0 likes•65 views
Report
Share
Download to read offline
At the heart of every event-driven architecture is a conduit for messages to flow through. AWS offers many services that can act as such conduit - EventBridge, SNS, SQS, Kinesis, DynamoDB streams, MSK, IOT Core and Amazon MQ just to name a few! These services have different characteristics and trade-offs around performance, scalability and cost. Picking the right service for your workload is not always easy. In this talk, let’s talk about how to pick the right messaging service to use in your event-driven architecture and play the game of trade-offs to your advantage.
Recommended
![[AWS Dev Day] 앱 현대화 | DevOps 개발자가 되기 위한 쿠버네티스 핵심 활용 예제 알아보기 - 정영준 AWS 솔루션즈 아키...](https://cdn.slidesharecdn.com/ss_thumbnails/appmodernizationdevopsshealth-190930044249-thumbnail.jpg?width=384&fit=bounds)
More Related Content
What's hot
![[Black Belt Online Seminar] AWS上でのログ管理](https://cdn.slidesharecdn.com/ss_thumbnails/aws-log-mgmt-161201043056-thumbnail.jpg?width=384&fit=bounds)
![[한국IBM] ServiceNow로 IT운영관리 최적화!](https://cdn.slidesharecdn.com/ss_thumbnails/ibm-190516012127-thumbnail.jpg?width=384&fit=bounds)
What's hot(20)
![[Black Belt Online Seminar] AWS上でのログ管理](https://cdn.slidesharecdn.com/ss_thumbnails/aws-log-mgmt-161201043056-thumbnail.jpg?width=768&fit=bounds)
![[한국IBM] ServiceNow로 IT운영관리 최적화!](https://cdn.slidesharecdn.com/ss_thumbnails/ibm-190516012127-thumbnail.jpg?width=768&fit=bounds)
![[REPEAT] Microsoft Active Directory Deep Dive (WIN303-R) - AWS re:Invent 2018](https://cdn.slidesharecdn.com/ss_thumbnails/win303-cully-aws-managed-micro-223d156e-6089-4328-8a84-188d50f004c1-1912633139-181211224501-thumbnail.jpg?width=768&fit=bounds)
![[Games on AWS 2019] AWS 사용자를 위한 만랩 달성 트랙 | AWS에서 분산 서비스 거부 공격(DDoS)을 고민하지 않는 ...](https://cdn.slidesharecdn.com/ss_thumbnails/gamesonaws-waf-191014023704-thumbnail.jpg?width=768&fit=bounds)
Similar to How to choose the right messaging service for your workload
Similar to How to choose the right messaging service for your workload(20)
More from Yan Cui
More from Yan Cui(20)
Recently uploaded
Recently uploaded(20)
How to choose the right messaging service for your workload
- 1. How to choose the right messaging service for your workload
- 6. Yan Cui http://theburningmonk.com @theburningmonk Independent Consultant advise training delivery
- 8. AWS services are the new data structures. Understanding their trade-offs is the new big O notation.
- 13. EventBridge
- 17. event
- 19. event FIFO Archiving Replays Schema Discovery Scalability Cost Performance Debugging Error Handling
- 21. picking the right AWS service is a valuable skill
- 23. Scaling
- 25. What are the scaling constraint for the messaging service?
- 26. Google “{service name} quotas”
- 30. How does Lambda’s concurrency scale with throughput?
- 38. more concurrency is not always better…
- 39. if you want… maximum throughput precise control over throughput SNS EventBridge Kinesis Provisioned
- 40. if you want… maximum throughput precise control over throughput SNS EventBridge Kinesis Provisioned
- 41. Downstream System SNS Lambda use Reserved Concurrency to limit concurrency
- 42. Reserved Concurrency are taken out of the available regional Lambda concurrency
- 43. managing Reserved Concurrency for many functions is dif fi cult and error prone, easy to create more problems than it solves
- 44. Costs
- 45. always factor scale into the equation
- 46. $10.836 1 msg/s for a month, 1KB per msg 1 x 60s x 60m x 24hr x 30days @ $0.014 per mil + 24hrs x 30days @ $0.015 per shard per hr $2.592 SNS SQS EventBridge Kinesis 1 x 60s x 60m x 24hr x 30days @ $1.00 per mil 1 x 60s x 60m x 24hr x 30days @ $0.40 per mil 1 x 60s x 60m x 24hr x 30days @ $0.50 per mil $1.037 $1.296
- 47. Kinesis on-demand mode pricing
- 48. $10.836 1 msg/s for a month, 1KB per msg 1 x 60s x 60m x 24hr x 30days @ $0.014 per mil + 24hrs x 30days @ $0.015 per shard per hr $2.592 SNS SQS EventBridge Kinesis Provisioned 1 x 60s x 60m x 24hr x 30days @ $1.00 per mil 1 x 60s x 60m x 24hr x 30days @ $0.40 per mil 1 x 60s x 60m x 24hr x 30days @ $0.50 per mil $1.037 $1.296 Kinesis On-Demand 1kb x 60s x 60m x 24hr x 30days @ $0.08 per GB ingested + 24hrs x 30days @ $0.04 per stream per hr $28.998
- 49. $47.088 1,000 msg/s for a month, 1KB per msg 1000 x 60s x 60m x 24hr x 30days @ $0.014 per mil + 24hrs x 30days @ $0.015 per shard per hr $2592.00 SNS SQS EventBridge Kinesis Provisioned 1000 x 60s x 60m x 24hr x 30days @ $1.00 per mil 1000 x 60s x 60m x 24hr x 30days @ $0.40 per mil 1000 x 60s x 60m x 24hr x 30days @ $0.50 per mil $1036.80 $1296.00 Kinesis On-Demand 1000kb x 60s x 60m x 24hr x 30days @ $0.08 per GB ingested + 24hrs x 30days @ $0.04 per stream per hr $226.55
- 50. services that charge by uptime are order(s) of magnitude cheaper when running at scale
- 51. National broadcaster of Finland
- 52. National broadcaster of Finland 500+ millions events per day, peaks at 600K+ messages/min
- 53. National broadcaster of Finland 500+ millions events per day, peaks at 600K+ messages/min Anahit Pogosova
- 54. National broadcaster of Finland 500+ millions events per day, peaks at 600K+ messages/min Anahit Pogosova decisions driven by cost ef fi ciency
- 55. National broadcaster of Finland 500+ millions events per day, peaks at 600K+ messages/min Anahit Pogosova
- 56. National broadcaster of Finland 500+ millions events per day, peaks at 600K+ messages/min Anahit Pogosova fallback in case Kinesis is down
- 57. Error Handling
- 59. 0, 1, 2
- 60. DLQs only capture the failed event
- 61. DLQ
- 68. prefer Lambda destination over DLQs (you can use both together, but there’s no clear reason for doing so)
- 72. DLQ
- 73. Observability
- 74. A measure of how well the internal state of an application can be inferred from its external outputs
- 75. X-Ray
- 78. X-Ray doesn’t trace through many popular messaging services.
- 79. X-Ray doesn’t trace through many popular messaging services. Need separate solution (e.g. correlation IDs) for Lambda logs.
- 81. all the indirect invocations are accounted for
- 83. all the Lambda logs from the transaction in chronological order
- 86. Step 1. Step 2.
- 87. No manual instrument. Better support for messaging services. Supports containers and Lambda.
- 88. “which messaging service should I use?”
- 90. task vs event
- 91. Task Event “Something happened” “Go do a thing”
- 92. Task Event “Something happened” “Go do a thing” Intended for a target receiver.
- 93. Task Event “Something happened” “Go do a thing” Intended for a target receiver. Often expects an answer.
- 94. Task Event “Something happened” “Go do a thing” Intended for a target receiver. Publishers are obvious to subscribers. Often expects an answer.
- 101. EventBridge SNS FIFO Do Stuff!
- 102. EventBridge SNS FIFO Do Stuff! order is lost
- 103. EventBridge SNS FIFO Do Stuff! order is lost
- 104. EventBridge SNS FIFO SQS FIFO Do Stuff (in order)! Other subscribers (not my problem!)
- 106. Kinesis Lambda
- 107. EventBridge Kinesis Lambda Other subscribers (not my problem!) send domain events
- 112. https://theburningmonk.com/hire-me Advise Training Delivery “Fundamentally, Yan has improved our team by increasing our ability to derive value from AWS and Lambda in particular.” Nick Blair Tech Lead