SQS ingress for AWS Lambda
•
0 likes•1,015 views
here's a proof-of-concept approach to process SQS messages with AWS Lambda and incorporate auto-scaling to elastically scale no. of instances of Lambda functions based on load
Recommended
More Related Content
Similar to SQS ingress for AWS Lambda
Similar to SQS ingress for AWS Lambda (20)
More from Yan Cui
More from Yan Cui (20)
Recently uploaded
Recently uploaded (20)
SQS ingress for AWS Lambda
- 1. for AWS LAMBDAAWS LAMBDA SQS ingressSQS ingress
- 2. hi,I’mYanCui
- 4. AWS user since 2009
- 7. can I haz SQS as ingress source?
- 8. DO IT we can DO IT ourselves!
- 11. retried 3 times dead letter queue configurable dead letter queue
- 12. retried 3 times dead letter queue one invocation per message configurable dead letter queue no. of pollers
- 13. retried 3 times dead letter queue one invocation per message highest parallelism! configurable dead letter queue no. of pollers
- 14. retried 3 times dead letter queue one invocation per message consider impact on downstream systems consider Lambda throttling which is per region, so other functions can get throttled too configurable dead letter queue no. of pollers
- 16. retried until success/expiry NO dead letter queue configurable dead letter queue
- 17. retried until success/expiry NO dead letter queue configurable dead letter queue
- 18. retried until success/expiry NO dead letter queue one invocation per shard configurable dead letter queue no. of pollers
- 19. retried until success/expiry NO dead letter queue one invocation per shard charged by shard hour + reqs configurable dead letter queue no. of pollers charged by reqs
- 20. retried until success/expiry configurable NO dead letter queue dead letter queue one invocation per shard no. of pollers charged by shard hour + reqs charged by reqs high baseline cost (1 shard) but order of magnitude cheaper at scale
- 21. 1 msg/s for a month (1K msgs) 1 x 60s x 60m x 24hr x 30days @ $0.4 per mil = $1.0368 1 x 60s x 60m x 24hr x 30days @ $0.014 per mil + 24hrs x 30days x $0.015 per hr = $10.836
- 22. 1000 msg/s for a month (1K msgs) 1000 x 60s x 60m x 24hr x 30days @ $0.4 per mil = $1036.80 1000 x 60s x 60m x 24hr x 30days @ $0.014 per mil + 24hrs x 30days x $0.015 per hr = $47.088
- 23. “none of them is the right choice for every situation”
- 24. SQS is right choice sometimes, no? “none of them is the right choice for every situation”
- 26. every X mins
- 27. simple cheap
- 28. simple cheap good for predictable traffic
- 29. unpredictable traffic leads to unpredictable latency
- 30. unpredictable traffic leads to unpredictable latency manual scaling
- 35. PROCESS
- 36. RECURSE
- 37. predictable latency even when traffic is unpredictable
- 38. unhandled exception kills loop
- 39. unhandled exception kills loop cost
- 43. token last_heartbeat my_loop_1 2017-05-17 20:47:32.142 my_loop_2 2017-05-17 20:47:29.923 … … passed to a recursive function at the start
- 44. every X mins look for dead loops
- 45. every X mins look for dead loops recurse start
- 47. too many messages add token
- 48. too many messages recurse start add token
- 52. Process SQS with Lambda part 1 : http://bit.ly/2rvXYdW part 2 : http://bit.ly/2pY0Grj
- 53. QUESTIONS?