Download as PDF, PPTX
Uploaded byYan Cui
Streams and serverless at DAZN
The document discusses serverless architectures using AWS Lambda and Amazon Kinesis. It describes how DAZN uses a serverless-first approach with services like API Gateway, SQS, SNS, Step Functions, and Kinesis. It provides examples of using Kinesis as a task queue, transaction log, and for building analytics pipelines. It also covers best practices for streaming data, security, observability, and challenges of building distributed systems with streams.
More Related Content
Similar to Streams and serverless at DAZN
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
Streams and serverless at DAZN
- 1.
- 2. Yan Cui http://theburningmonk.com @theburningmonk Principal Engineer@ Independent Consultant
- 3. “Netflix for sports” officesin London, Leeds, Katowice and Amsterdam
- 4. available in Austria,Switzerland, Germany, Japan, Canada, Italy, US, Spain and Brazil
- 5. available on 30+platforms
- 6.
- 7. “Netflix for sports” officesin London, Leeds, Katowice and Amsterdam We’re hiring! Visit engineering.dazn.com to learn more. follow @DAZN_ngnrs for updates about the engineering team.
- 9. we use aLOT
- 10.
- 11. there are nosilver bullets
- 12. serverless is notright for every use case (yet)
- 14.
- 16. we use aLOT
- 17. API Gateway SQSSNS Step Functions
- 18.
- 19. Kinesis as a taskqueue
- 20. Kinesis as a taskqueue as a transaction log
- 21. Kinesis as a taskqueue as a transaction log part of an analytics pipeline
- 22.
- 23.
- 24.
- 25.
- 26.
- 27.
- 28.
- 29. stream shard shard shard doX doX doX PutRecord: { PartitionKey: … Data:… } ordering is preserved within the same partition key
- 30. stream shard shard shard doX doX doX PutRecord: { PartitionKey: … Data:… } ordering is preserved within the same partition key data is retained for up to 7 days, default is 24 hours
- 31.
- 32.
- 33.
- 34.
- 35. stream doY doX shard doY doX doY doX shard shard every subscriber getsthe same data, in the same order
- 36. stream doY doX shard doY doX doY doX shard shard every subscriber getsthe same data, in the same order each subscriber process the stream at their own pace
- 37.
- 38. stream doY doX doY doX doY shard INGRESS 1MB/s data, 1000records/s shard EGRESS 2MB/s data per subscriber, 5 reads/s doX shard
- 39.
- 40. stream as atask queue
- 41.
- 42. ordering replay events Kinesis SQSSNS mode retry concurrency subscribers
- 43. ordering replay events Kinesis SQSSNS mode retry concurrency subscribers many one-to-one many
- 44. ordering replay events Kinesis SQSSNS by shard none (standard) global (FIFO) none mode retry concurrency subscribers many one-to-one many
- 45. ordering replay events Kinesis SQSSNS by shard none (standard) global (FIFO) none up to 7 days none none mode retry concurrency subscribers many one-to-one many
- 46. ordering replay events Kinesis SQSSNS by shard none (standard) global (FIFO) none up to 7 days none none mode retry batched batched (up to 10) singular concurrency subscribers many one-to-one many
- 47. ordering replay events Kinesis SQSSNS by shard none (standard) global (FIFO) none up to 7 days none none mode retry batched batched (up to 10) singular retried until success retry + DLQ retry + DLQ concurrency subscribers many one-to-one many
- 48. ordering replay events Kinesis SQSSNS by shard none (standard) global (FIFO) none up to 7 days none none mode retry batched batched (up to 10) singular retried until success retry + DLQ retry + DLQ concurrency 1 per shard auto-scaled fan-out!!! subscribers many one-to-one many
- 49. ordering replay events Kinesis SQSSNS by shard none (standard) global (FIFO) none up to 7 days none none mode retry batched batched (up to 10) singular retried until success retry + DLQ retry + DLQ concurrency 1 per shard auto-scaled fan-out!!! subscribers many one-to-one many
- 50. use the concurrencymodel to amortize spikes in traffic
- 51. leveraging the retry-until-successbehaviour
- 52.
- 53. 1 msg/s fora month, 1KB per msg 1 x 60s x 60m x 24hr x 30days @ $0.014 per mil + 24hrs x 30days @ $0.015 per hr $10.836 1 x 60s x 60m x 24hr x 30days @ $0.5 per mil $1.296 1 x 60s x 60m x 24hr x 30days @ $0.4 per mil (send) + 1 x 60s x 60m x 24hr x 30days @ 10 batch @ 20s long polling @ $0.4 per mil (receive) $1.089
- 54. 1K msg/s fora month, 1KB per msg 1k x 60s x 60m x 24hr x 30days @ $0.014 per mil + 24hrs x 30days @ $0.015 per hr $47.088 1k x 60s x 60m x 24hr x 30days @ $0.5 per mil $1296.00 1k x 60s x 60m x 24hr x 30days @ $0.4 per mil (send) + 1k x 60s x 60m x 24hr x 30days @ 10 batch @ 20s long polling @ $0.4 per mil (receive) $1041.98
- 55. stream as atransaction log
- 56.
- 57. events are anenabler for COMPOSABILITY
- 58.
- 59.
- 60. Kinesis API Gateway AWSLambda API GatewayAWS Lambda service-A service-B
- 61. Kinesis API Gateway AWSLambda API GatewayAWS Lambda service-A service-B
- 62. Kinesis API Gateway AWSLambda API GatewayAWS Lambda service-A service-B AWS Lambda AWS Lambda AWS Lambda
- 63. Kinesis API Gateway AWSLambda API GatewayAWS Lambda service-A service-B AWS Lambda AWS Lambda AWS Lambda DynamoDBIOT
- 64. Kinesis API Gateway AWSLambda API GatewayAWS Lambda service-A service-B AWS Lambda AWS Lambda AWS Lambda DynamoDBIOT
- 65. Kinesis API Gateway AWSLambda API GatewayAWS Lambda service-A service-B AWS Lambda AWS Lambda AWS Lambda DynamoDBIOT AWS Lambda AWS Lambda
- 66.
- 67. service A serviceB service C service D bounded context bounded context
- 68. service A serviceB service C service D bounded context bounded context
- 69. service A serviceB service C service D
- 71. service A serviceB service C service D
- 72. service A serviceB service C service D
- 73. service A serviceB service C service Dbackward-compatible?
- 75. bounded context DON’T useevents to orchestrate workflows within the same bounded context
- 76. bounded context adds unnecessary complexityto logging, tracing, and end-to-end reporting
- 77. bounded context the workflowdoesn’t exist as a standalone concept, but as the sum of a series of loosely connected parts
- 78. Step Functions use StepFunctions instead
- 79. Step Functions don’t forgetto emit events from the workflow
- 80. Step Functions so otherscan react to state changes that happened as part of the workflow
- 81.
- 82.
- 83. S3 BucketsKinesis StreamsKinesis Firehose
- 84. S3 BucketsKinesis Streams AWSLambda Kinesis Firehose
- 85. S3 BucketsKinesis Streams AWSLambda AWS Lambda Kinesis Firehose
- 86. S3 BucketsKinesis Streams AWSLambda AWS LambdaAWS Lambda Kinesis Firehose
- 87. S3 BucketsKinesis Streams AWSLambda AWS LambdaAWS Lambda DynamoDB ElasticSearch Kinesis Firehose
- 88. S3 BucketsKinesis Streams AWSLambda AWS Lambda Athena QuickSight AWS Lambda Kinesis Firehose
- 89. S3 BucketsKinesis Streams AWSLambda AWS Lambda Athena QuickSight AWS Lambda Kinesis Firehose
- 90. secure sensitive databoth at rest and in-transit
- 91.
- 92.
- 93.
- 94.
- 95.
- 96. principle of LeastPrivilege
- 97. keep functions single-purposed &apply per function policies
- 99.
- 100.
- 101. stream doY doX doY doX doY shard shard EGRESS 2MB/s data persubscriber, doX shard 5 reads/s
- 102. throughput is maintainedas you add more subscribers, but…
- 104. it’s difficult toadd more than 15 subscriber functions to a stream
- 106.
- 107.
- 108. user_events order_events promo_events my microservices still possibleto have too many subscribers for a stream
- 109. user_events order_events promo_events my microservices implicit couplingbetween publisher and subscriber
- 110. user_events order_events promo_events my microservices lose orderingof related events in different streams
- 111. lose ordering ofrelated events in different streams
- 113.
- 114. extra cost forthese consumer apps
- 115.
- 116.
- 119. { error: … message: … correlation-id:… } { message: … correlation-id: … }
- 120.
- 121.