Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Streams and serverless at DAZN

1,069 views

Published on

In this talk, we will look at how DAZN is using streams (Kinesis) and Serverless (AWS Lambda).

Published in: Technology

Streams and serverless at DAZN

  1. 1. serverless and streams at
  2. 2. Yan Cui http://theburningmonk.com @theburningmonk Principal Engineer @ Independent Consultant
  3. 3. “Netflix for sports” offices in London, Leeds, Katowice and Amsterdam
  4. 4. available in Austria, Switzerland, Germany, Japan, Canada, Italy, US, Spain and Brazil
  5. 5. available on 30+ platforms
  6. 6. ~1,000,000 concurrent viewers
  7. 7. “Netflix for sports” offices in London, Leeds, Katowice and Amsterdam We’re hiring! Visit engineering.dazn.com to learn more. follow @DAZN_ngnrs for updates about the engineering team.
  8. 8. we use a LOT
  9. 9. serverless-first
  10. 10. there are no silver bullets
  11. 11. serverless is not right for every use case (yet)
  12. 12. ~1,000,000 concurrent viewers
  13. 13. we use a LOT
  14. 14. API Gateway SQS SNS Step Functions
  15. 15. Kinesis
  16. 16. Kinesis as a task queue
  17. 17. Kinesis as a task queue as a transaction log
  18. 18. Kinesis as a task queue as a transaction log part of an analytics pipeline
  19. 19. https://noti.st/chrismunns/uqUWri/slides
  20. 20. https://noti.st/chrismunns/uqUWri/slides
  21. 21. stream
  22. 22. stream shard shard shard
  23. 23. stream shard shard shard PutRecord: { PartitionKey: … Data: … }
  24. 24. stream shard shard shard PutRecord: { PartitionKey: … Data: … }
  25. 25. stream shard shard shard doX doX doX PutRecord: { PartitionKey: … Data: … }
  26. 26. stream shard shard shard doX doX doX PutRecord: { PartitionKey: … Data: … } ordering is preserved within the same partition key
  27. 27. stream shard shard shard doX doX doX PutRecord: { PartitionKey: … Data: … } ordering is preserved within the same partition key data is retained for up to 7 days, default is 24 hours
  28. 28. doX oldest newest shard
  29. 29. doX oldest newest shard
  30. 30. stream shard shard shard doX doX doX
  31. 31. stream doY doX shard doY doX doY doX shard shard
  32. 32. stream doY doX shard doY doX doY doX shard shard every subscriber gets the same data, in the same order
  33. 33. stream doY doX shard doY doX doY doX shard shard every subscriber gets the same data, in the same order each subscriber process the stream at their own pace
  34. 34. stream doY doX doY doX doY doX shard shard INGRESS 1MB/s data, 1000 records/s shard
  35. 35. stream doY doX doY doX doY shard INGRESS 1MB/s data, 1000 records/s shard EGRESS 2MB/s data per subscriber, 5 reads/s doX shard
  36. 36. stream shard shard shard shard shard horizontalscalability
  37. 37. stream as a task queue
  38. 38. vs. vs. Kinesis SQS SNS
  39. 39. ordering replay events Kinesis SQS SNS mode retry concurrency subscribers
  40. 40. ordering replay events Kinesis SQS SNS mode retry concurrency subscribers many one-to-one many
  41. 41. ordering replay events Kinesis SQS SNS by shard none (standard) global (FIFO) none mode retry concurrency subscribers many one-to-one many
  42. 42. ordering replay events Kinesis SQS SNS by shard none (standard) global (FIFO) none up to 7 days none none mode retry concurrency subscribers many one-to-one many
  43. 43. ordering replay events Kinesis SQS SNS by shard none (standard) global (FIFO) none up to 7 days none none mode retry batched batched (up to 10) singular concurrency subscribers many one-to-one many
  44. 44. ordering replay events Kinesis SQS SNS by shard none (standard) global (FIFO) none up to 7 days none none mode retry batched batched (up to 10) singular retried until success retry + DLQ retry + DLQ concurrency subscribers many one-to-one many
  45. 45. ordering replay events Kinesis SQS SNS by shard none (standard) global (FIFO) none up to 7 days none none mode retry batched batched (up to 10) singular retried until success retry + DLQ retry + DLQ concurrency 1 per shard auto-scaled fan-out!!! subscribers many one-to-one many
  46. 46. ordering replay events Kinesis SQS SNS by shard none (standard) global (FIFO) none up to 7 days none none mode retry batched batched (up to 10) singular retried until success retry + DLQ retry + DLQ concurrency 1 per shard auto-scaled fan-out!!! subscribers many one-to-one many
  47. 47. use the concurrency model to amortize spikes in traffic
  48. 48. leveraging the retry-until-success behaviour
  49. 49. cost is scale-sensitive
  50. 50. 1 msg/s for a month, 1KB per msg 1 x 60s x 60m x 24hr x 30days @ $0.014 per mil + 24hrs x 30days @ $0.015 per hr $10.836 1 x 60s x 60m x 24hr x 30days @ $0.5 per mil $1.296 1 x 60s x 60m x 24hr x 30days @ $0.4 per mil (send) + 1 x 60s x 60m x 24hr x 30days @ 10 batch @ 20s long polling @ $0.4 per mil (receive) $1.089
  51. 51. 1K msg/s for a month, 1KB per msg 1k x 60s x 60m x 24hr x 30days @ $0.014 per mil + 24hrs x 30days @ $0.015 per hr $47.088 1k x 60s x 60m x 24hr x 30days @ $0.5 per mil $1296.00 1k x 60s x 60m x 24hr x 30days @ $0.4 per mil (send) + 1k x 60s x 60m x 24hr x 30days @ 10 batch @ 20s long polling @ $0.4 per mil (receive) $1041.98
  52. 52. stream as a transaction log
  53. 53. http://bit.ly/2Dpidje
  54. 54. events are an enabler for COMPOSABILITY
  55. 55. AWS LAMBDA is the...
  56. 56. Kinesis
  57. 57. Kinesis API Gateway AWS Lambda API GatewayAWS Lambda service-A service-B
  58. 58. Kinesis API Gateway AWS Lambda API GatewayAWS Lambda service-A service-B
  59. 59. Kinesis API Gateway AWS Lambda API GatewayAWS Lambda service-A service-B AWS Lambda AWS Lambda AWS Lambda
  60. 60. Kinesis API Gateway AWS Lambda API GatewayAWS Lambda service-A service-B AWS Lambda AWS Lambda AWS Lambda DynamoDBIOT
  61. 61. Kinesis API Gateway AWS Lambda API GatewayAWS Lambda service-A service-B AWS Lambda AWS Lambda AWS Lambda DynamoDBIOT
  62. 62. Kinesis API Gateway AWS Lambda API GatewayAWS Lambda service-A service-B AWS Lambda AWS Lambda AWS Lambda DynamoDBIOT AWS Lambda AWS Lambda
  63. 63. build loosely-coupled system through events
  64. 64. service A service B service C service D bounded context bounded context
  65. 65. service A service B service C service D bounded context bounded context
  66. 66. service A service B service C service D
  67. 67. service A service B service C service D
  68. 68. service A service B service C service D
  69. 69. service A service B service C service Dbackward-compatible?
  70. 70. bounded context DON’T use events to orchestrate workflows within the same bounded context
  71. 71. bounded context adds unnecessary complexity to logging, tracing, and end-to-end reporting
  72. 72. bounded context the workflow doesn’t exist as a standalone concept, but as the sum of a series of loosely connected parts
  73. 73. Step Functions use Step Functions instead
  74. 74. Step Functions don’t forget to emit events from the workflow
  75. 75. Step Functions so others can react to state changes that happened as part of the workflow
  76. 76. analytics pipeline
  77. 77. S3 Buckets
  78. 78. S3 BucketsKinesis Streams Kinesis Firehose
  79. 79. S3 BucketsKinesis Streams AWS Lambda Kinesis Firehose
  80. 80. S3 BucketsKinesis Streams AWS Lambda AWS Lambda Kinesis Firehose
  81. 81. S3 BucketsKinesis Streams AWS Lambda AWS LambdaAWS Lambda Kinesis Firehose
  82. 82. S3 BucketsKinesis Streams AWS Lambda AWS LambdaAWS Lambda DynamoDB ElasticSearch Kinesis Firehose
  83. 83. S3 BucketsKinesis Streams AWS Lambda AWS Lambda Athena QuickSight AWS Lambda Kinesis Firehose
  84. 84. S3 BucketsKinesis Streams AWS Lambda AWS Lambda Athena QuickSight AWS Lambda Kinesis Firehose
  85. 85. secure sensitive data both at rest and in-transit
  86. 86. leverage server-side encryption
  87. 87. http://amzn.to/1N3Twb8
  88. 88. http://amzn.to/1xF41eX
  89. 89. http://amzn.to/2tgvFR2
  90. 90. http://amzn.to/2GCMbAT
  91. 91. principle of Least Privilege
  92. 92. keep functions single-purposed & apply per function policies
  93. 93. http://bit.ly/2QIVYvL
  94. 94. https://lumigo.io/blog/lambda-and-kinesis-beware-of-hot-streams/
  95. 95. stream doY doX doY doX doY shard shard EGRESS 2MB/s data per subscriber, doX shard 5 reads/s
  96. 96. throughput is maintained as you add more subscribers, but…
  97. 97. it’s difficult to add more than 15 subscriber functions to a stream
  98. 98. my microservices
  99. 99. user_events order_events promo_events my microservices
  100. 100. user_events order_events promo_events my microservices still possible to have too many subscribers for a stream
  101. 101. user_events order_events promo_events my microservices implicit coupling between publisher and subscriber
  102. 102. user_events order_events promo_events my microservices lose ordering of related events in different streams
  103. 103. lose ordering of related events in different streams
  104. 104. preserve ordering of related events
  105. 105. extra cost for these consumer apps
  106. 106. http://bit.ly/2hxZGui
  107. 107. observability
  108. 108. { error: … message: … correlation-id: … } { message: … correlation-id: … }
  109. 109. https://github.com/getndazn/dazn-lambda-powertools
  110. 110. @theburningmonk theburningmonk.com github.com/theburningmonk

×