At Capital One we process billions of events everyday without any data loss and following exactly once processing semantics as well, some of these events are related to fraud detection, customer communications, payments processing...etc. Most of the times it is necessary for us to process these events in real time so we ReImagined auto scaling Kafka consumers to offer beyond predictable performance even during high volumes of inbound events as well as reduce resource utilization whenever possible. We are also making sure that there is no data loss and exactly once event processing when we are offering high tps. Unlike traditional applications, auto scaling an event driven real time stream consumer applications requires much more parameters and edge cases to consider. My talk will be covering all of these parameters, metrics, edge cases and steps to autoscale consumers safely..etc. Here are highlights of my talk: Streaming architecture high level overview which is supporting processing billions of events Our journey to ReImagine Autoscaling of streams Autoscaling benefits and how to Autoscale consumers to offer beyond predictable performance Lessons learned along the way and best practices In addition to this, giving an overview on observability and monitoring streams

1. #REIMAGINE
Autoscaling Stream Consumers in
Cloud Environments
Presenter: Sunil Kaitha
( Master Software Engineer @ Capital One )

3. CLIENT - SERVER APPLICATION

4. Scaling Types

5. SERVICE AUTO SCALING

6. EVENT DRIVEN ARCHITECTURE

7. Event Processing Delays & Resource Utilization
Approximate Calculation
Processed

8. PUSH vs PULL

9. PARTITIONS & CONSUMERS

10. Let’s look at Topic Metrics & Consumer lag

11. Streams Autoscaling

12. Streams Autoscaling

13. Consumers with Auto Scaling Capabilities
without Auto Scaling Capabilities with Auto Scaling Capabilities
Processed
Processed

14. Scaling Strategies
● Scaling out beyond partition count?
● Having minimum consumers zero?
● Ignoring CPU, Memory, Topic Volume..etc?
● Scaling Out and Scaling In in a flash?
● What about Vertical Scaling?

15. Key things to consider..
● Monitoring & Alerting
● Parallelism
● Partition count
● Partition rebalancing strategies (Static membership..etc)
● Resource allocation
● Functional & NFR Testing
● Scaling strategies
● Graceful shutdown of consumers
● Failures and retry
● Offset commits
● Exactly once semantics
● At least once processing

16. Q & A

17. Thank You!
Interested in a career ? Visit @ https://www.capitalonecareers.com/
THE POWER OF ONE
MAKES IT ALL POSSIBLE

18. REFERENCE
● ECS Auto scaling - https://aws.amazon.com/ecs/ ,
https://aws.amazon.com/about-aws/whats-new/2019/12/amazon-ecs-cluster-auto-scaling-now-available/
● Kafka: https://www.confluent.io/
● Capital One Awards and Recognition