Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Processing multi-million events per second Walmart
Kafka Streams at Scale Deepak Goyal Customer Backbone Walmart Labs
app-instance consumer processor rocks db producer akka server Kafka Streams @walmartlabs
Event Flow kafka cluster app cluster app-0 app-0’ stand-by app-0’’ stand-by change-log topic partition 0’’ partition 0’ pa...
Challenges 1. Fault Recovery 2. Horizontal Scalability 3. Cloud Readiness 4. Restricted RocksDB 5. Large Clusters @walmart...
keep streaming . . . . . . . . . . . . . . . . . . .
Upcoming SlideShare
Loading in …5
×

Kafka streams at Scale Preview

119 views

Published on

Walmart.com generates millions of events per second. At WalmartLabs, I’m working in a team called the Customer Backbone (CBB), where we wanted to upgrade to a platform capable of processing this event volume in real-time and store the state/knowledge of possibly all the Walmart Customers generated by the processing. Kafka streams’ event-driven architecture seemed like the only obvious choice.

However, there are a few challenges w.r.t. Walmart’s scale:

• the clusters need to be large and the problems thereof.

• infinite retention of changelog topics, wasting valuable disk.

• slow stand-by task recovery in case of a node failure (changelog topics have GBs of data)

• no repartitioning in Kafka Streams.



As part of the event-driven development and addressing the challenges above, I’m going to talk about some bold new ideas we developed as features/patches to Kafka Streams to deal with the scale required at Walmart.

• Cold Bootstrap: Where in case of a Kafka Streams node failure, how instead of recovering from the change-log topic, we bootstrap the standby from active’s RocksDB using JSch and zero event loss by careful offset management.

• Dynamic Repartitioning: We added support for repartitioning in Kafka Streams where state is distributed among the new partitions. We can now elastically scale to any number of partitions and any number of nodes.

• Cloud/Rack/AZ aware task assignment: No active and standby tasks of the same partition are assigned to the same rack.

• Decreased Partition Assignment Size: With large clusters like ours (>400 nodes and 3 stream threads per node), the size of Partition Assignment of the KS cluster being few 100MBs, it takes a lot of time to settle a rebalance.

Key Takeaways:

• Basic understanding of Kafka Streams.

• Productionizing Kafka Streams at scale.

• Using Kafka Streams as Distributed NoSQL DB

Published in: Engineering
no profile picture user

  • Login to see the comments

Kafka streams at Scale Preview

  1. 1. Processing multi-million events per second Walmart
  2. 2. Kafka Streams at Scale Deepak Goyal Customer Backbone Walmart Labs
  3. 3. app-instance consumer processor rocks db producer akka server Kafka Streams @walmartlabs
  4. 4. Event Flow kafka cluster app cluster app-0 app-0’ stand-by app-0’’ stand-by change-log topic partition 0’’ partition 0’ partition 0 input topic partition 0’’ partition 0’ partition 0 rocks db rocks db active rocks db Event Flow @walmartlabs
  5. 5. Challenges 1. Fault Recovery 2. Horizontal Scalability 3. Cloud Readiness 4. Restricted RocksDB 5. Large Clusters @walmartlabs
  6. 6. keep streaming . . . . . . . . . . . . . . . . . . .

×