SlideShare a Scribd company logo
© Walmart Inc. 2019
When Kafka Meets the
Scaling and Reliability
Needs of World’s Largest
Retailer
A Walmart Story
Presented By:
Suman Pattnaik, Walmart Labs
Prasanna Subburaj, Walmart Labs
© Walmart Inc. 2019
Any reference in this “When Kafka Meets the Scaling and Reliability
Needs of World’s Largest Retailer” presentation to any specific
commercial product, process, or service, or the use of any trade, firm
or corporation name is for information and convenience purposes
only, and does not constitute an endorsement or recommendation by
Walmart Inc.”
© Walmart Inc. 2019
Agenda1. About Us
2. Use Case Introduction
3. The Ask & Our Scale
4. Kafka’s Role in Our User
Case
5. Our Approach
6. Key Learnings
Agenda
© Walmart Inc. 2019
About Us
© Walmart Inc. 2019
Walmart Global Presence
© Walmart Inc. 2019
Meet our family in US.
ModCloth
San Francisco, CA
Walmart eCommerce
San Bruno, CA
Walmart Labs
Sunnyvale, CA
Vudu
Sunnyvale, CA
Hayneedle HQ
Omaha, NE
Moosejaw HQ
Madison Heights, MI
Walmart Home
Office
Bentonville, AR
Shoes.com
Boston, MA
Bonobos
New York, NY
Store No8
New York, NY
Parcel
New York, NY
Jet.com
Hoboken, NJ
© Walmart Inc. 2019
Our Use Case: Retail Inventory Management
© Walmart Inc. 2019
The Ask - Realtime and
Reliable Inventory
Visibility
• Accuracy
• Speed
• Reduced Complexity
• Elasticity and Scalability
• Improved resiliency
• Reduced Cost
© Walmart Inc. 2019
5000+ Stores
150+
Distribution
Center
1000+ Vendors
25 M+ B&M
Customer every
week
53 K+ Trailers
Owned
1M+ Online
Transactions
Walmart US - At Scale
© Walmart Inc. 2019
Our Use Case: At Scale
© Walmart Inc. 2019
MVP in 2015
© Walmart Inc. 2019
Kafka’s Role in Our Inventory Management
© Walmart Inc. 2019
Our Solution
© Walmart Inc. 2019
Realizing the Possibility…
© Walmart Inc. 2019
Interacts with 13
Teams Constantly
Ingests about 500
million events
daily
Daily Interaction - Fun Facts
Working with 6
other teams for
integration
Expected to add
650 million events
a day
© Walmart Inc. 2019
Key Learning – Kafka Streams
• Reasons for choosing Processor API
• Removal of commits in each processor
• num.stream.threads to increase parallelism
© Walmart Inc. 2019
Key Learning – Kafka Streams
• max.partition.fetch.bytes
• partitioner.class customization to avoid dirty writes
• retries, acks, max.in.flight.requests.per.connection
tuning to guarantee delivery
© Walmart Inc. 2019
Key Learning – Cassandra Kafka Sink
Connectors
• Number of partitions in a topic and parallelism
• Retry logic to handle failures in cassandra
• Kafka connect distributed properties
© Walmart Inc. 2019
Key Learning – HDFS Kafka Sink Connectors
• max.poll.interval.ms, max.poll.records
• session.timeout.ms
• Issues arising from # of partition in source topic
• More than one partition in config topic & rebalance
© Walmart Inc. 2019
Key Learning – Spark Streaming
• Issues maintaining consumer offsets in zookeeper
• Hadoop cluster resources issue and dedicated queue
• Small file issue and compaction
© Walmart Inc. 2019
Key Learning – Design & Architecture
• Choose design that satisfies Speed and Reliability
• Start building system and data resiliency early on
• Contracts should lends flexibility & manageability
• Telemetry, Logging, Alerting & Auditing
© Walmart Inc. 2019
Reference
• https://abcnews.go.com/Business/walmart-tops-fortunes-
list-500-largest-us-corporations/story?id=62852829
• https://en.wikipedia.org/wiki/Walmart
• https://corporate.walmart.com/our-story/our-locations
• https://corporate.walmart.com/our-story/our-history
© Walmart Inc. 2019
Questions

More Related Content

What's hot

Envoy and Kafka
Envoy and KafkaEnvoy and Kafka
Envoy and Kafka
Adam Kotwasinski
 
Deploying Confluent Platform for Production
Deploying Confluent Platform for ProductionDeploying Confluent Platform for Production
Deploying Confluent Platform for Production
confluent
 
Apache Flink, AWS Kinesis, Analytics
Apache Flink, AWS Kinesis, Analytics Apache Flink, AWS Kinesis, Analytics
Apache Flink, AWS Kinesis, Analytics
Araf Karsh Hamid
 
Introduction to Kafka Streams
Introduction to Kafka StreamsIntroduction to Kafka Streams
Introduction to Kafka Streams
Guozhang Wang
 
What is Apache Kafka and What is an Event Streaming Platform?
What is Apache Kafka and What is an Event Streaming Platform?What is Apache Kafka and What is an Event Streaming Platform?
What is Apache Kafka and What is an Event Streaming Platform?
confluent
 
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Kafka Tutorial - Introduction to Apache Kafka (Part 1)Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Jean-Paul Azar
 
Data Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to MeshData Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to Mesh
Jeffrey T. Pollock
 
Tuning kafka pipelines
Tuning kafka pipelinesTuning kafka pipelines
Tuning kafka pipelines
Sumant Tambe
 
Introduction to Kafka Cruise Control
Introduction to Kafka Cruise ControlIntroduction to Kafka Cruise Control
Introduction to Kafka Cruise Control
Jiangjie Qin
 
Apache Kafka in the Airline, Aviation and Travel Industry
Apache Kafka in the Airline, Aviation and Travel IndustryApache Kafka in the Airline, Aviation and Travel Industry
Apache Kafka in the Airline, Aviation and Travel Industry
Kai Wähner
 
Apache kafka
Apache kafkaApache kafka
Apache kafka
Kumar Shivam
 
Change Data Streaming Patterns for Microservices With Debezium
Change Data Streaming Patterns for Microservices With Debezium Change Data Streaming Patterns for Microservices With Debezium
Change Data Streaming Patterns for Microservices With Debezium
confluent
 
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaThe Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
Kai Wähner
 
Introduction to Apache Kafka
Introduction to Apache KafkaIntroduction to Apache Kafka
Introduction to Apache Kafka
Shiao-An Yuan
 
Stream Processing with Flink and Stream Sharing
Stream Processing with Flink and Stream SharingStream Processing with Flink and Stream Sharing
Stream Processing with Flink and Stream Sharing
confluent
 
Set your Data in Motion with Confluent & Apache Kafka Tech Talk Series LME
Set your Data in Motion with Confluent & Apache Kafka Tech Talk Series LMESet your Data in Motion with Confluent & Apache Kafka Tech Talk Series LME
Set your Data in Motion with Confluent & Apache Kafka Tech Talk Series LME
confluent
 
Securing Kafka
Securing Kafka Securing Kafka
Securing Kafka
confluent
 
Kafka At Scale in the Cloud
Kafka At Scale in the CloudKafka At Scale in the Cloud
Kafka At Scale in the Cloud
confluent
 
Tuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptxTuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptx
Flink Forward
 
SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...
SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...
SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...
HostedbyConfluent
 

What's hot (20)

Envoy and Kafka
Envoy and KafkaEnvoy and Kafka
Envoy and Kafka
 
Deploying Confluent Platform for Production
Deploying Confluent Platform for ProductionDeploying Confluent Platform for Production
Deploying Confluent Platform for Production
 
Apache Flink, AWS Kinesis, Analytics
Apache Flink, AWS Kinesis, Analytics Apache Flink, AWS Kinesis, Analytics
Apache Flink, AWS Kinesis, Analytics
 
Introduction to Kafka Streams
Introduction to Kafka StreamsIntroduction to Kafka Streams
Introduction to Kafka Streams
 
What is Apache Kafka and What is an Event Streaming Platform?
What is Apache Kafka and What is an Event Streaming Platform?What is Apache Kafka and What is an Event Streaming Platform?
What is Apache Kafka and What is an Event Streaming Platform?
 
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Kafka Tutorial - Introduction to Apache Kafka (Part 1)Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
 
Data Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to MeshData Mesh Part 4 Monolith to Mesh
Data Mesh Part 4 Monolith to Mesh
 
Tuning kafka pipelines
Tuning kafka pipelinesTuning kafka pipelines
Tuning kafka pipelines
 
Introduction to Kafka Cruise Control
Introduction to Kafka Cruise ControlIntroduction to Kafka Cruise Control
Introduction to Kafka Cruise Control
 
Apache Kafka in the Airline, Aviation and Travel Industry
Apache Kafka in the Airline, Aviation and Travel IndustryApache Kafka in the Airline, Aviation and Travel Industry
Apache Kafka in the Airline, Aviation and Travel Industry
 
Apache kafka
Apache kafkaApache kafka
Apache kafka
 
Change Data Streaming Patterns for Microservices With Debezium
Change Data Streaming Patterns for Microservices With Debezium Change Data Streaming Patterns for Microservices With Debezium
Change Data Streaming Patterns for Microservices With Debezium
 
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaThe Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
 
Introduction to Apache Kafka
Introduction to Apache KafkaIntroduction to Apache Kafka
Introduction to Apache Kafka
 
Stream Processing with Flink and Stream Sharing
Stream Processing with Flink and Stream SharingStream Processing with Flink and Stream Sharing
Stream Processing with Flink and Stream Sharing
 
Set your Data in Motion with Confluent & Apache Kafka Tech Talk Series LME
Set your Data in Motion with Confluent & Apache Kafka Tech Talk Series LMESet your Data in Motion with Confluent & Apache Kafka Tech Talk Series LME
Set your Data in Motion with Confluent & Apache Kafka Tech Talk Series LME
 
Securing Kafka
Securing Kafka Securing Kafka
Securing Kafka
 
Kafka At Scale in the Cloud
Kafka At Scale in the CloudKafka At Scale in the Cloud
Kafka At Scale in the Cloud
 
Tuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptxTuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptx
 
SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...
SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...
SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...
 

Similar to When Kafka Meets the Scaling and Reliability needs of World's Largest Retailer : A WALMART Story (Suman Pattnaik & Prasanna Subburaj, Walmart) Kafka Summit SF 2019

¿Son las bases de datos de contabilidad interesantes, o son parte del hype al...
¿Son las bases de datos de contabilidad interesantes, o son parte del hype al...¿Son las bases de datos de contabilidad interesantes, o son parte del hype al...
¿Son las bases de datos de contabilidad interesantes, o son parte del hype al...
javier ramirez
 
De un monolito a microservicios
De un monolito a microserviciosDe un monolito a microservicios
De un monolito a microservicios
Amazon Web Services
 
Introduction to AWS Travel by Massimo Morin
Introduction to AWS Travel by Massimo MorinIntroduction to AWS Travel by Massimo Morin
Introduction to AWS Travel by Massimo Morin
Sameer Kenkare
 
AWS Startup Day Bogotá - The Pitch
AWS Startup Day Bogotá - The PitchAWS Startup Day Bogotá - The Pitch
AWS Startup Day Bogotá - The Pitch
Amazon Web Services LATAM
 
AWS Startup Day Guadalajara - Pitch essentials
AWS Startup Day Guadalajara - Pitch essentialsAWS Startup Day Guadalajara - Pitch essentials
AWS Startup Day Guadalajara - Pitch essentials
Amazon Web Services LATAM
 
AWS AI and Machine Learning Journey
AWS AI and Machine Learning JourneyAWS AI and Machine Learning Journey
AWS AI and Machine Learning Journey
Amazon Web Services
 
Mastering Digital Channels with APIs
Mastering Digital Channels with APIsMastering Digital Channels with APIs
Mastering Digital Channels with APIs
CA API Management
 
AWS Startup Day Santiago - Taram: Fundraising Essentials
AWS Startup Day Santiago - Taram: Fundraising EssentialsAWS Startup Day Santiago - Taram: Fundraising Essentials
AWS Startup Day Santiago - Taram: Fundraising Essentials
Amazon Web Services LATAM
 
打造雲通路加速新零售轉型.pdf
打造雲通路加速新零售轉型.pdf打造雲通路加速新零售轉型.pdf
打造雲通路加速新零售轉型.pdf
Amazon Web Services
 
TECHTalks - Philadelphia PA - Brien Blandford
  TECHTalks - Philadelphia PA - Brien Blandford  TECHTalks - Philadelphia PA - Brien Blandford
TECHTalks - Philadelphia PA - Brien Blandford
EagleDream Technologies
 
Improving manufacturing operations is everything - MFG401 - Mexico City AWS S...
Improving manufacturing operations is everything - MFG401 - Mexico City AWS S...Improving manufacturing operations is everything - MFG401 - Mexico City AWS S...
Improving manufacturing operations is everything - MFG401 - Mexico City AWS S...
Amazon Web Services
 
Initiate Edinburgh 2019 - Moving to DevOps the Amazon Way
Initiate Edinburgh 2019 - Moving to DevOps the Amazon WayInitiate Edinburgh 2019 - Moving to DevOps the Amazon Way
Initiate Edinburgh 2019 - Moving to DevOps the Amazon Way
Amazon Web Services
 
AWS Initiate Day Manchester 2019 – Moving to DevOps the Amazon Way
AWS Initiate Day Manchester 2019 – Moving to DevOps the Amazon WayAWS Initiate Day Manchester 2019 – Moving to DevOps the Amazon Way
AWS Initiate Day Manchester 2019 – Moving to DevOps the Amazon Way
Amazon Web Services
 
The Pitch: Essentials for Success, and Blunders to Avoid
The Pitch: Essentials for Success, and Blunders to AvoidThe Pitch: Essentials for Success, and Blunders to Avoid
The Pitch: Essentials for Success, and Blunders to Avoid
Amazon Web Services
 
Gaining More Value from your Ariba Network Connection
Gaining More Value from your Ariba Network ConnectionGaining More Value from your Ariba Network Connection
Gaining More Value from your Ariba Network Connection
SAP Ariba
 
Fundraising Essentials for Every Entrepreneur
Fundraising Essentials for Every EntrepreneurFundraising Essentials for Every Entrepreneur
Fundraising Essentials for Every Entrepreneur
Amazon Web Services
 
[AWS summit 2019] 마이크로 서비스 패턴 데이터 베이스
[AWS summit 2019] 마이크로 서비스 패턴 데이터 베이스[AWS summit 2019] 마이크로 서비스 패턴 데이터 베이스
[AWS summit 2019] 마이크로 서비스 패턴 데이터 베이스
KyungHo Joo
 
AWS Summit Singapore 2019 | Driving Business Outcomes with Data Lake on AWS
AWS Summit Singapore 2019 | Driving Business Outcomes with Data Lake on AWSAWS Summit Singapore 2019 | Driving Business Outcomes with Data Lake on AWS
AWS Summit Singapore 2019 | Driving Business Outcomes with Data Lake on AWS
AWS Summits
 
AWS Cloud Experience CA: Receta del Éxito en la Nube: Cultura, Estrategia e I...
AWS Cloud Experience CA: Receta del Éxito en la Nube: Cultura, Estrategia e I...AWS Cloud Experience CA: Receta del Éxito en la Nube: Cultura, Estrategia e I...
AWS Cloud Experience CA: Receta del Éxito en la Nube: Cultura, Estrategia e I...
Amazon Web Services LATAM
 
Culture of Innovation at Amazon - AWS Startup Day Johannesburg.pdf
Culture of Innovation at Amazon - AWS Startup Day Johannesburg.pdfCulture of Innovation at Amazon - AWS Startup Day Johannesburg.pdf
Culture of Innovation at Amazon - AWS Startup Day Johannesburg.pdf
Amazon Web Services
 

Similar to When Kafka Meets the Scaling and Reliability needs of World's Largest Retailer : A WALMART Story (Suman Pattnaik & Prasanna Subburaj, Walmart) Kafka Summit SF 2019 (20)

¿Son las bases de datos de contabilidad interesantes, o son parte del hype al...
¿Son las bases de datos de contabilidad interesantes, o son parte del hype al...¿Son las bases de datos de contabilidad interesantes, o son parte del hype al...
¿Son las bases de datos de contabilidad interesantes, o son parte del hype al...
 
De un monolito a microservicios
De un monolito a microserviciosDe un monolito a microservicios
De un monolito a microservicios
 
Introduction to AWS Travel by Massimo Morin
Introduction to AWS Travel by Massimo MorinIntroduction to AWS Travel by Massimo Morin
Introduction to AWS Travel by Massimo Morin
 
AWS Startup Day Bogotá - The Pitch
AWS Startup Day Bogotá - The PitchAWS Startup Day Bogotá - The Pitch
AWS Startup Day Bogotá - The Pitch
 
AWS Startup Day Guadalajara - Pitch essentials
AWS Startup Day Guadalajara - Pitch essentialsAWS Startup Day Guadalajara - Pitch essentials
AWS Startup Day Guadalajara - Pitch essentials
 
AWS AI and Machine Learning Journey
AWS AI and Machine Learning JourneyAWS AI and Machine Learning Journey
AWS AI and Machine Learning Journey
 
Mastering Digital Channels with APIs
Mastering Digital Channels with APIsMastering Digital Channels with APIs
Mastering Digital Channels with APIs
 
AWS Startup Day Santiago - Taram: Fundraising Essentials
AWS Startup Day Santiago - Taram: Fundraising EssentialsAWS Startup Day Santiago - Taram: Fundraising Essentials
AWS Startup Day Santiago - Taram: Fundraising Essentials
 
打造雲通路加速新零售轉型.pdf
打造雲通路加速新零售轉型.pdf打造雲通路加速新零售轉型.pdf
打造雲通路加速新零售轉型.pdf
 
TECHTalks - Philadelphia PA - Brien Blandford
  TECHTalks - Philadelphia PA - Brien Blandford  TECHTalks - Philadelphia PA - Brien Blandford
TECHTalks - Philadelphia PA - Brien Blandford
 
Improving manufacturing operations is everything - MFG401 - Mexico City AWS S...
Improving manufacturing operations is everything - MFG401 - Mexico City AWS S...Improving manufacturing operations is everything - MFG401 - Mexico City AWS S...
Improving manufacturing operations is everything - MFG401 - Mexico City AWS S...
 
Initiate Edinburgh 2019 - Moving to DevOps the Amazon Way
Initiate Edinburgh 2019 - Moving to DevOps the Amazon WayInitiate Edinburgh 2019 - Moving to DevOps the Amazon Way
Initiate Edinburgh 2019 - Moving to DevOps the Amazon Way
 
AWS Initiate Day Manchester 2019 – Moving to DevOps the Amazon Way
AWS Initiate Day Manchester 2019 – Moving to DevOps the Amazon WayAWS Initiate Day Manchester 2019 – Moving to DevOps the Amazon Way
AWS Initiate Day Manchester 2019 – Moving to DevOps the Amazon Way
 
The Pitch: Essentials for Success, and Blunders to Avoid
The Pitch: Essentials for Success, and Blunders to AvoidThe Pitch: Essentials for Success, and Blunders to Avoid
The Pitch: Essentials for Success, and Blunders to Avoid
 
Gaining More Value from your Ariba Network Connection
Gaining More Value from your Ariba Network ConnectionGaining More Value from your Ariba Network Connection
Gaining More Value from your Ariba Network Connection
 
Fundraising Essentials for Every Entrepreneur
Fundraising Essentials for Every EntrepreneurFundraising Essentials for Every Entrepreneur
Fundraising Essentials for Every Entrepreneur
 
[AWS summit 2019] 마이크로 서비스 패턴 데이터 베이스
[AWS summit 2019] 마이크로 서비스 패턴 데이터 베이스[AWS summit 2019] 마이크로 서비스 패턴 데이터 베이스
[AWS summit 2019] 마이크로 서비스 패턴 데이터 베이스
 
AWS Summit Singapore 2019 | Driving Business Outcomes with Data Lake on AWS
AWS Summit Singapore 2019 | Driving Business Outcomes with Data Lake on AWSAWS Summit Singapore 2019 | Driving Business Outcomes with Data Lake on AWS
AWS Summit Singapore 2019 | Driving Business Outcomes with Data Lake on AWS
 
AWS Cloud Experience CA: Receta del Éxito en la Nube: Cultura, Estrategia e I...
AWS Cloud Experience CA: Receta del Éxito en la Nube: Cultura, Estrategia e I...AWS Cloud Experience CA: Receta del Éxito en la Nube: Cultura, Estrategia e I...
AWS Cloud Experience CA: Receta del Éxito en la Nube: Cultura, Estrategia e I...
 
Culture of Innovation at Amazon - AWS Startup Day Johannesburg.pdf
Culture of Innovation at Amazon - AWS Startup Day Johannesburg.pdfCulture of Innovation at Amazon - AWS Startup Day Johannesburg.pdf
Culture of Innovation at Amazon - AWS Startup Day Johannesburg.pdf
 

More from confluent

Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutes
confluent
 
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Era
confluent
 
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
confluent
 
Santander Stream Processing with Apache Flink
Santander Stream Processing with Apache FlinkSantander Stream Processing with Apache Flink
Santander Stream Processing with Apache Flink
confluent
 
Unlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insightsUnlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insights
confluent
 
Workshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con FlinkWorkshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con Flink
confluent
 
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
confluent
 
AWS Immersion Day Mapfre - Confluent
AWS Immersion Day Mapfre   -   ConfluentAWS Immersion Day Mapfre   -   Confluent
AWS Immersion Day Mapfre - Confluent
confluent
 
Eventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalkEventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalk
confluent
 
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent CloudQ&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
confluent
 
Citi TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep DiveCiti TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep Dive
confluent
 
Build real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with ConfluentBuild real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with Confluent
confluent
 
Q&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service MeshQ&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service Mesh
confluent
 
Citi Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka MicroservicesCiti Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka Microservices
confluent
 
Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3
confluent
 
Citi Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging ModernizationCiti Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging Modernization
confluent
 
Citi Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time dataCiti Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time data
confluent
 
Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2
confluent
 
Data In Motion Paris 2023
Data In Motion Paris 2023Data In Motion Paris 2023
Data In Motion Paris 2023
confluent
 
Confluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with SynthesisConfluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with Synthesis
confluent
 

More from confluent (20)

Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutes
 
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Era
 
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
 
Santander Stream Processing with Apache Flink
Santander Stream Processing with Apache FlinkSantander Stream Processing with Apache Flink
Santander Stream Processing with Apache Flink
 
Unlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insightsUnlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insights
 
Workshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con FlinkWorkshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con Flink
 
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
 
AWS Immersion Day Mapfre - Confluent
AWS Immersion Day Mapfre   -   ConfluentAWS Immersion Day Mapfre   -   Confluent
AWS Immersion Day Mapfre - Confluent
 
Eventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalkEventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalk
 
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent CloudQ&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
 
Citi TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep DiveCiti TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep Dive
 
Build real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with ConfluentBuild real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with Confluent
 
Q&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service MeshQ&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service Mesh
 
Citi Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka MicroservicesCiti Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka Microservices
 
Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3
 
Citi Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging ModernizationCiti Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging Modernization
 
Citi Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time dataCiti Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time data
 
Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2
 
Data In Motion Paris 2023
Data In Motion Paris 2023Data In Motion Paris 2023
Data In Motion Paris 2023
 
Confluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with SynthesisConfluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with Synthesis
 

Recently uploaded

Video Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the FutureVideo Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the Future
Alpen-Adria-Universität
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc
 
GenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizationsGenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizations
kumardaparthi1024
 
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024
Jason Packer
 
How to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For FlutterHow to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For Flutter
Daiki Mogmet Ito
 
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalizationFull-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
Zilliz
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
Tomaz Bratanic
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Brandon Minnick, MBA
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Speck&Tech
 
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
Zilliz
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
Kari Kakkonen
 
June Patch Tuesday
June Patch TuesdayJune Patch Tuesday
June Patch Tuesday
Ivanti
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
shyamraj55
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
Matthew Sinclair
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
innovationoecd
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Malak Abu Hammad
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
panagenda
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
Quotidiano Piemontese
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
Matthew Sinclair
 
Microsoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdfMicrosoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdf
Uni Systems S.M.S.A.
 

Recently uploaded (20)

Video Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the FutureVideo Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the Future
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
 
GenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizationsGenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizations
 
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024
 
How to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For FlutterHow to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For Flutter
 
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalizationFull-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
 
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
 
June Patch Tuesday
June Patch TuesdayJune Patch Tuesday
June Patch Tuesday
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
 
Microsoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdfMicrosoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdf
 

When Kafka Meets the Scaling and Reliability needs of World's Largest Retailer : A WALMART Story (Suman Pattnaik & Prasanna Subburaj, Walmart) Kafka Summit SF 2019

  • 1. © Walmart Inc. 2019 When Kafka Meets the Scaling and Reliability Needs of World’s Largest Retailer A Walmart Story Presented By: Suman Pattnaik, Walmart Labs Prasanna Subburaj, Walmart Labs
  • 2. © Walmart Inc. 2019 Any reference in this “When Kafka Meets the Scaling and Reliability Needs of World’s Largest Retailer” presentation to any specific commercial product, process, or service, or the use of any trade, firm or corporation name is for information and convenience purposes only, and does not constitute an endorsement or recommendation by Walmart Inc.”
  • 3. © Walmart Inc. 2019 Agenda1. About Us 2. Use Case Introduction 3. The Ask & Our Scale 4. Kafka’s Role in Our User Case 5. Our Approach 6. Key Learnings Agenda
  • 4. © Walmart Inc. 2019 About Us
  • 5. © Walmart Inc. 2019 Walmart Global Presence
  • 6. © Walmart Inc. 2019 Meet our family in US. ModCloth San Francisco, CA Walmart eCommerce San Bruno, CA Walmart Labs Sunnyvale, CA Vudu Sunnyvale, CA Hayneedle HQ Omaha, NE Moosejaw HQ Madison Heights, MI Walmart Home Office Bentonville, AR Shoes.com Boston, MA Bonobos New York, NY Store No8 New York, NY Parcel New York, NY Jet.com Hoboken, NJ
  • 7. © Walmart Inc. 2019 Our Use Case: Retail Inventory Management
  • 8. © Walmart Inc. 2019 The Ask - Realtime and Reliable Inventory Visibility • Accuracy • Speed • Reduced Complexity • Elasticity and Scalability • Improved resiliency • Reduced Cost
  • 9. © Walmart Inc. 2019 5000+ Stores 150+ Distribution Center 1000+ Vendors 25 M+ B&M Customer every week 53 K+ Trailers Owned 1M+ Online Transactions Walmart US - At Scale
  • 10. © Walmart Inc. 2019 Our Use Case: At Scale
  • 11. © Walmart Inc. 2019 MVP in 2015
  • 12. © Walmart Inc. 2019 Kafka’s Role in Our Inventory Management
  • 13. © Walmart Inc. 2019 Our Solution
  • 14. © Walmart Inc. 2019 Realizing the Possibility…
  • 15. © Walmart Inc. 2019 Interacts with 13 Teams Constantly Ingests about 500 million events daily Daily Interaction - Fun Facts Working with 6 other teams for integration Expected to add 650 million events a day
  • 16. © Walmart Inc. 2019 Key Learning – Kafka Streams • Reasons for choosing Processor API • Removal of commits in each processor • num.stream.threads to increase parallelism
  • 17. © Walmart Inc. 2019 Key Learning – Kafka Streams • max.partition.fetch.bytes • partitioner.class customization to avoid dirty writes • retries, acks, max.in.flight.requests.per.connection tuning to guarantee delivery
  • 18. © Walmart Inc. 2019 Key Learning – Cassandra Kafka Sink Connectors • Number of partitions in a topic and parallelism • Retry logic to handle failures in cassandra • Kafka connect distributed properties
  • 19. © Walmart Inc. 2019 Key Learning – HDFS Kafka Sink Connectors • max.poll.interval.ms, max.poll.records • session.timeout.ms • Issues arising from # of partition in source topic • More than one partition in config topic & rebalance
  • 20. © Walmart Inc. 2019 Key Learning – Spark Streaming • Issues maintaining consumer offsets in zookeeper • Hadoop cluster resources issue and dedicated queue • Small file issue and compaction
  • 21. © Walmart Inc. 2019 Key Learning – Design & Architecture • Choose design that satisfies Speed and Reliability • Start building system and data resiliency early on • Contracts should lends flexibility & manageability • Telemetry, Logging, Alerting & Auditing
  • 22. © Walmart Inc. 2019 Reference • https://abcnews.go.com/Business/walmart-tops-fortunes- list-500-largest-us-corporations/story?id=62852829 • https://en.wikipedia.org/wiki/Walmart • https://corporate.walmart.com/our-story/our-locations • https://corporate.walmart.com/our-story/our-history
  • 23. © Walmart Inc. 2019 Questions