SlideShare a Scribd company logo
1 of 25
Download to read offline
Cisco's eCommerce Transformation
using Kafka
Presented By:
Dharmesh Panchmatia (Sr. Director – Cisco Systems)
Gaurav Goyal (Principal Architect – Cisco Systems)
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Agenda
Kafka Architecture2
1 Kafka Use Cases
Kafka Monitoring3
Lessons Learnt4
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Orders booked
$50+B
138 Countries
63 Device types 16 Browsers
185KUsers16 Languages
6M Hits/day
6.9 M Estimates 5.3 M Quotes 1.9 M Orders 85.6% Orders
Orders
Autobook
Portal 71% B2B 29%
Cisco Commerce By The Numbers
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Ref Data
REFERENCE DATA SOURCE
DMPRD - RDBMS
Logging
Order Capture
DC1 - Tomcat
Order Capture
Transaction
Data
Downstream
Publish
X-Functional
Services (73)
DC3DC2DC1
TRANSACTION DATA STORE
P S S S S
N1 N2 N3 N4 N5
DC2 - Tomcat
1 2
3
4
Addresses Items
Preferences
Roles
Contacts Logging
DC1 & DC2
Commerce – Cloud Native
Kafka Use Cases
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Kafka – Use Cases
Data push to
downstreams
1. Avoid point to point
integration.
2. Avoid direct
connection to
transactional DB.
Elastic Search Data
Push
1. Reduce load on
transactional DB
2. Eliminates ES out of
sync in multi-DCs
Machine Learning Use
Cases using Spark
1. Recommendation
Engine
2. Most Popular
Configurations
3. Most popular products
for a given category
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Customer who bought X also bought Y.
Identify products which are
mostly bought together so we can create
bundles or promotions accordingly.
1
2
Algorithm: Apriori
ML Use Case
1. Recommendation Engine
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Provide visibility to most popular
configurations for a given product.
Provide visibility to a configuration which
Customer has recently bought for the
given product.
1
2
Allow selection of pre-configured products
instead of starting from scratch.
ML Use Case
2. Popular Product Configuration
Kafka Architecture
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Producer (Capture Order) Producer (Return Order)
Broker 1 Broker 2 Broker 3 Broker 4
ZK - 1 ZK - 2 ZK - 3 ZK - 4 ZK - 5
Consumer (Smart- SW SC)
Kafka Cluster
Zookeeper
DC1 DC2
DC1 DC2 DC3
DC1 – RCDN; DC2 – ALLEN; DC3 – RTP
Coordinates cluster membership
Commit Offset (v 0.10.x.x)
Kafka Architecture
Consumer (EDW)
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
RDBMS
ProducerCustom Code (DC1) Custom Code (DC2)FAULT
TOLERENT
Kafka
DC1 and DC2
Consumer Group - DC1 Consumer Group – DC2
Elastic Search – DC1 Elastic Search – DC2
Kafka Architecture – Elastic Search
Transaction Data
Order,
Estimate
Quote
RDBMS
Reference Data
Click Stream
Data
Data Visualization
Dynamic Querying
Data Science
Primary Analytics Data Store
MQL
Transaction Data
RDBMS
Subscriptions
Invoices
Kafka Architecture – ML & Analytics Use Case
Kafka Monitoring
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Monitoring: Kafka Manager and Kafdrop
Kafka Manager Kafdrop
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Kafka – Custom Scripts
1. Cron job to check Kafka
processes every minute. Restart
Kafka process
(and send email) in case it’s not
running.
2. Always take back up of logs
systematically when Kafka processes
are getting restarted.
3. Have a test topic and push test
message every minute. Trigger a
notification in case of failures.
Best Practices / Lessons Learnt
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Best
Practices
1
Have a mechanism to reset Kafka offsets on
demand.
4 Auto Re-push mechanism in case producer
gets error while pushing data into Kafka
2
Have a mechanism to re-push data to
Kafka topic,
3 Enable SSL for secure access
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
UI – Reset Offsets
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
UI – Re-push data
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Kafka Producer & Consumer Setup with SSL
Below properties are required to enable SSL for both Producer and Consumer
If client authentication is not required in
the broker then below configuration is
suffice, (kafka.client.truststore.jks will
be provided by kafka service host.)
1
If client authentication is required in the
broker then below configuration is
required. (kafka.client.keystore.jks will
be provided by kafka service host. )
2
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Auto Re-Push Mechanism
Failure
Source
Data Push
In case of failures
Offline
Scheduler
Failed records
Re - Push
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Lessons
Learnt
1
Have a while loop while subscribing to any
Kafka Topics instead of creating
consumer every time.
4
Data Size - consumer's
max.partition.fetch.bytes should be greater
or equals to the producers
producer.max.request.size Default is 1MB.
2
Always use key if you want all messages
for a particular key (e.g. order id) always
goes to a particular partition.
3
enable.auto.commit - Default is true. It is
better to set it false to get control over
when to commit the offset.
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Lessons
Learnt
5
Have a custom script deployed to monitor &
restart Kafka nodes in case of
any issues.
8
Reset offset: Make sure there is no active
consumer on this topic for that
consumer group.
6
heartbeat.interval.ms must be smaller
than session.timeout.ms.
session.timeout.ms : it controls the time it
takes to detect a consumer crash and
stop sending heartbeats.
heartbeat.interval.ms :The expected time
between heartbeats to the consumer
7 auto.offset.reset -default latest
Questions and Answers
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Kafka Architecture – ML Use Case
Quote
Stream
Order
Stream

More Related Content

What's hot

Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...
Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...
Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...
Timothy Spann
 

What's hot (20)

Introduction to Apache Kafka
Introduction to Apache KafkaIntroduction to Apache Kafka
Introduction to Apache Kafka
 
ksqlDB: A Stream-Relational Database System
ksqlDB: A Stream-Relational Database SystemksqlDB: A Stream-Relational Database System
ksqlDB: A Stream-Relational Database System
 
Kafka Retry and DLQ
Kafka Retry and DLQKafka Retry and DLQ
Kafka Retry and DLQ
 
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Kafka Tutorial - Introduction to Apache Kafka (Part 1)Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
 
Kafka Streams vs. KSQL for Stream Processing on top of Apache Kafka
Kafka Streams vs. KSQL for Stream Processing on top of Apache KafkaKafka Streams vs. KSQL for Stream Processing on top of Apache Kafka
Kafka Streams vs. KSQL for Stream Processing on top of Apache Kafka
 
Cloud Monitoring tool Grafana
Cloud Monitoring  tool Grafana Cloud Monitoring  tool Grafana
Cloud Monitoring tool Grafana
 
Apache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training - VerisignApache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training - Verisign
 
Apache Kafka Architecture & Fundamentals Explained
Apache Kafka Architecture & Fundamentals ExplainedApache Kafka Architecture & Fundamentals Explained
Apache Kafka Architecture & Fundamentals Explained
 
Centralized Logging System Using ELK Stack
Centralized Logging System Using ELK StackCentralized Logging System Using ELK Stack
Centralized Logging System Using ELK Stack
 
Data Pipelines with Apache Kafka
Data Pipelines with Apache KafkaData Pipelines with Apache Kafka
Data Pipelines with Apache Kafka
 
Kafka 101
Kafka 101Kafka 101
Kafka 101
 
Data Pipelines with Kafka Connect
Data Pipelines with Kafka ConnectData Pipelines with Kafka Connect
Data Pipelines with Kafka Connect
 
SSO introduction
SSO introductionSSO introduction
SSO introduction
 
Stream processing using Kafka
Stream processing using KafkaStream processing using Kafka
Stream processing using Kafka
 
Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...
Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...
Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...
 
The Rise Of Event Streaming – Why Apache Kafka Changes Everything
The Rise Of Event Streaming – Why Apache Kafka Changes EverythingThe Rise Of Event Streaming – Why Apache Kafka Changes Everything
The Rise Of Event Streaming – Why Apache Kafka Changes Everything
 
Getting Started with Confluent Schema Registry
Getting Started with Confluent Schema RegistryGetting Started with Confluent Schema Registry
Getting Started with Confluent Schema Registry
 
Apache Kafka - Overview
Apache Kafka - OverviewApache Kafka - Overview
Apache Kafka - Overview
 
Apache Kafka
Apache KafkaApache Kafka
Apache Kafka
 
Secure Spring Boot Microservices with Keycloak
Secure Spring Boot Microservices with KeycloakSecure Spring Boot Microservices with Keycloak
Secure Spring Boot Microservices with Keycloak
 

Similar to Cisco’s E-Commerce Transformation Using Kafka

How Cisco Migrated from MapReduce Jobs to Spark Jobs - StampedeCon 2015
How Cisco Migrated from MapReduce Jobs to Spark Jobs - StampedeCon 2015How Cisco Migrated from MapReduce Jobs to Spark Jobs - StampedeCon 2015
How Cisco Migrated from MapReduce Jobs to Spark Jobs - StampedeCon 2015
StampedeCon
 
L'azienda è più agile? Tutto merito del Data Center
L'azienda è più agile? Tutto merito del Data Center L'azienda è più agile? Tutto merito del Data Center
L'azienda è più agile? Tutto merito del Data Center
SMAU
 
How Cisco Provides World-Class Technology Conference Experiences Using Automa...
How Cisco Provides World-Class Technology Conference Experiences Using Automa...How Cisco Provides World-Class Technology Conference Experiences Using Automa...
How Cisco Provides World-Class Technology Conference Experiences Using Automa...
InfluxData
 
20120416 tf mms_feedback_slideshare
20120416 tf mms_feedback_slideshare20120416 tf mms_feedback_slideshare
20120416 tf mms_feedback_slideshare
Osamu Takazoe
 

Similar to Cisco’s E-Commerce Transformation Using Kafka (20)

TechWiseTV Workshop: ASR 9000
TechWiseTV Workshop: ASR 9000 TechWiseTV Workshop: ASR 9000
TechWiseTV Workshop: ASR 9000
 
Not Your Mother's Kafka - Deep Dive into Confluent Cloud Infrastructure | Gwe...
Not Your Mother's Kafka - Deep Dive into Confluent Cloud Infrastructure | Gwe...Not Your Mother's Kafka - Deep Dive into Confluent Cloud Infrastructure | Gwe...
Not Your Mother's Kafka - Deep Dive into Confluent Cloud Infrastructure | Gwe...
 
Elastic Cloud Enterprise @ Cisco
Elastic Cloud Enterprise @ CiscoElastic Cloud Enterprise @ Cisco
Elastic Cloud Enterprise @ Cisco
 
StampedeCon 2015 Keynote
StampedeCon 2015 KeynoteStampedeCon 2015 Keynote
StampedeCon 2015 Keynote
 
How Cisco Migrated from MapReduce Jobs to Spark Jobs - StampedeCon 2015
How Cisco Migrated from MapReduce Jobs to Spark Jobs - StampedeCon 2015How Cisco Migrated from MapReduce Jobs to Spark Jobs - StampedeCon 2015
How Cisco Migrated from MapReduce Jobs to Spark Jobs - StampedeCon 2015
 
Cisco connect montreal 2018 compute v final
Cisco connect montreal 2018   compute v finalCisco connect montreal 2018   compute v final
Cisco connect montreal 2018 compute v final
 
L'azienda è più agile? Tutto merito del Data Center
L'azienda è più agile? Tutto merito del Data Center L'azienda è più agile? Tutto merito del Data Center
L'azienda è più agile? Tutto merito del Data Center
 
Simplifying the secure data center
Simplifying the secure data centerSimplifying the secure data center
Simplifying the secure data center
 
Как развернуть кампусную сеть Cisco за 10 минут? Новые технологии для автомат...
Как развернуть кампусную сеть Cisco за 10 минут? Новые технологии для автомат...Как развернуть кампусную сеть Cisco за 10 минут? Новые технологии для автомат...
Как развернуть кампусную сеть Cisco за 10 минут? Новые технологии для автомат...
 
Big datadc skyfall_preso_v2
Big datadc skyfall_preso_v2Big datadc skyfall_preso_v2
Big datadc skyfall_preso_v2
 
Cisco connect winnipeg 2018 putting firepower into the next generation fire...
Cisco connect winnipeg 2018   putting firepower into the next generation fire...Cisco connect winnipeg 2018   putting firepower into the next generation fire...
Cisco connect winnipeg 2018 putting firepower into the next generation fire...
 
Cisco connect montreal 2018 secure dc
Cisco connect montreal 2018    secure dcCisco connect montreal 2018    secure dc
Cisco connect montreal 2018 secure dc
 
Citi Tech Talk: Hybrid Cloud
Citi Tech Talk: Hybrid CloudCiti Tech Talk: Hybrid Cloud
Citi Tech Talk: Hybrid Cloud
 
Gain Insight and Programmability with Cisco DC Networking
Gain Insight and Programmability with Cisco DC NetworkingGain Insight and Programmability with Cisco DC Networking
Gain Insight and Programmability with Cisco DC Networking
 
Gain Insight and Programmability with Cisco DC Networking
Gain Insight and Programmability with Cisco DC NetworkingGain Insight and Programmability with Cisco DC Networking
Gain Insight and Programmability with Cisco DC Networking
 
Cisco DC Networking: Gain Insight and Programmability with
Cisco DC Networking: Gain Insight and Programmability with Cisco DC Networking: Gain Insight and Programmability with
Cisco DC Networking: Gain Insight and Programmability with
 
Optimizing Performance in Rust for Low-Latency Database Drivers
Optimizing Performance in Rust for Low-Latency Database DriversOptimizing Performance in Rust for Low-Latency Database Drivers
Optimizing Performance in Rust for Low-Latency Database Drivers
 
How Cisco Provides World-Class Technology Conference Experiences Using Automa...
How Cisco Provides World-Class Technology Conference Experiences Using Automa...How Cisco Provides World-Class Technology Conference Experiences Using Automa...
How Cisco Provides World-Class Technology Conference Experiences Using Automa...
 
20120416 tf mms_feedback_slideshare
20120416 tf mms_feedback_slideshare20120416 tf mms_feedback_slideshare
20120416 tf mms_feedback_slideshare
 
New Approaches for Fraud Detection on Apache Kafka and KSQL
New Approaches for Fraud Detection on Apache Kafka and KSQLNew Approaches for Fraud Detection on Apache Kafka and KSQL
New Approaches for Fraud Detection on Apache Kafka and KSQL
 

More from confluent

More from confluent (20)

Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Era
 
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
 
Santander Stream Processing with Apache Flink
Santander Stream Processing with Apache FlinkSantander Stream Processing with Apache Flink
Santander Stream Processing with Apache Flink
 
Unlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insightsUnlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insights
 
Workshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con FlinkWorkshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con Flink
 
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...
 
AWS Immersion Day Mapfre - Confluent
AWS Immersion Day Mapfre   -   ConfluentAWS Immersion Day Mapfre   -   Confluent
AWS Immersion Day Mapfre - Confluent
 
Eventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalkEventos y Microservicios - Santander TechTalk
Eventos y Microservicios - Santander TechTalk
 
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent CloudQ&A with Confluent Experts: Navigating Networking in Confluent Cloud
Q&A with Confluent Experts: Navigating Networking in Confluent Cloud
 
Citi TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep DiveCiti TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep Dive
 
Build real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with ConfluentBuild real-time streaming data pipelines to AWS with Confluent
Build real-time streaming data pipelines to AWS with Confluent
 
Q&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service MeshQ&A with Confluent Professional Services: Confluent Service Mesh
Q&A with Confluent Professional Services: Confluent Service Mesh
 
Citi Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka MicroservicesCiti Tech Talk: Event Driven Kafka Microservices
Citi Tech Talk: Event Driven Kafka Microservices
 
Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3Confluent & GSI Webinars series - Session 3
Confluent & GSI Webinars series - Session 3
 
Citi Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging ModernizationCiti Tech Talk: Messaging Modernization
Citi Tech Talk: Messaging Modernization
 
Citi Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time dataCiti Tech Talk: Data Governance for streaming and real time data
Citi Tech Talk: Data Governance for streaming and real time data
 
Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2Confluent & GSI Webinars series: Session 2
Confluent & GSI Webinars series: Session 2
 
Data In Motion Paris 2023
Data In Motion Paris 2023Data In Motion Paris 2023
Data In Motion Paris 2023
 
Confluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with SynthesisConfluent Partner Tech Talk with Synthesis
Confluent Partner Tech Talk with Synthesis
 
The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023The Future of Application Development - API Days - Melbourne 2023
The Future of Application Development - API Days - Melbourne 2023
 

Recently uploaded

“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
Muhammad Subhan
 
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc
 

Recently uploaded (20)

Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdfWhere to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
 
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
 
TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024
 
Design and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data ScienceDesign and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data Science
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
 
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
 
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?
 
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
UiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewUiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overview
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptx
 
Generative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfGenerative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdf
 
WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM Performance
 
2024 May Patch Tuesday
2024 May Patch Tuesday2024 May Patch Tuesday
2024 May Patch Tuesday
 
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The Inside
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The InsideCollecting & Temporal Analysis of Behavioral Web Data - Tales From The Inside
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The Inside
 

Cisco’s E-Commerce Transformation Using Kafka

  • 1. Cisco's eCommerce Transformation using Kafka Presented By: Dharmesh Panchmatia (Sr. Director – Cisco Systems) Gaurav Goyal (Principal Architect – Cisco Systems)
  • 2. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Agenda Kafka Architecture2 1 Kafka Use Cases Kafka Monitoring3 Lessons Learnt4
  • 3. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Orders booked $50+B 138 Countries 63 Device types 16 Browsers 185KUsers16 Languages 6M Hits/day 6.9 M Estimates 5.3 M Quotes 1.9 M Orders 85.6% Orders Orders Autobook Portal 71% B2B 29% Cisco Commerce By The Numbers
  • 4. © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Ref Data REFERENCE DATA SOURCE DMPRD - RDBMS Logging Order Capture DC1 - Tomcat Order Capture Transaction Data Downstream Publish X-Functional Services (73) DC3DC2DC1 TRANSACTION DATA STORE P S S S S N1 N2 N3 N4 N5 DC2 - Tomcat 1 2 3 4 Addresses Items Preferences Roles Contacts Logging DC1 & DC2 Commerce – Cloud Native
  • 6. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Kafka – Use Cases Data push to downstreams 1. Avoid point to point integration. 2. Avoid direct connection to transactional DB. Elastic Search Data Push 1. Reduce load on transactional DB 2. Eliminates ES out of sync in multi-DCs Machine Learning Use Cases using Spark 1. Recommendation Engine 2. Most Popular Configurations 3. Most popular products for a given category
  • 7. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Customer who bought X also bought Y. Identify products which are mostly bought together so we can create bundles or promotions accordingly. 1 2 Algorithm: Apriori ML Use Case 1. Recommendation Engine
  • 8. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Provide visibility to most popular configurations for a given product. Provide visibility to a configuration which Customer has recently bought for the given product. 1 2 Allow selection of pre-configured products instead of starting from scratch. ML Use Case 2. Popular Product Configuration
  • 10. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Producer (Capture Order) Producer (Return Order) Broker 1 Broker 2 Broker 3 Broker 4 ZK - 1 ZK - 2 ZK - 3 ZK - 4 ZK - 5 Consumer (Smart- SW SC) Kafka Cluster Zookeeper DC1 DC2 DC1 DC2 DC3 DC1 – RCDN; DC2 – ALLEN; DC3 – RTP Coordinates cluster membership Commit Offset (v 0.10.x.x) Kafka Architecture Consumer (EDW)
  • 11. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential RDBMS ProducerCustom Code (DC1) Custom Code (DC2)FAULT TOLERENT Kafka DC1 and DC2 Consumer Group - DC1 Consumer Group – DC2 Elastic Search – DC1 Elastic Search – DC2 Kafka Architecture – Elastic Search
  • 12. Transaction Data Order, Estimate Quote RDBMS Reference Data Click Stream Data Data Visualization Dynamic Querying Data Science Primary Analytics Data Store MQL Transaction Data RDBMS Subscriptions Invoices Kafka Architecture – ML & Analytics Use Case
  • 14. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Monitoring: Kafka Manager and Kafdrop Kafka Manager Kafdrop
  • 15. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Kafka – Custom Scripts 1. Cron job to check Kafka processes every minute. Restart Kafka process (and send email) in case it’s not running. 2. Always take back up of logs systematically when Kafka processes are getting restarted. 3. Have a test topic and push test message every minute. Trigger a notification in case of failures.
  • 16. Best Practices / Lessons Learnt
  • 17. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Best Practices 1 Have a mechanism to reset Kafka offsets on demand. 4 Auto Re-push mechanism in case producer gets error while pushing data into Kafka 2 Have a mechanism to re-push data to Kafka topic, 3 Enable SSL for secure access
  • 18. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential UI – Reset Offsets
  • 19. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential UI – Re-push data
  • 20. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Kafka Producer & Consumer Setup with SSL Below properties are required to enable SSL for both Producer and Consumer If client authentication is not required in the broker then below configuration is suffice, (kafka.client.truststore.jks will be provided by kafka service host.) 1 If client authentication is required in the broker then below configuration is required. (kafka.client.keystore.jks will be provided by kafka service host. ) 2
  • 21. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Auto Re-Push Mechanism Failure Source Data Push In case of failures Offline Scheduler Failed records Re - Push
  • 22. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Lessons Learnt 1 Have a while loop while subscribing to any Kafka Topics instead of creating consumer every time. 4 Data Size - consumer's max.partition.fetch.bytes should be greater or equals to the producers producer.max.request.size Default is 1MB. 2 Always use key if you want all messages for a particular key (e.g. order id) always goes to a particular partition. 3 enable.auto.commit - Default is true. It is better to set it false to get control over when to commit the offset.
  • 23. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Lessons Learnt 5 Have a custom script deployed to monitor & restart Kafka nodes in case of any issues. 8 Reset offset: Make sure there is no active consumer on this topic for that consumer group. 6 heartbeat.interval.ms must be smaller than session.timeout.ms. session.timeout.ms : it controls the time it takes to detect a consumer crash and stop sending heartbeats. heartbeat.interval.ms :The expected time between heartbeats to the consumer 7 auto.offset.reset -default latest
  • 25. © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Kafka Architecture – ML Use Case Quote Stream Order Stream