SlideShare a Scribd company logo
1 of 16
Event Driven
Architectures with
Apache Kafka
MATT MASUDA
MMASUDA@GMAIL.COM
@MATTMASUDA
Agenda
OVERVIEW OF WEB-
BASED
ARCHITECTURES
EVENT DRIVEN
OVERVIEW
ADVANTAGES &
DISADVANTAGES
BRIEF KAFKA
OVERVIEW
EVENT DRIVEN WITH
KAFKA
DEMO
Microservices
Architecture
Web service
issues
• The service API is a form of coupling
• Service calls are synchronous
• All downstream calls must complete before your
operation can finish
• Load balancing and service location
Event Driven
Architectures
• Leverages asynchronous messaging platforms
• Maintains the micro in microservice
• Each service produces and/or consumes events
• Events are the contracts between services
• Asynchronous operation is tolerant of temporary service
disruptions
Example
Architecture
Saga Pattern
Saga Pattern –
Sad Path
Hard things
about event
driven
• Transactions are right out
• Getting your result out means polling or notification
through other means (Websockets, SSE, SignalR)
• Events are your contracts => design is important
• Messaging APIs can be less mature than web frameworks
Kafka Basics
• It is a distributed messaging platform.
• It lets you publish and subscribe to streams of records. In
this respect it is similar to a message queue or enterprise
messaging system.
• It lets you store streams of records in a fault-tolerant way.
• It lets you process streams of records as they occur.
Kafka Concepts
• Kafka is run as a cluster on one or more servers.
• The Kafka cluster stores streams of records in categories
called topics.
• Each record consists of a key, a value, and a timestamp.
• Producers publish streams of records to one or more
topics.
• Topics can be divided into partitions for scalability.
• Consumers subscribe to one or more topics and process
the streams of records produced to them.
• Consumers have control over their position in a topic.
• Consumers can be grouped for scalability.
Concepts,
illustrated
Consumer
Groups
Event driven
with Kafka
Solves service location
Solves load balancing
Pub/sub model makes it easy to add new services to existing
event streams
Durable storage – you can keep a record of system behavior
and trends
Read positioning means you can reprocess old events
Requires Kafka
Resources
Confluent blog: Journey to Event Driven
Saga pattern -
http://microservices.io/patterns/data/saga.html
Cloud Native Patterns (Manning)
Microservices Patterns (Manning)
Designing Data-Intensive Applications (O’Reilly)
Apache Kafka web site - https://kafka.apache.org/intro
Demo

More Related Content

What's hot

Operating Kafka on AutoPilot mode @ DBS Bank (Arpit Dubey, DBS Bank) Kafka Su...
Operating Kafka on AutoPilot mode @ DBS Bank (Arpit Dubey, DBS Bank) Kafka Su...Operating Kafka on AutoPilot mode @ DBS Bank (Arpit Dubey, DBS Bank) Kafka Su...
Operating Kafka on AutoPilot mode @ DBS Bank (Arpit Dubey, DBS Bank) Kafka Su...
confluent
 
SaaS App Development with Windows Cartridge
SaaS App Development with Windows Cartridge SaaS App Development with Windows Cartridge
SaaS App Development with Windows Cartridge
WSO2
 

What's hot (20)

Operating Kafka on AutoPilot mode @ DBS Bank (Arpit Dubey, DBS Bank) Kafka Su...
Operating Kafka on AutoPilot mode @ DBS Bank (Arpit Dubey, DBS Bank) Kafka Su...Operating Kafka on AutoPilot mode @ DBS Bank (Arpit Dubey, DBS Bank) Kafka Su...
Operating Kafka on AutoPilot mode @ DBS Bank (Arpit Dubey, DBS Bank) Kafka Su...
 
Locaweb cloud and sdn
Locaweb cloud and sdnLocaweb cloud and sdn
Locaweb cloud and sdn
 
Cloud inventory analytics Maestro server
Cloud inventory analytics   Maestro serverCloud inventory analytics   Maestro server
Cloud inventory analytics Maestro server
 
Designing microservices part2
Designing microservices part2Designing microservices part2
Designing microservices part2
 
A closer look to locaweb IaaS
A closer look to locaweb IaaSA closer look to locaweb IaaS
A closer look to locaweb IaaS
 
Apache kafka
Apache kafkaApache kafka
Apache kafka
 
Serverless
ServerlessServerless
Serverless
 
Modernizing with microservices and fast data
Modernizing with microservices and fast dataModernizing with microservices and fast data
Modernizing with microservices and fast data
 
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies...
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies...Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies...
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies...
 
Azure reference architectures
Azure reference architecturesAzure reference architectures
Azure reference architectures
 
Should we manage events like APIs? | Alan Chatt and Kim Clark, IBM
Should we manage events like APIs? | Alan Chatt and Kim Clark, IBMShould we manage events like APIs? | Alan Chatt and Kim Clark, IBM
Should we manage events like APIs? | Alan Chatt and Kim Clark, IBM
 
Microservices design patterns
Microservices design patternsMicroservices design patterns
Microservices design patterns
 
Azure Application Architecture Guide
Azure Application Architecture GuideAzure Application Architecture Guide
Azure Application Architecture Guide
 
Best Features of Azure Service Bus
Best Features of Azure Service BusBest Features of Azure Service Bus
Best Features of Azure Service Bus
 
Caching for Microservices Architectures: Session II - Caching Patterns
Caching for Microservices Architectures: Session II - Caching PatternsCaching for Microservices Architectures: Session II - Caching Patterns
Caching for Microservices Architectures: Session II - Caching Patterns
 
Microservices Using Docker Containers for Magento 2
Microservices Using Docker Containers for Magento 2Microservices Using Docker Containers for Magento 2
Microservices Using Docker Containers for Magento 2
 
Confluent Developer Training
Confluent Developer TrainingConfluent Developer Training
Confluent Developer Training
 
Event Driven Architectures with Apache Kafka on Heroku
Event Driven Architectures with Apache Kafka on HerokuEvent Driven Architectures with Apache Kafka on Heroku
Event Driven Architectures with Apache Kafka on Heroku
 
SaaS App Development with Windows Cartridge
SaaS App Development with Windows Cartridge SaaS App Development with Windows Cartridge
SaaS App Development with Windows Cartridge
 
Azure service bus based on cloud computing
Azure service bus based on cloud computingAzure service bus based on cloud computing
Azure service bus based on cloud computing
 

Similar to Event Driven Architectures with Apache Kafka

Kafka-and-event-driven-architecture-OGYatra20.ppt
Kafka-and-event-driven-architecture-OGYatra20.pptKafka-and-event-driven-architecture-OGYatra20.ppt
Kafka-and-event-driven-architecture-OGYatra20.ppt
Inam Bukhary
 

Similar to Event Driven Architectures with Apache Kafka (20)

Streaming with Spring Cloud Stream and Apache Kafka - Soby Chacko
Streaming with Spring Cloud Stream and Apache Kafka - Soby ChackoStreaming with Spring Cloud Stream and Apache Kafka - Soby Chacko
Streaming with Spring Cloud Stream and Apache Kafka - Soby Chacko
 
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...
 
Apache Kafka Introduction
Apache Kafka IntroductionApache Kafka Introduction
Apache Kafka Introduction
 
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
 
Unleashing Real-time Power with Kafka.pptx
Unleashing Real-time Power with Kafka.pptxUnleashing Real-time Power with Kafka.pptx
Unleashing Real-time Power with Kafka.pptx
 
Kafka and event driven architecture -og yatra20
Kafka and event driven architecture -og yatra20Kafka and event driven architecture -og yatra20
Kafka and event driven architecture -og yatra20
 
Kafka and event driven architecture -apacoug20
Kafka and event driven architecture -apacoug20Kafka and event driven architecture -apacoug20
Kafka and event driven architecture -apacoug20
 
Apache Kafka - Scalable Message-Processing and more !
Apache Kafka - Scalable Message-Processing and more !Apache Kafka - Scalable Message-Processing and more !
Apache Kafka - Scalable Message-Processing and more !
 
BBL KAPPA Lesfurets.com
BBL KAPPA Lesfurets.comBBL KAPPA Lesfurets.com
BBL KAPPA Lesfurets.com
 
AI&BigData Lab 2016. Сарапин Виктор: Размер имеет значение: анализ по требова...
AI&BigData Lab 2016. Сарапин Виктор: Размер имеет значение: анализ по требова...AI&BigData Lab 2016. Сарапин Виктор: Размер имеет значение: анализ по требова...
AI&BigData Lab 2016. Сарапин Виктор: Размер имеет значение: анализ по требова...
 
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with K...
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with K...Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with K...
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with K...
 
Building streaming data applications using Kafka*[Connect + Core + Streams] b...
Building streaming data applications using Kafka*[Connect + Core + Streams] b...Building streaming data applications using Kafka*[Connect + Core + Streams] b...
Building streaming data applications using Kafka*[Connect + Core + Streams] b...
 
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
 
Kafka presentation
Kafka presentationKafka presentation
Kafka presentation
 
Ten reasons to choose Apache Pulsar over Apache Kafka for Event Sourcing_Robe...
Ten reasons to choose Apache Pulsar over Apache Kafka for Event Sourcing_Robe...Ten reasons to choose Apache Pulsar over Apache Kafka for Event Sourcing_Robe...
Ten reasons to choose Apache Pulsar over Apache Kafka for Event Sourcing_Robe...
 
Kafka-and-event-driven-architecture-OGYatra20.ppt
Kafka-and-event-driven-architecture-OGYatra20.pptKafka-and-event-driven-architecture-OGYatra20.ppt
Kafka-and-event-driven-architecture-OGYatra20.ppt
 
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...
 
Connecting kafka message systems with scylla
Connecting kafka message systems with scylla   Connecting kafka message systems with scylla
Connecting kafka message systems with scylla
 
AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017
AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017
AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017
 
Introduction to kafka connector
Introduction to  kafka connectorIntroduction to  kafka connector
Introduction to kafka connector
 

Recently uploaded

%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
masabamasaba
 
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
masabamasaba
 
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
masabamasaba
 

Recently uploaded (20)

tonesoftg
tonesoftgtonesoftg
tonesoftg
 
%in Rustenburg+277-882-255-28 abortion pills for sale in Rustenburg
%in Rustenburg+277-882-255-28 abortion pills for sale in Rustenburg%in Rustenburg+277-882-255-28 abortion pills for sale in Rustenburg
%in Rustenburg+277-882-255-28 abortion pills for sale in Rustenburg
 
%in Midrand+277-882-255-28 abortion pills for sale in midrand
%in Midrand+277-882-255-28 abortion pills for sale in midrand%in Midrand+277-882-255-28 abortion pills for sale in midrand
%in Midrand+277-882-255-28 abortion pills for sale in midrand
 
WSO2Con2024 - Enabling Transactional System's Exponential Growth With Simplicity
WSO2Con2024 - Enabling Transactional System's Exponential Growth With SimplicityWSO2Con2024 - Enabling Transactional System's Exponential Growth With Simplicity
WSO2Con2024 - Enabling Transactional System's Exponential Growth With Simplicity
 
%in ivory park+277-882-255-28 abortion pills for sale in ivory park
%in ivory park+277-882-255-28 abortion pills for sale in ivory park %in ivory park+277-882-255-28 abortion pills for sale in ivory park
%in ivory park+277-882-255-28 abortion pills for sale in ivory park
 
WSO2CON 2024 - API Management Usage at La Poste and Its Impact on Business an...
WSO2CON 2024 - API Management Usage at La Poste and Its Impact on Business an...WSO2CON 2024 - API Management Usage at La Poste and Its Impact on Business an...
WSO2CON 2024 - API Management Usage at La Poste and Its Impact on Business an...
 
%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein
%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein
%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein
 
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
 
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital TransformationWSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
 
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
MarTech Trend 2024 Book : Marketing Technology Trends (2024 Edition) How Data...
 
Artyushina_Guest lecture_YorkU CS May 2024.pptx
Artyushina_Guest lecture_YorkU CS May 2024.pptxArtyushina_Guest lecture_YorkU CS May 2024.pptx
Artyushina_Guest lecture_YorkU CS May 2024.pptx
 
WSO2CON 2024 Slides - Open Source to SaaS
WSO2CON 2024 Slides - Open Source to SaaSWSO2CON 2024 Slides - Open Source to SaaS
WSO2CON 2024 Slides - Open Source to SaaS
 
Architecture decision records - How not to get lost in the past
Architecture decision records - How not to get lost in the pastArchitecture decision records - How not to get lost in the past
Architecture decision records - How not to get lost in the past
 
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
%+27788225528 love spells in Huntington Beach Psychic Readings, Attraction sp...
 
%in tembisa+277-882-255-28 abortion pills for sale in tembisa
%in tembisa+277-882-255-28 abortion pills for sale in tembisa%in tembisa+277-882-255-28 abortion pills for sale in tembisa
%in tembisa+277-882-255-28 abortion pills for sale in tembisa
 
%in Benoni+277-882-255-28 abortion pills for sale in Benoni
%in Benoni+277-882-255-28 abortion pills for sale in Benoni%in Benoni+277-882-255-28 abortion pills for sale in Benoni
%in Benoni+277-882-255-28 abortion pills for sale in Benoni
 
VTU technical seminar 8Th Sem on Scikit-learn
VTU technical seminar 8Th Sem on Scikit-learnVTU technical seminar 8Th Sem on Scikit-learn
VTU technical seminar 8Th Sem on Scikit-learn
 
%in tembisa+277-882-255-28 abortion pills for sale in tembisa
%in tembisa+277-882-255-28 abortion pills for sale in tembisa%in tembisa+277-882-255-28 abortion pills for sale in tembisa
%in tembisa+277-882-255-28 abortion pills for sale in tembisa
 
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
%+27788225528 love spells in Knoxville Psychic Readings, Attraction spells,Br...
 
WSO2CON 2024 - How to Run a Security Program
WSO2CON 2024 - How to Run a Security ProgramWSO2CON 2024 - How to Run a Security Program
WSO2CON 2024 - How to Run a Security Program
 

Event Driven Architectures with Apache Kafka

  • 1. Event Driven Architectures with Apache Kafka MATT MASUDA MMASUDA@GMAIL.COM @MATTMASUDA
  • 2. Agenda OVERVIEW OF WEB- BASED ARCHITECTURES EVENT DRIVEN OVERVIEW ADVANTAGES & DISADVANTAGES BRIEF KAFKA OVERVIEW EVENT DRIVEN WITH KAFKA DEMO
  • 4. Web service issues • The service API is a form of coupling • Service calls are synchronous • All downstream calls must complete before your operation can finish • Load balancing and service location
  • 5. Event Driven Architectures • Leverages asynchronous messaging platforms • Maintains the micro in microservice • Each service produces and/or consumes events • Events are the contracts between services • Asynchronous operation is tolerant of temporary service disruptions
  • 9. Hard things about event driven • Transactions are right out • Getting your result out means polling or notification through other means (Websockets, SSE, SignalR) • Events are your contracts => design is important • Messaging APIs can be less mature than web frameworks
  • 10. Kafka Basics • It is a distributed messaging platform. • It lets you publish and subscribe to streams of records. In this respect it is similar to a message queue or enterprise messaging system. • It lets you store streams of records in a fault-tolerant way. • It lets you process streams of records as they occur.
  • 11. Kafka Concepts • Kafka is run as a cluster on one or more servers. • The Kafka cluster stores streams of records in categories called topics. • Each record consists of a key, a value, and a timestamp. • Producers publish streams of records to one or more topics. • Topics can be divided into partitions for scalability. • Consumers subscribe to one or more topics and process the streams of records produced to them. • Consumers have control over their position in a topic. • Consumers can be grouped for scalability.
  • 14. Event driven with Kafka Solves service location Solves load balancing Pub/sub model makes it easy to add new services to existing event streams Durable storage – you can keep a record of system behavior and trends Read positioning means you can reprocess old events Requires Kafka
  • 15. Resources Confluent blog: Journey to Event Driven Saga pattern - http://microservices.io/patterns/data/saga.html Cloud Native Patterns (Manning) Microservices Patterns (Manning) Designing Data-Intensive Applications (O’Reilly) Apache Kafka web site - https://kafka.apache.org/intro
  • 16. Demo

Editor's Notes

  1. test
  2. This is your example site, talk about what it does