This document compares RabbitMQ and Apache Kafka messaging systems. It provides an overview of core concepts for each including queues/topics, exchanges/partitions, and consumer groups. It also includes example messaging patterns and topologies for handling orders in an e-commerce system, demonstrating how each system could be used to implement request/response and publish-subscribe messaging across services.
Join guest speaker Jack Vanlightly in Part II of this series of three webinars where we explore what RabbitMQ and Apache Kafka are and their approach to messaging.
Each technology has made very different decisions regarding every aspect of their design, each with strengths and weaknesses, enabling different architectural patterns.
WEBINAR LIVE DATE: Thursday 30 August 2018 | 17:30 CEST / 16:30 BST / 11:30 EDT / 08:30 PDT
———————————————————————
SPEAKER CONTACT DETAILS
JACK VANLIGHTLY - Jack Vanlightly is a cloud software architect based in Barcelona specialising in event-driven architectures, data processing pipelines and data stores both relational and non-relational.
Twitter: https://twitter.com/vanlightly
———————————————————————
COMPANY CONTACT DETAILS
ERLANG SOLUTIONS
- Website: https://www.erlang-solutions.com
- Twitter: https://www.twitter.com/ErlangSolutions
- LinkedIn: http://www.linkedin.com/company/erlan…
- GitHub: https://github.com/esl
PPT concetrates on :
1. Introduction to RabbitMQ .
2. Difference between AMQP and JMS
3. Different types of exchanges
4. Routing Key and Binding concept
RabbitMQ is an open source message-broker software that originally implemented the Advanced Message Queuing Protocol (AMQP).it accepts and forwards messages.
Join guest speaker Jack Vanlightly in Part II of this series of three webinars where we explore what RabbitMQ and Apache Kafka are and their approach to messaging.
Each technology has made very different decisions regarding every aspect of their design, each with strengths and weaknesses, enabling different architectural patterns.
WEBINAR LIVE DATE: Thursday 30 August 2018 | 17:30 CEST / 16:30 BST / 11:30 EDT / 08:30 PDT
———————————————————————
SPEAKER CONTACT DETAILS
JACK VANLIGHTLY - Jack Vanlightly is a cloud software architect based in Barcelona specialising in event-driven architectures, data processing pipelines and data stores both relational and non-relational.
Twitter: https://twitter.com/vanlightly
———————————————————————
COMPANY CONTACT DETAILS
ERLANG SOLUTIONS
- Website: https://www.erlang-solutions.com
- Twitter: https://www.twitter.com/ErlangSolutions
- LinkedIn: http://www.linkedin.com/company/erlan…
- GitHub: https://github.com/esl
PPT concetrates on :
1. Introduction to RabbitMQ .
2. Difference between AMQP and JMS
3. Different types of exchanges
4. Routing Key and Binding concept
RabbitMQ is an open source message-broker software that originally implemented the Advanced Message Queuing Protocol (AMQP).it accepts and forwards messages.
The session presents in the details the RabbitMQ message broker along with demonstrations using the Java client, the Spring integration for RabbitMQ and the administration tools provided as part of the RabbitMQ installation.
The session discusses on how companies are using Apache Kafka & also covers under the hood details like partitions, brokers, replication.
About apache kafka: Apache Kafka is a distributed a streaming platform, Apache Kafka provides low-latency, high-throughput, fault-tolerant publish and subscribe pipelines and is able to process streams of events. Kafka provides reliable, millisecond responses to support both customer-facing applications and connecting downstream systems with real-time data.
At Hootsuite, we've been transitioning from a single monolithic PHP application to a set of scalable Scala-based microservices. To avoid excessive coupling between services, we've implemented an event system using Apache Kafka that allows events to be reliably produced + consumed asynchronously from services as well as data stores.
In this presentation, I talk about:
- Why we chose Kafka
- How we set up our Kafka clusters to be scalable, highly available, and multi-data-center aware.
- How we produce + consume events
- How we ensure that events can be understood by all parts of our system (Some that are implemented in other programming languages like PHP and Python) and how we handle evolving event payload data.
This session goes through the understanding of Apache Kafka, its components and working with best practices to achieve fault tolerant system with high availability and consistency by tuning Kafka brokers and producer to achieve the best result.
A presentation from internal meeting on Message Broker System and RabbitMQ. RabbitMQ is open source message broker software that implements the Advanced Message Queuing Protocol (AMQP).
Apache Kafka vs RabbitMQ: Fit For Purpose / Decision TreeSlim Baltagi
Â
Kafka as a streaming data platform is becoming the successor to traditional messaging systems such as RabbitMQ. Nevertheless, there are still some use cases where they could be a good fit. This one single slide tries to answer in a concise and unbiased way where to use Apache Kafka and where to use RabbitMQ. Your comments and feedback are much appreciated.
Jay Kreps is a Principal Staff Engineer at LinkedIn where he is the lead architect for online data infrastructure. He is among the original authors of several open source projects including a distributed key-value store called Project Voldemort, a messaging system called Kafka, and a stream processing system called Samza. This talk gives an introduction to Apache Kafka, a distributed messaging system. It will cover both how Kafka works, as well as how it is used at LinkedIn for log aggregation, messaging, ETL, and real-time stream processing.
The session presents in the details the RabbitMQ message broker along with demonstrations using the Java client, the Spring integration for RabbitMQ and the administration tools provided as part of the RabbitMQ installation.
The session discusses on how companies are using Apache Kafka & also covers under the hood details like partitions, brokers, replication.
About apache kafka: Apache Kafka is a distributed a streaming platform, Apache Kafka provides low-latency, high-throughput, fault-tolerant publish and subscribe pipelines and is able to process streams of events. Kafka provides reliable, millisecond responses to support both customer-facing applications and connecting downstream systems with real-time data.
At Hootsuite, we've been transitioning from a single monolithic PHP application to a set of scalable Scala-based microservices. To avoid excessive coupling between services, we've implemented an event system using Apache Kafka that allows events to be reliably produced + consumed asynchronously from services as well as data stores.
In this presentation, I talk about:
- Why we chose Kafka
- How we set up our Kafka clusters to be scalable, highly available, and multi-data-center aware.
- How we produce + consume events
- How we ensure that events can be understood by all parts of our system (Some that are implemented in other programming languages like PHP and Python) and how we handle evolving event payload data.
This session goes through the understanding of Apache Kafka, its components and working with best practices to achieve fault tolerant system with high availability and consistency by tuning Kafka brokers and producer to achieve the best result.
A presentation from internal meeting on Message Broker System and RabbitMQ. RabbitMQ is open source message broker software that implements the Advanced Message Queuing Protocol (AMQP).
Apache Kafka vs RabbitMQ: Fit For Purpose / Decision TreeSlim Baltagi
Â
Kafka as a streaming data platform is becoming the successor to traditional messaging systems such as RabbitMQ. Nevertheless, there are still some use cases where they could be a good fit. This one single slide tries to answer in a concise and unbiased way where to use Apache Kafka and where to use RabbitMQ. Your comments and feedback are much appreciated.
Jay Kreps is a Principal Staff Engineer at LinkedIn where he is the lead architect for online data infrastructure. He is among the original authors of several open source projects including a distributed key-value store called Project Voldemort, a messaging system called Kafka, and a stream processing system called Samza. This talk gives an introduction to Apache Kafka, a distributed messaging system. It will cover both how Kafka works, as well as how it is used at LinkedIn for log aggregation, messaging, ETL, and real-time stream processing.
Full recorded presentation at https://www.youtube.com/watch?v=2UfAgCSKPZo for Tetrate Tech Talks on 2022/05/13.
Envoy's support for Kafka protocol, in form of broker-filter and mesh-filter.
Contents:
- overview of Kafka (usecases, partitioning, producer/consumer, protocol);
- proxying Kafka (non-Envoy specific);
- proxying Kafka with Envoy;
- handling Kafka protocol in Envoy;
- Kafka-broker-filter for per-connection proxying;
- Kafka-mesh-filter to provide front proxy for multiple Kafka clusters.
References:
- https://adam-kotwasinski.medium.com/deploying-envoy-and-kafka-8aa7513ec0a0
- https://adam-kotwasinski.medium.com/kafka-mesh-filter-in-envoy-a70b3aefcdef
bigdata 2022_ FLiP Into Pulsar Apps
In this session, Timothy will introduce you to the world of Apache Pulsar and how to build real-time messaging and streaming applications with a variety of OSS libraries, schemas, languages, frameworks, and tools.
FLiP Into Pulsar Apps
08:30 – 09:15
•
23 Nov, 2022
In this session, Timothy will introduce you to the world of Apache Pulsar and how to build real-time messaging and streaming applications with a variety of OSS libraries, schemas, languages, frameworks, and tools.
Apache Kafka is the most used data streaming broker by companies. It could manage millions of messages easily and it is the base of many architectures based in events, micro-services, orchestration, ... and now cloud environments. OpenShift is the most extended Platform as a Service (PaaS). It is based in Kubernetes and it helps the companies to deploy easily any kind of workload in a cloud environment. Thanks many of its features it is the base for many architectures based in stateless applications to build new Cloud Native Applications. Strimzi is an open source community that implements a set of Kubernetes Operators to help you to manage and deploy Apache Kafka brokers in OpenShift environments.
These slides will introduce you Strimzi as a new component on OpenShift to manage your Apache Kafka clusters.
Slides used at OpenShift Meetup Spain:
- https://www.meetup.com/es-ES/openshift_spain/events/261284764/
JConf.dev 2022 - Apache Pulsar Development 101 with JavaTimothy Spann
Â
JConf.dev 2022 - Apache Pulsar Development 101 with Java
https://2022.jconf.dev/
In this session I will get you started with real-time cloud native streaming programming with Java. We will start off with a gentle introduction to Apache Pulsar and setting up your first easy standalone cluster. We will then l show you how to produce and consume message to Pulsar using several different Java libraries including native Java client, AMQP/RabbitMQ, MQTT and even Kafka. After this session you will building real-time streaming and messaging applications with Java. We will also touch on Apache Spark and Apache Flink.
Timothy Spann
Tim Spann is a Developer Advocate @ StreamNative where he works with Apache Pulsar, Apache Flink, Apache NiFi, Apache MXNet, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Principal Field Engineer at Cloudera, a Senior Solutions Architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science. https://www.datainmotion.dev/p/about-me.html https://dzone.com/users/297029/bunkertor.html https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/speaker/185963
AMIS SIG - Introducing Apache Kafka - Scalable, reliable Event Bus & Message ...Lucas Jellema
Â
Introduction of Apache Kafka - the open source platform for real time message queuing and reliable, scalable, distributed event handling and high volume pub/sub implementation.
see GitHub https://github.com/MaartenSmeets/kafka-workshop for the workshop resources.
📗 Fintech Trends for 2022 – Building for Resilience & Security
Since we first published our white paper on the state of play for technology in modern financial services, a lot has gone on.
However, the overarching themes of this paper remain relevant: financial services and products must be based on trusted, secure and resilient tech for the exciting innovations in the industry to take hold.
Our Fintech Marketing Lead, Michael Jaiyeola, produces this paper in collaboration with our global fintech clients, internal engineering and project management teams and influential subject matter experts [see below].
👉 Discover 5 key tech trends shaping modern financial services.
👉 Learn how to bridge the gap between business and tech functions for strategic success.
“Here we bring you a report that describes some of the technologies needed to be competitive, agile and innovative in this new age of human-centric technology.”
@Phil Harrison, CCO Fintech Trifork & Erlang Solutions.
#emergingtech #financialservices #diversityandinclusion #womenintechnology #futureofwork #distributedcomputing #systemresilience #cybersecurity #web3 #blockchain #cbdc #payments #ai #embeddedfinance #opensourcesoftware #functionalprogramming #erlang #elixirlang
Join us for September's webinar with Bleacher Report's Ben Marx as he walks us through how to "successfully adopt Elixir". This includes managing your team, learning a new language and bountiful resources to explore.
This webinar is part of our #OpenErlang campaign, celebrating 20 years of Erlang being an open source programming language.
WEBINAR LIVE DATE: Monday 17 September 2018 | 17:00 CEST / 16:00 BST / 11:00 EDT / 08:00 PDT
———————————————————————
SPEAKER CONTACT DETAILS
BEN MARX - Ben Marx is a software architect based in San Francisco for sports culture website Bleacher Report. He is also co-author of Adopting Elixir and co-organizer of the Erlang Elixir meet up in San Francisco. In his free time, he's also programming in Rust.
Twitter: https://twitter.com/bgmarx
———————————————————————
COMPANY CONTACT DETAILS
ERLANG SOLUTIONS
- Website: https://www.erlang-solutions.com
- Twitter: https://www.twitter.com/ErlangSolutions
- LinkedIn: http://www.linkedin.com/company/erlan…
- GitHub: https://github.com/esl
Designing & architecting RabbitMQ engineered systems - Ayanda Dube @ London R...Erlang Solutions
Â
The architecture, design and development of RabbitMQ based computer systems has, up until now, habitually been carried out in an ad-hoc and informal manner. This leads to design misses during the early system development phases, which later manifest as various problems in performance on live RabbitMQ installations. In this talk Ayanda Dube introduces and discusses a structured approach/guide to designing and implementing systems which make use of RabbitMQ as a messaging and queueing engine. We get into the depths of each design phase and points of consideration which aid system developers and solution architects in implementing robust, reliable and stable engineered systems which, under the hood, make use of RabbitMQ, for simple requirements, to some of the most complex deployments and use cases.
Building the ideal betting stack | London Erlang User Group presentationErlang Solutions
Â
Speaker: Chandru Mullaparthi, Chairman of the Erlang Industrial User Group; Former Head of Software Architecture at bet365.
Expert industry analysis and proposal of the ideal components available today to make up a betting tech stack. A creative session to visualise building a betting stack from the ground up with no concern for a legacy, instead focusing on traditional and innovative technologies.
Chandru will draw on his experience building highly available, reliable, and concurrent systems for the online gambling & betting industry to outline his dreamteam of programming languages, middleware, operations, configuration management, and databases.
Empowering mobile first workers in emerging-markets using messagingErlang Solutions
Â
Smartphone messaging has taken over emerging-markets since the birth of Whatsapp in 2009. Peer-to-peer messaging grown in popularity and created new communication habits for the majority of consumers, especially in Latin America and Africa. Today, Business-to-Consumer messaging is transforming the way companies and individuals are communicating nowadays.
Carlos Sierra, CEO and Founder of Twnel, shares the lessons he learned during Twnel's first 18 months of operation in Latin America, helping businesses use the power of messaging to connect with their networks of individual business partners.
Enhancing Research Orchestration Capabilities at ORNL.pdfGlobus
Â
Cross-facility research orchestration comes with ever-changing constraints regarding the availability and suitability of various compute and data resources. In short, a flexible data and processing fabric is needed to enable the dynamic redirection of data and compute tasks throughout the lifecycle of an experiment. In this talk, we illustrate how we easily leveraged Globus services to instrument the ACE research testbed at the Oak Ridge Leadership Computing Facility with flexible data and task orchestration capabilities.
Into the Box Keynote Day 2: Unveiling amazing updates and announcements for modern CFML developers! Get ready for exciting releases and updates on Ortus tools and products. Stay tuned for cutting-edge innovations designed to boost your productivity.
AI Pilot Review: The World’s First Virtual Assistant Marketing SuiteGoogle
Â
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
👉👉 Click Here To Get More Info 👇👇
https://sumonreview.com/ai-pilot-review/
AI Pilot Review: Key Features
âś…Deploy AI expert bots in Any Niche With Just A Click
âś…With one keyword, generate complete funnels, websites, landing pages, and more.
âś…More than 85 AI features are included in the AI pilot.
âś…No setup or configuration; use your voice (like Siri) to do whatever you want.
✅You Can Use AI Pilot To Create your version of AI Pilot And Charge People For It…
âś…ZERO Manual Work With AI Pilot. Never write, Design, Or Code Again.
âś…ZERO Limits On Features Or Usages
âś…Use Our AI-powered Traffic To Get Hundreds Of Customers
âś…No Complicated Setup: Get Up And Running In 2 Minutes
âś…99.99% Up-Time Guaranteed
âś…30 Days Money-Back Guarantee
âś…ZERO Upfront Cost
See My Other Reviews Article:
(1) TubeTrivia AI Review: https://sumonreview.com/tubetrivia-ai-review
(2) SocioWave Review: https://sumonreview.com/sociowave-review
(3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review
(4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review
Enterprise Resource Planning System includes various modules that reduce any business's workload. Additionally, it organizes the workflows, which drives towards enhancing productivity. Here are a detailed explanation of the ERP modules. Going through the points will help you understand how the software is changing the work dynamics.
To know more details here: https://blogs.nyggs.com/nyggs/enterprise-resource-planning-erp-system-modules/
Navigating the Metaverse: A Journey into Virtual Evolution"Donna Lenk
Â
Join us for an exploration of the Metaverse's evolution, where innovation meets imagination. Discover new dimensions of virtual events, engage with thought-provoking discussions, and witness the transformative power of digital realms."
Developing Distributed High-performance Computing Capabilities of an Open Sci...Globus
Â
COVID-19 had an unprecedented impact on scientific collaboration. The pandemic and its broad response from the scientific community has forged new relationships among public health practitioners, mathematical modelers, and scientific computing specialists, while revealing critical gaps in exploiting advanced computing systems to support urgent decision making. Informed by our team’s work in applying high-performance computing in support of public health decision makers during the COVID-19 pandemic, we present how Globus technologies are enabling the development of an open science platform for robust epidemic analysis, with the goal of collaborative, secure, distributed, on-demand, and fast time-to-solution analyses to support public health.
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoamtakuyayamamoto1800
Â
In this slide, we show the simulation example and the way to compile this solver.
In this solver, the Helmholtz equation can be solved by helmholtzFoam. Also, the Helmholtz equation with uniformly dispersed bubbles can be simulated by helmholtzBubbleFoam.
Unleash Unlimited Potential with One-Time Purchase
BoxLang is more than just a language; it's a community. By choosing a Visionary License, you're not just investing in your success, you're actively contributing to the ongoing development and support of BoxLang.
How to Position Your Globus Data Portal for Success Ten Good PracticesGlobus
Â
Science gateways allow science and engineering communities to access shared data, software, computing services, and instruments. Science gateways have gained a lot of traction in the last twenty years, as evidenced by projects such as the Science Gateways Community Institute (SGCI) and the Center of Excellence on Science Gateways (SGX3) in the US, The Australian Research Data Commons (ARDC) and its platforms in Australia, and the projects around Virtual Research Environments in Europe. A few mature frameworks have evolved with their different strengths and foci and have been taken up by a larger community such as the Globus Data Portal, Hubzero, Tapis, and Galaxy. However, even when gateways are built on successful frameworks, they continue to face the challenges of ongoing maintenance costs and how to meet the ever-expanding needs of the community they serve with enhanced features. It is not uncommon that gateways with compelling use cases are nonetheless unable to get past the prototype phase and become a full production service, or if they do, they don't survive more than a couple of years. While there is no guaranteed pathway to success, it seems likely that for any gateway there is a need for a strong community and/or solid funding streams to create and sustain its success. With over twenty years of examples to draw from, this presentation goes into detail for ten factors common to successful and enduring gateways that effectively serve as best practices for any new or developing gateway.
Globus Connect Server Deep Dive - GlobusWorld 2024Globus
Â
We explore the Globus Connect Server (GCS) architecture and experiment with advanced configuration options and use cases. This content is targeted at system administrators who are familiar with GCS and currently operate—or are planning to operate—broader deployments at their institution.
May Marketo Masterclass, London MUG May 22 2024.pdfAdele Miller
Â
Can't make Adobe Summit in Vegas? No sweat because the EMEA Marketo Engage Champions are coming to London to share their Summit sessions, insights and more!
This is a MUG with a twist you don't want to miss.
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns
Â
Unlocking Business Potential: Tailored Technology Solutions by Prosigns
Discover how Prosigns, a leading technology solutions provider, partners with businesses to drive innovation and success. Our presentation showcases our comprehensive range of services, including custom software development, web and mobile app development, AI & ML solutions, blockchain integration, DevOps services, and Microsoft Dynamics 365 support.
Custom Software Development: Prosigns specializes in creating bespoke software solutions that cater to your unique business needs. Our team of experts works closely with you to understand your requirements and deliver tailor-made software that enhances efficiency and drives growth.
Web and Mobile App Development: From responsive websites to intuitive mobile applications, Prosigns develops cutting-edge solutions that engage users and deliver seamless experiences across devices.
AI & ML Solutions: Harnessing the power of Artificial Intelligence and Machine Learning, Prosigns provides smart solutions that automate processes, provide valuable insights, and drive informed decision-making.
Blockchain Integration: Prosigns offers comprehensive blockchain solutions, including development, integration, and consulting services, enabling businesses to leverage blockchain technology for enhanced security, transparency, and efficiency.
DevOps Services: Prosigns' DevOps services streamline development and operations processes, ensuring faster and more reliable software delivery through automation and continuous integration.
Microsoft Dynamics 365 Support: Prosigns provides comprehensive support and maintenance services for Microsoft Dynamics 365, ensuring your system is always up-to-date, secure, and running smoothly.
Learn how our collaborative approach and dedication to excellence help businesses achieve their goals and stay ahead in today's digital landscape. From concept to deployment, Prosigns is your trusted partner for transforming ideas into reality and unlocking the full potential of your business.
Join us on a journey of innovation and growth. Let's partner for success with Prosigns.
Experience our free, in-depth three-part Tendenci Platform Corporate Membership Management workshop series! In Session 1 on May 14th, 2024, we began with an Introduction and Setup, mastering the configuration of your Corporate Membership Module settings to establish membership types, applications, and more. Then, on May 16th, 2024, in Session 2, we focused on binding individual members to a Corporate Membership and Corporate Reps, teaching you how to add individual members and assign Corporate Representatives to manage dues, renewals, and associated members. Finally, on May 28th, 2024, in Session 3, we covered questions and concerns, addressing any queries or issues you may have.
For more Tendenci AMS events, check out www.tendenci.com/events
First Steps with Globus Compute Multi-User EndpointsGlobus
Â
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
Accelerate Enterprise Software Engineering with PlatformlessWSO2
Â
Key takeaways:
Challenges of building platforms and the benefits of platformless.
Key principles of platformless, including API-first, cloud-native middleware, platform engineering, and developer experience.
How Choreo enables the platformless experience.
How key concepts like application architecture, domain-driven design, zero trust, and cell-based architecture are inherently a part of Choreo.
Demo of an end-to-end app built and deployed on Choreo.
Custom Healthcare Software for Managing Chronic Conditions and Remote Patient...Mind IT Systems
Â
Healthcare providers often struggle with the complexities of chronic conditions and remote patient monitoring, as each patient requires personalized care and ongoing monitoring. Off-the-shelf solutions may not meet these diverse needs, leading to inefficiencies and gaps in care. It’s here, custom healthcare software offers a tailored solution, ensuring improved care and effectiveness.
Software Engineering, Software Consulting, Tech Lead.
Spring Boot, Spring Cloud, Spring Core, Spring JDBC, Spring Security,
Spring Transaction, Spring MVC,
Log4j, REST/SOAP WEB-SERVICES.
2. Background
• Jack Vanlightly
• Software Architect of SII Concatel, Barcelona
• Event-Driven Architectures
• Messaging Systems
• Data Integrations
• https://jack-vanlightly.com
4. The Log
(Apache Kafka)
The Queue
(RabbitMQ)
The Queue
• First In First Out data structure
• Optimized for efficient write and read operations from either end of
the sequence
• Stored data is transient in nature
• Not shared between applications
7 6 5 4 3 2 1Write Read
VS The Log
• Also known as Write-Ahead Logs, Transaction Logs and Commit
Logs.
• Append-only. Ordered By Time.
• Unique log sequence numbers.
• Stored data is persistent in nature
• Shared between applications
1 2 3 4 5 6 7
Reader
Reader
5. RabbitMQ
The Building Blocks
Routing
Fanout Exchange
Direct Exchange
Topic Exchange
Header Exchange
Consistent Hashing Exchange
Random Exchange
Sharding
Queues and Consumers
Queues
Competing Consumers
Non-Competing Consumers
Ephemeral Queues
Lazy Queues
Priority Queues
More
Deadletter Exchange
Alternate Exchange
Virtual Hosts
6. Exchanges, Queues and Bindings
• Publishers send messages to Exchanges
• Exchanges route messages to Queues and even other
Exchanges.
• Consumers read from Queues
• Bindings link Exchanges to Queues and even Exchanges to
Exchanges
Publisher Exchange Queue Consumer
binding
RabbitMQ
Building Blocks
Traditional queues on
steroids
8. Publish – Subscribe with Routing
Direct Exchange
Routing by Routing Key
Exact match routing key = binding key
Publisher
Direct
Exchange
Queue B Consumer
Queue A
Queue C Consumer
Consumer
created
modified
cancelled
Routing Key
Binding Key: created
Binding Key: modified
Binding Key: cancelled
9. Publish – Subscribe with Routing
Topic Exchange
Wildcard routing by Routing Key
* = match 1 word
# = match multiple words
booking.baggage.added
booking.baggage.removed
booking.passenger.added
booking.passenger.removed
payment.received
Publisher
Topic
Exchange
Queue B Consumer
Queue A
Queue C Consumer
Consumer
Routing Key
booking.#
*.passenger.*
payment.received
10. Publish – Subscribe with Routing
Header Exchange
Routing by message headers
Publisher
Header
Exchange
Queue B Consumer
Queue A
Queue C Consumer
Consumer
VIP=true, Country=UK, All
VIP=False
VIP: true
Country: UK
VIP=true, Country=UK, Any
11. Point-to-Point Messaging
Default Exchange
Direct exchange type
All queues have implicit binding to the default exchange.
Routes messages by Routing Key to Queue Name
Publisher
Default
Exchange
Orders Consumer
Notifications
Shipping Consumer
Consumer
Routing Key
12. Protecting Against Routing Failures
Alternate Exchange
Not an exchange type but a configured relationship
between two exchanges.
Route unrouteable messages to an alternate exchange
Publisher
Topic
Exchange
Queue Consumer
Queue
Unrouteable
messages
Consumer
Consumer
created
modified
Fanout
Exchangeunrouteable
13. Scaling Out Consumers
Fanout
Exchange
Shipping Shipping
Non-Competing Consumers
Each consumer receives every message
Scale Out with Competing Consumers
Each Shipping consumer receives a subset of the messages.
Notifications
Billing Billing
Notifier
Sales
Publisher
Fanout
Exchange
Shipping Shipping
Shipping
Shipping
Notifications Notifier
Billing Billing
Sales
Publisher
14. Scaling Out Queues
Random Exchange
Load balance messages across queues randomly
Publisher
Random
Exchange
Shipping.2 Consumer
Shipping.1
Shipping.3 Consumer
Consumer
15. Scaling Out Queues With Data Locality
Consistent Hashing Exchange
Partition messages across queues via hashing function over routing key, message header or message property
Publisher
Hashing
Exchange
Shipping.2 Consumer
Shipping.1
Shipping.3 Consumer
Consumer
Routing Key
1/3 hash space
1/3 hash space
1/3 hash space
16. Scaling Out Queues With Data Locality
Sharding Plugin and Modulus Exchange
Partition messages across queues over multiple hosts via hashing function on the routing key.
Publisher
Modulus
Exchange
Host 1 Shard 2 Consumer
Host 1 Shard 1
Host 1 Shard 3 Consumer
Consumer
Routing Key
Publisher
Modulus
Exchange
Host 2 Shard 2 Consumer
Host 2 Shard 1
Host 2 Shard 3 Consumer
Consumer
Routing Key
17. Apply limits to queues
• Length
• Size
• Time limits (TTL)
Eject messages from a queue when:
• A queue size/length limit reached.
• A message has spent longer than the TTL time limit in the
queue
Route to a deadletter exchange to avoid message loss.
Queue Limits
and
Deadletter
Exchanges
7 6 5 4 3 2 1
X
Queue
Exchange
Consumer
Queue
18. Ephemeral
• Auto-Delete Queue
• Queue Expiration (TTL)
• Exclusive Queues
• Auto-Delete Exchanges
Lazy Queues
• Memory optimized queues.
Priority Queues
• No longer FIFO.
• Publisher sets priority on messages
• Priority Queue moves higher priority messages ahead of lower
• Drawbacks – blocked low priority messages, low priority can
eject high priority when queue is full
Ephemeral Exchanges
and Queues
Lazy Queues
Priority
Queues
19. • Allows Multi-Tenant architecture
• Isolation of groups of exchanges, queues and users
• No routing between two virtual hosts
Virtual Hosts
21. • Distributed because Kafka is deployed as a cluster of nodes, for
both fault tolerance and scale
• Partitioned because each topic can be sharded into multiple
partitions, with each partition storing a disjoint set of the data.
• Replicated because partitions are usually replicated across
multiple nodes (servers).
• Commit Log because messages are stored in an append only
fashion.
Apache Kafka
Core Concepts
The Rise of the
Distributed,
Partitioned,
Replicated,
Commit Log
22. • Kafka has a simple Publish Subscribe model
• Topics
• Each Topic has one or more partitions
• Each partition has a replication factor of one or more
Topics,
Partitions
and
Consumer
Groups
Producer ConsumerTopic
Producer ConsumerPartition 2
Partition 1
Partition 3
27. • Producers append messages to the end of partitions.
• The partition is chosen by the producer (round robin or partition
key).
• Messages are removed according to the data retention policy.
• Consumers keep track their position in the partitions they
consume from – called the consumer offset
Topics,
Partitions
and
Consumer
Groups
1 2 3 4 5 6 7
Producer
Consumer 1
Write
Read at offsetData retention policyX
28. Topics,
Partitions
and
Consumer
Groups
1 2 3 4 5 6 7
Producer
Consumer 1
CG ARead at offset
Consumer 4
CG BRead at offset
Partition 1
1 2 3 4 5 6 7
Consumer 2
CG ARead at offset
Consumer 5
CG B
Read
at offset
Partition 28
1 2 3 4 5 6
Consumer 3
CG ARead at offset
Consumer 6
CG BRead at offset
Partition 3
30. Log Compaction
• Clean up old versions of a record (by record key). Keep latest
version only.
• Periodic process
• Kafka as an event store
Apache Kafka
Log Compaction
A X G U D W J X E A E O P X
A X
G U
D
W J
X E
A E O D X
G U W J A E O D X
31. Avro Schema Registry
• Send messages in a compact binary format – Avro
• Schema Registry service automates sharing of Avro schemas
between producers and consumers
• Schema Registry to safely control evolution of schemas
Apache Kafka
Schema Registry
Producer ConsumerTopic
Schema
Registry
Send
schema
Get
schema
32. Data Replication / Event Sourcing
• Kafka as a single source of truth
• Sits at the center of your architecture
• Kafka Connect
Apache Kafka
Data Replication
Event Sourcing
Kafka Topics
34. Example Scenario
eCommerce
Site
Sales and
Inventory
Billing
Shipping
PlaceOrder OrderPlaced
OrderBilledOrderPlaced
eCommerce
Site
Sales and
Inventory
Billing
Shipping
ModifyOrder OrderModified
ModificationBilledOrderModified
eCommerce
Site
Sales and
Inventory
Billing
Shipping
CancelOrder OrderCancelled
OrderCancelled
35. Example Scenario – RabbitMQ Fanout Exchanges
eCommerce
Site
PlaceOrder
SalesSales
OrderPlaced
BillingBilling
Shipping Shipping
OrderRefunded
Fanout Exchange per Event, Single Queue per Application
ModifyOrder
CancelOrder
OrderModified
OrderCancelled
OrderBilled
Modification
Billed
38. Example Scenario – Kafka Topics
eCommerce
Site
SalesOrder Requests Topic
Orders Topic
Related Events in One Topic
Billing
Shipping
OrderPlaced Event
OrderModified Event
OrderCancelled EventOrderBilled Event
ModificationBilled Event
OrderRefunded Event
OrderShipped Event
OrderShippingCancelled Event
PlaceOrder Request
ModifyOrder Request
CancelOrder Request
39. Example Scenario – Kafka Topics
Related Events in a Few Topics
eCommerce
Site
SalesOrder Requests Topic
Orders Topic
Billing
Shipping
OrderPlaced Event
OrderModified Event
OrderCancelled Event
OrderBilled Event
ModificationBilled Event
OrderRefunded Event
OrderShipped Event
OrderShippingCancelled Event
PlaceOrder Request
ModifyOrder Request
CancelOrder Request
Billing Topic
Shipments Topic
40. Example Scenario – Kafka Topics
One Topic per Event
eCommerce
Site
SalesModify Order Requests Topic
OrdersModified Topic
Billing
Shipping
OrderPlaced Event
OrderModified Event
OrderCancelled Event
OrderShipped Event
OrderShippingCancelled Event
PlaceOrder Request
ModifyOrder Request
CancelOrder Request
Order Refunded Topic
Shipments Topic
Order Billed Topic
ModificationBilled Topic
OrdersPlaced Topic
OrdersCancelled Topic
CancelledShipments Topic
OrderRefunded Event
OrderBilled Event
ModificationBilled Event
Place Order Requests Topic
Cancel Order Requests Topic
41. Modifying the Scenario – Class of Service
eCommerce
Site
Sales and
Inventory
Billing
Shipping
PlaceOrder-Priority
PlaceOrder-Standard
OrderPlaced-Priority
OrderPlaced-Standard
OrderBilled-Priority
OrderBilled-Standard
OrderPlaced-Priority
OrderPlaced-Standard
eCommerce
Site
Sales and
Inventory
Billing
Shipping
ModifyOrder-Priority
ModifyOrder-Standard
OrderModified-Priority
OrderModified-Standard
ModificationBilled-Priority
ModificationBilled-Standard
OrderModified-Priority
OrderModified-Standard
42. Scenario #2 – RabbitMQ Topic Exchanges
eCommerce
Site
PlaceOrder
Sales
Sales-Standard
OrderPlaced
Billing-Priority
Billing
Shipping
Shipping-Priority
OrderRefunded
Direct Exchange per Event, Routing Key with Class of Service,
Queue per Class of Service
ModifyOrder
CancelOrder
OrderModified
OrderCancelled
OrderBilled
Modification
Billed
Sales-Priority
Billing-Standard
Shipping-Standard
priority
standard
43. Scenario #2 – RabbitMQ Two Layered Topic Exchanges
eCommerce
Site
amq.topic
Sales
Sales-Standard
Shipping-Priority
place.order.standard
modify.order.standard
cancel.order.standard
Billing-Priority
Billing
order.placed.priority
order.modified.priority
order.cancelled.priority
Single Topic Exchange (amq.topic)
Routing Key with event name
Class of Service as Message Header
Sales-Priority
Billing-Standard
order.placed.*
order.modified.*
order.cancelled.*
Shipping-Standard
order.placed.*
order.modified.*
order.cancelled.*
order.billed.*
modification.billed.*
Shipping
Sales (Topic)
place.order.*
modify.order.*
cancel.order.*
Billing
(Topic)
Shipping
(Topic)
order.billed.standard
modification.billed.standard
order.refunded.standard
order.billed.priority
modification.billed.priority
order.refunded.priority
#.priority
#.standard
#.priority
#.priority
#.standard
#.standard
place.order.priority
modify.order.priority
cancel.order.priority
order.placed.standard
order.modified.standard
order.cancelled.standard
44. Scenario #2 – RabbitMQ Topic and Header Exchanges
eCommerce
Site
amq.topic
Sales
Sales-Standard
Shipping-Priority
place.order
modify.order
cancel.order
Billing-Priority
Billing
order.placed
order.modified
order.cancelled
Single Topic Exchange (amq.topic)
Routing Key with event name
Class of Service as Message Header
Sales-Priority
Billing-Standard
order.placed
order.modified
order.cancelled
Shipping-Standard
order.placed
order.modified
order.cancelled,
order.billed
modification.billed
Shipping
Sales
(Header)
place.order
modify.order
cancel.order
Billing
(Header)
Shipping
(Header)order.billed
modification.billed
order.refunded
45. Scenario #2 – RabbitMQ Topic Exchanges
Priority VHost
Replicate preferred routing topology without Class of Service,
in two separate virtual hosts
Standard VHost
46. Scenario #2 – Kafka Topics
eCommerce
Site
Sales-Priority
Order Requests Topic
Orders Topic
Related Events in One Topic – Dedicated Application Instances
Billing-Priority
Shipping-Standard
OrderPlaced Event
OrderModified Event
OrderCancelled Event
OrderBilled Event
ModificationBilled Event
OrderRefunded Event
OrderShipped Event
OrderShippingCancelled Event
PlaceOrder Request
ModifyOrder Request
CancelOrder Request
Sales-Standard
Billing-Standard
Shipping-Priority
47. Scenario #2 – Kafka Topics
eCommerce
Site
Priority Orders Topic
Related Events in Class of Service Based Topics – Dedicated Instances
Billing-Priority
Shipping-Standard
OrderPlaced Event
OrderModified Event
OrderCancelled Event
OrderBilled Event, ModificationBilled Event, OrderRefunded Event
PlaceOrder Request
ModifyOrder Request
CancelOrder Request
Sales-Priority
Shipping-Priority
Billing-Standard
Standard Orders Topic
OrderShipped Event, OrderShippingCancelled Event
OrderShipped Event, OrderShippingCancelled Event
OrderBilled Event, ModificationBilled Event, OrderRefunded Event
Priority Order
Requests Topic
Sales-Standard
Standard Order
Requests Topic OrderPlaced Event
OrderModified Event
OrderCancelled Event
48. Scenario #2 – Replicate Events to Other Systems
eCommerce
Site
Sales-Priority
Order Requests Topic
Orders Topic
Kafka Topic as an Event Store and Data Replication Source
Billing-Priority
Shipping-Standard
Sales-Standard
Billing-Standard
Shipping-Priority
49. Decoupling of publishers from subscribers
Endpoint Decoupling
RabbitMQ: The endpoint we publish to is decoupled from
the endpoint we consume from.
Kafka: The endpoint we publish to is the same as we
subscribe from.
Temporal Decoupling
RabbitMQ: Consumers read a message once, most likely
close to the time of publishing.
Kafka: Consumers are decoupled temporally from the
publisher. Consumers can read messages multiple times,
and go back in history to read old messages.
50. RabbitMQ KafkaSimple Pub Sub
Easily Evolvable
Topologies
Transient Messages
Persistent Messages
AMQP
Time Travel
Data Replication
Event Store
Simple Scaling
Flexible Message
Routing