SlideShare a Scribd company logo
Classification: Internal
May 2024
Transition to Apache Kafka
on Kubernetes with Strimzi
Classification: Internal
Speakers
.
Steffen Wirenfeldt Karlsson
Lead Software Engineer
Streaming Services, Maersk
steffen.karlsson@maersk.com
Sai Charan Madhvaraj
Lead Software Engineer
RETinA Team, Maersk
sai.madhvaraj@maersk.com
Ravikanth Mallappa
Lead Platform Architect
Streaming Services, Maersk
ravi.mallappa@maersk.com
Classification: Internal
Agenda
• What is Maersk?
• Past
• Purpose
• Architecture
• Present
• Purpose
• Architecture & Features
• Tools
• Observability & Monitoring
• Alerting
• Contributions
• Future
• On-Prem
• Stream Processing
A.P. Moller - Maersk
3
Classification: Internal
A.P. Moller - Maersk
4
START
Meet the customer
Booking Set for reliable and cost-efficient
shipping to the final destination.
Collect the goods
Production: Picked up at the
customers’ facilities at any
place in the world.
Store the goods
Warehouse: Goods are stored or
managed throughout the supply
chain, based on customers’ needs.
Transport the goods
Transportation: Moved by the world’s
most sustainable fleet through Maersk's
global transport network.
Clear the goods at arrival
Import terminal: Taken through customs
promptly and efficiently.
Store the goods
Warehouse: Stored and managed for
optimisation of stock, costs and inventory days.
END
Deliver the goods
Customer warehouse or shop
Seamlessly delivered at the
destination of the customers'
preference.
Clear the goods for departure
Export terminal: Taken
through customs promptly
and efficiently.
ALL THE WAY
Connecting and simplifying
our customers’ supply chains
Rail Freight
Connecting and
simplifying
global supply chains
A.P. Moller - Maersk enables its customers to trade
and grow by transporting goods anywhere.
Maersk works to provide customers with a simple end-
to-end offering of products and services, seamless
customer engagement and a superior end-to-end
delivery network, taking the complexity out of global
supply chains.
Past
Based on Confluent
Classification: Internal
Why
A.P. Moller - Maersk
6
Past Present Future
Strategy to move from
batch to event driven
enterprise architecture
Fast and easy
bootstrap with
PaaS and SaaS
Gain trust of the
business
Cost-benefit analysis:
Pay-as-you-go
Classification: Internal
Architecture
• Confluent Cloud and Enterprise (on-prem)
• Datadog for observability and monitoring
• Azure for KeyVault, Container Registry, DevOps and AD
• Confluent Replicators from G2C and C2G
• Confluent Connectors from MQ applications to Cloud
• Confluent Control Center for management
• Automated GitOps-like self-service using GitHub and Azure DevOps
pipelines to deploy topics, users etc.
• Custom reconciler as Azure Pipeline, to bring cluster to up to state
with expected self-service
A.P. Moller - Maersk
7
Past Present Future
Cloud
Enterprise
On-Prem
Internal
On-Prem
Present
Based on Strimzi
Classification: Internal
Why
A.P. Moller - Maersk
9
Open-source adoption
strategy, benefitting from
the economy of scale
Building up internal
capabilities and inner-
sourcing for a more
tailored solution
Internal knowledge on
how to manage and
administer a Kafka
ecosystem
Rapidly increasing
adoption of the platform,
point of no return
Present
Past Future
Classification: Internal
API
RETinA Architecture
A.P. Moller - Maersk
10
Present
Past Future
Replication
Region X Region Y
Self-service
External
Systems
Runtime
Classification: Internal
Features
• Support for various Cluster Topologies
o Shared: Central cluster hosting topics of multiple platforms/tenants with required governance using quotas and schemas
o Dedicated: Independent Kafka clusters for platform teams with varying requirements
o Hub and Spoke: Aggregates data between local and central Kafka clusters
• Highly available with active-active muti-region setup and 99.99% SLA/SLO, implemented using Mirror Maker 2
• Enforced schema management on shared cluster (AVRO, JSON and Protobuf)
• External Data connectivity using Kafka Connect and Debezium
• API first architecture with automated self-service capabilities on top, using GitHub Actions
• Clusters constantly up-to-date by reconciliation using Flux
• Fully monitored and observed using Grafana and Prometheus stack
• Secret management with HashiCorp Vault and synced to the clusters using External Secrets
A.P. Moller - Maersk
11
Present
Past Future
Classification: Internal
Tenants
Number of teams using our
Strimzi-based solution +210
Number of Strimzi-based clusters
actively running
Number of brokers in all
our clusters
Topics
Total number of topics on
all our clusters
Total number of partitions
on all our topics +177K
Number of AVRO and JSON schema
versions in Schema Registry +22K
M A ER SK
Events
Total number of messages
produced per day, avg +250M
Bytes in per second, avg ~65Mb
~90Mb
Improving life
for all by integrating
the world
OUR PURPOSE
The integration illustrated by five years of Automatic Identification
System (AIS) transponder data from A.P. Moller - Maersk vessels
registered in the company’s scheduling system GSIS
Gateway and hub terminals
63
+300
+13K
Bytes out per second, avg
Classification: Internal
RETinA Manager
A.P. Moller - Maersk
13
Present
Past Future
Unique users
per day
+300
Requests
per day
+7.5K
https://github.com/provectus/kafka-ui
Classification: Internal
Schema Compatibility UI
A.P. Moller - Maersk
14
Present
Past Future
https://github.com/steffen-karlsson/schema-compatibility-ui
• Standalone and open-source
Apache-2.0 license
• Fail fast and enable tenants to be
more independent to decrease
time-to-market
• Full self-service and transparency in
schema compatibility and
comparison
• Schema Types:
Avro, JSON and Protobuf
• Compatibility levels:
Backwards, Forwards, Full and None
Classification: Internal
Observability & Monitoring
• Fully internal and open-source
based observability platform
• Based on the Prometheus Grafana
stack and OpenTelemetry
• Real User Monitoring
• Synthetic Monitoring
A.P. Moller - Maersk
15
Present
Past Future
Classification: Internal
Alerting
• Aggregated alerting by cluster and namespace
• Alerts and Dashboards as code
• Automated deployment and synchronization using GitHub Actions
• Alert prediction per environment
• SLA/SLO 99.99%
A.P. Moller - Maersk
16
Present
Past Future
Classification: Internal
Strimzi Contributions
A.P. Moller - Maersk
17
Present
Past Future
• #3761 Provide metrics to monitor certificates expiration
• #2779 CrdGenerator validate @JsonPropertyOrder
• #8732 Enhance KafkaBridge resource with consumer inactivity
timeout and HTTP consumer/producer parts enablement
• #9537 Kafka Exporter Grafana dashboard too long URL error
• #7374 Improve the Kafka brokers Grafana dashboard
Future
... still based on Strimzi
Classification: Internal
A.P. Moller - Maersk
19
Future
Present
Past
Replication
Replication
API
Self-service
• Future will be to migrate current Kafka
infrastructure running on VMs to K8S based
deployment using Strimzi
• Migration of existing tenants to Strimzi based
cluster will be done using MM2
• On-prem infrastructure to be fully monitored
and observed using the Grafana and
Prometheus stack
RETinA On-Prem
Classification: Internal
Stream Processing
A.P. Moller - Maersk
20
• Automated self-serviceable stream processing
on-top of Strimzi with Flink and GitHub Actions
• API-first architecture for better system-to-
system integrations
• Predefined template jobs for better overall
performance on the cluster
• Deployed using open-source community operator
for better stability and configuration of Flink
• Fully monitored and observed using the Grafana
and Prometheus stack
Future
Present
Past
Replication
API
Self-service
Driven by Purpose
Guided by Values
ALL THE WAY
Thank you
22

More Related Content

Similar to StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi

Kubernetes connectivity to Cloud Native Kafka | Evan Shortiss and Hugo Guerre...
Kubernetes connectivity to Cloud Native Kafka | Evan Shortiss and Hugo Guerre...Kubernetes connectivity to Cloud Native Kafka | Evan Shortiss and Hugo Guerre...
Kubernetes connectivity to Cloud Native Kafka | Evan Shortiss and Hugo Guerre...
HostedbyConfluent
 
Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...
Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...
Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...
HostedbyConfluent
 
Centralizing Kubernetes and Container Operations
Centralizing Kubernetes and Container OperationsCentralizing Kubernetes and Container Operations
Centralizing Kubernetes and Container Operations
Kublr
 
Building Real-time Pipelines with FLaNK_ A Case Study with Transit Data
Building Real-time Pipelines with FLaNK_ A Case Study with Transit DataBuilding Real-time Pipelines with FLaNK_ A Case Study with Transit Data
Building Real-time Pipelines with FLaNK_ A Case Study with Transit Data
Timothy Spann
 
AWS re:Invent 2016: Discovery Channel's Broadcast Workflows and Channel Origi...
AWS re:Invent 2016: Discovery Channel's Broadcast Workflows and Channel Origi...AWS re:Invent 2016: Discovery Channel's Broadcast Workflows and Channel Origi...
AWS re:Invent 2016: Discovery Channel's Broadcast Workflows and Channel Origi...
Amazon Web Services
 
Devops with Alibaba Cloud
Devops with Alibaba CloudDevops with Alibaba Cloud
Devops with Alibaba Cloud
gavaskar s
 
SpringOne Tour: An Introduction to Azure Spring Apps Enterprise
SpringOne Tour: An Introduction to Azure Spring Apps EnterpriseSpringOne Tour: An Introduction to Azure Spring Apps Enterprise
SpringOne Tour: An Introduction to Azure Spring Apps Enterprise
VMware Tanzu
 
CNCF Webinar - How to Gain Insights from Istio by leveraging CNCF projects
CNCF Webinar -  How to Gain Insights from Istio by leveraging CNCF projectsCNCF Webinar -  How to Gain Insights from Istio by leveraging CNCF projects
CNCF Webinar - How to Gain Insights from Istio by leveraging CNCF projects
Neeraj Poddar
 
Kappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology ComparisonKappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology Comparison
Kai Wähner
 
Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...
Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...
Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...
confluent
 
Apache Kafka® at Dropbox
Apache Kafka® at DropboxApache Kafka® at Dropbox
Apache Kafka® at Dropbox
confluent
 
OSO Confluent GitOps Demo
OSO Confluent GitOps DemoOSO Confluent GitOps Demo
OSO Confluent GitOps Demo
Sion Smith
 
Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022
Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022
Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022
HostedbyConfluent
 
HBase Meetup @ Cask HQ 09/25
HBase Meetup @ Cask HQ 09/25HBase Meetup @ Cask HQ 09/25
HBase Meetup @ Cask HQ 09/25
Cask Data
 
Kafka talk
Kafka talkKafka talk
Kafka talk
Maheedhar Gunturu
 
The Age of the Clusters: Offering Kafka as a Service in Your Organisation wit...
The Age of the Clusters: Offering Kafka as a Service in Your Organisation wit...The Age of the Clusters: Offering Kafka as a Service in Your Organisation wit...
The Age of the Clusters: Offering Kafka as a Service in Your Organisation wit...
HostedbyConfluent
 
Analyzing the World's Largest Security Data Lake!
Analyzing the World's Largest Security Data Lake!Analyzing the World's Largest Security Data Lake!
Analyzing the World's Largest Security Data Lake!
DataWorks Summit
 
Kafka as a service in your organsation
Kafka as a service in your organsationKafka as a service in your organsation
Kafka as a service in your organsation
Sion Smith
 
Webinar: What's new in CDAP 3.5?
Webinar: What's new in CDAP 3.5?Webinar: What's new in CDAP 3.5?
Webinar: What's new in CDAP 3.5?
Cask Data
 
What's New in IBM Streams V4.1
What's New in IBM Streams V4.1What's New in IBM Streams V4.1
What's New in IBM Streams V4.1
lisanl
 

Similar to StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi (20)

Kubernetes connectivity to Cloud Native Kafka | Evan Shortiss and Hugo Guerre...
Kubernetes connectivity to Cloud Native Kafka | Evan Shortiss and Hugo Guerre...Kubernetes connectivity to Cloud Native Kafka | Evan Shortiss and Hugo Guerre...
Kubernetes connectivity to Cloud Native Kafka | Evan Shortiss and Hugo Guerre...
 
Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...
Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...
Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...
 
Centralizing Kubernetes and Container Operations
Centralizing Kubernetes and Container OperationsCentralizing Kubernetes and Container Operations
Centralizing Kubernetes and Container Operations
 
Building Real-time Pipelines with FLaNK_ A Case Study with Transit Data
Building Real-time Pipelines with FLaNK_ A Case Study with Transit DataBuilding Real-time Pipelines with FLaNK_ A Case Study with Transit Data
Building Real-time Pipelines with FLaNK_ A Case Study with Transit Data
 
AWS re:Invent 2016: Discovery Channel's Broadcast Workflows and Channel Origi...
AWS re:Invent 2016: Discovery Channel's Broadcast Workflows and Channel Origi...AWS re:Invent 2016: Discovery Channel's Broadcast Workflows and Channel Origi...
AWS re:Invent 2016: Discovery Channel's Broadcast Workflows and Channel Origi...
 
Devops with Alibaba Cloud
Devops with Alibaba CloudDevops with Alibaba Cloud
Devops with Alibaba Cloud
 
SpringOne Tour: An Introduction to Azure Spring Apps Enterprise
SpringOne Tour: An Introduction to Azure Spring Apps EnterpriseSpringOne Tour: An Introduction to Azure Spring Apps Enterprise
SpringOne Tour: An Introduction to Azure Spring Apps Enterprise
 
CNCF Webinar - How to Gain Insights from Istio by leveraging CNCF projects
CNCF Webinar -  How to Gain Insights from Istio by leveraging CNCF projectsCNCF Webinar -  How to Gain Insights from Istio by leveraging CNCF projects
CNCF Webinar - How to Gain Insights from Istio by leveraging CNCF projects
 
Kappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology ComparisonKappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology Comparison
 
Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...
Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...
Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...
 
Apache Kafka® at Dropbox
Apache Kafka® at DropboxApache Kafka® at Dropbox
Apache Kafka® at Dropbox
 
OSO Confluent GitOps Demo
OSO Confluent GitOps DemoOSO Confluent GitOps Demo
OSO Confluent GitOps Demo
 
Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022
Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022
Streaming Time Series Data With Kenny Gorman and Elena Cuevas | Current 2022
 
HBase Meetup @ Cask HQ 09/25
HBase Meetup @ Cask HQ 09/25HBase Meetup @ Cask HQ 09/25
HBase Meetup @ Cask HQ 09/25
 
Kafka talk
Kafka talkKafka talk
Kafka talk
 
The Age of the Clusters: Offering Kafka as a Service in Your Organisation wit...
The Age of the Clusters: Offering Kafka as a Service in Your Organisation wit...The Age of the Clusters: Offering Kafka as a Service in Your Organisation wit...
The Age of the Clusters: Offering Kafka as a Service in Your Organisation wit...
 
Analyzing the World's Largest Security Data Lake!
Analyzing the World's Largest Security Data Lake!Analyzing the World's Largest Security Data Lake!
Analyzing the World's Largest Security Data Lake!
 
Kafka as a service in your organsation
Kafka as a service in your organsationKafka as a service in your organsation
Kafka as a service in your organsation
 
Webinar: What's new in CDAP 3.5?
Webinar: What's new in CDAP 3.5?Webinar: What's new in CDAP 3.5?
Webinar: What's new in CDAP 3.5?
 
What's New in IBM Streams V4.1
What's New in IBM Streams V4.1What's New in IBM Streams V4.1
What's New in IBM Streams V4.1
 

Recently uploaded

Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Need for Speed: Removing speed bumps from your Symfony projects ⚡️Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Łukasz Chruściel
 
Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Neo4j - Product Vision and Knowledge Graphs - GraphSummit ParisNeo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Neo4j
 
A Sighting of filterA in Typelevel Rite of Passage
A Sighting of filterA in Typelevel Rite of PassageA Sighting of filterA in Typelevel Rite of Passage
A Sighting of filterA in Typelevel Rite of Passage
Philip Schwarz
 
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptx
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxTop Features to Include in Your Winzo Clone App for Business Growth (4).pptx
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptx
rickgrimesss22
 
What is Augmented Reality Image Tracking
What is Augmented Reality Image TrackingWhat is Augmented Reality Image Tracking
What is Augmented Reality Image Tracking
pavan998932
 
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
mz5nrf0n
 
Cracking the code review at SpringIO 2024
Cracking the code review at SpringIO 2024Cracking the code review at SpringIO 2024
Cracking the code review at SpringIO 2024
Paco van Beckhoven
 
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOMLORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
lorraineandreiamcidl
 
Launch Your Streaming Platforms in Minutes
Launch Your Streaming Platforms in MinutesLaunch Your Streaming Platforms in Minutes
Launch Your Streaming Platforms in Minutes
Roshan Dwivedi
 
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
kalichargn70th171
 
SWEBOK and Education at FUSE Okinawa 2024
SWEBOK and Education at FUSE Okinawa 2024SWEBOK and Education at FUSE Okinawa 2024
SWEBOK and Education at FUSE Okinawa 2024
Hironori Washizaki
 
Vitthal Shirke Java Microservices Resume.pdf
Vitthal Shirke Java Microservices Resume.pdfVitthal Shirke Java Microservices Resume.pdf
Vitthal Shirke Java Microservices Resume.pdf
Vitthal Shirke
 
Graspan: A Big Data System for Big Code Analysis
Graspan: A Big Data System for Big Code AnalysisGraspan: A Big Data System for Big Code Analysis
Graspan: A Big Data System for Big Code Analysis
Aftab Hussain
 
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI AppAI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
Google
 
A Study of Variable-Role-based Feature Enrichment in Neural Models of Code
A Study of Variable-Role-based Feature Enrichment in Neural Models of CodeA Study of Variable-Role-based Feature Enrichment in Neural Models of Code
A Study of Variable-Role-based Feature Enrichment in Neural Models of Code
Aftab Hussain
 
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissancesAtelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Neo4j
 
Preparing Non - Technical Founders for Engaging a Tech Agency
Preparing Non - Technical Founders for Engaging  a  Tech AgencyPreparing Non - Technical Founders for Engaging  a  Tech Agency
Preparing Non - Technical Founders for Engaging a Tech Agency
ISH Technologies
 
Hand Rolled Applicative User Validation Code Kata
Hand Rolled Applicative User ValidationCode KataHand Rolled Applicative User ValidationCode Kata
Hand Rolled Applicative User Validation Code Kata
Philip Schwarz
 
Fundamentals of Programming and Language Processors
Fundamentals of Programming and Language ProcessorsFundamentals of Programming and Language Processors
Fundamentals of Programming and Language Processors
Rakesh Kumar R
 
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona InfotechMobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
Drona Infotech
 

Recently uploaded (20)

Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Need for Speed: Removing speed bumps from your Symfony projects ⚡️Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
 
Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Neo4j - Product Vision and Knowledge Graphs - GraphSummit ParisNeo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
 
A Sighting of filterA in Typelevel Rite of Passage
A Sighting of filterA in Typelevel Rite of PassageA Sighting of filterA in Typelevel Rite of Passage
A Sighting of filterA in Typelevel Rite of Passage
 
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptx
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxTop Features to Include in Your Winzo Clone App for Business Growth (4).pptx
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptx
 
What is Augmented Reality Image Tracking
What is Augmented Reality Image TrackingWhat is Augmented Reality Image Tracking
What is Augmented Reality Image Tracking
 
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
 
Cracking the code review at SpringIO 2024
Cracking the code review at SpringIO 2024Cracking the code review at SpringIO 2024
Cracking the code review at SpringIO 2024
 
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOMLORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
 
Launch Your Streaming Platforms in Minutes
Launch Your Streaming Platforms in MinutesLaunch Your Streaming Platforms in Minutes
Launch Your Streaming Platforms in Minutes
 
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
Why Mobile App Regression Testing is Critical for Sustained Success_ A Detail...
 
SWEBOK and Education at FUSE Okinawa 2024
SWEBOK and Education at FUSE Okinawa 2024SWEBOK and Education at FUSE Okinawa 2024
SWEBOK and Education at FUSE Okinawa 2024
 
Vitthal Shirke Java Microservices Resume.pdf
Vitthal Shirke Java Microservices Resume.pdfVitthal Shirke Java Microservices Resume.pdf
Vitthal Shirke Java Microservices Resume.pdf
 
Graspan: A Big Data System for Big Code Analysis
Graspan: A Big Data System for Big Code AnalysisGraspan: A Big Data System for Big Code Analysis
Graspan: A Big Data System for Big Code Analysis
 
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI AppAI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
 
A Study of Variable-Role-based Feature Enrichment in Neural Models of Code
A Study of Variable-Role-based Feature Enrichment in Neural Models of CodeA Study of Variable-Role-based Feature Enrichment in Neural Models of Code
A Study of Variable-Role-based Feature Enrichment in Neural Models of Code
 
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissancesAtelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissances
 
Preparing Non - Technical Founders for Engaging a Tech Agency
Preparing Non - Technical Founders for Engaging  a  Tech AgencyPreparing Non - Technical Founders for Engaging  a  Tech Agency
Preparing Non - Technical Founders for Engaging a Tech Agency
 
Hand Rolled Applicative User Validation Code Kata
Hand Rolled Applicative User ValidationCode KataHand Rolled Applicative User ValidationCode Kata
Hand Rolled Applicative User Validation Code Kata
 
Fundamentals of Programming and Language Processors
Fundamentals of Programming and Language ProcessorsFundamentals of Programming and Language Processors
Fundamentals of Programming and Language Processors
 
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona InfotechMobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
 

StrimziCon 2024 - Transition to Apache Kafka on Kubernetes with Strimzi

  • 1. Classification: Internal May 2024 Transition to Apache Kafka on Kubernetes with Strimzi
  • 2. Classification: Internal Speakers . Steffen Wirenfeldt Karlsson Lead Software Engineer Streaming Services, Maersk steffen.karlsson@maersk.com Sai Charan Madhvaraj Lead Software Engineer RETinA Team, Maersk sai.madhvaraj@maersk.com Ravikanth Mallappa Lead Platform Architect Streaming Services, Maersk ravi.mallappa@maersk.com
  • 3. Classification: Internal Agenda • What is Maersk? • Past • Purpose • Architecture • Present • Purpose • Architecture & Features • Tools • Observability & Monitoring • Alerting • Contributions • Future • On-Prem • Stream Processing A.P. Moller - Maersk 3
  • 4. Classification: Internal A.P. Moller - Maersk 4 START Meet the customer Booking Set for reliable and cost-efficient shipping to the final destination. Collect the goods Production: Picked up at the customers’ facilities at any place in the world. Store the goods Warehouse: Goods are stored or managed throughout the supply chain, based on customers’ needs. Transport the goods Transportation: Moved by the world’s most sustainable fleet through Maersk's global transport network. Clear the goods at arrival Import terminal: Taken through customs promptly and efficiently. Store the goods Warehouse: Stored and managed for optimisation of stock, costs and inventory days. END Deliver the goods Customer warehouse or shop Seamlessly delivered at the destination of the customers' preference. Clear the goods for departure Export terminal: Taken through customs promptly and efficiently. ALL THE WAY Connecting and simplifying our customers’ supply chains Rail Freight Connecting and simplifying global supply chains A.P. Moller - Maersk enables its customers to trade and grow by transporting goods anywhere. Maersk works to provide customers with a simple end- to-end offering of products and services, seamless customer engagement and a superior end-to-end delivery network, taking the complexity out of global supply chains.
  • 6. Classification: Internal Why A.P. Moller - Maersk 6 Past Present Future Strategy to move from batch to event driven enterprise architecture Fast and easy bootstrap with PaaS and SaaS Gain trust of the business Cost-benefit analysis: Pay-as-you-go
  • 7. Classification: Internal Architecture • Confluent Cloud and Enterprise (on-prem) • Datadog for observability and monitoring • Azure for KeyVault, Container Registry, DevOps and AD • Confluent Replicators from G2C and C2G • Confluent Connectors from MQ applications to Cloud • Confluent Control Center for management • Automated GitOps-like self-service using GitHub and Azure DevOps pipelines to deploy topics, users etc. • Custom reconciler as Azure Pipeline, to bring cluster to up to state with expected self-service A.P. Moller - Maersk 7 Past Present Future Cloud Enterprise On-Prem Internal On-Prem
  • 9. Classification: Internal Why A.P. Moller - Maersk 9 Open-source adoption strategy, benefitting from the economy of scale Building up internal capabilities and inner- sourcing for a more tailored solution Internal knowledge on how to manage and administer a Kafka ecosystem Rapidly increasing adoption of the platform, point of no return Present Past Future
  • 10. Classification: Internal API RETinA Architecture A.P. Moller - Maersk 10 Present Past Future Replication Region X Region Y Self-service External Systems Runtime
  • 11. Classification: Internal Features • Support for various Cluster Topologies o Shared: Central cluster hosting topics of multiple platforms/tenants with required governance using quotas and schemas o Dedicated: Independent Kafka clusters for platform teams with varying requirements o Hub and Spoke: Aggregates data between local and central Kafka clusters • Highly available with active-active muti-region setup and 99.99% SLA/SLO, implemented using Mirror Maker 2 • Enforced schema management on shared cluster (AVRO, JSON and Protobuf) • External Data connectivity using Kafka Connect and Debezium • API first architecture with automated self-service capabilities on top, using GitHub Actions • Clusters constantly up-to-date by reconciliation using Flux • Fully monitored and observed using Grafana and Prometheus stack • Secret management with HashiCorp Vault and synced to the clusters using External Secrets A.P. Moller - Maersk 11 Present Past Future
  • 12. Classification: Internal Tenants Number of teams using our Strimzi-based solution +210 Number of Strimzi-based clusters actively running Number of brokers in all our clusters Topics Total number of topics on all our clusters Total number of partitions on all our topics +177K Number of AVRO and JSON schema versions in Schema Registry +22K M A ER SK Events Total number of messages produced per day, avg +250M Bytes in per second, avg ~65Mb ~90Mb Improving life for all by integrating the world OUR PURPOSE The integration illustrated by five years of Automatic Identification System (AIS) transponder data from A.P. Moller - Maersk vessels registered in the company’s scheduling system GSIS Gateway and hub terminals 63 +300 +13K Bytes out per second, avg
  • 13. Classification: Internal RETinA Manager A.P. Moller - Maersk 13 Present Past Future Unique users per day +300 Requests per day +7.5K https://github.com/provectus/kafka-ui
  • 14. Classification: Internal Schema Compatibility UI A.P. Moller - Maersk 14 Present Past Future https://github.com/steffen-karlsson/schema-compatibility-ui • Standalone and open-source Apache-2.0 license • Fail fast and enable tenants to be more independent to decrease time-to-market • Full self-service and transparency in schema compatibility and comparison • Schema Types: Avro, JSON and Protobuf • Compatibility levels: Backwards, Forwards, Full and None
  • 15. Classification: Internal Observability & Monitoring • Fully internal and open-source based observability platform • Based on the Prometheus Grafana stack and OpenTelemetry • Real User Monitoring • Synthetic Monitoring A.P. Moller - Maersk 15 Present Past Future
  • 16. Classification: Internal Alerting • Aggregated alerting by cluster and namespace • Alerts and Dashboards as code • Automated deployment and synchronization using GitHub Actions • Alert prediction per environment • SLA/SLO 99.99% A.P. Moller - Maersk 16 Present Past Future
  • 17. Classification: Internal Strimzi Contributions A.P. Moller - Maersk 17 Present Past Future • #3761 Provide metrics to monitor certificates expiration • #2779 CrdGenerator validate @JsonPropertyOrder • #8732 Enhance KafkaBridge resource with consumer inactivity timeout and HTTP consumer/producer parts enablement • #9537 Kafka Exporter Grafana dashboard too long URL error • #7374 Improve the Kafka brokers Grafana dashboard
  • 19. Classification: Internal A.P. Moller - Maersk 19 Future Present Past Replication Replication API Self-service • Future will be to migrate current Kafka infrastructure running on VMs to K8S based deployment using Strimzi • Migration of existing tenants to Strimzi based cluster will be done using MM2 • On-prem infrastructure to be fully monitored and observed using the Grafana and Prometheus stack RETinA On-Prem
  • 20. Classification: Internal Stream Processing A.P. Moller - Maersk 20 • Automated self-serviceable stream processing on-top of Strimzi with Flink and GitHub Actions • API-first architecture for better system-to- system integrations • Predefined template jobs for better overall performance on the cluster • Deployed using open-source community operator for better stability and configuration of Flink • Fully monitored and observed using the Grafana and Prometheus stack Future Present Past Replication API Self-service
  • 21. Driven by Purpose Guided by Values ALL THE WAY