Apache Kafka and Kubernetes in the Telco Industry - Cloud-Native 5G, MEC and OSS/BSS/OTT Real-World Use Cases.
This session explores architectures and use cases for event streaming with the open-source framework Apache Kafka in the Telco industry. Telcos modernize their edge and hybrid cloud infrastructure with Kafka and Kubernetes to provide an elastic scalable real-time infrastructure for high volumes of data. Demonstrated use cases include building 5G networks, proactive OSS network monitoring, integration with hybrid and multi-cloud BSS and OTT services.
Video recording of the presentation:
https://youtu.be/yoRUC0sFbJo
More details about the event streaming and cloud-native architectures and use cases in the telecom sector:
https://www.kai-waehner.de/blog/2020/03/06/event-streaming-apache-kafka-telecommunications-industry-telco-business/
https://www.kai-waehner.de/blog/2020/07/03/telco-ott-applications-with-apache-kafka-telecom-sector-oss-bss-hybrid-cloud/
https://www.kai-waehner.de/blog/2021/05/23/apache-kafka-cloud-native-telco-infrastructure-low-latency-data-streaming-5g-aws-wavelength/
https://www.kai-waehner.de/blog/2020/10/14/use-cases-architectures-apache-kafka-edge-computing-industrial-iot-retail-store-cell-tower-train-factory/
https://www.kai-waehner.de/blog/2021/04/20/comparison-open-source-apache-kafka-vs-confluent-cloudera-red-hat-amazon-msk-cloud/
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Kai Wähner
Decentralized finance with crypto and NFTs is a huge topic these days. It becomes a powerful combination with the coming metaverse platforms across industries. This session explores the relationship between crypto technologies and modern enterprise architecture.
I discuss how data streaming and Apache Kafka help build innovation and scalable real-time applications of a future metaverse. Let's skip the buzz (and NFT bubble) and instead review existing real-world deployments in the crypto and blockchain world powered by Kafka and its ecosystem.
Apache Kafka is the de facto standard for data streaming to process data in motion. With its significant adoption growth across all industries, I get a very valid question every week: When NOT to use Apache Kafka? What limitations does the event streaming platform have? When does Kafka simply not provide the needed capabilities? How to qualify Kafka out as it is not the right tool for the job?
This session explores the DOs and DONTs. Separate sections explain when to use Kafka, when NOT to use Kafka, and when to MAYBE use Kafka.
No matter if you think about open source Apache Kafka, a cloud service like Confluent Cloud, or another technology using the Kafka protocol like Redpanda or Pulsar, check out this slide deck.
A detailed article about this topic:
https://www.kai-waehner.de/blog/2022/01/04/when-not-to-use-apache-kafka/
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKai Wähner
Live commerce combines instant purchasing of a featured product and audience participation.
This talk explores the need for real-time data streaming with Apache Kafka between applications to enable live commerce across online stores and brick & mortar stores across regions, countries, and continents in any retail business.
The discussion covers several building blocks of a live commerce enterprise architecture, including transactional data processing, omnichannel, natural language processing, augmented reality, edge computing, and more.
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaKai Wähner
If there were a buzzword of the hour, it would certainly be "data mesh"! This new architectural paradigm unlocks analytic data at scale and enables rapid access to an ever-growing number of distributed domain datasets for various usage scenarios.
As such, the data mesh addresses the most common weaknesses of the traditional centralized data lake or data platform architecture. And the heart of a data mesh infrastructure must be real-time, decoupled, reliable, and scalable.
This presentation explores how Apache Kafka, as an open and scalable decentralized real-time platform, can be the basis of a data mesh infrastructure and - complemented by many other data platforms like a data warehouse, data lake, and lakehouse - solve real business problems.
There is no silver bullet or single technology/product/cloud service for implementing a data mesh. The key outcome of a data mesh architecture is the ability to build data products; with the right tool for the job.
A good data mesh combines data streaming technology like Apache Kafka or Confluent Cloud with cloud-native data warehouse and data lake architectures from Snowflake, Databricks, Google BigQuery, et al.
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareKai Wähner
Enterprise integration is more challenging than ever before. The IT evolution requires the integration of more and more technologies. Applications are deployed across the edge, hybrid, and multi-cloud architectures. Traditional middleware such as MQ, ETL, ESB does not scale well enough or only processes data in batch instead of real-time.
This presentation explores why Apache Kafka is the new black for integration projects, how Kafka fits into the discussion around cloud-native iPaaS (Integration Platform as a Service) solutions, and why event streaming is a new software category.
A concrete real-world example shows the difference between event streaming and traditional integration platforms respectively cloud-native iPaaS.
Video Recording of this presentation:
https://www.youtube.com/watch?v=I8yZwKg_IJc&t=2842s
Blog post about this topic:
https://www.kai-waehner.de/blog/2021/11/03/apache-kafka-cloud-native-ipaas-versus-mq-etl-esb-middleware/
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?Kai Wähner
The concepts and architectures of a data warehouse, a data lake, and data streaming are complementary to solving business problems.
Unfortunately, the underlying technologies are often misunderstood, overused for monolithic and inflexible architectures, and pitched for wrong use cases by vendors. Let’s explore this dilemma in a presentation.
The slides cover technologies such as Apache Kafka, Apache Spark, Confluent, Databricks, Snowflake, Elasticsearch, AWS Redshift, GCP with Google Bigquery, and Azure Synapse.
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureKai Wähner
Apache Kafka in conjunction with Apache Spark became the de facto standard for processing and analyzing data. Both frameworks are open, flexible, and scalable.
Unfortunately, the latter makes operations a challenge for many teams. Ideally, teams can use serverless SaaS offerings to focus on business logic. However, hybrid and multi-cloud scenarios require a cloud-native platform that provides automated and elastic tooling to reduce the operations burden.
This session explores different architectures to build serverless Apache Kafka and Apache Spark multi-cloud architectures across regions and continents.
We start from the analytics perspective of a data lake and explore its relation to a fully integrated data streaming layer with Kafka to build a modern data Data Lakehouse.
Real-world use cases show the joint value and explore the benefit of the "delta lake" integration.
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...Kai Wähner
Hybrid cloud architectures are the new black for most companies. A cloud-first strategy is evident for many new enterprise architectures, but some use cases require resiliency across edge sites and multiple cloud regions. Data streaming with the Apache Kafka ecosystem is a perfect technology for building resilient and hybrid real-time applications at any scale. This talk explores different architectures and their trade-offs for transactional and analytical workloads. Real-world examples include financial services, retail, and the automotive industry.
Video recording:
https://qconlondon.com/london2022/presentation/resilient-real-time-data-streaming-across-the-edge-and-hybrid-cloud
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Kai Wähner
Decentralized finance with crypto and NFTs is a huge topic these days. It becomes a powerful combination with the coming metaverse platforms across industries. This session explores the relationship between crypto technologies and modern enterprise architecture.
I discuss how data streaming and Apache Kafka help build innovation and scalable real-time applications of a future metaverse. Let's skip the buzz (and NFT bubble) and instead review existing real-world deployments in the crypto and blockchain world powered by Kafka and its ecosystem.
Apache Kafka is the de facto standard for data streaming to process data in motion. With its significant adoption growth across all industries, I get a very valid question every week: When NOT to use Apache Kafka? What limitations does the event streaming platform have? When does Kafka simply not provide the needed capabilities? How to qualify Kafka out as it is not the right tool for the job?
This session explores the DOs and DONTs. Separate sections explain when to use Kafka, when NOT to use Kafka, and when to MAYBE use Kafka.
No matter if you think about open source Apache Kafka, a cloud service like Confluent Cloud, or another technology using the Kafka protocol like Redpanda or Pulsar, check out this slide deck.
A detailed article about this topic:
https://www.kai-waehner.de/blog/2022/01/04/when-not-to-use-apache-kafka/
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKai Wähner
Live commerce combines instant purchasing of a featured product and audience participation.
This talk explores the need for real-time data streaming with Apache Kafka between applications to enable live commerce across online stores and brick & mortar stores across regions, countries, and continents in any retail business.
The discussion covers several building blocks of a live commerce enterprise architecture, including transactional data processing, omnichannel, natural language processing, augmented reality, edge computing, and more.
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaKai Wähner
If there were a buzzword of the hour, it would certainly be "data mesh"! This new architectural paradigm unlocks analytic data at scale and enables rapid access to an ever-growing number of distributed domain datasets for various usage scenarios.
As such, the data mesh addresses the most common weaknesses of the traditional centralized data lake or data platform architecture. And the heart of a data mesh infrastructure must be real-time, decoupled, reliable, and scalable.
This presentation explores how Apache Kafka, as an open and scalable decentralized real-time platform, can be the basis of a data mesh infrastructure and - complemented by many other data platforms like a data warehouse, data lake, and lakehouse - solve real business problems.
There is no silver bullet or single technology/product/cloud service for implementing a data mesh. The key outcome of a data mesh architecture is the ability to build data products; with the right tool for the job.
A good data mesh combines data streaming technology like Apache Kafka or Confluent Cloud with cloud-native data warehouse and data lake architectures from Snowflake, Databricks, Google BigQuery, et al.
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareKai Wähner
Enterprise integration is more challenging than ever before. The IT evolution requires the integration of more and more technologies. Applications are deployed across the edge, hybrid, and multi-cloud architectures. Traditional middleware such as MQ, ETL, ESB does not scale well enough or only processes data in batch instead of real-time.
This presentation explores why Apache Kafka is the new black for integration projects, how Kafka fits into the discussion around cloud-native iPaaS (Integration Platform as a Service) solutions, and why event streaming is a new software category.
A concrete real-world example shows the difference between event streaming and traditional integration platforms respectively cloud-native iPaaS.
Video Recording of this presentation:
https://www.youtube.com/watch?v=I8yZwKg_IJc&t=2842s
Blog post about this topic:
https://www.kai-waehner.de/blog/2021/11/03/apache-kafka-cloud-native-ipaas-versus-mq-etl-esb-middleware/
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?Kai Wähner
The concepts and architectures of a data warehouse, a data lake, and data streaming are complementary to solving business problems.
Unfortunately, the underlying technologies are often misunderstood, overused for monolithic and inflexible architectures, and pitched for wrong use cases by vendors. Let’s explore this dilemma in a presentation.
The slides cover technologies such as Apache Kafka, Apache Spark, Confluent, Databricks, Snowflake, Elasticsearch, AWS Redshift, GCP with Google Bigquery, and Azure Synapse.
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureKai Wähner
Apache Kafka in conjunction with Apache Spark became the de facto standard for processing and analyzing data. Both frameworks are open, flexible, and scalable.
Unfortunately, the latter makes operations a challenge for many teams. Ideally, teams can use serverless SaaS offerings to focus on business logic. However, hybrid and multi-cloud scenarios require a cloud-native platform that provides automated and elastic tooling to reduce the operations burden.
This session explores different architectures to build serverless Apache Kafka and Apache Spark multi-cloud architectures across regions and continents.
We start from the analytics perspective of a data lake and explore its relation to a fully integrated data streaming layer with Kafka to build a modern data Data Lakehouse.
Real-world use cases show the joint value and explore the benefit of the "delta lake" integration.
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apac...Kai Wähner
Hybrid cloud architectures are the new black for most companies. A cloud-first strategy is evident for many new enterprise architectures, but some use cases require resiliency across edge sites and multiple cloud regions. Data streaming with the Apache Kafka ecosystem is a perfect technology for building resilient and hybrid real-time applications at any scale. This talk explores different architectures and their trade-offs for transactional and analytical workloads. Real-world examples include financial services, retail, and the automotive industry.
Video recording:
https://qconlondon.com/london2022/presentation/resilient-real-time-data-streaming-across-the-edge-and-hybrid-cloud
Data Streaming with Apache Kafka in the Defence and Cybersecurity IndustryKai Wähner
Agenda:
1) Defence, Modern Warfare, and Cybersecurity in 202X
2) Data in Motion with Apache Kafka as Defence Backbone
3) Situational Awareness
4) Threat Intelligence
5) Forensics and AI / Machine Learning
6) Air-Gapped and Zero Trust Environments
7) SIEM / SOAR Modernization
Technologies discussed in the presentation include Apache Kafka, Kafka Streams, kqlDB, Kafka Connect, Elasticsearch, Splunk, IBM QRadar, Zeek, Netflow, PCAP, TensorFlow, AWS, Azure, GCP, Sigma, Confluent Cloud,
Real-World Deployments of Data Streaming with Apache Kafka across the Healthcare Value Chain using open source and cloud-native technologies and serverless SaaS:
1) Legacy Modernization and Hybrid Cloud: Optum (UnitedHealth Group, Centene, Bayer)
2) Streaming ETL (Bayer, Babylon Health)
3) Real-time Analytics (Cerner, Celmatix, CDC/Centers for Disease Control and Prevention)
4) Machine Learning and Data Science (Recursion, Humana)
5) Open API and Omnichannel (Care.com, Invitae)
The Rise of Data in Motion in the Healthcare Industry - Use Cases, Architectures and Examples powered by Apache Kafka.
Use Cases for Data in Motion in the Healthcare Industry:
- Know Your Patient (= “Customer 360”)
- Operations (Healthcare 4.0 including Drug R&D, Patient Care, etc.)
- IT Perspective (Cybersecurity, Mainframe Offload, Hybrid Cloud, Streaming ETL, etc)
Real-world examples include Covid-19 Electronic Lab Reporting, Cerner, Optum, Centene, Humana, Invitae, Bayer, Celmatix, Care.com.
Apache Kafka for Real-time Supply Chainin the Food and Retail IndustryKai Wähner
Use Cases, Architectures, and Real-World Examples for data in motion and real-time event streaming powered by Apache Kafka across the supply chain and logistics. Case studies and deployments include Baader, Walmart, Migros, Albertsons, Domino's Pizza, Instacart, Grab, Royal Caribbean, and more.
Kafka for Real-Time Replication between Edge and Hybrid CloudKai Wähner
Not all workloads allow cloud computing. Low latency, cybersecurity, and cost-efficiency require a suitable combination of edge computing and cloud integration.
This session explores architectures and design patterns for software and hardware considerations to deploy hybrid data streaming with Apache Kafka anywhere. A live demo shows data synchronization from the edge to the public cloud across continents with Kafka on Hivecell and Confluent Cloud.
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0Kai Wähner
The manufacturing industry is moving away from just selling machinery, devices, and other hardware. Software and services increase revenue and margins. Equipment-as-a-Service (EaaS) even outsources the maintenance to the vendor.
This paradigm shift is only possible with reliable and scalable real-time data processing leveraging an event streaming platform such as Apache Kafka. This talk explores how Kafka-native Condition Monitoring and Predictive Maintenance help with this innovation.
More details:
https://www.kai-waehner.de/blog/2021/10/25/apache-kafka-condition-monitoring-predictive-maintenance-industrial-iot-digital-twin/
Video recording:
https://youtu.be/tfOuN5KeI9w
Apache Kafka Landscape for Automotive and ManufacturingKai Wähner
Today, in 2022, Apache Kafka is the central nervous system of many applications in various areas related to the automotive and manufacturing industry for processing analytical and transactional data in motion across edge, hybrid, and multi-cloud deployments.
This presentation explores the automotive event streaming landscape, including connected vehicles, smart manufacturing, supply chain optimization, aftersales, mobility services, and innovative new business models.
Afterwards, many real-world examples are shown from companies such as Audi, BMW, Porsche, Tesla, Uber, Grab, and FREENOW.
More detail in the blog post:
https://www.kai-waehner.de/blog/2022/01/12/apache-kafka-landscape-for-automotive-and-manufacturing/
Kappa vs Lambda Architectures and Technology ComparisonKai Wähner
Real-time data beats slow data. That’s true for almost every use case. Nevertheless, enterprise architects build new infrastructures with the Lambda architecture that includes separate batch and real-time layers.
This video explores why a single real-time pipeline, called Kappa architecture, is the better fit for many enterprise architectures. Real-world examples from companies such as Disney, Shopify, Uber, and Twitter explore the benefits of Kappa but also show how batch processing fits into this discussion positively without the need for a Lambda architecture.
The main focus of the discussion is on Apache Kafka (and its ecosystem) as the de facto standard for event streaming to process data in motion (the key concept of Kappa), but the video also compares various technologies and vendors such as Confluent, Cloudera, IBM Red Hat, Apache Flink, Apache Pulsar, AWS Kinesis, Amazon MSK, Azure Event Hubs, Google Pub Sub, and more.
Video recording of this presentation:
https://youtu.be/j7D29eyysDw
Further reading:
https://www.kai-waehner.de/blog/2021/09/23/real-time-kappa-architecture-mainstream-replacing-batch-lambda/
https://www.kai-waehner.de/blog/2021/04/20/comparison-open-source-apache-kafka-vs-confluent-cloudera-red-hat-amazon-msk-cloud/
https://www.kai-waehner.de/blog/2021/05/09/kafka-api-de-facto-standard-event-streaming-like-amazon-s3-object-storage/
The Top 5 Apache Kafka Use Cases and Architectures in 2022Kai Wähner
This document discusses the top 5 use cases and architectures for data in motion in 2022. It describes:
1) The Kappa architecture as an alternative to the Lambda architecture that uses a single stream to handle both real-time and batch data.
2) Hyper-personalized omnichannel experiences that integrate customer data from multiple sources in real-time to provide personalized experiences across channels.
3) Multi-cloud deployments using Apache Kafka and data mesh architectures to share data across different cloud platforms.
4) Edge analytics that deploy stream processing and Kafka brokers at the edge to enable low-latency use cases and offline functionality.
5) Real-time cybersecurity applications that use streaming data
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesKai Wähner
Technical thought leadership presentation to discuss how leading organizations move to real-time architecture to support business growth and enhance customer experience. This is a forum to discuss use cases with your peers to understand how other digital-native companies are utilizing data in motion to drive competitive advantage.
Agenda:
- Data in Motion with Event Streaming and Apache Kafka
- Streaming ETL Pipelines
- IT Modernisation and Hybrid Multi-Cloud
- Customer Experience and Customer 360
- IoT and Big Data Processing
- Machine Learning and Analytics
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...Kai Wähner
The Rise of Data in Motion in the Public Sector powered by event streaming with Apache Kafka.
Citizen Services:
- Health services, e.g. hospital modernization, track & trace - Covid distance control
- Public administration - reduce bureaucracy, data democratization across government departments
- eGovernment - Efficient and digital citizen engagement, e.g. personal ID application process
Smart City
- Smart driving, parking, buildings, environment
Waste management
- Open exchange – e.g. mobility services (1st and 3rd party)
Energy
- Smart grid and utilities infrastructure (energy distribution, smart home, smart meters, smart water, etc.)
- National Security
Law enforcement, surveillance, police/interior security data exchange
- Defense and military (border control, intelligent solider)
Cybersecurity for situational awareness and threat intelligence
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Kai Wähner
The Era of Telco 4.0: Embracing Digital Transformation with Data in Motion. Learn about Payment and FinServ Integration for Data in Motion with 5G and Apache Kafka.
1) The rise of Telco 4.0 and the future forward
2) Data in Motion in the Telco industry
3) Real-world Fintech and Payment examples powered by Data in Motion
Apache Kafka in the Transportation and LogisticsKai Wähner
Event Streaming with Apache Kafka in the Transportation and Logistics.
Track & Trace, Real-time Locating System, Customer 360, Open API, and more…
Examples include Swiss Post, SBB, Deutsche Bahn, Hermes, Migros, Here Technologies, Otonomo, Lyft, Uber, Free Now, Lufthansa, Air France, Singapore Airlines, Amadeus Group, and more.
Apache Kafka for Cybersecurity and SIEM / SOAR ModernizationKai Wähner
Data in Motion powered by the Apache Kafka ecosystem for Situational Awareness, Threat Detection, Forensics, Zero Trust Zones and Air-Gapped Environments.
Agenda:
1) Cybersecurity in 202X
2) Data in Motion as Cybersecurity Backbone
3) Situational Awareness
4) Threat Intelligence
5) Forensics
6) Air-Gapped and Zero Trust Environments
7) SIEM / SOAR Modernization
More details in the "Kafka for Cybersecurity" blog series:
https://www.kai-waehner.de/blog/2021/07/02/kafka-cybersecurity-siem-soar-part-1-of-6-data-in-motion-as-backbone/
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....Kai Wähner
Connect all the things: An intro to event streaming for the automotive industry including connected cars, mobility services, and manufacturing / industrial IoT.
Video recording of this talk: https://www.youtube.com/watch?v=rBfBFrcO-WU
The Fourth Industrial Revolution (also known as Industry 4.0) is the ongoing automation of traditional manufacturing and industrial practices, using modern smart technology. Event Streaming with Apache Kafka plays a massive role in processing massive volumes of data in real-time in a reliable, scalable, and flexible way using integrating with various legacy and modern data sources and sinks.
Other industries—retail, healthcare, government, financial services, energy, and more—also lean into Industry 4.0 technology to take advantage of IoT devices, sensors, smart machines, robotics, and connected data. The variety of these deployments goes from disconnected edge use cases across hybrid architectures to global multi-cloud deployments.
In this presentation, I want to give you an overview of existing use cases for event streaming technology in a connected world across supply chains, industries and customer experiences that come along with these interdisciplinary data intersections:
- The Automotive Industry (and it’s not only Connected Cars)
- Mobility Services across verticals (transportation, logistics, travel industry, retailing, …)
- Smart Cities (including citizen health services, communication infrastructure, …)
Real-world examples include use cases from car makers such as Audi, BMW, Porsche, Tesla, plus many examples from mobility services such as Uber, Lyft, Here Technologies, and more.
Serverless Kafka on AWS as Part of a Cloud-native Data Lake ArchitectureKai Wähner
AWS Data Lake / Lake House + Confluent Cloud for Serverless Apache Kafka. Learn about use cases, architectures, and features.
Data must be continuously collected, processed, and reactively used in applications across the entire enterprise - some in real time, some in batch mode. In other words: As an enterprise becomes increasingly software-defined, it needs a data platform designed primarily for "data in motion" rather than "data at rest."
Apache Kafka is now mainstream when it comes to data in motion! The Kafka API has become the de facto standard for event-driven architectures and event streaming. Unfortunately, the cost of running it yourself is very often too expensive when you add factors like scaling, administration, support, security, creating connectors...and everything else that goes with it. Resources in enterprises are scarce: this applies to both the best team members and the budget.
The cloud - as we all know - offers the perfect solution to such challenges.
Most likely, fully-managed cloud services such as AWS S3, DynamoDB or Redshift are already in use. Now it is time to implement "fully-managed" for Kafka as well - with Confluent Cloud on AWS.
Building a central integration layer that doesn't care where or how much data is coming from.
Implementing scalable data stream processing to gain real-time insights
Leveraging fully managed connectors (like S3, Redshift, Kinesis, MongoDB Atlas & more) to quickly access data
Confluent Cloud in action? Let's show how ao.com made it happen!
Translated with www.DeepL.com/Translator (free version)
IBM Cloud Pak for Integration with Confluent Platform powered by Apache KafkaKai Wähner
The Rise of Data in Motion powered by Event Streaming - Use Cases and Architecture for IBM Cloud Pak with Confluent Platform. Including screenshots of the live demo (integration between IBM and Kafka via Confluent Platform and Kafka Connect connectors).
Learn about the integration capabilities of IBM Cloud Pak for Integration, now with the industry’s leading event streaming platform from Confluent Platform powered by Apache Kafka.
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?Kai Wähner
Microservices became the new black in enterprise architectures. APIs provide functions to other applications or end users. Even if your architecture uses another pattern than microservices, like SOA (Service-Oriented Architecture) or Client-Server communication, APIs are used between the different applications and end users.
Apache Kafka plays a key role in modern microservice architectures to build open, scalable, flexible and decoupled real time applications. API Management complements Kafka by providing a way to implement and govern the full life cycle of the APIs.
This session explores how event streaming with Apache Kafka and API Management (including API Gateway and Service Mesh technologies) complement and compete with each other depending on the use case and point of view of the project team. The session concludes exploring the vision of event streaming APIs instead of RPC calls.
Understand how event streaming with Kafka and Confluent complements tools and frameworks such as Kong, Mulesoft, Apigee, Envoy, Istio, Linkerd, Software AG, TIBCO Mashery, IBM, Axway, etc.
A Streaming API Data Exchangeprovides streaming replication between business units and companies. API Management with REST/HTTP is not appropriate for streaming data.
The rise of data in motion in the insurance industry is visible across all lines of business including life, healthcare, travel, vehicle, and others. Apache Kafka changes how enterprises rethink data. This blog post explores use cases and architectures for event streaming. Real-world examples from Generali, Centene, Humana, and Telsa show innovative insurance-related data integration and stream processing in real-time.
Apache Kafka and MQTT - Overview, Comparison, Use Cases, ArchitecturesKai Wähner
Apache Kafka and MQTT are a perfect combination for many IoT use cases. This presentation covers the pros and cons of both technologies. Various use cases across industries, including connected vehicles, manufacturing, mobility services, and smart city are explored. The examples use different architectures, including lightweight edge scenarios, hybrid integrations, and serverless cloud solutions.
Blog series with more details here:
https://www.kai-waehner.de/blog/2021/03/15/apache-kafka-mqtt-sparkplug-iot-blog-series-part-1-of-5-overview-comparison/
Consistent toolbox talks are critical for maintaining workplace safety, as they provide regular opportunities to address specific hazards and reinforce safe practices.
These brief, focused sessions ensure that safety is a continual conversation rather than a one-time event, which helps keep safety protocols fresh in employees' minds. Studies have shown that shorter, more frequent training sessions are more effective for retention and behavior change compared to longer, infrequent sessions.
Engaging workers regularly, toolbox talks promote a culture of safety, empower employees to voice concerns, and ultimately reduce the likelihood of accidents and injuries on site.
The traditional method of conducting safety talks with paper documents and lengthy meetings is not only time-consuming but also less effective. Manual tracking of attendance and compliance is prone to errors and inconsistencies, leading to gaps in safety communication and potential non-compliance with OSHA regulations. Switching to a digital solution like Safelyio offers significant advantages.
Safelyio automates the delivery and documentation of safety talks, ensuring consistency and accessibility. The microlearning approach breaks down complex safety protocols into manageable, bite-sized pieces, making it easier for employees to absorb and retain information.
This method minimizes disruptions to work schedules, eliminates the hassle of paperwork, and ensures that all safety communications are tracked and recorded accurately. Ultimately, using a digital platform like Safelyio enhances engagement, compliance, and overall safety performance on site. https://safelyio.com/
Hyperledger Besu 빨리 따라하기 (Private Networks)wonyong hwang
Hyperledger Besu의 Private Networks에서 진행하는 실습입니다. 주요 내용은 공식 문서인https://besu.hyperledger.org/private-networks/tutorials 의 내용에서 발췌하였으며, Privacy Enabled Network와 Permissioned Network까지 다루고 있습니다.
This is a training session at Hyperledger Besu's Private Networks, with the main content excerpts from the official document besu.hyperledger.org/private-networks/tutorials and even covers the Private Enabled and Permitted Networks.
Data Streaming with Apache Kafka in the Defence and Cybersecurity IndustryKai Wähner
Agenda:
1) Defence, Modern Warfare, and Cybersecurity in 202X
2) Data in Motion with Apache Kafka as Defence Backbone
3) Situational Awareness
4) Threat Intelligence
5) Forensics and AI / Machine Learning
6) Air-Gapped and Zero Trust Environments
7) SIEM / SOAR Modernization
Technologies discussed in the presentation include Apache Kafka, Kafka Streams, kqlDB, Kafka Connect, Elasticsearch, Splunk, IBM QRadar, Zeek, Netflow, PCAP, TensorFlow, AWS, Azure, GCP, Sigma, Confluent Cloud,
Real-World Deployments of Data Streaming with Apache Kafka across the Healthcare Value Chain using open source and cloud-native technologies and serverless SaaS:
1) Legacy Modernization and Hybrid Cloud: Optum (UnitedHealth Group, Centene, Bayer)
2) Streaming ETL (Bayer, Babylon Health)
3) Real-time Analytics (Cerner, Celmatix, CDC/Centers for Disease Control and Prevention)
4) Machine Learning and Data Science (Recursion, Humana)
5) Open API and Omnichannel (Care.com, Invitae)
The Rise of Data in Motion in the Healthcare Industry - Use Cases, Architectures and Examples powered by Apache Kafka.
Use Cases for Data in Motion in the Healthcare Industry:
- Know Your Patient (= “Customer 360”)
- Operations (Healthcare 4.0 including Drug R&D, Patient Care, etc.)
- IT Perspective (Cybersecurity, Mainframe Offload, Hybrid Cloud, Streaming ETL, etc)
Real-world examples include Covid-19 Electronic Lab Reporting, Cerner, Optum, Centene, Humana, Invitae, Bayer, Celmatix, Care.com.
Apache Kafka for Real-time Supply Chainin the Food and Retail IndustryKai Wähner
Use Cases, Architectures, and Real-World Examples for data in motion and real-time event streaming powered by Apache Kafka across the supply chain and logistics. Case studies and deployments include Baader, Walmart, Migros, Albertsons, Domino's Pizza, Instacart, Grab, Royal Caribbean, and more.
Kafka for Real-Time Replication between Edge and Hybrid CloudKai Wähner
Not all workloads allow cloud computing. Low latency, cybersecurity, and cost-efficiency require a suitable combination of edge computing and cloud integration.
This session explores architectures and design patterns for software and hardware considerations to deploy hybrid data streaming with Apache Kafka anywhere. A live demo shows data synchronization from the edge to the public cloud across continents with Kafka on Hivecell and Confluent Cloud.
Apache Kafka for Predictive Maintenance in Industrial IoT / Industry 4.0Kai Wähner
The manufacturing industry is moving away from just selling machinery, devices, and other hardware. Software and services increase revenue and margins. Equipment-as-a-Service (EaaS) even outsources the maintenance to the vendor.
This paradigm shift is only possible with reliable and scalable real-time data processing leveraging an event streaming platform such as Apache Kafka. This talk explores how Kafka-native Condition Monitoring and Predictive Maintenance help with this innovation.
More details:
https://www.kai-waehner.de/blog/2021/10/25/apache-kafka-condition-monitoring-predictive-maintenance-industrial-iot-digital-twin/
Video recording:
https://youtu.be/tfOuN5KeI9w
Apache Kafka Landscape for Automotive and ManufacturingKai Wähner
Today, in 2022, Apache Kafka is the central nervous system of many applications in various areas related to the automotive and manufacturing industry for processing analytical and transactional data in motion across edge, hybrid, and multi-cloud deployments.
This presentation explores the automotive event streaming landscape, including connected vehicles, smart manufacturing, supply chain optimization, aftersales, mobility services, and innovative new business models.
Afterwards, many real-world examples are shown from companies such as Audi, BMW, Porsche, Tesla, Uber, Grab, and FREENOW.
More detail in the blog post:
https://www.kai-waehner.de/blog/2022/01/12/apache-kafka-landscape-for-automotive-and-manufacturing/
Kappa vs Lambda Architectures and Technology ComparisonKai Wähner
Real-time data beats slow data. That’s true for almost every use case. Nevertheless, enterprise architects build new infrastructures with the Lambda architecture that includes separate batch and real-time layers.
This video explores why a single real-time pipeline, called Kappa architecture, is the better fit for many enterprise architectures. Real-world examples from companies such as Disney, Shopify, Uber, and Twitter explore the benefits of Kappa but also show how batch processing fits into this discussion positively without the need for a Lambda architecture.
The main focus of the discussion is on Apache Kafka (and its ecosystem) as the de facto standard for event streaming to process data in motion (the key concept of Kappa), but the video also compares various technologies and vendors such as Confluent, Cloudera, IBM Red Hat, Apache Flink, Apache Pulsar, AWS Kinesis, Amazon MSK, Azure Event Hubs, Google Pub Sub, and more.
Video recording of this presentation:
https://youtu.be/j7D29eyysDw
Further reading:
https://www.kai-waehner.de/blog/2021/09/23/real-time-kappa-architecture-mainstream-replacing-batch-lambda/
https://www.kai-waehner.de/blog/2021/04/20/comparison-open-source-apache-kafka-vs-confluent-cloudera-red-hat-amazon-msk-cloud/
https://www.kai-waehner.de/blog/2021/05/09/kafka-api-de-facto-standard-event-streaming-like-amazon-s3-object-storage/
The Top 5 Apache Kafka Use Cases and Architectures in 2022Kai Wähner
This document discusses the top 5 use cases and architectures for data in motion in 2022. It describes:
1) The Kappa architecture as an alternative to the Lambda architecture that uses a single stream to handle both real-time and batch data.
2) Hyper-personalized omnichannel experiences that integrate customer data from multiple sources in real-time to provide personalized experiences across channels.
3) Multi-cloud deployments using Apache Kafka and data mesh architectures to share data across different cloud platforms.
4) Edge analytics that deploy stream processing and Kafka brokers at the edge to enable low-latency use cases and offline functionality.
5) Real-time cybersecurity applications that use streaming data
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesKai Wähner
Technical thought leadership presentation to discuss how leading organizations move to real-time architecture to support business growth and enhance customer experience. This is a forum to discuss use cases with your peers to understand how other digital-native companies are utilizing data in motion to drive competitive advantage.
Agenda:
- Data in Motion with Event Streaming and Apache Kafka
- Streaming ETL Pipelines
- IT Modernisation and Hybrid Multi-Cloud
- Customer Experience and Customer 360
- IoT and Big Data Processing
- Machine Learning and Analytics
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...Kai Wähner
The Rise of Data in Motion in the Public Sector powered by event streaming with Apache Kafka.
Citizen Services:
- Health services, e.g. hospital modernization, track & trace - Covid distance control
- Public administration - reduce bureaucracy, data democratization across government departments
- eGovernment - Efficient and digital citizen engagement, e.g. personal ID application process
Smart City
- Smart driving, parking, buildings, environment
Waste management
- Open exchange – e.g. mobility services (1st and 3rd party)
Energy
- Smart grid and utilities infrastructure (energy distribution, smart home, smart meters, smart water, etc.)
- National Security
Law enforcement, surveillance, police/interior security data exchange
- Defense and military (border control, intelligent solider)
Cybersecurity for situational awareness and threat intelligence
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Kai Wähner
The Era of Telco 4.0: Embracing Digital Transformation with Data in Motion. Learn about Payment and FinServ Integration for Data in Motion with 5G and Apache Kafka.
1) The rise of Telco 4.0 and the future forward
2) Data in Motion in the Telco industry
3) Real-world Fintech and Payment examples powered by Data in Motion
Apache Kafka in the Transportation and LogisticsKai Wähner
Event Streaming with Apache Kafka in the Transportation and Logistics.
Track & Trace, Real-time Locating System, Customer 360, Open API, and more…
Examples include Swiss Post, SBB, Deutsche Bahn, Hermes, Migros, Here Technologies, Otonomo, Lyft, Uber, Free Now, Lufthansa, Air France, Singapore Airlines, Amadeus Group, and more.
Apache Kafka for Cybersecurity and SIEM / SOAR ModernizationKai Wähner
Data in Motion powered by the Apache Kafka ecosystem for Situational Awareness, Threat Detection, Forensics, Zero Trust Zones and Air-Gapped Environments.
Agenda:
1) Cybersecurity in 202X
2) Data in Motion as Cybersecurity Backbone
3) Situational Awareness
4) Threat Intelligence
5) Forensics
6) Air-Gapped and Zero Trust Environments
7) SIEM / SOAR Modernization
More details in the "Kafka for Cybersecurity" blog series:
https://www.kai-waehner.de/blog/2021/07/02/kafka-cybersecurity-siem-soar-part-1-of-6-data-in-motion-as-backbone/
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....Kai Wähner
Connect all the things: An intro to event streaming for the automotive industry including connected cars, mobility services, and manufacturing / industrial IoT.
Video recording of this talk: https://www.youtube.com/watch?v=rBfBFrcO-WU
The Fourth Industrial Revolution (also known as Industry 4.0) is the ongoing automation of traditional manufacturing and industrial practices, using modern smart technology. Event Streaming with Apache Kafka plays a massive role in processing massive volumes of data in real-time in a reliable, scalable, and flexible way using integrating with various legacy and modern data sources and sinks.
Other industries—retail, healthcare, government, financial services, energy, and more—also lean into Industry 4.0 technology to take advantage of IoT devices, sensors, smart machines, robotics, and connected data. The variety of these deployments goes from disconnected edge use cases across hybrid architectures to global multi-cloud deployments.
In this presentation, I want to give you an overview of existing use cases for event streaming technology in a connected world across supply chains, industries and customer experiences that come along with these interdisciplinary data intersections:
- The Automotive Industry (and it’s not only Connected Cars)
- Mobility Services across verticals (transportation, logistics, travel industry, retailing, …)
- Smart Cities (including citizen health services, communication infrastructure, …)
Real-world examples include use cases from car makers such as Audi, BMW, Porsche, Tesla, plus many examples from mobility services such as Uber, Lyft, Here Technologies, and more.
Serverless Kafka on AWS as Part of a Cloud-native Data Lake ArchitectureKai Wähner
AWS Data Lake / Lake House + Confluent Cloud for Serverless Apache Kafka. Learn about use cases, architectures, and features.
Data must be continuously collected, processed, and reactively used in applications across the entire enterprise - some in real time, some in batch mode. In other words: As an enterprise becomes increasingly software-defined, it needs a data platform designed primarily for "data in motion" rather than "data at rest."
Apache Kafka is now mainstream when it comes to data in motion! The Kafka API has become the de facto standard for event-driven architectures and event streaming. Unfortunately, the cost of running it yourself is very often too expensive when you add factors like scaling, administration, support, security, creating connectors...and everything else that goes with it. Resources in enterprises are scarce: this applies to both the best team members and the budget.
The cloud - as we all know - offers the perfect solution to such challenges.
Most likely, fully-managed cloud services such as AWS S3, DynamoDB or Redshift are already in use. Now it is time to implement "fully-managed" for Kafka as well - with Confluent Cloud on AWS.
Building a central integration layer that doesn't care where or how much data is coming from.
Implementing scalable data stream processing to gain real-time insights
Leveraging fully managed connectors (like S3, Redshift, Kinesis, MongoDB Atlas & more) to quickly access data
Confluent Cloud in action? Let's show how ao.com made it happen!
Translated with www.DeepL.com/Translator (free version)
IBM Cloud Pak for Integration with Confluent Platform powered by Apache KafkaKai Wähner
The Rise of Data in Motion powered by Event Streaming - Use Cases and Architecture for IBM Cloud Pak with Confluent Platform. Including screenshots of the live demo (integration between IBM and Kafka via Confluent Platform and Kafka Connect connectors).
Learn about the integration capabilities of IBM Cloud Pak for Integration, now with the industry’s leading event streaming platform from Confluent Platform powered by Apache Kafka.
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?Kai Wähner
Microservices became the new black in enterprise architectures. APIs provide functions to other applications or end users. Even if your architecture uses another pattern than microservices, like SOA (Service-Oriented Architecture) or Client-Server communication, APIs are used between the different applications and end users.
Apache Kafka plays a key role in modern microservice architectures to build open, scalable, flexible and decoupled real time applications. API Management complements Kafka by providing a way to implement and govern the full life cycle of the APIs.
This session explores how event streaming with Apache Kafka and API Management (including API Gateway and Service Mesh technologies) complement and compete with each other depending on the use case and point of view of the project team. The session concludes exploring the vision of event streaming APIs instead of RPC calls.
Understand how event streaming with Kafka and Confluent complements tools and frameworks such as Kong, Mulesoft, Apigee, Envoy, Istio, Linkerd, Software AG, TIBCO Mashery, IBM, Axway, etc.
A Streaming API Data Exchangeprovides streaming replication between business units and companies. API Management with REST/HTTP is not appropriate for streaming data.
The rise of data in motion in the insurance industry is visible across all lines of business including life, healthcare, travel, vehicle, and others. Apache Kafka changes how enterprises rethink data. This blog post explores use cases and architectures for event streaming. Real-world examples from Generali, Centene, Humana, and Telsa show innovative insurance-related data integration and stream processing in real-time.
Apache Kafka and MQTT - Overview, Comparison, Use Cases, ArchitecturesKai Wähner
Apache Kafka and MQTT are a perfect combination for many IoT use cases. This presentation covers the pros and cons of both technologies. Various use cases across industries, including connected vehicles, manufacturing, mobility services, and smart city are explored. The examples use different architectures, including lightweight edge scenarios, hybrid integrations, and serverless cloud solutions.
Blog series with more details here:
https://www.kai-waehner.de/blog/2021/03/15/apache-kafka-mqtt-sparkplug-iot-blog-series-part-1-of-5-overview-comparison/
Consistent toolbox talks are critical for maintaining workplace safety, as they provide regular opportunities to address specific hazards and reinforce safe practices.
These brief, focused sessions ensure that safety is a continual conversation rather than a one-time event, which helps keep safety protocols fresh in employees' minds. Studies have shown that shorter, more frequent training sessions are more effective for retention and behavior change compared to longer, infrequent sessions.
Engaging workers regularly, toolbox talks promote a culture of safety, empower employees to voice concerns, and ultimately reduce the likelihood of accidents and injuries on site.
The traditional method of conducting safety talks with paper documents and lengthy meetings is not only time-consuming but also less effective. Manual tracking of attendance and compliance is prone to errors and inconsistencies, leading to gaps in safety communication and potential non-compliance with OSHA regulations. Switching to a digital solution like Safelyio offers significant advantages.
Safelyio automates the delivery and documentation of safety talks, ensuring consistency and accessibility. The microlearning approach breaks down complex safety protocols into manageable, bite-sized pieces, making it easier for employees to absorb and retain information.
This method minimizes disruptions to work schedules, eliminates the hassle of paperwork, and ensures that all safety communications are tracked and recorded accurately. Ultimately, using a digital platform like Safelyio enhances engagement, compliance, and overall safety performance on site. https://safelyio.com/
Hyperledger Besu 빨리 따라하기 (Private Networks)wonyong hwang
Hyperledger Besu의 Private Networks에서 진행하는 실습입니다. 주요 내용은 공식 문서인https://besu.hyperledger.org/private-networks/tutorials 의 내용에서 발췌하였으며, Privacy Enabled Network와 Permissioned Network까지 다루고 있습니다.
This is a training session at Hyperledger Besu's Private Networks, with the main content excerpts from the official document besu.hyperledger.org/private-networks/tutorials and even covers the Private Enabled and Permitted Networks.
Enhanced Screen Flows UI/UX using SLDS with Tom KittPeter Caitens
Join us for an engaging session led by Flow Champion, Tom Kitt. This session will dive into a technique of enhancing the user interfaces and user experiences within Screen Flows using the Salesforce Lightning Design System (SLDS). This technique uses Native functionality, with No Apex Code, No Custom Components and No Managed Packages required.
14 th Edition of International conference on computer visionShulagnaSarkar2
About the event
14th Edition of International conference on computer vision
Computer conferences organized by ScienceFather group. ScienceFather takes the privilege to invite speakers participants students delegates and exhibitors from across the globe to its International Conference on computer conferences to be held in the Various Beautiful cites of the world. computer conferences are a discussion of common Inventions-related issues and additionally trade information share proof thoughts and insight into advanced developments in the science inventions service system. New technology may create many materials and devices with a vast range of applications such as in Science medicine electronics biomaterials energy production and consumer products.
Nomination are Open!! Don't Miss it
Visit: computer.scifat.com
Award Nomination: https://x-i.me/ishnom
Conference Submission: https://x-i.me/anicon
For Enquiry: Computer@scifat.com
🏎️Tech Transformation: DevOps Insights from the Experts 👩💻campbellclarkson
Connect with fellow Trailblazers, learn from industry experts Glenda Thomson (Salesforce, Principal Technical Architect) and Will Dinn (Judo Bank, Salesforce Development Lead), and discover how to harness DevOps tools with Salesforce.
Boost Your Savings with These Money Management AppsJhone kinadey
A money management app can transform your financial life by tracking expenses, creating budgets, and setting financial goals. These apps offer features like real-time expense tracking, bill reminders, and personalized insights to help you save and manage money effectively. With a user-friendly interface, they simplify financial planning, making it easier to stay on top of your finances and achieve long-term financial stability.
These are the slides of the presentation given during the Q2 2024 Virtual VictoriaMetrics Meetup. View the recording here: https://www.youtube.com/watch?v=hzlMA_Ae9_4&t=206s
Topics covered:
1. What is VictoriaLogs
Open source database for logs
● Easy to setup and operate - just a single executable with sane default configs
● Works great with both structured and plaintext logs
● Uses up to 30x less RAM and up to 15x disk space than Elasticsearch
● Provides simple yet powerful query language for logs - LogsQL
2. Improved querying HTTP API
3. Data ingestion via Syslog protocol
* Automatic parsing of Syslog fields
* Supported transports:
○ UDP
○ TCP
○ TCP+TLS
* Gzip and deflate compression support
* Ability to configure distinct TCP and UDP ports with distinct settings
* Automatic log streams with (hostname, app_name, app_id) fields
4. LogsQL improvements
● Filtering shorthands
● week_range and day_range filters
● Limiters
● Log analytics
● Data extraction and transformation
● Additional filtering
● Sorting
5. VictoriaLogs Roadmap
● Accept logs via OpenTelemetry protocol
● VMUI improvements based on HTTP querying API
● Improve Grafana plugin for VictoriaLogs -
https://github.com/VictoriaMetrics/victorialogs-datasource
● Cluster version
○ Try single-node VictoriaLogs - it can replace 30-node Elasticsearch cluster in production
● Transparent historical data migration to object storage
○ Try single-node VictoriaLogs with persistent volumes - it compresses 1TB of production logs from
Kubernetes to 20GB
● See https://docs.victoriametrics.com/victorialogs/roadmap/
Try it out: https://victoriametrics.com/products/victorialogs/
How GenAI Can Improve Supplier Performance Management.pdfZycus
Data Collection and Analysis with GenAI enables organizations to gather, analyze, and visualize vast amounts of supplier data, identifying key performance indicators and trends. Predictive analytics forecast future supplier performance, mitigating risks and seizing opportunities. Supplier segmentation allows for tailored management strategies, optimizing resource allocation. Automated scorecards and reporting provide real-time insights, enhancing transparency and tracking progress. Collaboration is fostered through GenAI-powered platforms, driving continuous improvement. NLP analyzes unstructured feedback, uncovering deeper insights into supplier relationships. Simulation and scenario planning tools anticipate supply chain disruptions, supporting informed decision-making. Integration with existing systems enhances data accuracy and consistency. McKinsey estimates GenAI could deliver $2.6 trillion to $4.4 trillion in economic benefits annually across industries, revolutionizing procurement processes and delivering significant ROI.
The Rising Future of CPaaS in the Middle East 2024Yara Milbes
Explore "The Rising Future of CPaaS in the Middle East in 2024" with this comprehensive PPT presentation. Discover how Communication Platforms as a Service (CPaaS) is transforming communication across various sectors in the Middle East.
Flutter vs. React Native: A Detailed Comparison for App Development in 2024dhavalvaghelanectarb
Choosing the right framework for your cross-platform mobile app can be a tough decision. Both Flutter and React Native offer compelling features and have earned their place in the development world. Here is a detailed comparison to help you weigh their strengths and weaknesses. Here are the pros and cons of developing mobile apps in React Native vs Flutter.
Strengthening Web Development with CommandBox 6: Seamless Transition and Scal...Ortus Solutions, Corp
Join us for a session exploring CommandBox 6’s smooth website transition and efficient deployment. CommandBox revolutionizes web development, simplifying tasks across Linux, Windows, and Mac platforms. Gain insights and practical tips to enhance your development workflow.
Come join us for an enlightening session where we delve into the smooth transition of current websites and the efficient deployment of new ones using CommandBox 6. CommandBox has revolutionized web development, consistently introducing user-friendly enhancements that catalyze progress in the field. During this presentation, we’ll explore CommandBox’s rich history and showcase its unmatched capabilities within the realm of ColdFusion, covering both major variations.
The journey of CommandBox has been one of continuous innovation, constantly pushing boundaries to simplify and optimize development processes. Regardless of whether you’re working on Linux, Windows, or Mac platforms, CommandBox empowers developers to streamline tasks with unparalleled ease.
In our session, we’ll illustrate the simple process of transitioning existing websites to CommandBox 6, highlighting its intuitive features and seamless integration. Moreover, we’ll unveil the potential for effortlessly deploying multiple websites, demonstrating CommandBox’s versatility and adaptability.
Join us on this journey through the evolution of web development, guided by the transformative power of CommandBox 6. Gain invaluable insights, practical tips, and firsthand experiences that will enhance your development workflow and embolden your projects.
DECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSISTier1 app
Are you ready to unlock the secrets hidden within Java thread dumps? Join us for a hands-on session where we'll delve into effective troubleshooting patterns to swiftly identify the root causes of production problems. Discover the right tools, techniques, and best practices while exploring *real-world case studies of major outages* in Fortune 500 enterprises. Engage in interactive lab exercises where you'll have the opportunity to troubleshoot thread dumps and uncover performance issues firsthand. Join us and become a master of Java thread dump analysis!
Why Apache Kafka Clusters Are Like Galaxies (And Other Cosmic Kafka Quandarie...Paul Brebner
Closing talk for the Performance Engineering track at Community Over Code EU (Bratislava, Slovakia, June 5 2024) https://eu.communityovercode.org/sessions/2024/why-apache-kafka-clusters-are-like-galaxies-and-other-cosmic-kafka-quandaries-explored/ Instaclustr (now part of NetApp) manages 100s of Apache Kafka clusters of many different sizes, for a variety of use cases and customers. For the last 7 years I’ve been focused outwardly on exploring Kafka application development challenges, but recently I decided to look inward and see what I could discover about the performance, scalability and resource characteristics of the Kafka clusters themselves. Using a suite of Performance Engineering techniques, I will reveal some surprising discoveries about cosmic Kafka mysteries in our data centres, related to: cluster sizes and distribution (using Zipf’s Law), horizontal vs. vertical scalability, and predicting Kafka performance using metrics, modelling and regression techniques. These insights are relevant to Kafka developers and operators.
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...The Third Creative Media
"Navigating Invideo: A Comprehensive Guide" is an essential resource for anyone looking to master Invideo, an AI-powered video creation tool. This guide provides step-by-step instructions, helpful tips, and comparisons with other AI video creators. Whether you're a beginner or an experienced video editor, you'll find valuable insights to enhance your video projects and bring your creative ideas to life.
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...
Cloud-Native 5G, MEC and OSS/BSS/OTT Telco with Apache Kafka and Kubernetes
1. Apache Kafka and Kubernetes in the Telco Industry
Cloud-Native 5G, MEC and OSS/BSS/OTT Real-World Use Cases
Kai Waehner
Field CTO
contact@kai-waehner.de
linkedin.com/in/kaiwaehner
@KaiWaehner
www.confluent.io
www.kai-waehner.de
2. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
1. The Evolution of the Telco Industry
2. Data in Motion with Event Streaming
3. Edge, Hybrid and Cloud Architectures
4. Event Streaming in OSS
5. Event Streaming in BSS
6. Event Streaming in OTT
7. Case Study for Infrastructure Modernization
Agenda
3. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
1. The Evolution of the Telco Industry
2. Data in Motion with Event Streaming
3. Edge, Hybrid and Cloud Architectures
4. Event Streaming in OSS
5. Event Streaming in BSS
6. Event Streaming in OTT
7. Case Study for Infrastructure Modernization
Agenda
4. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Telco need to transform everything
“The biggest hurdle to
overcome with digital
transformation is to
understand that it isn’t just a
simple technology challenge
– it covers every part of the
telco business.”
https://www.huawei.com/en/about-huawei/publications/winwin-magazine/plus-intelligence/telco-digital-transformation
5. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
5G enables new business models
https://www.youtube.com/watch?v=A9DDe0alvGo
6. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
5G Network Slicing Use Cases
https://www.youtube.com/watch?v=A9DDe0alvGo
7. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
5G Success Criteria
https://medium.com/open-5g-hypercore/open-5g-hypercore-episode-i-166a89805c08
8. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Dish builds a Cloud-Based 5G Network
Greenfield standalone network with pure 5G and cloud native from the start, rather than retrofitting a 4G core
Combination of Dish’ OSS and BSS with AWS Local Zones / Outposts, Kubernetes, ML and 3rd party services
Free from the security limitations of traditional 2G, 3G and 4G technologies.
“As DISH deploys its network, the company is
partnering exclusively with vendors offering
cloud-native technology, bringing them
together on AWS to provide DISH customers
greater flexibility and control of their 5G-
enabled solutions.
As a new carrier, leveraging AWS and its
extensive network of partners enables us to
differentiate ourselves by operating our 5G
network with a high degree of automation.”
https://awsinsider.net/articles/2021/04/26/dish-taps-aws-for-its-5g-network.aspx
9. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
https://www2.deloitte.com/content/dam/Deloitte/it/Documents/technology-media-telecommunications/Next%20Gen%20Telco%20Architecture_2017_final.pdf
10. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
1. The Evolution of the Telco Industry
2. Data in Motion with Event Streaming
3. Edge, Hybrid and Cloud Architectures
4. Event Streaming in OSS
5. Event Streaming in BSS
6. Event Streaming in OTT
7. Case Study for Infrastructure Modernization
Agenda
11. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
This is a fundamental paradigm shift...
11
Infrastructure
as code
Data in motion
as continuous
streams of events
Future of the
datacenter
Future of data
Cloud-native
Event
Streaming
12. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Data in Motion in the Telco Industry
Your Business as Streams of Events, powered by Kafka
Continuous
Proactive Network
Monitoring
Incident
Management
Reporting
Data Normalization
Middleware
Telemetry
Data
OSS
Infrastructure
Example
13. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Apache Kafka is the Platform for Data in Motion
MES
ERP
Sensors
Mobile
Customer 360
Real-time Alerting
System
Data warehouse
Producers
Consumers
Streams and storage of real time events
Stream
processing
apps
Connectors
Connectors
Stream
processing
apps
Supplier
Alert
Forecast
Inventory Customer
Order
13
14. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Truly CLOUD-NATIVE experience
at the edge, in the data center, and in the cloud
Confluent Cloud
A fully managed, cloud-native service for Apache Kafka
Confluent Platform
A complete, enterprise-grade distribution of Apache Kafka
Confluent for
Kubernetes
Ansible
Playbooks
Packages:
Docker, RPMs,
Tarball
Public Cloud Workloads Edge and On-Premise Workloads
On Kubernetes On VMs / Bare Metal
15. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Kafka Connect
Kafka Cluster
CRM Integration
Domain-Driven Design (DDD) and decoupled Telco services
Legacy Integration
Proactive
Monitoring
Mainframe
Connector
Java / C++ /
Go / ksqlDB
Schema Registry
Event Streaming Platform
BSS Domain Legacy Customer Domain OSS Domain
Audit Logs,
RBAC, etc.
16. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
From purpose-built hardware to generic hardware and elastic scale
From monoliths to decoupled, independent services
https://www2.deloitte.com/content/dam/Deloitte/it/Documents/technology-media-telecommunications/Next%20Gen%20Telco%20Architecture_2017_final.pdf
(Network Function Virtualization)
17. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Event Streaming for
traditional and new innovative use cases
in the telecommunications industry
Real Time Processing Digital Transformation Strategic Goals
Move data from OSS fixed,
mobile, cable into
various systems
Innovative OTT Services
Proactive network
monitoring
CDR
(Call Detail Record)
processing
Fleet management and
connected car solutions
Customer service
(e.g. chat bots) and
social Integration
Cloud-native network
simulation, planning
and deployment
Personalized
messages and
recommendations
Fraud detection
Predictive maintenance
across networks
and infrastructure
Nationwide
security
infrastructure
(SIEM / SOAR)
Augmented
reality
Spanning different
business units
(TV, analytics, video
platform, sales and
services)
OSS-BSS integration
and middleware
modernization
Regulatory
reporting
Customer churn and
cross- / up-selling
Merger after
acquisitions
Omni-channel
customer 360
Converged
billing solution
18. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
1. The Evolution of the Telco Industry
2. Data in Motion with Event Streaming
3. Edge, Hybrid and Cloud Architectures
4. Event Streaming in OSS
5. Event Streaming in BSS
6. Event Streaming in OTT
7. Case Study for Infrastructure Modernization
Agenda
19. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Digital service at the edge
Single
Kafka Broker
or Cluster
Digital Service
Self-managed or
certified OEM Hardware
Kafka Cluster
in DC / Cloud
Replicator
SNMP
Monitoring
Syslog
Troubleshooting
TCP
Command & Control
20. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Hybrid Architectures in the Telco Industry
Digital Service
21. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Global Event Streaming
Streaming Replication between Kafka Clusters
Bridge to Databases, Data Lakes, Apps, APIs, SaaS
Aggregate Small Footprint Edge
Deployments with Replication
(Aggregation)
Simplify Disaster Recovery
Operations with
Multi-Region Clusters
with RPO=0 and RTO~0
Stream Data Globally with
Replication and Cluster Linking
21
22. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Low Latency 5G Use Cases for MEC
with AWS Wavelength (based on AWS Outposts) and Confluent
23. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
CRM
3rd party
payment
provider
Context-specific
real-time upsell
Customer data
Payment processing and
fraud detection as a service
Manager
Get report
API
Customer Customer
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Hybrid Retail Architecture
24. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Point of Sale
(POS) Loyalty
System
Local Inventory
Management
Payment Discount
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Global Inventory
Management
Event Streaming at the Edge
in the Smart Retail Store
Item Availability
25. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Disconnected Edge
Time
P
C3 C2
C1
Context-specific
Advertisement
Real-time
(Milliseconds)
Location-based
Customer Action
Always on (even “offline”)
Replayability
Reduced traffic cost
Better latency
Payment Processing
Near Real-time
(Seconds)
Replication to Cloud
Batch
(Depending on Network Bandwidth)
26. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
1. The Evolution of the Telco Industry
2. Data in Motion with Event Streaming
3. Edge, Hybrid and Cloud Architectures
4. Event Streaming in OSS
5. Event Streaming in BSS
6. Event Streaming in OTT
7. Case Study for Infrastructure Modernization
Agenda
27. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
OSM - Open Source MANO
(NFV Management and Orchestration)
Addressing challenges with orchestration, interoperability and performance optimization
Based on ETSI (European Telecommunication Standards Institute) NFV standards and Information Model
Interoperability among different service provider NFV infrastructures or OSS systems
Apache Kafka asynchronous communication between OSM components and decoupling introduction of new pluggable modules
https://osm.etsi.org/news-events/blog/30-open-source-mano-addressing-interoperability-challenge-in-nfv
28. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Global IP Network @ NTT Com
https://imply.io/post/kappa-architecture-at-ntt
Challenges
• Blackbox
• Not scalable
• Limited ad-hoc analysis capabilities
Real-time visibility into global traffic
• Technical: Capacity planning, traffic engineering, etc.
• Economical: Resource optimization, revenue leak detection, etc.
• Security
Traffic matrix
Representation of all traffic among all possible pairs of sources and destinations in a given
internet domain
29. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
1. The Evolution of the Telco Industry
2. Data in Motion with Event Streaming
3. Edge, Hybrid and Cloud Architectures
4. Event Streaming in OSS
5. Event Streaming in BSS
6. Event Streaming in OTT
7. Case Study for Infrastructure Modernization
Agenda
30. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
compax – BSS and CRM
AAX3 - A flexible, microservice-based BSS product
Powered by a cloud-native Kafka backbone
31. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
8x8 – Customer 360
Voice over IP (VoIP) provider
Cloud-based voice, contact center, video, mobile and
unified communications for >1 million business users
• Confluent to build a multi-tenant, scalable, elastic SaaS
contact center solution
• Strict data separation and secure, GDPR compliant data environment to
handle sensitive caller information
• Collection and processing of billions of metrics in real-time
• Replication between data centers for disaster recovery
• Real-time analytics for Dashboards and Machine
Learning
• Key customer experience metrics such as customer wait time and first-call
resolution rates
• Proactive alerts (for example, an issue with a local ISP)
• Prediction of staffing needs
• Predict contact center call volumes for the next 12 hours based on current
and historical data
https://www.confluent.io/customers/8x8/
32. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Omnichannel Retail
Time
P
App1 App2
App3
Sales Talk on site in
Car Dealership
Right now
Location-based
Customer Action
Customer 360
(Website, Mobile App, On Site in Store, In-Car)
Car Configurator
10 and 8 days ago
Context-specific
Marketing Campaign
90 and 60 days ago
33. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Omnichannel Retail
Time
P
App1 App2
App3
Machine Learning
Context-specific
Recommendations
Location-based
Customer Action
Customer 360
(Business Intelligence, Machine Learning)
Machine Learning
Train Recommendation Engine
Reporting
All Customer Interactions
34. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
1. The Evolution of the Telco Industry
2. Data in Motion with Event Streaming
3. Edge, Hybrid and Cloud Architectures
4. Event Streaming in OSS
5. Event Streaming in BSS
6. Event Streaming in OTT
7. Case Study for Infrastructure Modernization
Agenda
35. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Intrado
Tech-enabled real-time communication solutions
Resilient and reliable service for customers
Enterprise collaboration, digital media, healthcare, public safety
Personalized experiences through market intelligence and social sensing
https://www.confluent.io/online-talks/how-intrado-transformed-conferencing/
36. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Hotstar – OTT for millions of cricket fans in India
https://www.confluent.io/kafka-summit-san-francisco-2019/scaling-for-indias-cricket-hungry-population/
https://www.confluent.io/customers/disneyhotstar/
@
37. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
1. The Evolution of the Telco Industry
2. Data in Motion with Event Streaming
3. Edge, Hybrid and Cloud Architectures
4. Event Streaming in OSS
5. Event Streaming in BSS
6. Event Streaming in OTT
7. Case Study for Infrastructure Modernization
Agenda
38. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Replace old mainframe with Confluent
(instead of buying a new mainframe)
https://community.ibm.com/community/user/ibmz-and-linuxone/blogs/destination-z1/2019/12/23/capacity-is-king-how-the-ibm-z14-addresses-telecom-volume-concerns
39. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Strangler design pattern
https://paulhammant.com/2013/07/14/legacy-application-strangulation-case-studies/
https://martinfowler.com/bliki/StranglerFigApplication.html
“The most important
reason to consider a
strangler fig application
over a cut-over rewrite
is reduced risk.”
Martin Fowler
40. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Year 0: Direct communication between mainframe and app
Application
1) Direct Legacy Mainframe Communication to App
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Customer Service System
‘1980’ (Mainframe)
41. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Year 1: Kafka for decoupling between mainframe and app
Application
1) Direct Legacy Mainframe Communication to App
2) Kafka for Decoupling between Mainframe and App
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Mainframe Integration
- Change Data Capture (IIDR)
- Kafka Connect (JMS, MQ, JDBC)
- REST Proxy
- Kafka Client
- 3rd Party Tool
Customer Service System
‘1980’ (Mainframe)
42. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Year 2 to 4: New projects and applications
Application
Microservices
Agile, Lightweight
(but Scalable, Robust)
Applications
Big Data Project (Elastic,
Spark,
AWS Services, …)
1) Direct Legacy Mainframe Communication to App
2) Kafka for Decoupling between Mainframe and App
3) New Projects and Applications
External
Solution
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Mainframe Integration
- Change Data Capture (IIDR)
- Kafka Connect (JMS, MQ, JDBC)
- REST Proxy
- Kafka Client
- 3rd Party Tool
Customer Service System
‘1980’ (Mainframe)
43. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Year 5: Mainframe replacement
Application
Microservices
Agile, Lightweight
(but Scalable, Robust)
Applications
Big Data Project (Elastic,
Spark,
AWS Services, …)
1) Direct Legacy Mainframe Communication to App
2) Kafka for Decoupling between Mainframe and App
3) New Projects and Applications
4) Mainframe Replacement
External
Solution
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Customer Service System
‘2020’ (Modern Technology)
44. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Why Confluent?
45. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
The Rise of Data in Motion
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka
52
46. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
I N V E S T M E N T & T I M E
V
A
L
U
E
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka Cluster)
Start to Build Pipeline /
Deliver 1 New Outcome
(1 Kafka Cluster)
Mission-Critical Deployment
(Stretched, Hybrid,
Multi-Region)
Build Contextual
Event-Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
53
47. Apache Kafka in the Telco Industry – @KaiWaehner - www.kai-waehner.de
Car Engine Car Self-driving Car
Confluent completes Apache Kafka. Cloud-native. Everywhere.