Data integration with Apache Kafka

Introduction to Kafka connect

Knoldus Inc.

Kafka presentation

Mohammed Fazuluddin

Stream processing using Kafka

Knoldus Inc.

Shift Deployment Security Left with Weave GitOps & Upbound’s Universal Crossp...

Weaveworks

In this session, we’ve partnered with Upbound to showcase how to effectively manage application delivery while maintaining a high level of security using Weave GitOps and Upbound. Managing a stateful application deployment with a relational database, Weave GitOps can recognize if there is a policy violation and correct it before deploying the application. Join us as we demonstrate the scenarios where: All changes to application configuration are managed through Git workflows Upbound’s Universal Crossplane allows you to build, deploy, and manage your cloud platforms GitOps provides an extra layer of security by removing the need for direct access to Kubernetes clusters Policy-as-Code guarantees security, resilience and coding standards compliance Watch the recording: xx

Watch this talk here: https://www.confluent.io/online-talks/apache-kafka-architecture-and-fundamentals-explained-on-demand This session explains Apache Kafka’s internal design and architecture. Companies like LinkedIn are now sending more than 1 trillion messages per day to Apache Kafka. Learn about the underlying design in Kafka that leads to such high throughput. This talk provides a comprehensive overview of Kafka architecture and internal functions, including: -Topics, partitions and segments -The commit log and streams -Brokers and broker replication -Producer basics -Consumers, consumer groups and offsets This session is part 2 of 4 in our Fundamentals for Apache Kafka series.

Service Mesh with Apache Kafka, Kubernetes, Envoy, Istio and Linkerd

Microservice architectures are not free lunch! Microservices need to be decoupled, flexible, operationally transparent, data aware and elastic. Most material from last years only discusses point-to-point architectures with inflexible and non-scalable technologies like REST / HTTP. This video takes a look at cutting edge technologies like Apache Kafka, Kubernetes, Envoy, Linkerd and Istio to implement a cloud-native service mesh to solve these challenges and bring microservices to the next level of scale, speed and efficiency. Key takeaways: - Apache Kafka decouples services, including event streams and request-response - Kubernetes provides a cloud-native infrastructure for the Kafka ecosystem - Service Mesh helps with security and observability at ecosystem / organization scale - Envoy and Istio sit in the layer above Kafka and are orthogonal to the goals Kafka addresses Blog post: http://www.kai-waehner.de/blog/2019/09/24/cloud-native-apache-kafka-kubernetes-envoy-istio-linkerd-service-mesh Video recording of this slide deck: https://youtu.be/Us_C4RFOUrA

Introduction to Apache Kafka

Jeff Holoman

Apache Kafka 0.8 basic training - Verisign

Michael Noll

Apache Kafka 0.8 basic training (120 slides) covering: 1. Introducing Kafka: history, Kafka at LinkedIn, Kafka adoption in the industry, why Kafka 2. Kafka core concepts: topics, partitions, replicas, producers, consumers, brokers 3. Operating Kafka: architecture, hardware specs, deploying, monitoring, P&S tuning 4. Developing Kafka apps: writing to Kafka, reading from Kafka, testing, serialization, compression, example apps 5. Playing with Kafka using Wirbelsturm Audience: developers, operations, architects Created by Michael G. Noll, Data Architect, Verisign, https://www.verisigninc.com/ Verisign is a global leader in domain names and internet security. Tools mentioned: - Wirbelsturm (https://github.com/miguno/wirbelsturm) - kafka-storm-starter (https://github.com/miguno/kafka-storm-starter) Blog post at: http://www.michael-noll.com/blog/2014/08/18/apache-kafka-training-deck-and-tutorial/ Many thanks to the LinkedIn Engineering team (the creators of Kafka) and the Apache Kafka open source community!

Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)

Stream Processing with Apache Kafka and .NET

Presentation from South Bay.NET meetup on 3/30. Speaker: Matt Howlett, Software Engineer at Confluent Apache Kafka is a scalable streaming platform that forms a key part of the infrastructure at many companies including Uber, Netflix, Walmart, Airbnb, Goldman Sachs and LinkedIn. In this talk Matt will give a technical overview of Kafka, discuss some typical use cases (from surge pricing to fraud detection to web analytics) and show you how to use Kafka from within your C#/.NET applications.

Introduction to Kafka Streams

Kafka Streams is a new stream processing library natively integrated with Kafka. It has a very low barrier to entry, easy operationalization, and a natural DSL for writing stream processing applications. As such it is the most convenient yet scalable option to analyze, transform, or otherwise process data that is backed by Kafka. We will provide the audience with an overview of Kafka Streams including its design and API, typical use cases, code examples, and an outlook of its upcoming roadmap. We will also compare Kafka Streams' light-weight library approach with heavier, framework-based tools such as Spark Streaming or Storm, which require you to understand and operate a whole different infrastructure for processing real-time data in Kafka.

Introduction to Apache Kafka

AIMDek Technologies

Introduction to Kubernetes

rajdeep

Apache kafka

Kumar Shivam

Containers Docker Kind Kubernetes Istio

Araf Karsh Hamid

Fundamentals of Apache Kafka

Chhavi Parasher

Deep Dive into Apache Kafka

In the last few years, Apache Kafka has been used extensively in enterprises for real-time data collecting, delivering, and processing. In this presentation, Jun Rao, Co-founder, Confluent, gives a deep dive on some of the key internals that help make Kafka popular. - Companies like LinkedIn are now sending more than 1 trillion messages per day to Kafka. Learn about the underlying design in Kafka that leads to such high throughput. - Many companies (e.g., financial institutions) are now storing mission critical data in Kafka. Learn how Kafka supports high availability and durability through its built-in replication mechanism. - One common use case of Kafka is for propagating updatable database records. Learn how a unique feature called compaction in Apache Kafka is designed to solve this kind of problem more naturally.

What is Apache Kafka and What is an Event Streaming Platform?

Speaker: Gabriel Schenker, Lead Curriculum Developer, Confluent Streaming platforms have emerged as a popular, new trend, but what exactly is a streaming platform? Part messaging system, part Hadoop made fast, part fast ETL and scalable data integration. With Apache Kafka® at the core, event streaming platforms offer an entirely new perspective on managing the flow of data. This talk will explain what an event streaming platform such as Apache Kafka is and some of the use cases and design patterns around its use—including several examples of where it is solving real business problems. New developments in this area such as KSQL will also be discussed.

How Apache Kafka® Works

Watch this talk here: https://www.confluent.io/online-talks/how-apache-kafka-works-on-demand Pick up best practices for developing applications that use Apache Kafka, beginning with a high level code overview for a basic producer and consumer. From there we’ll cover strategies for building powerful stream processing applications, including high availability through replication, data retention policies, producer design and producer guarantees. We’ll delve into the details of delivery guarantees, including exactly-once semantics, partition strategies and consumer group rebalances. The talk will finish with a discussion of compacted topics, troubleshooting strategies and a security overview. This session is part 3 of 4 in our Fundamentals for Apache Kafka series.

Streaming all over the world Real life use cases with Kafka Streams

Introducing Kafka's Streams API

Modern businesses have data at their core, and this data is changing continuously. How can we harness this torrent of information in real-time? The answer is stream processing, and the technology that has since become the core platform for streaming data is Apache Kafka. Among the thousands of companies that use Kafka to transform and reshape their industries are the likes of Netflix, Uber, PayPal, and AirBnB, but also established players such as Goldman Sachs, Cisco, and Oracle. Unfortunately, today’s common architectures for real-time data processing at scale suffer from complexity: there are many technologies that need to be stitched and operated together, and each individual technology is often complex by itself. This has led to a strong discrepancy between how we, as engineers, would like to work vs. how we actually end up working in practice. In this session we talk about how Apache Kafka helps you to radically simplify your data processing architectures. We cover how you can now build normal applications to serve your real-time processing needs — rather than building clusters or similar special-purpose infrastructure — and still benefit from properties such as high scalability, distributed computing, and fault-tolerance, which are typically associated exclusively with cluster technologies. Notably, we introduce Kafka’s Streams API, its abstractions for streams and tables, and its recently introduced Interactive Queries functionality. As we will see, Kafka makes such architectures equally viable for small, medium, and large scale use cases.

Exactly-once Stream Processing with Kafka Streams

An Introduction to Confluent Cloud: Apache Kafka as a Service

Business breakout during Confluent’s streaming event in Munich, presented by Hans Jespersen, VP WW Systems Engineering at Confluent. This three-day hands-on course focused on how to build, manage, and monitor clusters using industry best-practices developed by the world’s foremost Apache Kafka™ experts. The sessions focused on how Kafka and the Confluent Platform work, how their main subsystems interact, and how to set up, manage, monitor, and tune your cluster.

Building Event Driven (Micro)services with Apache Kafka

Guido Schmutz

What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to bind services together? Or is it better to use a richer, more loosely-coupled protocol? This talk will start with quick recap of how we created systems over the past 20 years and how different architectures evolved from it. The talk will show how we piece services together in event driven systems, how we use a distributed log (event hub) to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk will show the difference between a request-driven and event-driven communication and show when to use which. It highlights how the modern stream processing systems can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.

Streaming in Practice - Putting Apache Kafka in Production

This presentation focuses on how to integrate all these components into an enterprise environment and what things you need to consider as you move into production. We will touch on the following topics: - Patterns for integrating with existing data systems and applications - Metadata management at enterprise scale - Tradeoffs in performance, cost, availability and fault tolerance - Choosing which cross-datacenter replication patterns fit with your application - Considerations for operating Kafka-based data pipelines in production

Data Streaming with Apache Kafka & MongoDB

What's hot

Apache Kafka Architecture & Fundamentals Explained

Service Mesh with Apache Kafka, Kubernetes, Envoy, Istio and Linkerd

Introduction to Apache Kafka

Jeff Holoman

Apache Kafka 0.8 basic training - Verisign

Michael Noll

Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)

Stream Processing with Apache Kafka and .NET

Introduction to Kafka Streams

Introduction to Apache Kafka

AIMDek Technologies

Introduction to Kubernetes

rajdeep

Apache kafka

Kumar Shivam

Containers Docker Kind Kubernetes Istio

Araf Karsh Hamid

Fundamentals of Apache Kafka

Chhavi Parasher

Deep Dive into Apache Kafka

What is Apache Kafka and What is an Event Streaming Platform?

How Apache Kafka® Works

Streaming all over the world Real life use cases with Kafka Streams

Introducing Kafka's Streams API

Exactly-once Stream Processing with Kafka Streams

An Introduction to Confluent Cloud: Apache Kafka as a Service

Building Event Driven (Micro)services with Apache Kafka

Guido Schmutz

What's hot (20)

Apache Kafka Architecture & Fundamentals Explained

Service Mesh with Apache Kafka, Kubernetes, Envoy, Istio and Linkerd

Introduction to Apache Kafka

Apache Kafka 0.8 basic training - Verisign

Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)

Stream Processing with Apache Kafka and .NET

Introduction to Kafka Streams

Introduction to Apache Kafka

Introduction to Kubernetes

Apache kafka

Containers Docker Kind Kubernetes Istio

Fundamentals of Apache Kafka

Deep Dive into Apache Kafka

What is Apache Kafka and What is an Event Streaming Platform?

How Apache Kafka® Works

Streaming all over the world Real life use cases with Kafka Streams

Introducing Kafka's Streams API

Exactly-once Stream Processing with Kafka Streams

An Introduction to Confluent Cloud: Apache Kafka as a Service

Building Event Driven (Micro)services with Apache Kafka

Viewers also liked

Streaming in Practice - Putting Apache Kafka in Production

Data Streaming with Apache Kafka & MongoDB

Introduction To Streaming Data and Stream Processing with Apache Kafka

Modern businesses have data at their core, and this data is changing continuously. How can we harness this torrent of continuously changing data in real time? The answer is stream processing, and one system that has become a core hub for streaming data is Apache Kafka. This presentation will give a brief introduction to Apache Kafka and describe its usage as a platform for streaming data. It will explain how Kafka serves as a foundation for both streaming data pipelines and applications that consume and process real-time data streams. It will introduce some of the newer components of Kafka that help make this possible, including Kafka Connect, a framework for capturing continuous data streams, and Kafka Streams, a lightweight stream processing library. This is talk 1 out of 6 from the Kafka Talk Series. http://www.confluent.io/apache-kafka-talk-series/introduction-to-stream-processing-with-apache-kafka

Apache kafka-a distributed streaming platform

Demystifying Stream Processing with Apache Kafka

www.confluent.io/online-talks/ This talk will introduce Kafka Streams and help you understand how to map practical data problems to stream processing and how to write applications that process streams of data at scale using Kafka Streams. We will also cover what is stream processing, why one should care about stream processing, where Apache Kafka and Kafka Streams fit in, the hard parts of stream processing, and how Kafka Streams solves those problems; along with a concrete example of how these ideas tie together in Kafka Streams and in the big picture of your data center.

The Data Dichotomy- Rethinking the Way We Treat Data and Services

A Practical Guide to Selecting a Stream Processing Technology

Presented by Michael Noll, Product Manager, Confluent. Why are there so many stream processing frameworks that each define their own terminology? Are the components of each comparable? Why do you need to know about spouts or DStreams just to process a simple sequence of records? Depending on your application’s requirements, you may not need a full framework at all. Processing and understanding your data to create business value is the ultimate goal of a stream data platform. In this talk we will survey the stream processing landscape, the dimensions along which to evaluate stream processing technologies, and how they integrate with Apache Kafka. Particularly, we will learn how Kafka Streams, the built-in stream processing engine of Apache Kafka, compares to other stream processing systems that require a separate processing infrastructure.

What's new in Confluent 3.2 and Apache Kafka 0.10.2

With the introduction of connect and streams API in 2016, Apache Kafka is becoming the defacto solution for anyone looking to build a streaming platform. The community continues to add additional capabilities to make it the complete solution for streaming data. Join us as we review the latest additions in Apache Kafka 0.10.2. In addition, we’ll cover what’s new in Confluent Enterprise 3.2 that makes it possible for running Kafka at scale.

Power of the Log: LSM & Append Only Data Structures

This talk is about the beauty of sequential access and append-only data structures. We'll do this in the context of a little-known paper entitled “Log Structured Merge Trees”. LSM describes a surprisingly counterintuitive approach to storing and accessing data in a sequential fashion. It came to prominence in Google's Big Table paper and today, the use of Logs, LSM and append-only data structures drive many of the world's most influential storage systems: Cassandra, HBase, RocksDB, Kafka and more. Finally, we'll look at how the beauty of sequential access goes beyond database internals, right through to how applications communicate, share data and scale.

Leveraging Mainframe Data for Modern Analytics

“The mainframe is going away” is as true now as it was 10, 20 and 30 years ago. Mainframes are still crucial in handling critical business transactions, they were however built for an era where batch data movement was the norm and can be difficult to integrate into today’s data-driven, real-time, analytics-focused business processes as well as the environments that support them. Until now. Join experts from Confluent, Attunity, and Capgemini for a one-hour online talk session where you’ll learn how to: Unlock your mainframe data with unique change data capture (CDC) functionality without incurring the complexity and expense that come with sending ongoing queries into the mainframe database How using CDC benefits advanced analytics approaches such as deep machine learning and predictive analytics Deliver ongoing streams of data in real-time to the most demanding analytics environments Ensure that your analytics environment includes the broadest possible range of data sources and destinations while ensuring true enterprise-grade functionality Identify use cases that can help you get started delivering value to the business moving from POC to Pilot to Production

Microservices in the Apache Kafka Ecosystem

Strata+Hadoop 2017 San Jose - The Rise of Real Time: Apache Kafka and the Str...

Monitoring Apache Kafka with Confluent Control Center

Presentation by Nick Dearden, Direct, Product and Engineering, Confluent It’s 3 am. Do you know how your Kafka cluster is doing? With over 150 metrics to think about, operating a Kafka cluster can be daunting, particularly as a deployment grows. Confluent Control Center is the only complete monitoring and administration product for Apache Kafka and is designed specifically for making the Kafka operators life easier. Join Confluent as we cover how Control Center is used to simplify deployment, operability, and ensure message delivery. Watch the recording: https://www.confluent.io/online-talk/monitoring-and-alerting-apache-kafka-with-confluent-control-center/

Data Pipelines Made Simple with Apache Kafka

Presentation by Ewen Cheslack-Postava, Engineer, Apache Kafka Committer, Confluent In streaming workloads, often times data produced at the source is not useful down the pipeline or it requires some transformation to get it into usable shape. Similarly, where sensitive data is concerned, filtering of topics is helpful to ensure that the wrong data doesn't get to the wrong place. The newest release of Apache Kafka now offers the ability to do transformations on individual messages, making is possible to implement finer grained transformations customized to your unique needs. In this session we’ll talk about the new single message transform capabilities, how to use them to implement things like data masking and advanced partitioning, and when you’ll need to use more complex tools like the Kafka Streams API instead.

Strata+Hadoop 2017 San Jose: Lessons from a year of supporting Apache Kafka

The number of deployments of Apache Kafka at enterprise scale has greatly increased in the years since Kafka’s original development in 2010. Along with this rapid growth has come a wide variety of use cases and deployment strategies that transcend what Kafka’s creators imagined when they originally developed the technology. As the scope and reach of streaming data platforms based on Apache Kafka has grown, the need to understand monitoring and troubleshooting strategies has as well. Dustin Cote and Ryan Pridgeon share their experience supporting Apache Kafka at enterprise-scale and explore monitoring and troubleshooting techniques to help you avoid pitfalls when scaling large-scale Kafka deployments. Topics include: - Effective use of JMX for Kafka - Tools for preventing small problems from becoming big ones - Efficient architectures proven in the wild - Finding and storing the right information when it all goes wrong Visit www.confluent.io for more information.

Distributed stream processing with Apache Kafka

A modern business operates 24/7 and generates data continuously. Shouldn’t we process it continuously too? A rich ecosystem of real-time data-processing frameworks, tools and systems has been forming around Apache Kafka that allows data to be processed continuously as it occurs. Jay Kreps will introduce Kafka and explain why it has become the de facto standard for streaming data. He will draw on practical experience building stream-processing applications to discuss the difference between architectures and the challenges each presents. Jay will then outline the Kafka Streams API, which offers new stream processing functionality in Kafka, and explain how it helps tame some of the complexity in real-time architectures. Visit www.confluent.io for more information

user Behavior Analysis with Session Windows and Apache Kafka's Streams API

For many industries the need to group together related events based on a period of activity or inactivity is key. Advertising businesses, content producers are just a few examples of where session windows can be used to better understand user behavior. While such sessionization has been possible in Apache Kafka up to this point, implementing it has been rather complex and required leveraging low-level APIs. In the most recent release of Kafka, however, new capabilities have been added making session windows much easier to implement. In this online talk, we’ll introduce the concept of a session window, talk about common use cases, and walk through how Apache Kafka can be used for session-oriented use cases.

Building Event-Driven Services with Apache Kafka

Building Large-Scale Stream Infrastructures Across Multiple Data Centers with...

BY Jun Rao From the Bay Area Apache Kafka September 2016 Meetup. Abstract: To manage the ever-increasing volume and velocity of data within your company you have successfully made the transition from single machines and one-off solutions to large, distributed stream infrastructures in your data center powered by Apache Kafka. But what needs to be done if one data center is not enough? In this session we describe building resilient data pipelines with Apache Kafka that span multiple data centers and points of presence. We provide an overview of best practices and common patterns while covering key areas such as architecture guidelines, data replication and mirroring as well as disaster scenarios and failure handling.

Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout

Viewers also liked (20)

Streaming in Practice - Putting Apache Kafka in Production

Data Streaming with Apache Kafka & MongoDB

Introduction To Streaming Data and Stream Processing with Apache Kafka

Apache kafka-a distributed streaming platform

Demystifying Stream Processing with Apache Kafka

The Data Dichotomy- Rethinking the Way We Treat Data and Services

A Practical Guide to Selecting a Stream Processing Technology

What's new in Confluent 3.2 and Apache Kafka 0.10.2

Power of the Log: LSM & Append Only Data Structures

Leveraging Mainframe Data for Modern Analytics

Microservices in the Apache Kafka Ecosystem

Strata+Hadoop 2017 San Jose - The Rise of Real Time: Apache Kafka and the Str...

Monitoring Apache Kafka with Confluent Control Center

Data Pipelines Made Simple with Apache Kafka

Strata+Hadoop 2017 San Jose: Lessons from a year of supporting Apache Kafka

Distributed stream processing with Apache Kafka

user Behavior Analysis with Session Windows and Apache Kafka's Streams API

Building Event-Driven Services with Apache Kafka

Building Large-Scale Stream Infrastructures Across Multiple Data Centers with...

Kafka + Uber- The World’s Realtime Transit Infrastructure, Aaron Schildkrout

Similar to Data integration with Apache Kafka

Data Pipelines with Kafka Connect

Kaufman Ng

In this presentation we describe the design and implementation of Kafka Connect, Kafka’s new tool for scalable, fault-tolerant data import and export. First we’ll discuss some existing tools in the space and why they fall short when applied to data integration at large scale. Next, we will explore Kafka Connect’s design and how it compares to systems with similar goals, discussing key design decisions that trade off between ease of use for connector developers, operational complexity, and reuse of existing connectors. Finally, we’ll discuss how standardizing on Kafka Connect can ultimately lead to simplifying your entire data pipeline, making ETL into your data warehouse and enabling stream processing applications as simple as adding another Kafka connector.

The Never Landing Stream with HTAP and Streaming

Confluent kafka meetupseattle jan2017

Nitin Kumar

GSJUG: Mastering Data Streaming Pipelines 09May2023

GSJUG: Mastering Data Streaming Pipelines 09May2023 https://www.meetup.com/futureofdata-princeton/events/293233881/ This is a repost from the Garden State Java Users Group Event. Join me at https://www.meetup.com/garden-state-java-user-group/events/293229660/ See: https://www.eventbrite.com/e/mastering-data-streaming-pipelines-tickets-627677218457?_ga=2.253257801.1787151623.1682868226-741104479.1678110925 Please note that registration via EventBrite is required to attend either in-person or online. We are happy to announce that Tim Spann will be our special guest for the May 9, 2023 meeting! Abstract: In this session, Tim will show you some best practices that he has discovered over the last seven years in building data streaming applications including IoT, CDC, Logs, and more. In his modern approach, we utilize several Apache frameworks to maximize the best features of all. We often start with Apache NiFi as the orchestrator of streams flowing into Apache Kafka. From there we build streaming ETL with Apache Flink, enhance events with NiFi enrichment. We build continuous queries against our topics with Flink SQL. We will show where Java fits in as sources, enrichments, NiFi processors and sinks. We hope to see you on May 9! Speaker Timothy Spann Tim Spann is a Principal Developer Advocate in Data In Motion for Cloudera. He works with Apache NiFi, Apache Pulsar, Apache Kafka, Apache Flink, Flink SQL, Apache Pinot, Trino, Apache Iceberg, DeltaLake, Apache Spark, Big Data, IoT, Cloud, AI/DL, machine learning, and deep learning. Tim has over ten years of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton & NYC on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. He holds a BS and MS in computer science. In this session, Tim will show you some best practices that he has discovered over the last seven years in building data streaming applications, including IoT, CDC, Logs, and more. In his modern approach, we utilize several Apache frameworks to maximize the best features of all. We often start with Apache NiFi as the orchestrator of streams flowing into Apache Kafka. From there, we build streaming ETL with Apache Flink, enhance events with NiFi enrichment. We build continuous queries against our topics with Flink SQL. We will show where Java fits in as sources, enrichments, NiFi processors, and sinks. https://www.eventbrite.com/e/mastering-data-streaming-pipelines-tickets-627677218457?_ga=2.253257801.178

Building real time data-driven products

Lars Albertsson

This presentation will describe how to go beyond a "Hello world" stream application and build a real-time data-driven product. We will present architectural patterns, go through tradeoffs and considerations when deciding on technology and implementation strategy, and describe how to put the pieces together. We will also cover necessary practical pieces for building real products: testing streaming applications, and how to evolve products over time. Presented at highloadstrategy.com 2016 by Øyvind Løkling (Schibsted Products & Technology), joint work with Lars Albertsson (independent, www.mapflat.com).

Cloud-native Data: Every Microservice Needs a Cache

cornelia davis

Presented at the Pivotal Toronto Users Group, March 2017 Cloud-native applications form the foundation for modern, cloud-scale digital solutions, and the patterns and practices for cloud-native at the app tier are becoming widely understood – statelessness, service discovery, circuit breakers and more. But little has changed in the data tier. Our modern apps are often connected to monolithic shared databases that have monolithic practices wrapped around them. As a result, the autonomy promised by moving to a microservices application architecture is compromised. With lessons from the application tier to guide us, the industry is now figuring out what the cloud-native architectural patterns are at the data tier. Join us to explore some of these with Cornelia Davis, a five year Cloud Foundry veteran who is now focused on cloud-native data. As it happens, every microservice needs a cache and this evening will drill deep on that topic. She’ll cover a variety of caching patterns and use cases, and demonstrate how their use helps preserve the autonomy that is driving agile software delivery practices today.

Building streaming data applications using Kafka*[Connect + Core + Streams] b...

Data Con LA

Abstract:- Apache Kafka evolved from an enterprise messaging system to a fully distributed streaming data platform for building real-time streaming data pipelines and streaming data applications without the need for other tools/clusters for data ingestion, storage and stream processing. In this talk you will learn more about: A quick introduction to Kafka Core, Kafka Connect and Kafka Streams through code examples, key concepts and key features. A reference architecture for building such Kafka-based streaming data applications. A demo of an end-to-end Kafka-based streaming data application.

Being Ready for Apache Kafka - Apache: Big Data Europe 2015

Michael Noll

These are the slides of my Kafka talk at Apache: Big Data Europe in Budapest, Hungary. Enjoy! --Michael Apache Kafka is a high-throughput distributed messaging system that has become a mission-critical infrastructure component for modern data platforms. Kafka is used across a wide range of industries by thousands of companies such as Twitter, Netflix, Cisco, PayPal, and many others. After a brief introduction to Kafka this talk will provide an update on the growth and status of the Kafka project community. Rest of the talk will focus on walking the audience through what's required to put Kafka in production. We’ll give an overview of the current ecosystem of Kafka, including: client libraries for creating your own apps; operational tools; peripheral components required for running Kafka in production and for integration with other systems like Hadoop. We will cover the upcoming project roadmap, which adds key features to make Kafka even more convenient to use and more robust in production.

Budapest Data/ML - Building Modern Data Streaming Apps with NiFi, Flink and K...

Budapest Data/ML - Building Modern Data Streaming Apps with NiFi, Flink and Kafka Apache NiFi, Apache Flink, Apache Kafka Timothy Spann Principal Developer Advocate Cloudera Data in Motion https://budapestdata.hu/2023/en/speakers/timothy-spann/ Timothy Spann Principal Developer Advocate Cloudera (US) LinkedIn · GitHub · datainmotion.dev June 8 · Online · English talk Building Modern Data Streaming Apps with NiFi, Flink and Kafka In my session, I will show you some best practices I have discovered over the last 7 years in building data streaming applications including IoT, CDC, Logs, and more. In my modern approach, we utilize several open-source frameworks to maximize the best features of all. We often start with Apache NiFi as the orchestrator of streams flowing into Apache Kafka. From there we build streaming ETL with Apache Flink SQL. We will stream data into Apache Iceberg. We use the best streaming tools for the current applications with FLaNK. flankstack.dev BIO Tim Spann is a Principal Developer Advocate in Data In Motion for Cloudera. He works with Apache NiFi, Apache Pulsar, Apache Kafka, Apache Flink, Flink SQL, Apache Pinot, Trino, Apache Iceberg, DeltaLake, Apache Spark, Big Data, IoT, Cloud, AI/DL, machine learning, and deep learning. Tim has over ten years of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton & NYC on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. He holds a BS and MS in computer science.

Au delà des brokers, un tour de l’environnement Kafka | Florent Ramière

Unconference Round Table Notes

Unconference Round Table Notes The future of real-time stream processing WASM (Web Assembly) Petabyte, 5000 Node Clusters, Smart Hyper Scaling Multi-language support (Python, Rust, Kotlin, Golang, Carbon, JVM) Machine Learning, Deep Learning, AI and Advanced Math Low Code Development like Apache NiFi, DataFlow Designer, SQL Dynamic Hybrid Deployment Citizen Stream Engineer IoT, Edge Streaming and Hybrid Edge Streaming Java 20, 21; Java Loom Virtual Threading Ultra low latency, trillions of events per second, massive RAM/network Current challenges of real-time stream processing and proposed solutions Deployment, Automation and Scaling Choosing right project/sizing for use case Simple Event Processing vs Complex Event Processing Leveraging existing applications Developer Skills Self management and monitoring Cost issues -> autoscaling, optimizing, performance, hybrid deployment Performance / Benchmarking real-time stream processing Kafka/Pulsar: https://openmessaging.cloud/docs/benchmarks/ NiFi: https://blog.cloudera.com/benchmarking-nifi-performance-and-scalability/ Flink: https://github.com/ververica/flink-sql-benchmark Hazelcast: https://hazelcast.com/press-release/hazelcast-demonstrates-cloud-efficiency-real-time-stream-processing-of-one-billion-events-per-second/ Current trends of real-time stream processing in 2023 Current challenges of real-time stream processing and proposed solutions Performance / Benchmarking real-time stream processing The future of real-time stream processing Current trends of real-time stream processing in 2023 Lightweight serverless Hazelcast SQL Flink Kafka or Pulsar as Messaging Hub Java 17+ Managed Clusters, Containers and Environments Real-Time Analytics Fast Storage Options

How a Data Mesh is Driving our Platform | Trey Hicks, Gloo

HostedbyConfluent

At Gloo.us, we face a challenge in providing platform data to heterogeneous applications in a way that eliminates access contention, avoids high latency ETLs, and ensures consistency for many teams. We're solving this problem by adopting Data Mesh principles and leveraging Kafka, Kafka Connect, and Kafka streams to build an event driven architecture to connect applications to the data they need. A domain driven design keeps the boundaries between specialized process domains and singularly focused data domains clear, distinct, and disciplined. Applying the principles of a Data Mesh, process domains assume the responsibility of transforming, enriching, or aggregating data rather than relying on these changes at the source of truth -- the data domains. Architecturally, we've broken centralized big data lakes into smaller data stores that can be consumed into storage managed by process domains. This session covers how we’re applying Kafka tools to enable our data mesh architecture. This includes how we interpret and apply the data mesh paradigm, the role of Kafka as the backbone for a mesh of connectivity, the role of Kafka Connect to generate and consume data events, and the use of KSQL to perform minor transformations for consumers.

ETL as a Platform: Pandora Plays Nicely Everywhere with Real-Time Data Pipelines

ETL can be painful with dirty data and outdated batch processes slowing you down; there has to be a better way. In this talk we’ll discuss the benefits of introducing a streaming platform to your architecture including how it can greatly simplify complexity, speed up performance, and help your team deliver the features they need with real-time data integration. Pandora’s Lawrence Weikum will discuss what they’ve done to bring real-time data integration to the team. We’ll review their Kafka-powered data pipelines and how they make the most of Kafka’s Connect API to make it surprisingly system to keep systems in sync. Presented by: Lawrence Weikum, Senior Software Engineer, Pandora Gehrig Kunz, Technical Product Marketing Manager, Confluent

Introduction to Apache Mesos and DC/OS

Steve Wong

Building Streaming Data Applications Using Apache Kafka

Slim Baltagi

Apache Kafka evolved from an enterprise messaging system to a fully distributed streaming data platform for building real-time streaming data pipelines and streaming data applications without the need for other tools/clusters for data ingestion, storage and stream processing. In this talk you will learn more about: 1. A quick introduction to Kafka Core, Kafka Connect and Kafka Streams: What is and why? 2. Code and step-by-step instructions to build an end-to-end streaming data application using Apache Kafka

Bridge to Cloud: Using Apache Kafka to Migrate to AWS

Watch this talk here: https://www.confluent.io/online-talks/bridge-to-cloud-apache-kafka-migrate-aws Speakers: Priya Shivakumar, Director of Product, Confluent + Konstantine Karantasis, Software Engineer, Confluent + Rohit Pujari, Partner Solutions Architect, AWS Most companies start their cloud journey with a new use case, or a new application. Sometimes these applications can run independently in the cloud, but often times they need data from the on premises datacenter. Existing applications will slowly migrate, but will need a strategy and the technology to enable a multi-year migration. In this session, we will share how companies around the world are using Confluent Cloud, a fully managed Apache Kafka service, to migrate to AWS. By implementing a central-pipeline architecture using Apache Kafka to sync on-prem and cloud deployments, companies can accelerate migration times and reduce costs. In this online talk we will cover: •How to take the first step in migrating to AWS •How to reliably sync your on premises applications using a persistent bridge to cloud •Learn how Confluent Cloud can make this daunting task simple, reliable and performant •See a demo of the hybrid-cloud and multi-region deployment of Apache Kafka

DIMT '23 Session_Demo_ Latest Innovations Breakout.pdf

Confluent Partner Tech Talk with Reply

Cloud-native Data

cornelia davis

<November 2017 Updated from earlier presentations on Cloud-native Data> Cloud-native applications form the foundation for modern, cloud-scale digital solutions, and the patterns and practices for cloud-native at the app tier are becoming widely understood – statelessness, service discovery, circuit breakers and more. But little has changed in the data tier. Our modern apps are often connected to monolithic shared databases that have monolithic practices wrapped around them. As a result, the autonomy promised by moving to a microservices application architecture is compromised. What we need are patterns and practices for cloud-native data. The anti-patterns of shared databases and simple proxy-style web services to front them give way to approaches that include use of caches (Netflix calls caching their hidden microservice), database per service and polyglot persistence, modern versions of ETL and data integration and more. In this session, aimed at the application developer/architect, Cornelia will look at those patterns and see how they serve the needs of the cloud-native application.

Cloud-Native-Data with Cornelia Davis

VMware Tanzu

Similar to Data integration with Apache Kafka (20)

Data Pipelines with Kafka Connect

The Never Landing Stream with HTAP and Streaming

Confluent kafka meetupseattle jan2017

GSJUG: Mastering Data Streaming Pipelines 09May2023

Building real time data-driven products

Cloud-native Data: Every Microservice Needs a Cache

Building streaming data applications using Kafka*[Connect + Core + Streams] b...

Being Ready for Apache Kafka - Apache: Big Data Europe 2015

Budapest Data/ML - Building Modern Data Streaming Apps with NiFi, Flink and K...

Au delà des brokers, un tour de l’environnement Kafka | Florent Ramière

Unconference Round Table Notes

How a Data Mesh is Driving our Platform | Trey Hicks, Gloo

ETL as a Platform: Pandora Plays Nicely Everywhere with Real-Time Data Pipelines

Introduction to Apache Mesos and DC/OS

Building Streaming Data Applications Using Apache Kafka

Bridge to Cloud: Using Apache Kafka to Migrate to AWS

DIMT '23 Session_Demo_ Latest Innovations Breakout.pdf

Confluent Partner Tech Talk with Reply

Cloud-native Data

Cloud-Native-Data with Cornelia Davis

More from confluent

Speed Wins: From Kafka to APIs in Minutes

Evolving Data Governance for the Real-time Streaming and AI Era

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...

Santander Stream Processing with Apache Flink

Unlocking the Power of IoT: A comprehensive approach to real-time insights

Workshop híbrido: Stream Processing con Flink

El Stream processing es un requisito previo de la pila de data streaming, que impulsa aplicaciones y pipelines en tiempo real. Permite una mayor portabilidad de datos, una utilización optimizada de recursos y una mejor experiencia del cliente al procesar flujos de datos en tiempo real. En nuestro taller práctico híbrido, aprenderás cómo filtrar, unir y enriquecer fácilmente datos en tiempo real dentro de Confluent Cloud utilizando nuestro servicio Flink sin servidor.

Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...

Our talk will explore the transformative impact of integrating Confluent, HiveMQ, and SparkPlug in Industry 4.0, emphasizing the creation of a Unified Namespace. In addition to the creation of a Unified Namespace, our webinar will also delve into Stream Governance and Scaling, highlighting how these aspects are crucial for managing complex data flows and ensuring robust, scalable IIoT-Platforms. You will learn how to ensure data accuracy and reliability, expand your data processing capabilities, and optimize your data management processes. Don't miss out on this opportunity to learn from industry experts and take your business to the next level.

AWS Immersion Day Mapfre - Confluent

La arquitectura impulsada por eventos (EDA) será el corazón del ecosistema de MAPFRE. Para seguir siendo competitivas, las empresas de hoy dependen cada vez más del análisis de datos en tiempo real, lo que les permite obtener información y tiempos de respuesta más rápidos. Los negocios con datos en tiempo real consisten en tomar conciencia de la situación, detectar y responder a lo que está sucediendo en el mundo ahora.

Eventos y Microservicios - Santander TechTalk

Q&A with Confluent Experts: Navigating Networking in Confluent Cloud

Citi TechTalk Session 2: Kafka Deep Dive

Build real-time streaming data pipelines to AWS with Confluent

Traditional data pipelines often face scalability issues and challenges related to cost, their monolithic design, and reliance on batch data processing. They also typically operate under the premise that all data needs to be stored in a single centralized data source before it's put to practical use. Confluent Cloud on Amazon Web Services (AWS) provides a fully managed cloud-native platform that helps you simplify the way you build real-time data flows using streaming data pipelines and Apache Kafka.

Q&A with Confluent Professional Services: Confluent Service Mesh

Citi Tech Talk: Event Driven Kafka Microservices

Confluent & GSI Webinars series - Session 3

An in depth look at how Confluent is being used in the financial services industry. Gain an understanding of how organisations are utilising data in motion to solve common problems and gain benefits from their real time data capabilities. It will look more deeply into some specific use cases and show how Confluent technology is used to manage costs and mitigate risks. This session is aimed at Solutions Architects, Sales Engineers and Pre Sales, and also the more technically minded business aligned people. Whilst this is not a deeply technical session, a level of knowledge around Kafka would be helpful.

Citi Tech Talk: Messaging Modernization

Transforming applications built with traditional messaging solutions such as TIBCO, MQ and Solace to be scalable, reliable and ready for the move to cloud How can applications built with traditional messaging technologies like TIBCO, Solace and IBM MQ be modernised and be made cloud ready? What are the advantages to Event Streaming approaches to pub/sub vs traditional message queues? What are the strengeths and weaknesses of both approaches, and what use cases and requirements are actually a better fit for messaging than Kafka?

Citi Tech Talk: Data Governance for streaming and real time data

Confluent & GSI Webinars series: Session 2

Data In Motion Paris 2023

Vous apprendrez également à : • Créer plus rapidement des produits et fonctionnalités à l’aide d’une suite complète de connecteurs et d’outils de gestion des flux, et à connecter vos environnements à des pipelines de données • Protéger vos données et charges de travail les plus critiques grâce à des garanties intégrées en matière de sécurité, de gouvernance et de résilience • Déployer Kafka à grande échelle en quelques minutes tout en réduisant les coûts et la charge opérationnelle associés

Confluent Partner Tech Talk with Synthesis