Azure Labs: Confluent on Azure Container Services & Real-time Search with Redis | Alicia Moniz, Confluent and Ramya Orunganti, Microsoft

•

1 like•569 views

Technology

Kafka Summit
Presented by: Alicia Moniz
#kafkaonazure
Evangelista
#kafkaonAzure
Confluent.io

Kafka to Cloud Lifecycle
Azure First Development
Customers have streamlined their
infrastructure and are leveraging the
advantages of cloud-native
services across all their applications
Azure Adoption
Data migration and initial application
to Azure
• On-premises to Azure
• Multi-cloud AWS, GCP, Azure
Hybrid Cloud with Azure at Scale
Customers set up ongoing
synchronization between on-prem and
cloud environments

Hybrid Design Pattern
Bridge to Cloud
LEGACY EDW
DBs
Data Streams
Apps
Azure Data Lake Gen2
Azure Blob Storage
Azure Synapse
Azure Database for MySQL
Azure Database for PostgreSQL
Snowflake on Azure
MAINFRAME
Replicator
On-prem
Replicator
Azure SQL Database
Azure Cosmos Database
Source
Connectors
Sink
Connectors
KSQL DB
Schema Registry
Replicator

What is a
Connector?
Confluent Platform offers 120+
pre-built connectors to help you
quickly and reliably integrate
with Apache Kafka®.
Connectors import and export
data from some of the most
commonly used data systems.
Connectors either run as a
managed resource on Confluent
Cloud or as a self managed
resource on a self managed
kafka cluster.

Azure Connectors
Power Apps
Web Apps
Azure
Functions
Azure
Databricks
Azure Data Lake Gen2
Azure Blob Storage
Azure Synapse
Azure SQL Database
Azure Cosmos Database
Storage Databases App Services
Event/IOT
Edge
IOT Hub
Logic Apps
Event Hubs
Kafka APIs: Java Python, C#
(.NET), REST

Demo
Set up a Managed Connector :
Azure Data Lake Storage
Azure Data Lake Storage Gen2 Sink Connector
• The Azure Data Lake Gen2 Sink Connector
integrates Azure Data Lake Gen2 with Apache
Kafka. The connector can export data from
Apache Kafka® topics to Azure Data Lake Gen2
files in either Avro or JSON formats.
• Depending on your environment, the Azure Data
Lake Gen2 connector can export data by
guaranteeing exactly-once delivery semantics to
consumers of the Azure Data Lake Gen2 files it
produces. The Azure Data Lake Gen2 sink
connector periodically polls data from Kafka and
in turn uploads it to Azure Data Lake Gen2.

Demo
Set up a Self Managed Connector :
Cosmos DB on Azure Container Services
• Azure Cosmos DB Connector (Source and
Sink)
• The Azure Cosmos DB connectors allow
moving data between Azure Cosmos DB and
Kafka. The Cosmos DB Sink connector writes
data from a Kafka topic to a Cosmos DB
container.
• The Cosmos DB Source connector writes
changes from a Cosmos DB container to a
Kafka topic.

Next Steps
Visit
confluentinc/confluent-azure-
examples on github
Sign Up for a trial
Confluent Cloud on Azure
Marketplace

What's hot

Legacy migration is a journey. Mainframes cannot be replaced in a single project. A big bang will fail. This has to be planned long-term. Mainframe offloading and replacement with Apache Kafka and its ecosystem can be used to keep a more modern data store in real-time sync with the mainframe, while at the same time persisting the event data on the bus to enable microservices, and deliver the data to other systems such as data warehouses and search indexes. This session walks through the different steps some companies are already gone through. Technical options like Change Data Capture (CDC), MQ, and third-party tools for mainframe integration, offloading and replacement are explored.

Mainframe Integration, Offloading and Replacement with Apache Kafka | Kai Wae...

HostedbyConfluent

Confluent Enterprise Datasheet

confluent

Spring Cloud Stream is a framework built on top of the foundations of Spring Boot, the foremost JVM framework for developing microservice applications. It brings the familiar patterns and philosophies that Spring has championed for years through its programming model by allowing developers to focus primarily on the business logic of their applications. Kafka Streams is a powerful stream processing library built on top of Apache Kafka and attracts many developers because of its simplicity and deployment models as microservice applications. By developing Kafka Streams applications using Spring Cloud Stream, application developers get the best of both worlds - simpler stream processing execution models of Kafka Streams and battle-tested microservices foundations of Spring Boot via Spring Cloud Stream. This talk will explore: The integration points and various capabilities of Spring Cloud Stream touchpoints with Kafka Streams How to build event streaming applications using Spring’s programming model built on top of Kafka Streams, including a demo of a stateful application using Kafka Streams and Spring Cloud Stream’s functional support How to use interactive queries to expose materialized views from the state stores in the application How this Kafka Streams application can run as part of a data pipeline using Spring Cloud Data Flow in Kubernetes

Event Streaming with Kafka Streams and Spring Cloud Stream | Soby Chacko, VMware

HostedbyConfluent

Building Out Your Kafka Developer CDC Ecosystem

confluent

Scaling spark on kubernetes at Lyft

Li Gao

"There's little talk about capacity planning Kafka clusters, it's very much learn as you go, every cluster is different. In this talk Kafka DevOps Engineer Jason Bell takes you through the things that will help you, from broker capacity, thinking about topics and how the other Confluent components can affect throughput and performance. With a number of production deployments under his watchful gaze for over six years Jason has plenty of experience, stories and useful information that will help you. By the end of the talk you'll have a good understanding of designing the cluster for various scenarios, where the points of latency are to watch and monitor. And also how to prevent teams breaking the cluster behind your back. This talk is designed for everyone, anyone who is just starting to those who are operating Kafka on a daily basis."

Capacity Planning Your Kafka Cluster | Jason Bell, Digitalis

HostedbyConfluent

(Benoit Carrière, Expedia) Kafka Summit SF 2018 As Expedia, the world’s largest online travel agency, moved to a multi Virtual Private Cloud (VPC) strategy in AWS, we faced the challenge of making our systems accessible, or using other systems, across many VPCs. In most cases, a secure internet-facing endpoint or doing VPC Peering should do the work, right? But what if the system isn’t a typical HTTP-based microservice? What if it’s a distributed, partitioned and binary protocol-based system, where anyone talks to everyone all the time? That’s exactly what we encountered when we tried to make our Kafka accessible to our clients. We solved this problem by leveraging Apache Kafka®‘s distributive nature, using AWS’ new VPC Endpoint technology and their recent Network Load Balancer, some Route53 records and a bit of creativity! In this session, I’ll dive into: -Our use case: Kafka accessible to other VPCs -Why we didn’t go with internet-facing endpoint or use VPC Peering -A brief description on how VPC endpoints work -Our solution to the problem: That’s where the fun starts.

Connecting Kafka Across Multiple AWS VPCs

confluent

Data insights and data-driven strategies create the competitive differentiators companies thrive off today. The need for unified messaging and streaming has never been more apparent. Pulsar started with the goal of building a global, geo-replicated infrastructure to serve Yahoo!’s messaging needs. With the increased need to process both business events (such as payment request, billing request) and operational events (such as log data, click events, etc), the team at Yahoo! set out to build a true unified infrastructure platform to handle all in-motion data. That technology became Apache Pulsar. In this talk, Matteo Merli and Sijie Guo will dive into the landscape of unified messaging and streaming, how Pulsar helps companies achieve this vision, and what the future of Pulsar will look like.

Apache Pulsar: Why Unified Messaging and Streaming Is the Future - Pulsar Sum...

StreamNative

Gwen Shapira, Confluent | Kafka Summit 2020 Keynote | Kafka’s New Architecture

confluent

Providing a great media consumption experience to customers is crucial to maximizing audience engagement. To do that, it is important that you make content available for consumption anytime, anywhere, on any device, with a personalized and interactive experience. This session explores the power of big data log analytics (real-time and batched), using technologies like Spark, Shark, Kafka, Amazon Elastic MapReduce, Amazon Redshift and other AWS services. Such analytics are useful for content personalization, recommendations, personalized dynamic ad-insertions, interactivity, and streaming quality. This session also includes a discussion from Netflix, which explores personalized content search and discovery with the power of metadata.

Maximizing Audience Engagement in Media Delivery (MED303) | AWS re:Invent 2013

Amazon Web Services

Streaming with Spring Cloud Stream and Apache Kafka - Soby Chacko

VMware Tanzu

How to over-engineer things and have fun? | Oto Brglez, OPALAB

HostedbyConfluent

Can and should Apache Kafka replace a database? How long can and should I store data in Kafka? How can I query and process data in Kafka? These are common questions that come up more and more. This session explains the idea behind databases and different features like storage, queries, transactions, and processing to evaluate when Kafka is a good fit, and when it is not. The discussion includes different Kafka-native add-ons like Tiered Storage for long-term, cost-efficient storage, and ksqlDB as an event streaming database. The relation and trade-offs between Kafka and other databases are explored to complement each other instead of thinking about a replacement. This includes different options for pull and push-based bi-directional integration.

Can Apache Kafka Replace a Database? – The 2021 Update | Kai Waehner, Confluent

HostedbyConfluent

Application teams in JPMC have started shifting towards building event driven architectures and real time steaming pipelines and Kafka has been at core in this journey. As application teams have started adopting Kafka rapidly, need for a centrally managed Kafka as a service has emerged. We have started delivering Kafka as a service in early 2018 and running in production for more than an year now operating 80+ clusters (and growing) in all environments together. One of the key requirements is to provide truly segregated, secured multi-tenant environment with RBAC model while satisfying financial regulations and controls at the same time. Operating clusters at large scale requires scalable self-service capabilities and cluster management orchestration. In this talk we will present - Our experiences in delivering and operating secured, multi-tenant and resilient Kafka clusters at scale. - Internals of our service framework/control plane which enables self-service capabilities for application teams, cluster build/patch orchestration and capacity management capabilities for TSE/admin teams. - Our approach in enabling automated Cross Datacenter failover for application teams using service framework and confluent replicator.

Secure Kafka at scale in true multi-tenant environment ( Vishnu Balusu & Asho...

confluent

In this talk by Sean Glover, Principal Engineer at Lightbend, we will review how the Strimzi Kafka Operator, a supported technology in Lightbend Platform, makes many operational tasks in Kafka easy, such as the initial deployment and updates of a Kafka and ZooKeeper cluster. See the blog post containing the YouTube video here: https://www.lightbend.com/blog/running-kafka-on-kubernetes-with-strimzi-for-real-time-streaming-applications

Running Kafka On Kubernetes With Strimzi For Real-Time Streaming Applications

Lightbend

Serverless is a misnomer, your future cloud native applications will consist of both microservices and functions, wrapped as Linux containers, but in many cases where you the developer ignore the operational aspects of managing that infrastructure. In this session we get started using Function as a Service (FaaS) engine with Apache OpenWhisk deployed on Kubernetes and OpenShift. With Kubernetes/OpenShift being de-facto platform for Cloud Native Java Applications, we will explore on to see how to make Cloud Native Java Applications a.k.a Microservices can complement the serverless Functions. This technical intensive session will open up one of the possible serverless web application architecture where we deploy an API Gateway into the FaaS platform to draw the curtains up to see the microservices talking to the serverless functions. This session ends with us seeing how Event Sinks and Event Sources map in the Serverless World.

Serverless and Servicefull Applications - Where Microservices complements Ser...

Red Hat Developers

Milano Apache Kafka Meetup by Confluent (First Italian Kafka Meetup) on Wednesday, November 29th 2017. Il talk introduce Apache Kafka (incluse le APIs Kafka Connect e Kafka Streams), Confluent (la società creata dai creatori di Kafka) e spiega perché Kafka è un'ottima e semplice soluzione per la gestione di stream di dati nel contesto di due delle principali forze trainanti e trend industriali: Internet of Things (IoT) e Microservices.

Introduction to Apache Kafka and Confluent... and why they matter

confluent

Streams Change data capture (CDC) via Debezium is liberation for your data: By capturing changes from the log files of the database, it enables a wide range of use cases such as reliable microservices data exchange, the creation of audit logs, invalidating caches and much more. In this talk we're taking CDC to the next level by exploring the benefits of integrating Debezium with streaming queries via Kafka Streams. Come and join us to learn * How to run low-latency, time-windowed queries on your operational data * How to enrich audit logs with application-provided metadata * How to materialize aggregate views based on multiple change data streams, ensuring transactional boundaries of the source database We'll also show how to leverage the Quarkus stack for running your Kafka Streams applications on the JVM, as well as natively via GraalVM, many goodies included, such as its live coding feature for instant feedback during development, health checks, metrics and more.

Change Data Capture Pipelines with Debezium and Kafka Streams (Gunnar Morling...

HostedbyConfluent

You have built an event-driven system leveraging Apache Kafka. Now you face the challenge of integrating traditional synchronous request-response capabilities, such as user interaction, through an HTTP web service. There are various techniques, each with advantages and disadvantages. This talk discusses multiple options on how to do a request-response over Kafka — showcasing producers and consumers using single and multiple topics, and more advanced considerations using the interactive queries of ksqlDB and Kafka Streams. Advanced considerations discussed: What a consumer rebalance means to your active request-responses. Discuss options for blocking for the async response in the web-service. How can the CQRS (Command Query Responsibility Segregation) be leveraged with the interactive state stores of Kafka Streams and ksqlDB? Interactive queries of the ksqlDB and Kafka Streams state stores are not available during a rebalance. What is the active Kafka development happening that will make interactive queries a more feasible option? Would a custom state store help with rebalancing limitations? Can custom partitioning be used for proper routing, and what impacts could that have to the other services in your ecosystem? We will explore the above considerations with an interactive quiz application built using Apache Kafka, Kafka Streams, and ksqlDB. With a proper implementation in place, your request-response application can scale and be performant along with handling all of the requests.

Synchronous Commands over Apache Kafka (Neil Buesing, Object Partners, Inc) K...

confluent

apidays LIVE New York 2021 - Service reliability through autoscaling workload...

apidays

What's hot (20)