Keeping Analytics Data Fresh in a Streaming Architecture | John Neal, Qlik

•Download as PPTX, PDF•

0 likes•430 views

Qlik is an industry leader across its solution stack, both on the Data Integration side of things with Qlik Replicate (real-time CDC) and Qlik Compose (data warehouse and data lake automation), and on the Analytics side with Qlik Sense. These two “sides” of Qlik are coming together more frequently these days as the need for “always fresh” data increases across organizations. When real-time streaming applications are the topic du jour, those companies are looking to Apache Kafka to provide the architectural backbone those applications require. Those same companies turn to Qlik Replicate to put the data from their enterprise database systems into motion at scale, whether that data resides in “legacy” mainframe databases; traditional relational databases such as Oracle, MySQL, or SQL Server; or applications such as SAP and SalesForce. In this session we will look in depth at how Qlik Replicate can be used to continuously stream changes from a source database into Apache Kafka. From there, we will explore how a purpose-built consumer can be used to provide the bridge between Apache Kafka and an analytics application such as Qlik Sense.

Technology

Keeping Analytics Data
Fresh in a Streaming
Architecture
Bridging the gap between Qlik
Replicate and Qlik Sense using
Apache Kafka
John Neal
Principal Solution Architect
Qlik Partner Engineering

2
What is Data Streaming ?
Data streams are generated by all types of sources,
in various formats and volumes.
“Streaming data” – the continuous flow
of data generated by various sources
“Kafka” - a stream processing technology
data streams can be processed, stored,
analyzed, and acted upon in real-time

3
Lambda-Style Architectures
Streaming and batch working together
NoSQL
IoT
Mobile
Apps
Web
Legacy
DB/DW
Incoming Data
Streaming (Speed) Layer
Serving Layer
Batch Layer
Stream Processing
(Spark Streaming,
Storm, Flink, …)
Incremental
Views
All Data Pre-Compute
Views
(Spark, M/R, HQL, …)
Real-time Views
Batch Views
Queries /
ML /
Analytics
Ingest & Store Prepare / Curate Publish Consume
Data

4
Kappa-Style Architectures
Where everything is a stream
Streaming Data
Streaming Layer
Stream Processing
(Spark Streaming,
Storm, Flink, …)
Real-time Results
Serving Layer
Real-time View
Queries /
ML /
Analytics
Mirror events
to long term
storage
Storage Layer
Raw Data History
Re-compute
events from
storage if
needed
Historical View
Ingest & Store Prepare / Curate Publish Consume
Data

5
CDC & Streaming Go Together
But Use the Right Tool for the Job at Hand
• Kafka is the glue that stitches streaming
components together to solve problems
• Change Data Capture can get data
moving again after it has landed in a
database, feeding streaming
architectures
• Kafka is not a replacement for straight on
database-to-database replication

6
TARGET SCHEMA
CREATION
SAP
RDBMS
EDW
FILE
MAINFRAME
HETEROGENEOUS
DATA TYPE MAPPING
BATCH TO CDC
TRANSITION
DDL CHANGE
PROPAGATION
FILTERING
TRANSFORMATIONS
RDBMS
EDW
FILES
STREAMING
DATA LAKE
Log Based
CDC
BATCH
IN-MEMORY
Replicate
Qlik Replicate
Automated Real-time Data Delivery

7
Deliver Any Source To Kafka
• Easy to use GUI
- Develop
- Manage and Monitor
• Log-based change data capture
- No triggers, no queries
• Supports databases on prem and in the cloud
• One solution for all sources
• Transform data in flight
• Filter at both table and column level
• Proven solution that supports production loads
Qlik Replicate: CDC at Enterprise Scale
Amazon RDS
Azure SQL
Managed Instance Amazon Aurora
Amazon RDS
Google Cloud SQL
Amazon Aurora
Amazon RDS
Google Cloud SQL
Amazon RDS
Amazon RDS
DB2 for zOS
DB2 for iSeries
DB2 LUW

8
Keep Analytics Data Fresh
Using CDC, Kafka, and Confluent
Replicate
Confluent Cloud
(Managed Kafka)
Confluent
Platform
ksqlDB
Machine
Learning
Sense

12
Summarizing Key Points
Qlik Replicate delivers
data from databases to
Kafka in real-time at
enterprise scale.
Purpose-built Kafka
Consumers bridge gaps
Kafka is the glue that
holds streaming
architectures together
Kafka is not a
replacement for direct
database to database
replication

https://www.qlik.com/products/data-integration-products
john.neal@qlik.com

One of the key metrics to monitor when working with Apache Kafka, as a data pipeline or a streaming platform, is Consumer Groups Lag. Lag is the delta between the last produced message and the last committed message of a partition. In other words, lag indicates how far behind your application is in processing up-to-date information. For a long time, we used our own service to keep track of these metrics, collect them and visualize them. But this didn’t scale well. You had to perform many manual operations, redeploy it and to do other tedious manual tasks, but most importantly, the biggest gap for us, was that its output was represented in absolute numbers (e.g - your lag is 30K), which basically tells you nothing as a human being. We understood that we had to find a more suitable solution that will give us better visibility and will allow us to measure the lag in a time-based format that we all understand. In this talk, I’m going to go over the core concepts of Kafka offsets and lags, and explain why lag even matters and is an important KPI to measure. I’ll also talk about the kind of research we did to find the right tool, what the options in the market were at the time, and eventually why we chose Linkedin’s Burrow as the right tool for us. And finally, I’ll take a closer look at Burrow, its building blocks, how we build and deploy it, how we monitor better with it, and eventually the most important improvement - how we transformed its output from numbers to time-based metrics.

Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...

HostedbyConfluent

One of the great things about running applications in the cloud is that you only pay for the resources that you use. But that also makes it more important than ever for our applications to be resource-efficient. This becomes even more critical when we use serverless functions. Micronaut is an application framework that provides dependency injection, developer productivity features, and excellent support for Apache Kafka. By performing dependency injection, AOP, and other productivity-enhancing magic at compile time, Micronaut allows us to build smaller, more efficient microservices and serverless functions. In this session, we'll explore the ways that Apache Kafka and Micronaut work together to enable us to build fast, efficient, event-driven applications. Then we'll see it in action, using the AWS Lambda Sink Connector for Confluent Cloud.

Kafka Excellence at Scale – Cloud, Kubernetes, Infrastructure as Code (Vik Wa...

HostedbyConfluent

Cloud is changing the world; Kubernetes is changing the world; real-time event streaming is changing the world. In this talk we explore some of best practices to synergistically combine the power of these paradigm shifts to achieve a much greater return on your Kafka investments. From declarative deployments, zero-downtime upgrades, elastic scaling to self-healing and automated governance, learn how you can bring the next level of speed, agility, resilience, and security to your Kafka implementations.

Gwen Shapira, Confluent | Kafka Summit 2020 Keynote | Kafka’s New Architecture

confluent

Distributed Data Storage & Streaming for Real-time Decisioning Using Kafka, S...

HostedbyConfluent

Real-time connectivity of databases and systems is critical in enterprises adopting digital transformation to support super-fast decisioning to drive applications like fraud detection, digital payments, recommendation engines. This talk will focus on the many functions that database streaming serves with Kafka, Spark and Aerospike. We will explore how to eliminate the wall between transaction processing and analytics by synthesizing streaming data with system of record data, to gain key insights in real-time.

Confluent On Azure: Why you should add Confluent to your Azure toolkit | Alic...

HostedbyConfluent

As a data professional, you are the glue that makes cross-platform integrations possible. With the increase in adoption of hybrid cloud architectures, Kafka is an increasingly relevant tool for building data pipelines between platforms and accelerating delivery on cloud projects. Early exposure to Kafka on Azure capabilities gives you an edge to build better mousetraps at the design phase. Customers already running Kafka on premises and are looking to extend Kafka systems to Azure can get started quickly with Confluent Cloud. Additionally, DevOps for self-managed options can be easily scalable with Ansible for Virtual Machines or containers via Azure Kubernetes Services or Azure Container Instances. This session is presented from the Microsoft Solution Architect perspective by Israel Ekpo, Microsoft Cloud Solution Architect and Alicia Moniz, Microsoft MVP. They will cover use cases and scenarios, along with key Azure integration points and architecture patterns.

Azure Cosmos DB Kafka Connectors | Abinav Rameesh, Microsoft

HostedbyConfluent

The document discusses Kafka connectors for Cosmos DB that allow for seamless integration between the two services without requiring complex application code. It provides an overview of Kafka Connect and connectors, use cases for integrating Cosmos DB and Kafka, and the architecture of source and sink connectors that can read from and write to Cosmos DB and Kafka. It also previews a demo of the connectors and suggests ways to take integration further.

Using Kafka as a Database For Real-Time Transaction Processing | Chad Preisle...

HostedbyConfluent

You have learned about Kafka event sourcing with streams and using Kafka as a database, but you may be having a tough time wrapping your head around what that means and what challenges you will face. Kafka’s exactly once semantics, data retention rules, and stream DSL make it a great database for real-time transaction processing. This talk will focus on how to use Kafka events as a database. We will talk about using KTables vs GlobalKTables, and how to apply them to patterns we use with traditional databases. We will go over a real-world example of joining events against existing data and some issues to be aware of. We will finish covering some important things to remember about state stores, partitions, and streams to help you avoid problems when your data sets become large.

Robinhood uses Kafka in every line of its business, from stock and crypto trading to clearing and data analytics. One interesting aspect of our architecture is that many of our microservices leveraging Kafka are written in Python. When you combine Python's relatively slow performance coupled, its reliance on process-based parallelism and Robinhood’s scale, the result is a massive fleet of application processes producing to and consuming from our Kafka clusters. This fleet generates an atypical workload on Kafka that warrants a deeper investment in scalability and reliability. This talk discusses our investments in Kafka infrastructure for a large-scale Python-based environment: kafkahood: our librdkafka-based client library wrapper that codifies best practices, sane defaults and deep client-side observability. kafkaproxy: a Rust-based sidecar proxy that reduces connection fan-in from Python gunicorn worker pools to our Kafka clusters. We'll also present challenges we encountered along the way and share our learnings with the audience.

The New Way of Configuring Grace Periods for Windowed Operations in Kafka Str...

HostedbyConfluent

This document discusses KIP-633, which proposes changing the default grace period for windowed operations in Kafka Streams from 24 hours to 0. The motivation is that the 24-hour default was causing confusion for users and suppressing results. The KIP was worked on by a team of contributors and introduces new APIs in Kafka Streams 3.0 that give developers better control over grace periods. It provides resources for migrating code to the new APIs and understanding the compatibility changes.

0-330km/h: Porsche's Data Streaming Journey | Sridhar Mamella, Porsche

HostedbyConfluent

The auto industry is midst a data revolution that is transforming how companies do business. Once a scarce resource, data has now become abundant and cheap. What are the new technologies that change the way we produce, collect, process, store, and analyze data. What new streams of data are being created with Industry 4.0 and the Internet of Things on the horizon, is there significant value to taking a strategic approach to Fast Data. How is Porsche building the next level Data Streaming Platform with open source technologies and how we are using CI/CD pipelines amongst others in order to serve our use cases.

Mind the App: How to Monitor Your Kafka Streams Applications | Bruno Cadonna,...

HostedbyConfluent

You cannot operate what you cannot measure. In this talk, I am going to present the built-in metrics framework of Kafka Streams that supports monitoring Kafka Streams applications. You will learn how to setup monitoring of metrics for your Kafka Streams applications and you will hear about the following recent improvements to the metrics framework that aim to extend and simplify monitoring. KIP-444 aims to simplify and extend the built-in metrics framework. The RocksDB metrics introduced in KIP-471 and KIP-607 allow you to look directly into the built-in persistent state stores of your Kafka Streams applications. Finally, KIP-613 specifies metrics that measure end-to-end latencies in your applications. This talk will help you collect intel about the behavior of your Kafka Streams applications, and will allow you to reason about the deployment. In the end, you will be able to better understand your applications and run them in a more robust manner.

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...

HostedbyConfluent

This document summarizes Activision Data's transition from a batch data pipeline to a real-time streaming data pipeline using Apache Kafka and Kafka Streams. Some key points: - The new pipeline ingests, processes, and stores game telemetry data from over 200k messages per second and over 5PB of data across 9 years of games. - Kafka Streams is used to transform the raw streaming data through multiple microservices with low 10-second end-to-end latency, compared to 6-24 hours previously. - Kafka Connect integrates the streaming data with data stores like AWS S3, Cassandra, and Elasticsearch. - The new pipeline provides real-time and historical access to structured

How Confluent Completes the Event Streaming Platform (Addison Huddy & Dan Ros...

HostedbyConfluent

Confluent Platform 6.0 and Project Metamorphosis complete the event streaming platform by providing elastic scalability, infinite storage, global access, and transforming Kafka. Key features include self-balancing clusters and dynamic scaling on Confluent Cloud, tiered storage and infinite retention on the platform, and cluster linking to simplify hybrid and multi-cloud deployments. These new capabilities help remove limitations on scale, storage, and deployment that traditionally challenged Kafka applications.

Keep your Metadata Repository Current with Event-Driven Updates using CDC and...

confluent

This document discusses challenges with keeping a metadata repository current using event-driven updates from data sources. It describes how using Apache Kafka and the Debezium connector to capture changes from database "outbox" tables that mirror system catalog metadata tables allows pushing metadata deltas to the repository in real-time. This overcomes limitations of log-based and query-based CDC approaches when applied directly to database system tables.

The Road Most Traveled: A Kafka Story | Heikki Nousiainen, Aiven

HostedbyConfluent

When moving to a cloud native architecture Moogsoft knew they needed more scale than Rabbit could provide. Moogsoft moved into Kafka which is known for quick writing and driving heavy event driven workloads on top of niceties such as replayability. Choosing the tool was easy, finding a vendor that ticked all their boxes was not. They needed to ensure scalability, upgradability, builds via existing IAC pipelines, and observability via existing tools. When Moogsoft found Aiven, they were impressed with their offering and ability to scale on demand. During this presentation we will explore how Moogsoft used Aiven for Kafka to manage and scale their data in the cloud.

Building Event Streaming Microservices with Spring Boot and Apache Kafka | Ja...

HostedbyConfluent

This document summarizes Jan Svoboda's presentation on building event streaming microservices with Spring Boot and Apache Kafka. Jan discusses his motivation for learning Kafka and microservices development. He outlines a typical journey from strangling monoliths to building event-driven architectures using Kafka. Jan explains how Spring and Kafka integrate well together. The remainder of the presentation demonstrates building a sample application using these techniques and dives into common architecture patterns like event sourcing, CQRS, scaling stateful services, and reverse proxy routing.

Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...

HostedbyConfluent

Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...

HostedbyConfluent

The Apache Kafka ecosystem is very rich with components and pieces that make for designing and implementing secure, efficient, fault-tolerant and scalable event stream processing (ESP) systems. Using real-world examples, this talk covers why Apache Kafka is an excellent choice for cloud-native and hybrid architectures, how to go about designing, implementing and maintaining ESP systems, best practices and patterns for migrating to the cloud or hybrid configurations, when to go with PaaS or IaaS, what options are available for running Kafka in cloud or hybrid environments and what you need to build and maintain successful ESP systems that are secure, performant, reliable, highly-available and scalable.

Building Retry Architectures in Kafka with Compacted Topics | Matthew Zhou, V...

HostedbyConfluent

In this talk, we'll discuss how VillageMD is able to use Kafka topic compaction for rapidly scaling our reprocessing pipelines to encompass hundreds of feeds. Within healthcare data ecosystems, privacy and data minimalism are key design priorities. Being able to handle data deletion in a reliable, timely manner within event-driven architectures is becoming more and more necessary with key governance frameworks like the GDPR and HIPAA. We'll be giving an overview of the building and governance of dead-letter queues for streaming data processing. We'll discuss: 1. How to architect a data sink for failed records. 2. How topic compaction can reduce duplicate data and enable idempotency. 3. Building a tombstoning system for removing successfully reprocessed records from the queues. 4. Considerations for monitoring a reprocessing system in production -- what metrics, dataops, and SLAs are useful?

Kafka error handling patterns and best practices | Hemant Desale and Aruna Ka...

HostedbyConfluent

Transaction Banking from Goldman Sachs is a high volume, latency sensitive digital banking platform offering. We have chosen an event driven architecture to build highly decoupled and independent microservices in a cloud native manner and are designed to meet the objectives of Security, Availability Latency and Scalability. Kafka was a natural choice – to decouple producers and consumers and to scale easily for high volume processing. However, there are certain aspects that require careful consideration – handling errors and partial failures, managing downtime of consumers, secure communication between brokers and producers / consumers. In this session, we will present the patterns and best practices that helped us build robust event driven applications. We will also present our solution approach that has been reused across multiple application domains. We hope that by sharing our experience, we can establish a reference implementation that application developers can benefit from.

5 lessons learned for successful migration to Confluent cloud | Natan Silinit...

HostedbyConfluent

Confluent Cloud makes Devops engineers lives a lot more easier. Yet moving 1500 microservices, 10K topics and 100K partitions to a multi-cluster Confluent cloud can be a challenge. In this talk you will hear about 5 lessons that Wix has learned in order to successfully meet this challenge. These lessons include: 1. Automation, Automation, Automation - all the process has to be completely automated at such scale 2. Prefer a gradual approach - E.g. migrate topics in small chunks and not all at once. Reduces risks if things go bad 3. Cleanup first - avoid migrating unused topics or topics with too many unnecessary partitions

Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...

HostedbyConfluent

At Wells-Fargo, we move 150 TB of logs data from our syslogs to Splunk forwarders that get indexed and organized for analytic queries. As we modernize and migrate our applications to our hybrid cloud the performance expectations for this infrastructure will proportionately increase. Those improvements include the resilience of the end to end infrastructure. First, we decoupled the applications from their logging interface through a loglibrary which split the streams of logs from their sources to KAFKA which routed them to two separate destinations Splunk and ELK respectively. We also used prometheus and grafana for monitoring the metrics. We also deployed KAFKA, Splunk, ELK, Prometheus and Grafana on the Kubernetes clusters. Confluent had released a version of KAFKA without Zookeeper and replaced its functionality with Quorum Controller. The Quorum-Controller version exhibited better disposability one of the 12factors that's important for Cloud-Nativeness. We packaged this version into a Kubernetes operator called Keda and deployed this for auto-scaling. We tested this to simulate the amount of logdata that we typically generate in production. Based on the above we have also implemented distributed tracing and help make it just as resilient. We will share our lessons learnt, the patterns and practices to modernize both our underlying runtime platforms and our applications with highly performing and resilient event-driven architectures.

Navigating the obdervability storm with Kafka | Jose Manuel Cristobal, Adidas

HostedbyConfluent

When all your stores are closed, e-commerce becomes your bigger store… and the most challenging. That means a myriad of systems orchestrated to make it happen, all of them scaling out accordingly and implementing Observability and SRE practices to support this growth, preserving stability and reliability. How can we detect problems, root causes and react? How can we predict those problems? HOLMES is the adidas solution to accelerate problem detection, giving a holistic view of technical systems through metrics and logs democratisation. In this talk, we'll show how Kafka technology stack allows adidas to support the ingestion and offload of all logs and metrics of the company. A platform which adoption has skyrocketed during 2020, supporting 100 Billion messages per day. The main takeaway will be the explanation of a cutting-edge solution based on kafka technology stack (kafka, Kafka Streams and Kafka Connect) for demanding throughput ecosystem.

How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...

HostedbyConfluent

As Kafka deployments grow within your organization, so do the challenges around lifecycle management. For instance, do you really know what streams exist, who is producing and consuming them? What is the effect of upstream changes? How is this information kept up to date, so it is relevant and consistent to others looking to reuse these streams? Ever wish you had a way to view and visualize graphically the relationships between schemas, topics and applications? In this talk we will show you how to do that and get more value from your Kafka Streaming infrastructure using an event portal. It’s like an API portal but specialized for event streams and publish/subscribe patterns. Join us to see how you can automatically discover event streams from your Kafka clusters, import them to a catalog and then leverage code gen capabilities to ease development of new applications.

Creating a Kafka Topic. Super easy? | Andrew Stevenson and Marios Andreopoulo...

HostedbyConfluent

Kafka at the Edge: an IoT scenario with OpenShift Streams for Apache Kafka | ...

Red Hat Developers

This document discusses Apache Kafka and Red Hat OpenShift Streams for Apache Kafka. It begins with an overview of what Apache Kafka is and its common use cases. It then demonstrates how Red Hat OpenShift Streams provides a managed Apache Kafka cluster as a service, including a dedicated cluster, configuration management, metrics, monitoring and other features to provide a streamlined developer experience. It concludes with information on trying OpenShift Streams for Apache Kafka and additional resources.

An Introduction to Confluent Cloud: Apache Kafka as a Service

confluent

Business breakout during Confluent’s streaming event in Munich, presented by Hans Jespersen, VP WW Systems Engineering at Confluent. This three-day hands-on course focused on how to build, manage, and monitor clusters using industry best-practices developed by the world’s foremost Apache Kafka™ experts. The sessions focused on how Kafka and the Confluent Platform work, how their main subsystems interact, and how to set up, manage, monitor, and tune your cluster.

Otimizações de Projetos de Big Data, Dw e AI no Microsoft Azure

Luan Moreno Medeiros Maciel

Databricks Platform.pptx

Alex Ivy

The document provides an overview of the Databricks platform, which offers a unified environment for data engineering, analytics, and AI. It describes how Databricks addresses the complexity of managing data across siloed systems by providing a single "data lakehouse" platform where all data and analytics workloads can be run. Key features highlighted include Delta Lake for ACID transactions on data lakes, auto loader for streaming data ingestion, notebooks for interactive coding, and governance tools to securely share and catalog data and models.

What's hot

Taming a massive fleet of Python-based Kafka apps at Robinhood | Chandra Kuch...

HostedbyConfluent

The New Way of Configuring Grace Periods for Windowed Operations in Kafka Str...

HostedbyConfluent

0-330km/h: Porsche's Data Streaming Journey | Sridhar Mamella, Porsche

HostedbyConfluent

Mind the App: How to Monitor Your Kafka Streams Applications | Bruno Cadonna,...

HostedbyConfluent

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...

HostedbyConfluent

How Confluent Completes the Event Streaming Platform (Addison Huddy & Dan Ros...

HostedbyConfluent

Keep your Metadata Repository Current with Event-Driven Updates using CDC and...

confluent

The Road Most Traveled: A Kafka Story | Heikki Nousiainen, Aiven

HostedbyConfluent

Building Event Streaming Microservices with Spring Boot and Apache Kafka | Ja...

HostedbyConfluent

Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...

HostedbyConfluent

Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...

HostedbyConfluent

Building Retry Architectures in Kafka with Compacted Topics | Matthew Zhou, V...

HostedbyConfluent

Kafka error handling patterns and best practices | Hemant Desale and Aruna Ka...

HostedbyConfluent

5 lessons learned for successful migration to Confluent cloud | Natan Silinit...

HostedbyConfluent

Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...

HostedbyConfluent

Navigating the obdervability storm with Kafka | Jose Manuel Cristobal, Adidas

HostedbyConfluent

How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...

HostedbyConfluent

Creating a Kafka Topic. Super easy? | Andrew Stevenson and Marios Andreopoulo...

HostedbyConfluent

Kafka at the Edge: an IoT scenario with OpenShift Streams for Apache Kafka | ...

Red Hat Developers

An Introduction to Confluent Cloud: Apache Kafka as a Service

confluent

What's hot (20)