Use Apache Gradle to Build and Automate KSQL and Kafka Streams (Stewart Bryson, Red Pill Analytics) Kafka Summit NYC 2019

•

1 like•860 views

KSQL is an easy-to-use and easy-to-understand streaming SQL engine for Apache Kafka built on top of Kafka Streams. The ability to write streaming applications using only SQL makes Apache Kafka available to a whole range of new developers and potential use cases, either as a stand-alone solution, or as a single component to a broader Kafka Streams implementation. Inspired by a customer project now in production, experience the lifecycle of a streaming application developed using KSQL and Kafka Streams. With Apache Gradle as our build framework, we’ll explore the open-source Gradle plugin we built during this project to improve developer efficiency and automate the deployment of KSQL pipelines, user-defined functions, and Kafka Streams microservices. We’ll demonstrate the deployment process live, and discuss design decisions around incorporating SQL-based processes into an overall streaming application. Key Takeaways 1. KSQL is a natural choice for expressing data-driven applications, but it may not naturally fit into established DevOps processes and automations. 2. We built an open-source Gradle plugin to handle all aspects of deploying a Kafka-based streaming application: KSQL pipelines, KSQL user-defined functions, and Kafka Streams microservices. 3. KSQL pipelines can be deployed using either a server start script, or the KSQL REST API, and our Gradle plugin fully supports both options.

Technology

Use Apache Gradle to Build
and Automate KSQL and
Apache Kafka Streams

twitter: @stewartbryson
medium: @stewartbryson
linkedin: stewartbryson
Owner & CEO
Red Pill Analytics

@redpilla
What We Do
Data Warehouse Analytics
ANALYTICS
Data-engineering & ETL

@redpilla
Project Overview
KSQL
KSQL User Defined Functions
(demo-only)
Kafka Streams (demo-only)
Agenda

@redpilla
Data Integration: Streaming In

@redpilla
Data Integration: Streaming Out

@redpilla
Packaging of Payloads for OWMC

@redpilla
KSQL Registration Queries
CREATE STREAM clickstream
( _time bigint,
time varchar,
ip varchar,
request varchar,
status int,
userid int,
bytes bigint,
agent varchar )
with
( kafka_topic = 'clickstream',
value_format = 'json' );

@redpilla
KSQL Persistent Queries
CREATE TABLE events_per_min AS
SELECT userid,
count(*) AS events
FROM clickstream window
TUMBLING (size 60 second)
GROUP BY userid;

@redpilla
A pipeline is a group of SQL
statements that work together to
define an end-to-end process.

@redpilla
KSQL Pipelines
clickstream
clickstream_codes
Streams and TablesSQL Scripts
enriched_error_codes
enriched_error_codes_count
customer_clickstream
user_clickstream
web_users
click_user_sessions

@redpilla
Multiple statements per pipeline
Statements are in script files
Script files are in Git
Developers need to iterate

@redpilla
What is Apache Gradle?
Gradle is an open-source build automation tool for the Age
of Continuous Delivery (CD). Building software is no longer
just about compiling, linking and packaging.

@redpilla
gradle-confluent for KSQL
pipelines
shadow for KSQL user defined
functions
application for Kafka Streams
Gradle Plugins

@redpilla
Plugin
github.com/RedPillAnalytics/gradle-confluent

@redpilla
Unfortunately…
I only have 40 minutes

@redpilla
Unfortunately…
I only have 33 minutes

@redpilla
Demo
...using a Jupyter Notebook

Use Apache Gradle to Build and Automate KSQL and Kafka Streams (Stewart Bryson, Red Pill Analytics) Kafka Summit NYC 2019

Tinder’s Quickfire Pipeline powers all things data at Tinder. It was originally built using AWS Kinesis Firehoses and has since been extended to use both Kafka and other event buses. It is the core of Tinder’s data infrastructure. This rich data flow of both client and backend data has been extended to service a variety of needs at Tinder, including Experimentation, ML, CRM, and Observability, allowing backend developers easier access to shared client side data. We perform this using many systems, including Kafka, Spark, Flink, Kubernetes, and Prometheus. Many of Tinder’s systems were natively designed in an RPC first architecture. Things we’ll discuss decoupling your system at scale via event-driven architectures include: – Powering ML, backend, observability, and analytical applications at scale, including an end to end walk through of our processes that allow non-programmers to write and deploy event-driven data flows. – Show end to end the usage of dynamic event processing that creates other stream processes, via a dynamic control plane topology pattern and broadcasted state pattern – How to manage the unavailability of cached data that would normally come from repeated API calls for data that’s being backfilled into Kafka, all online! (and why this is not necessarily a “good” idea) – Integrating common OSS frameworks and libraries like Kafka Streams, Flink, Spark and friends to encourage the best design patterns for developers coming from traditional service oriented architectures, including pitfalls and lessons learned along the way. – Why and how to avoid overloading microservices with excessive RPC calls from event-driven streaming systems – Best practices in common data flow patterns, such as shared state via RocksDB + Kafka Streams as well as the complementary tools in the Apache Ecosystem. – The simplicity and power of streaming SQL with microservices

Data Transformations on Ops Metrics using Kafka Streams (Srividhya Ramachandr...

confluent

This document discusses using Kafka Streams to transform operational metrics data from Priceline applications before loading it into Splunk. It describes how the legacy monitoring system worked, and the motives for moving to Kafka and Kafka Streams. It then explains how data is collected from applications into Kafka topics, and how various transformations like formatting, key application, and aggregation are performed using Kafka Streams before loading to Splunk. It also discusses testing, monitoring, and debugging Kafka Streams applications.

Bank of China Tech Talk 2: Introduction to Streaming Data and Stream Processi...

confluent

Choose Right Stream Storage: Amazon Kinesis Data Streams vs MSK

Sungmin Kim

Closing Keynote: The Physics of Streaming | Tim Berglund, Confluent | Kafka S...

HostedbyConfluent

Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...

HostedbyConfluent

Apache Druid is a high-performance distributed analytics store for modern analytics applications. It supports ingesting millions of events per second and sub-second query processing. Druid supports various types of data sources for ingestion, including Apache Kafka. You can immediately query on stream events once they get ingested into Druid. Since Kafka provides scalable and robust data delivery while Druid supports advanced complex analysis on streams, Kafka and Druid are widely used together for BI and operational analytics use cases, which require interactivity, scalability, real-time, and performance. This talk is based on our real-world experiences building out streaming analytics stacks powering production use cases across many industries.

KSQL: Open Source Streaming for Apache Kafka

confluent

Maximize the Business Value of Machine Learning and Data Science with Kafka (...

confluent

Today, many companies that have lots of data are still struggling to derive value from machine learning (ML) and data science investments. Why? Accessing the data may be difficult. Or maybe it’s poorly labeled. Or vital context is missing. Or there are questions around data integrity. Or standing up an ML service can be cumbersome and complex. At Nuuly, we offer an innovative clothing rental subscription model and are continually evolving our ML solutions to gain insight into the behaviors of our unique customer base as well as provide personalized services. In this session, I’ll share how we used event streaming with Apache Kafka® and Confluent Cloud to address many of the challenges that may be keeping your organization from maximizing the business value of machine learning and data science. First, you’ll see how we ensure that every customer interaction and its business context is collected. Next, I’ll explain how we can replay entire interaction histories using Kafka as a transport layer as well as a persistence layer and a business application processing layer. Order management, inventory management, logistics, subscription management – all of it integrates with Kafka as the common backbone. These data streams enable Nuuly to rapidly prototype and deploy dynamic ML models to support various domains, including pricing, recommendations, product similarity, and warehouse optimization. Join us and learn how Kafka can help improve machine learning and data science initiatives that may not be delivered to their full potential.

The graphical curve of ‘getting things done with Kafka’ vs. ‘time spent with Kafka’ rises pretty quickly before starting to plateau. Kafka, as we know, is great to get started out of the box. Eventually though, anyone who has used Kafka for a reasonable amount of time would come across problems, both trivial and hardcore. More often than not, these problems feel tedious and repetitive which could be addressed better by the Kafka community. Issues like skewed partitions and brokers, partition reassignment for a large number of topics, working with messed up consumer offsets, distributing workload evenly to partitions, monitor and view the Kafka cluster better, etc. sound familiar among the community. Come, learn from our experiences trying to overcome and automate some of the most common problems surrounding the scaling Kafka clusters at Tinder. Let’s create an environment of sharing the best practices for usage of tools around Kafka.

Keynote: Jay Kreps, Confluent | Kafka ♥ Cloud | Kafka Summit 2020

confluent

Modeling the Smart and Connected City of the Future with Kafka and Spark

SingleStore

- Modeling the Smart and Connected City of the Future with Kafka and Spark discusses using Kafka, Spark, and MemSQL to build a real-time data pipeline for a hypothetical "MemCity" that captures data from 1.4 million households. - The document outlines the components of the "Real-Time Trinity" - Kafka for a high-throughput message queue, Spark for data transformation, and MemSQL for real-time data serving and analytics. - It also introduces MemSQL Streamliner, which is designed to simplify the creation of real-time data pipelines through a graphical interface and one-click deployment of integrated Apache Spark clusters.

Elastically Scaling Kafka Using Confluent

confluent

This document discusses how Confluent Platform provides elastic scaling for Apache Kafka. It offers fully managed cloud services through Confluent Cloud or self-managed software. Confluent Cloud allows users to easily scale Kafka workloads from 0 MBps to GBps without complex provisioning. It also offers pay-for-use pricing where customers only pay for the data streamed, with the ability to scale to zero. For self-managed deployments, Confluent Platform enables dynamic scaling of Kafka clusters on Kubernetes through features like tiered storage and self-balancing clusters that can rebalance partitions in seconds versus hours for other Kafka services.

Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...

HostedbyConfluent

The Apache Kafka ecosystem is very rich with components and pieces that make for designing and implementing secure, efficient, fault-tolerant and scalable event stream processing (ESP) systems. Using real-world examples, this talk covers why Apache Kafka is an excellent choice for cloud-native and hybrid architectures, how to go about designing, implementing and maintaining ESP systems, best practices and patterns for migrating to the cloud or hybrid configurations, when to go with PaaS or IaaS, what options are available for running Kafka in cloud or hybrid environments and what you need to build and maintain successful ESP systems that are secure, performant, reliable, highly-available and scalable.

Building a real-time data processing pipeline using Apache Kafka, Kafka Conne...

Paul Brebner

With the rapid onset of the global Covid-19 Pandemic from the start of this year the USA Centers for Disease Control and Prevention (CDC) had to quickly implement a new Covid-19 specific pipeline to collect testing data from all of the USA’s states and territories, and carry out other critical steps including integration, cleaning, checking, enrichment, analysis, and enforcing data governance and privacy etc. The pipeline then produces multiple consumable results for federal and public agencies. They did this in under 30 days, using Apache Kafka. In this presentation we'll build a similar (but simpler) pipeline for ingesting, integrating, indexing, searching/analysing and visualising some publicly available tidal data. We'll briefly introduce each technology and component, and walk through the steps of using Apache Kafka, Kafka Connect, Elasticsearch and Kibana to build the pipeline and visualise the results.

DataOps Automation for a Kafka Streaming Platform (Andrew Stevenson + Spiros ...

HostedbyConfluent

DataOps challenges us to build data experiences in a repeatable way. For those with Kafka, this means finding a means of deploying flows in an automated and consistent fashion. The challenge is to make the deployment of Kafka flows consistent across different technologies and systems: the topics, the schemas, the monitoring rules, the credentials, the connectors, the stream processing apps. And ideally not coupled to a particular infrastructure stack. In this talk we will discuss the different approaches and benefits/disadvantages to automating the deployment of Kafka flows including Git operators and Kubernetes operators. We will walk through and demo deploying a flow on AWS EKS with MSK and Kafka Connect using GitOps practices: including a stream processing application, S3 connector with credentials held in AWS Secrets Manager.

Kafka in Context, Cloud, & Community (Simon Elliston Ball, Cloudera) Kafka Su...

HostedbyConfluent

The document discusses Kafka in the context of cloud platforms and open source communities. It describes several Apache projects that can be used with Kafka, such as Apache NiFi for data collection, Apache Flink for stream processing, and Apache Ranger for security. It also outlines features of Cloudera's platform for managing Kafka deployments, including unified governance tools, monitoring, and services to simplify operations. Finally, it discusses how Kafka can be deployed across cloud, on-premise, and hybrid environments with auto-scaling and other management capabilities.

Kafka as an Event Store (Guido Schmutz, Trivadis) Kafka Summit NYC 2019

confluent

Event Sourcing and CQRS are two popular patterns for implementing a Microservices architectures. With Event Sourcing we do not store the state of an object, but instead store all the events impacting its state. Then to retrieve an object state, we have to read the different events related to a certain object and apply them one by one. CQRS (Command Query Responsibility Segregation) on the other hand is a way to dissociate writes (Command) and reads (Query). Event Sourcing and CQRS are frequently grouped and used together to form something bigger. While it is possible to implement CQRS without Event Sourcing, the opposite is not necessarily correct. In order to implement Event Sourcing, an efficient Event Store is needed. But is that also true when combining Event Sourcing and CQRS? And what is an event store in the first place and what features should it implement? This presentation will first discuss what functionalities an event store should offer and then present how Apache Kafka can be used to implement an event store. But is Kafka good enough or do specific event store solutions such as AxonDB or Event Store provide a better solution?

Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...

HostedbyConfluent

Apache Kafka users who want to leverage Google Cloud Platform's (GCPs) data analytics platform and open source hosting capabilities can bridge their existing Kafka infrastructure on-premise or in other clouds to GCP using Confluent's replicator tool and managed Kafka service on GCP. Using actual customer examples and a reference architecture, we'll showcase how existing Kafka users can stream data to GCP and use it in popular tools like Apache Beam on Dataflow, BigQuery, Google Cloud Storage (GCS), Spark on Dataproc, and Tensorflow for data warehousing, data processing, data storage, and advanced analytics using AI and ML.

O'Reilly Media Webcast: Building Real-Time Data Pipelines

SingleStore

As our customers tap into new sources of data or modify to existing data pipelines, we are often asked questions like: What technologies should we consider? Where can we reduce data latency? How can we simplify our data architecture? To eliminate the guesswork, we teamed up with Ben Lorica, Chief Data Scientist at O’Reilly Media to host a webcast centered around building real-time data pipelines.

First Steps with Apache Kafka on Google Cloud Platform

confluent

Speakers: Jay Smith, Cloud Customer Engineer, Google Cloud + Gwen Shapira, Product Manager, Confluent Curious about Apache Kafka®? Find out why you would want to use the de facto standard for real-time streaming, the easiest way to get started and how to leverage the extensive Apache Kafka ecosystem. In this chat, we'll talk about three common use cases, review stream processing patterns and discuss integration with important GCP services such as BigQuery. We'll also demo how to implement real-time clickstream analytics on Confluent Cloud, fully managed Apache Kafka as a service.

Event Streaming CTO Roundtable for Cloud-native Kafka Architectures

Kai Wähner

Technical thought leadership presentation to discuss how leading organizations move to real-time architecture to support business growth and enhance customer experience. This is a forum to discuss use cases with your peers to understand how other digital-native companies are utilizing data in motion to drive competitive advantage. Agenda: - Data in Motion with Event Streaming and Apache Kafka - Streaming ETL Pipelines - IT Modernisation and Hybrid Multi-Cloud - Customer Experience and Customer 360 - IoT and Big Data Processing - Machine Learning and Analytics

INTRODUCING: CREATE PIPELINE

SingleStore

This document introduces MemSQL Pipelines, which allow for exactly-once data ingestion semantics when streaming data from Kafka into MemSQL. MemSQL Pipelines provide a native way to extract, transform, and load external data into MemSQL tables. They offer a scalable and highly performant ETL process across a distributed cluster. The document explains streaming semantics like at least once and exactly once delivery, and how MemSQL Pipelines coordinate with Kafka to enable exactly-once ingestion through offset tracking. It presents the architecture of MemSQL Pipelines and demonstrates their use through a live demo.

Microservice Plumbing - Glynn Bird - Codemotion Rome 2017

Codemotion

Bridge to Cloud: Using Apache Kafka to Migrate to GCP

confluent

Watch this talk here: https://www.confluent.io/online-talks/bridge-to-cloud-apache-kafka-migrate-gcp Most companies start their cloud journey with a new use case, or a new application. Sometimes these applications can run independently in the cloud, but often times they need data from the on premises datacenter. Existing applications will slowly migrate, but will need a strategy and the technology to enable a multi-year migration. In this session, we will share how companies around the world are using Confluent Cloud, a fully managed Apache Kafka® service, to migrate to Google Cloud Platform. By implementing a central-pipeline architecture using Apache Kafka to sync on-prem and cloud deployments, companies can accelerate migration times and reduce costs. Register now to learn: -How to take the first step in migrating to GCP -How to reliably sync your on premises applications using a persistent bridge to cloud -How Confluent Cloud can make this daunting task simple, reliable and performant

From Kafka to BigQuery - Strata Singapore

Ofir Sharony

Kai Waehner - KSQL – The Open Source SQL Streaming Engine for Apache Kafka - ...

Codemotion

Kai Waehner - KSQL – The Open Source SQL Streaming Engine for Apache Kafka - ...

Codemotion

KSQL is a streaming SQL engine that allows users to analyze streaming data in Apache Kafka using SQL queries. It provides a simple interactive interface using SQL to explore, transform, and perform real-time analytics on data in Kafka. KSQL can be used for tasks like data exploration, transformation, ETL, enrichment, monitoring, anomaly detection, and more. It builds on Kafka Streams to provide scalability, fault tolerance, and exactly-once processing semantics out of the box.

What's hot

Machine Learning on Streaming Data using Kafka, Beam, and TensorFlow (Mikhail...

confluent

A guide through the Azure Messaging services - Update Conference

Eldert Grootenboer

PowerStream Demo

SingleStore

Troubleshooting as Your Kafka Clusters Grow (Krunal Vora, Tinder) Kafka Summi...

confluent

Keynote: Jay Kreps, Confluent | Kafka ♥ Cloud | Kafka Summit 2020

confluent

Modeling the Smart and Connected City of the Future with Kafka and Spark

SingleStore

Elastically Scaling Kafka Using Confluent

confluent

Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...

HostedbyConfluent

Building a real-time data processing pipeline using Apache Kafka, Kafka Conne...

Paul Brebner

DataOps Automation for a Kafka Streaming Platform (Andrew Stevenson + Spiros ...

HostedbyConfluent

Kafka in Context, Cloud, & Community (Simon Elliston Ball, Cloudera) Kafka Su...

HostedbyConfluent

Kafka as an Event Store (Guido Schmutz, Trivadis) Kafka Summit NYC 2019

confluent

Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...

HostedbyConfluent

O'Reilly Media Webcast: Building Real-Time Data Pipelines

SingleStore

First Steps with Apache Kafka on Google Cloud Platform

confluent

Event Streaming CTO Roundtable for Cloud-native Kafka Architectures

Kai Wähner

INTRODUCING: CREATE PIPELINE

SingleStore

Microservice Plumbing - Glynn Bird - Codemotion Rome 2017

Codemotion

Bridge to Cloud: Using Apache Kafka to Migrate to GCP

confluent

From Kafka to BigQuery - Strata Singapore

Ofir Sharony

What's hot (20)

Machine Learning on Streaming Data using Kafka, Beam, and TensorFlow (Mikhail...

A guide through the Azure Messaging services - Update Conference

PowerStream Demo

Troubleshooting as Your Kafka Clusters Grow (Krunal Vora, Tinder) Kafka Summi...

Keynote: Jay Kreps, Confluent | Kafka ♥ Cloud | Kafka Summit 2020

Modeling the Smart and Connected City of the Future with Kafka and Spark

Elastically Scaling Kafka Using Confluent

Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...

Building a real-time data processing pipeline using Apache Kafka, Kafka Conne...

DataOps Automation for a Kafka Streaming Platform (Andrew Stevenson + Spiros ...

Kafka in Context, Cloud, & Community (Simon Elliston Ball, Cloudera) Kafka Su...

Kafka as an Event Store (Guido Schmutz, Trivadis) Kafka Summit NYC 2019

Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...

O'Reilly Media Webcast: Building Real-Time Data Pipelines

First Steps with Apache Kafka on Google Cloud Platform

Event Streaming CTO Roundtable for Cloud-native Kafka Architectures

INTRODUCING: CREATE PIPELINE

Microservice Plumbing - Glynn Bird - Codemotion Rome 2017

Bridge to Cloud: Using Apache Kafka to Migrate to GCP

From Kafka to BigQuery - Strata Singapore

Similar to Use Apache Gradle to Build and Automate KSQL and Kafka Streams (Stewart Bryson, Red Pill Analytics) Kafka Summit NYC 2019

Kai Waehner - KSQL – The Open Source SQL Streaming Engine for Apache Kafka - ...

Codemotion

Kai Waehner - KSQL – The Open Source SQL Streaming Engine for Apache Kafka - ...

Codemotion

Real-Time Stream Processing with KSQL and Apache Kafka

confluent

Real Time Stream Processing with KSQL and Kafka David Peterson, Confluent APAC APIdays Melbourne 2018 Unordered, unbounded and massive datasets are increasingly common in day-to-day business. Using this to your advantage is incredibly difficult with current system designs. We are stuck in a model where we can only take advantage of this *after* it has happened. Many times, this is too late to be useful in the enterprise. KSQL is a streaming SQL engine for Apache Kafka. KSQL lowers the entry bar to the world of stream processing, providing a simple and completely interactive SQL interface for processing data in Kafka. KSQL (like Kafka) is open-source, distributed, scalable, and reliable. A real time Kafka platform moves your data up the stack, closer to the heart of your business, allowing you to build scalable, mission-critical services by quickly deploying SQL-like queries in a severless pattern. This talk will highlight key use cases for real time data, and stream processing with KSQL: Real time analytics, security and anomaly detection, real time ETL / data integration, Internet of Things, application development, and deploying Machine Learning models with KSQ. Real time data and stream processing means that Kafka is just as important to the disrupted as it is to the disruptors.

Real Time Stream Processing with KSQL and Kafka

David Peterson

Unordered, unbounded and massive datasets are increasingly common in day-to-day business. Using this to your advantage is incredibly difficult with current system designs. We are stuck in a model where we can only take advantage of this *after* it has happened. Many times, this is too late to be useful in the enterprise. KSQL is a streaming SQL engine for Apache Kafka. KSQL lowers the entry bar to the world of stream processing, providing a simple and completely interactive SQL interface for processing data in Kafka. KSQL (like Kafka) is open-source, distributed, scalable, and reliable. A real time Kafka platform moves your data up the stack, closer to the heart of your business, allowing you to build scalable, mission-critical services by quickly deploying SQL-like queries in a severless pattern. This talk will highlight key use cases for real time data, stream processing with KSQL: Real time analytics, security and anomaly detection, real time ETL / data integration, Internet of Things, application development, and deploying Machine Learning models with KSQL. Real time data and stream processing means that Kafka is just as important to the disrupted as it is to the disruptors.

KSQL – The Open Source SQL Streaming Engine for Apache Kafka (Big Data Spain ...

Kai Wähner

KSQL – The Open Source SQL Streaming Engine for Apache Kafka (Talk at Big Data Spain 2018 in Madrid). - KSQL includes access to the rich Apache Kafka ecosystem and is suitable for various use cases, including Streaming ETL, Real Time Stream Monitoring and Anomaly Detection - KSQL allows to realize stream processing without coding and without additional analytics cluster Description: The rapidly expanding world of stream processing can be daunting, with new concepts such as various types of time semantics, windowed aggregates, changelogs, and programming frameworks to master. KSQL is an open-source, Apache 2.0 licensed streaming SQL engine on top of Apache Kafka which aims to simplify all this and make stream processing available to everyone. Even though it is simple to use, KSQL is built for mission-critical and scalable production deployments (using Kafka Streams under the hood). Benefits of using KSQL include: No coding required; no additional analytics cluster needed; streams and tables as first-class constructs; access to the rich Kafka ecosystem. This session introduces the concepts and architecture of KSQL. Use cases such as Streaming ETL, Real Time Stream Monitoring or Anomaly Detection are discussed. A live demo shows how to setup and use KSQL quickly and easily on top of your Kafka ecosystem.

Streaming etl in practice with postgre sql, apache kafka, and ksql mic

Bas van Oudenaarde

This document discusses using Apache Kafka, Kafka Connect, and KSQL to perform streaming ETL from a PostgreSQL database. It describes how to integrate PostgreSQL with Kafka using Kafka Connect and the Debezium PostgreSQL connector. It then shows how to use KSQL to perform streaming transformations on the PostgreSQL data in Kafka, including joining, filtering, and aggregating data streams in real-time. The document promotes using this approach for applications like real-time monitoring, anomaly detection, and analytics to enable streaming data pipelines and transform data once for multiple uses.

Expanding Apache Spark Use Cases in 2.2 and Beyond with Matei Zaharia and dem...

Databricks

2017 continues to be an exciting year for big data and Apache Spark. I will talk about two major initiatives that Databricks has been building: Structured Streaming, the new high-level API for stream processing, and new libraries that we are developing for machine learning. These initiatives can provide order of magnitude performance improvements over current open source systems while making stream processing and machine learning more accessible than ever before.

Introduction to apache kafka, confluent and why they matter

Paolo Castagna

Saving Money by Optimizing Your Cloud Add-On Infrastructure

Atlassian

People love the freedom and control that comes with hosting an add-on in the cloud, but financially speaking, that freedom doesn't come for free. You'll end up paying for your servers whether they serve requests or not, and of course, someone is needed to monitor and upgrade these servers. In this talk, we will cover best practices on how to get a simple and inexpensive cloud add-on going for Hipchat, Bitbucket, Confluence and JIRA without all the overhead of running your own servers. We'll cover serverless technologies like AWS Lambda, static frontends, and how you can use JIRA and Confluence to host your add-on data. Products covered: JIRA Software, JIRA Core, Confluence, HipChat

KSQL – An Open Source Streaming Engine for Apache Kafka

Kai Wähner

The rapidly expanding world of stream processing can be daunting, with new concepts such as various types of time semantics, windowed aggregates, changelogs, and programming frameworks to master. KSQL is an open-source, Apache 2.0 licensed streaming SQL engine on top of Apache Kafka which aims to simplify all this and make stream processing available to everyone. The project is managed and open sourced by Confluent. KSQL makes it easy to read, write, and process streaming data in real-time, at scale, using SQL-like semantics. It offers an easy way to express stream processing logic as an alternative to writing an application in a programming language such as Java, Python or Go. Benefits of using KSQL include: No coding required; no additional analytics cluster needed; streams and tables as first-class constructs; access to the rich Kafka ecosystem. This session introduces the concepts and architecture of KSQL. Use cases such as Streaming ETL, Real Time Stream Monitoring or Anomaly Detection are discussed. A live demo shows how to setup and use KSQL quickly and easily on top of your Kafka ecosystem.

KSQL Deep Dive - The Open Source Streaming Engine for Apache Kafka

Kai Wähner

This document provides an agenda for a deep dive on KSQL, the streaming SQL engine for Apache Kafka. It begins with an overview of the Apache Kafka ecosystem and how Kafka Streams serves as the foundation for KSQL. It then discusses the motivations for using KSQL and covers KSQL concepts like streams, tables, and windowing. The agenda also includes two live demos - an introduction to KSQL and a clickstream analysis example. It will discuss building user-defined functions with KSQL and machine learning. Finally, it covers getting started with KSQL.

Apache Kafka and KSQL in Action: Let's Build a Streaming Data Pipeline!

confluent

This document provides an overview and introduction to Apache Kafka and KSQL for building streaming data pipelines. It discusses how Kafka is an event streaming platform that can be used for messaging, streaming data, and stream processing. It then introduces KSQL, which is a streaming SQL engine for Apache Kafka that allows users to perform stream processing by writing SQL-like queries against Kafka topics. The document uses diagrams and examples to illustrate how to build a streaming data pipeline using Kafka Connect to ingest data, Kafka to store and transport streams, and KSQL to perform stream processing, enrichment, and analytics.

Un'introduzione a Kafka Streams e KSQL... and why they matter!

Paolo Castagna

This document provides an introduction to Kafka Streams and KSQL and why they are important tools for building real-time applications. It discusses how Kafka Streams is a Java API that allows developers to build scalable, fault-tolerant, and stateful applications using stream processing. KSQL is introduced as a streaming SQL engine that builds on top of Kafka Streams and provides a simple way to analyze, transform, and aggregate streaming data using SQL-like queries. Examples are given of different use cases for KSQL, such as data exploration, streaming ETL, anomaly detection, and real-time monitoring.

MLOps with a Feature Store: Filling the Gap in ML Infrastructure

Data Science Milan

A Feature Store enables machine learning (ML) features to be registered, discovered, and used as part of ML pipelines, thus making it easier to transform and validate the training data that is fed into machine learning systems. Feature stores can also enable consistent engineering of features between training and inference, but to do so, they need a common data processing platform. The first Feature Stores, developed at hyperscale AI companies such as Uber, Airbnb, and Facebook, enabled feature engineering using domain specific languages, providing abstractions tailored to the companies’ feature engineering domains. However, a general purpose Feature Store needs a general purpose feature engineering, feature selection, and feature transformation platform. In this talk, we describe how we built a general purpose, open-source Feature Store for ML around dataframes and Apache Spark. We will demonstrate how data engineers can transform and engineers features from backend databases and data lakes, while data scientists can use PySpark to select and transform features into train/test data in a file format of choice (.tfrecords, .npy, .petastorm, etc) on a file system of choice (S3, HDFS). Finally, we will show how the Feature Store enables end-to-end ML pipelines to be factored into feature engineering and data science stages that each can run at different cadences. Bio: Fabio Buso is the head of engineering at Logical Clocks AB, where he leads the Feature Store development. Fabio holds a master's degree in cloud computing and services with a focus on data intensive applications, awarded by a joint program between KTH Stockholm and TU Berlin. Topics: feature store, MLOps.

AI at Scale

Adi Polak

This document discusses AI at scale using Apache Spark on Azure. It provides an overview of Apache Spark, how it can be used for machine learning with tools like MLlib and Databricks, and how cognitive services can be combined with Spark. It also discusses using Azure services like Databricks, HDInsight and AKS for running Spark workloads at scale, and the roles of data engineers and data scientists.

KSQL: Streaming SQL for Kafka

confluent

KSQL and Kafka Streams – When to Use Which, and When to Use Both

confluent

Technical breakout during Confluent’s streaming event in Munich, presented by Michael Noll, Product Manager at Confluent. This three-day hands-on course focused on how to build, manage, and monitor clusters using industry best-practices developed by the world’s foremost Apache Kafka™ experts. The sessions focused on how Kafka and the Confluent Platform work, how their main subsystems interact, and how to set up, manage, monitor, and tune your cluster.

Witsml data processing with kafka and spark streaming

Mark Kerzner

This document summarizes a presentation about using Kafka and Spark Streaming to process real-time well data in WITSML format. It discusses WITSML data standards, using Kafka as a messaging system to ingest WITSML data from rigs and service companies, and Spark Streaming to consume Kafka topics and apply rules to detect anomalies and send alerts. Visualizing the data in real-time using Highcharts javascript is also covered. Lessons learned focus on improving data partitioning and managing producer/consumer services.

APAC ksqlDB Workshop

confluent

Riviera Jug - 20/03/2018 - KSQL

Florent Ramiere

Similar to Use Apache Gradle to Build and Automate KSQL and Kafka Streams (Stewart Bryson, Red Pill Analytics) Kafka Summit NYC 2019 (20)

Kai Waehner - KSQL – The Open Source SQL Streaming Engine for Apache Kafka - ...

Real-Time Stream Processing with KSQL and Apache Kafka

Real Time Stream Processing with KSQL and Kafka

KSQL – The Open Source SQL Streaming Engine for Apache Kafka (Big Data Spain ...

Streaming etl in practice with postgre sql, apache kafka, and ksql mic

Expanding Apache Spark Use Cases in 2.2 and Beyond with Matei Zaharia and dem...

Introduction to apache kafka, confluent and why they matter

Saving Money by Optimizing Your Cloud Add-On Infrastructure

KSQL – An Open Source Streaming Engine for Apache Kafka

KSQL Deep Dive - The Open Source Streaming Engine for Apache Kafka

Apache Kafka and KSQL in Action: Let's Build a Streaming Data Pipeline!

Un'introduzione a Kafka Streams e KSQL... and why they matter!

MLOps with a Feature Store: Filling the Gap in ML Infrastructure

AI at Scale

KSQL: Streaming SQL for Kafka

KSQL and Kafka Streams – When to Use Which, and When to Use Both

Witsml data processing with kafka and spark streaming

APAC ksqlDB Workshop

Riviera Jug - 20/03/2018 - KSQL

More from confluent

Building API data products on top of your real-time data infrastructure

confluent

This talk and live demonstration will examine how Confluent and Gravitee.io integrate to unlock value from streaming data through API products. You will learn how data owners and API providers can document, secure data products on top of Confluent brokers, including schema validation, topic routing and message filtering. You will also see how data and API consumers can discover and subscribe to products in a developer portal, as well as how they can integrate with Confluent topics through protocols like REST, Websockets, Server-sent Events and Webhooks. Whether you want to monetize your real-time data, enable new integrations with partners, or provide self-service access to topics through various protocols, this webinar is for you!

Speed Wins: From Kafka to APIs in Minutes

confluent

Evolving Data Governance for the Real-time Streaming and AI Era

confluent

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...

confluent

Santander Stream Processing with Apache Flink

confluent

Unlocking the Power of IoT: A comprehensive approach to real-time insights

confluent

Workshop híbrido: Stream Processing con Flink

confluent

El Stream processing es un requisito previo de la pila de data streaming, que impulsa aplicaciones y pipelines en tiempo real. Permite una mayor portabilidad de datos, una utilización optimizada de recursos y una mejor experiencia del cliente al procesar flujos de datos en tiempo real. En nuestro taller práctico híbrido, aprenderás cómo filtrar, unir y enriquecer fácilmente datos en tiempo real dentro de Confluent Cloud utilizando nuestro servicio Flink sin servidor.

Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...

confluent

Our talk will explore the transformative impact of integrating Confluent, HiveMQ, and SparkPlug in Industry 4.0, emphasizing the creation of a Unified Namespace. In addition to the creation of a Unified Namespace, our webinar will also delve into Stream Governance and Scaling, highlighting how these aspects are crucial for managing complex data flows and ensuring robust, scalable IIoT-Platforms. You will learn how to ensure data accuracy and reliability, expand your data processing capabilities, and optimize your data management processes. Don't miss out on this opportunity to learn from industry experts and take your business to the next level.

AWS Immersion Day Mapfre - Confluent

confluent

La arquitectura impulsada por eventos (EDA) será el corazón del ecosistema de MAPFRE. Para seguir siendo competitivas, las empresas de hoy dependen cada vez más del análisis de datos en tiempo real, lo que les permite obtener información y tiempos de respuesta más rápidos. Los negocios con datos en tiempo real consisten en tomar conciencia de la situación, detectar y responder a lo que está sucediendo en el mundo ahora.

Eventos y Microservicios - Santander TechTalk

confluent

Q&A with Confluent Experts: Navigating Networking in Confluent Cloud

confluent

This document discusses networking options and best practices for Confluent Cloud. It provides an overview of public endpoints, private link, and peering options. It then discusses best practices for private networking architectures on Azure using hub-and-spoke and private link designs. Finally, it addresses networking considerations and challenges for Kafka Connect managed connectors, as well as planned enhancements for DNS peering and outbound private link support.

Citi TechTalk Session 2: Kafka Deep Dive

confluent

Build real-time streaming data pipelines to AWS with Confluent

confluent

Traditional data pipelines often face scalability issues and challenges related to cost, their monolithic design, and reliance on batch data processing. They also typically operate under the premise that all data needs to be stored in a single centralized data source before it's put to practical use. Confluent Cloud on Amazon Web Services (AWS) provides a fully managed cloud-native platform that helps you simplify the way you build real-time data flows using streaming data pipelines and Apache Kafka.

Q&A with Confluent Professional Services: Confluent Service Mesh

confluent

Citi Tech Talk: Event Driven Kafka Microservices

confluent

Confluent & GSI Webinars series - Session 3

confluent

An in depth look at how Confluent is being used in the financial services industry. Gain an understanding of how organisations are utilising data in motion to solve common problems and gain benefits from their real time data capabilities. It will look more deeply into some specific use cases and show how Confluent technology is used to manage costs and mitigate risks. This session is aimed at Solutions Architects, Sales Engineers and Pre Sales, and also the more technically minded business aligned people. Whilst this is not a deeply technical session, a level of knowledge around Kafka would be helpful.

Citi Tech Talk: Messaging Modernization

confluent

This document discusses moving to an event-driven architecture using Confluent. It begins by outlining some of the limitations of traditional messaging middleware approaches. Confluent provides benefits like stream processing, persistence, scalability and reliability while avoiding issues like lack of structure, slow consumers, and technical debt. The document then discusses how Confluent can help modernize architectures, enable new real-time use cases, and reduce costs through migration. It provides examples of how companies like Advance Auto Parts and Nord/LB have benefitted from implementing Confluent platforms.

Citi Tech Talk: Data Governance for streaming and real time data

confluent

Confluent & GSI Webinars series: Session 2

confluent

Data In Motion Paris 2023

confluent

Vous apprendrez également à : • Créer plus rapidement des produits et fonctionnalités à l’aide d’une suite complète de connecteurs et d’outils de gestion des flux, et à connecter vos environnements à des pipelines de données • Protéger vos données et charges de travail les plus critiques grâce à des garanties intégrées en matière de sécurité, de gouvernance et de résilience • Déployer Kafka à grande échelle en quelques minutes tout en réduisant les coûts et la charge opérationnelle associés

More from confluent (20)

Building API data products on top of your real-time data infrastructure

Speed Wins: From Kafka to APIs in Minutes

Evolving Data Governance for the Real-time Streaming and AI Era

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...

Santander Stream Processing with Apache Flink

Unlocking the Power of IoT: A comprehensive approach to real-time insights

Workshop híbrido: Stream Processing con Flink

Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...

AWS Immersion Day Mapfre - Confluent

Eventos y Microservicios - Santander TechTalk

Q&A with Confluent Experts: Navigating Networking in Confluent Cloud

Citi TechTalk Session 2: Kafka Deep Dive

Build real-time streaming data pipelines to AWS with Confluent

Q&A with Confluent Professional Services: Confluent Service Mesh

Citi Tech Talk: Event Driven Kafka Microservices

Confluent & GSI Webinars series - Session 3

Citi Tech Talk: Messaging Modernization

Citi Tech Talk: Data Governance for streaming and real time data

Confluent & GSI Webinars series: Session 2

Data In Motion Paris 2023

Recently uploaded

The Microsoft 365 Migration Tutorial For Beginner.pptx

operationspcvita

QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...

AlexanderRichford

QR Secure: A Hybrid Approach Using Machine Learning and Security Validation Functions to Prevent Interaction with Malicious QR Codes. Aim of the Study: The goal of this research was to develop a robust hybrid approach for identifying malicious and insecure URLs derived from QR codes, ensuring safe interactions. This is achieved through: Machine Learning Model: Predicts the likelihood of a URL being malicious. Security Validation Functions: Ensures the derived URL has a valid certificate and proper URL format. This innovative blend of technology aims to enhance cybersecurity measures and protect users from potential threats hidden within QR codes 🖥 🔒 This study was my first introduction to using ML which has shown me the immense potential of ML in creating more secure digital environments!

AWS Certified Solutions Architect Associate (SAA-C03)

HarpalGohil4

QA or the Highway - Component Testing: Bridging the gap between frontend appl...

zjhamm304

[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...

Jason Yip

The typical problem in product engineering is not bad strategy, so much as “no strategy”. This leads to confusion, lack of motivation, and incoherent action. The next time you look for a strategy and find an empty space, instead of waiting for it to be filled, I will show you how to fill it in yourself. If you’re wrong, it forces a correction. If you’re right, it helps create focus. I’ll share how I’ve approached this in the past, both what works and lessons for what didn’t work so well.

Northern Engraving | Nameplate Manufacturing Process - 2024

Northern Engraving

Manufacturing custom quality metal nameplates and badges involves several standard operations. Processes include sheet prep, lithography, screening, coating, punch press and inspection. All decoration is completed in the flat sheet with adhesive and tooling operations following. The possibilities for creating unique durable nameplates are endless. How will you create your brand identity? We can help!

AI in the Workplace Reskilling, Upskilling, and Future Work.pptx

Sunil Jagani

Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors

DianaGray10

Join us to learn how UiPath Apps can directly and easily interact with prebuilt connectors via Integration Service--including Salesforce, ServiceNow, Open GenAI, and more. The best part is you can achieve this without building a custom workflow! Say goodbye to the hassle of using separate automations to call APIs. By seamlessly integrating within App Studio, you can now easily streamline your workflow, while gaining direct access to our Connector Catalog of popular applications. We’ll discuss and demo the benefits of UiPath Apps and connectors including: Creating a compelling user experience for any software, without the limitations of APIs. Accelerating the app creation process, saving time and effort Enjoying high-performance CRUD (create, read, update, delete) operations, for seamless data management. Speakers: Russell Alfeche, Technology Leader, RPA at qBotic and UiPath MVP Charlie Greenberg, host

Lee Barnes - Path to Becoming an Effective Test Automation Engineer.pdf

leebarnesutopia

So… you want to become a Test Automation Engineer (or hire and develop one)? While there’s quite a bit of information available about important technical and tool skills to master, there’s not enough discussion around the path to becoming an effective Test Automation Engineer that knows how to add VALUE. In my experience this had led to a proliferation of engineers who are proficient with tools and building frameworks but have skill and knowledge gaps, especially in software testing, that reduce the value they deliver with test automation. In this talk, Lee will share his lessons learned from over 30 years of working with, and mentoring, hundreds of Test Automation Engineers. Whether you’re looking to get started in test automation or just want to improve your trade, this talk will give you a solid foundation and roadmap for ensuring your test automation efforts continuously add value. This talk is equally valuable for both aspiring Test Automation Engineers and those managing them! All attendees will take away a set of key foundational knowledge and a high-level learning path for leveling up test automation skills and ensuring they add value to their organizations.

Harnessing the Power of NLP and Knowledge Graphs for Opioid Research

Neo4j

Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck

FilipTomaszewski5

Apps Break Data

Ivo Velitchkov

How information systems are built or acquired puts information, which is what they should be about, in a secondary place. Our language adapted accordingly, and we no longer talk about information systems but applications. Applications evolved in a way to break data into diverse fragments, tightly coupled with applications and expensive to integrate. The result is technical debt, which is re-paid by taking even bigger "loans", resulting in an ever-increasing technical debt. Software engineering and procurement practices work in sync with market forces to maintain this trend. This talk demonstrates how natural this situation is. The question is: can something be done to reverse the trend?

"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba

Fwdays

This is a session that details how PostgreSQL's features and Azure AI Services can be effectively used to significantly enhance the search functionality in any application. In this session, we'll share insights on how we used PostgreSQL to facilitate precise searches across multiple fields in our mobile application. The techniques include using LIKE and ILIKE operators and integrating a trigram-based search to handle potential misspellings, thereby increasing the search accuracy. We'll also discuss how the azure_ai extension on PostgreSQL databases in Azure and Azure AI Services were utilized to create vectors from user input, a feature beneficial when users wish to find specific items based on text prompts. While our application's case study involves a drug search, the techniques and principles shared in this session can be adapted to improve search functionality in a wide range of applications. Join us to learn how PostgreSQL and Azure AI can be harnessed to enhance your application's search capability.

Day 2 - Intro to UiPath Studio Fundamentals

UiPathCommunity

In our second session, we shall learn all about the main features and fundamentals of UiPath Studio that enable us to use the building blocks for any automation project. 📕 Detailed agenda: Variables and Datatypes Workflow Layouts Arguments Control Flows and Loops Conditional Statements 💻 Extra training through UiPath Academy: Variables, Constants, and Arguments in Studio Control Flow in Studio

Mutation Testing for Task-Oriented Chatbots

Pablo Gómez Abajo

Conversational agents, or chatbots, are increasingly used to access all sorts of services using natural language. While open-domain chatbots - like ChatGPT - can converse on any topic, task-oriented chatbots - the focus of this paper - are designed for specific tasks, like booking a flight, obtaining customer support, or setting an appointment. Like any other software, task-oriented chatbots need to be properly tested, usually by defining and executing test scenarios (i.e., sequences of user-chatbot interactions). However, there is currently a lack of methods to quantify the completeness and strength of such test scenarios, which can lead to low-quality tests, and hence to buggy chatbots. To fill this gap, we propose adapting mutation testing (MuT) for task-oriented chatbots. To this end, we introduce a set of mutation operators that emulate faults in chatbot designs, an architecture that enables MuT on chatbots built using heterogeneous technologies, and a practical realisation as an Eclipse plugin. Moreover, we evaluate the applicability, effectiveness and efficiency of our approach on open-source chatbots, with promising results.

Must Know Postgres Extension for DBA and Developer during Migration

Mydbops

Mydbops Opensource Database Meetup 16 Topic: Must-Know PostgreSQL Extensions for Developers and DBAs During Migration Speaker: Deepak Mahto, Founder of DataCloudGaze Consulting Date & Time: 8th June | 10 AM - 1 PM IST Venue: Bangalore International Centre, Bangalore Abstract: Discover how PostgreSQL extensions can be your secret weapon! This talk explores how key extensions enhance database capabilities and streamline the migration process for users moving from other relational databases like Oracle. Key Takeaways: * Learn about crucial extensions like oracle_fdw, pgtt, and pg_audit that ease migration complexities. * Gain valuable strategies for implementing these extensions in PostgreSQL to achieve license freedom. * Discover how these key extensions can empower both developers and DBAs during the migration process. * Don't miss this chance to gain practical knowledge from an industry expert and stay updated on the latest open-source database trends. Mydbops Managed Services specializes in taking the pain out of database management while optimizing performance. Since 2015, we have been providing top-notch support and assistance for the top three open-source databases: MySQL, MongoDB, and PostgreSQL. Our team offers a wide range of services, including assistance, support, consulting, 24/7 operations, and expertise in all relevant technologies. We help organizations improve their database's performance, scalability, efficiency, and availability. Contact us: info@mydbops.com Visit: https://www.mydbops.com/ Follow us on LinkedIn: https://in.linkedin.com/company/mydbops For more details and updates, please follow up the below links. Meetup Page : https://www.meetup.com/mydbops-databa... Twitter: https://twitter.com/mydbopsofficial Blogs: https://www.mydbops.com/blog/ Facebook(Meta): https://www.facebook.com/mydbops/

What is an RPA CoE? Session 1 – CoE Vision

DianaGray10

A Deep Dive into ScyllaDB's Architecture

ScyllaDB

inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill

LizaNolte

HERE IS YOUR WEBINAR CONTENT! 'Mastering Customer Journey Management with Dr. Graham Hill'. We hope you find the webinar recording both insightful and enjoyable. In this webinar, we explored essential aspects of Customer Journey Management and personalization. Here’s a summary of the key insights and topics discussed: Key Takeaways: Understanding the Customer Journey: Dr. Hill emphasized the importance of mapping and understanding the complete customer journey to identify touchpoints and opportunities for improvement. Personalization Strategies: We discussed how to leverage data and insights to create personalized experiences that resonate with customers. Technology Integration: Insights were shared on how inQuba’s advanced technology can streamline customer interactions and drive operational efficiency.

Leveraging the Graph for Clinical Trials and Standards

Neo4j

Recently uploaded (20)

The Microsoft 365 Migration Tutorial For Beginner.pptx

QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...

AWS Certified Solutions Architect Associate (SAA-C03)

QA or the Highway - Component Testing: Bridging the gap between frontend appl...

[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...

Northern Engraving | Nameplate Manufacturing Process - 2024

AI in the Workplace Reskilling, Upskilling, and Future Work.pptx

Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors

Lee Barnes - Path to Becoming an Effective Test Automation Engineer.pdf

Harnessing the Power of NLP and Knowledge Graphs for Opioid Research

Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck

Apps Break Data

"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba

Day 2 - Intro to UiPath Studio Fundamentals

Mutation Testing for Task-Oriented Chatbots

Must Know Postgres Extension for DBA and Developer during Migration

What is an RPA CoE? Session 1 – CoE Vision

A Deep Dive into ScyllaDB's Architecture

inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill

Leveraging the Graph for Clinical Trials and Standards

Use Apache Gradle to Build and Automate KSQL and Kafka Streams (Stewart Bryson, Red Pill Analytics) Kafka Summit NYC 2019

1. Use Apache Gradle to Build and Automate KSQL and Apache Kafka Streams

2. twitter: @stewartbryson medium: @stewartbryson linkedin: stewartbryson Owner & CEO Red Pill Analytics

3. @redpilla What We Do Data Warehouse Analytics ANALYTICS Data-engineering & ETL

4. @redpilla Project Overview KSQL KSQL User Defined Functions (demo-only) Kafka Streams (demo-only) Agenda

5. @redpilla

6. @redpilla Project Inspiration

7. @redpilla Data Integration: Streaming In

8. @redpilla Data Integration: Streaming Out

9. @redpilla Packaging of Payloads for OWMC

10. @redpilla KSQL Primer

11. @redpilla KSQL Registration Queries CREATE STREAM clickstream ( _time bigint, time varchar, ip varchar, request varchar, status int, userid int, bytes bigint, agent varchar ) with ( kafka_topic = 'clickstream', value_format = 'json' );

12. @redpilla KSQL Persistent Queries CREATE TABLE events_per_min AS SELECT userid, count(*) AS events FROM clickstream window TUMBLING (size 60 second) GROUP BY userid;

13. @redpilla A pipeline is a group of SQL statements that work together to define an end-to-end process.

14. @redpilla KSQL Dependencies

15. @redpilla KSQL Pipelines clickstream clickstream_codes Streams and TablesSQL Scripts enriched_error_codes enriched_error_codes_count customer_clickstream user_clickstream web_users click_user_sessions

16. @redpilla KSQL Pipelines clickstream clickstream_codes Streams and TablesSQL Scripts enriched_error_codes enriched_error_codes_count customer_clickstream user_clickstream web_users click_user_sessions Dependencies within a pipeline

17. @redpilla KSQL Pipelines clickstream clickstream_codes Streams and TablesSQL Scripts enriched_error_codes enriched_error_codes_count customer_clickstream user_clickstream web_users click_user_sessions Dependencies across pipelines

18. @redpilla KSQL Pipelines clickstream clickstream_codes Streams and TablesSQL Scripts enriched_error_codes enriched_error_codes_count customer_clickstream user_clickstream web_users click_user_sessions Manage all KSQL dependencies in one place

19. @redpilla Multiple statements per pipeline Statements are in script files Script files are in Git Developers need to iterate

20. @redpilla What is Apache Gradle? Gradle is an open-source build automation tool for the Age of Continuous Delivery (CD). Building software is no longer just about compiling, linking and packaging.

21. @redpilla What is Apache Gradle?

22. @redpilla gradle-confluent for KSQL pipelines shadow for KSQL user defined functions application for Kafka Streams Gradle Plugins

23. @redpilla Plugin

24. @redpilla Plugin github.com/RedPillAnalytics/gradle-confluent

25. @redpilla Unfortunately… I only have 40 minutes

26. @redpilla Unfortunately… I only have 33 minutes

27. @redpilla Demo

28. @redpilla Demo ...using a Jupyter Notebook

Use Apache Gradle to Build and Automate KSQL and Kafka Streams (Stewart Bryson, Red Pill Analytics) Kafka Summit NYC 2019

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Use Apache Gradle to Build and Automate KSQL and Kafka Streams (Stewart Bryson, Red Pill Analytics) Kafka Summit NYC 2019

Similar to Use Apache Gradle to Build and Automate KSQL and Kafka Streams (Stewart Bryson, Red Pill Analytics) Kafka Summit NYC 2019 (20)

More from confluent

More from confluent (20)

Recently uploaded

Recently uploaded (20)

Use Apache Gradle to Build and Automate KSQL and Kafka Streams (Stewart Bryson, Red Pill Analytics) Kafka Summit NYC 2019