Introducing Kafka Connect and Implementing Custom Connectors

•

1 like•622 views

Kobi Hikri (Independent Software Architect and Consultant): Kobi provides a short intro to Kafka Connect, and then shows an actual code example of developing and dockerizing a custom connector.

Data & Analytics

Introducing Kafka
Connect and
ImplementingCustom
Connectors
Kobi Hikri
Kafka Connect

Agenda
 What (is Kafka Connect)
 The World of Data
 Why (do we need it) Kafka Connect
 When (will we consider using) Kafka Connect
 How Kafka Connect
 Setting up the infrastructure
 Interacting with Kafka Connect
 Implementing a custom source connector
 Working with connectors
 What will not be covered in this talk (will be covered per request)
 Kafka ConnectTransformations
 Working with custom schemas

What is Kafka
Connect
 A Kafka integration tool
 Allows importing data into Kafka
 And exporting data out of Kafka
 Runs as a cluster of one or more system processes
 Each such process is referred to as a Worker
 Workers are grouped together in Worker Groups
 Connector Instances are managed by theConnect cluster
 Tasks are configured by connectors and are distributed across workers
 Data in being pulled by SourceTasks (configured by Source Connectors)
 Or Pushed by Sink Tasks (configured by Sink Connectors)
Worker
Worker
Worker
Worker
Worker Group Worker Group

TheWorld of Data
Dataisscatteredallover!
AJigsawandplentyofglueareneeded

Why KafkaConnect
 In one word
 Integration
 Of different data oriented technologies
 Of a multitude of changing APIs and communication protocols
 In more than one word
 Fault tolerant, scalable Integration with various technologies

When Kafka
Connect
 Implementing a Kafka Producer or Consumer within the
desired data source / target is impossible or expensive (e.g.
the source code is unavailable)
 Even when we can modify the source code of the source /
target system – but prefer to have a scalable and fault tolerant
integration with it (e.g. not to be limited in bandwidth by the
direct connection between the technology and our Kafka
cluster).

How KafkaConnect
Demo
Time
Docker
Compose
Kafka
Connect
Rest API
Custom
Source
Connector

What's hot

How to Build a Machine Learning Infrastructure with Kafka, Connect, Streams, KSQL, etc… This talk shows how to build Machine Learning models at extreme scale and how to productionize the built models in mission-critical real time applications by leveraging open source components in the public cloud. The session discusses the relation between TensorFlow and the Apache Kafka ecosystem - and why this is a great fit for machine learning at extreme scale. The Machine Learning architecture includes: Kafka Connect for continuous high volume data ingestion into the public cloud, TensorFlow leveraging Deep Learning algorithms to build an analytic model on powerful GPUs, Kafka Streams for model deployment and inference in real time, and KSQL for real time analytics of predictions, alerts and model accuracy. Sensor analytics for predictive alerting in real time is used as real world example from Internet of Things scenarios. A live demo shows the out-of-the-box integration and dynamic scalability of these components on Google Cloud. Key takeaways for the audience • Learn how to build a Machine Learning infrastructure at extreme scale and how to productionize the built models in mission-critical real time applications • Understand the benefits of a machine learning platform on the public cloud • Learn about an extreme scale Machine Learning architecture around the Apache Kafka open source ecosystem including Kafka Connect, Kafka Streams and KSQL • See a live demo for an Internet of Things use case: Sensor analytics for predictive alerting in real time

Deep Learning at Extreme Scale (in the Cloud)  with the Apache Kafka Open Sou...

Kai Wähner

Java for Recruiters

ph7 -

IBM Integration Bus High Availability Overview

Peter Broadhurst

Tech JAM 2016 TEC 11 実践 SAP HANA 大解剖

Koji Shinkubo

Garbage First Garbage Collector (G1 GC) - Migration to, Expectations and Adva...

Monica Beckwith

Database Cloud Services Office Hours : Oracle sharding hyperscale globally d...

Tammy Bednar

Document Imaging and the SAP Content Server 101

Verbella CMG

ssis lab

rasheedam

Hana sql

Manuel Zárate

Managing a large multitenant database with many pluggable database can be a difficult task. Many PDBs fight for the server resources like I/O, CPU and memory. It can be difficult to keep the SLAs agreed with your customers. Oracle Database 12c Release 2 offers improvments in resource management for PDBs and a new feature called „Lockdown Profiles“ which helps you to limit the available features on PDB level. The talk shows the various areas of these two features in a CDB environment and shows how they will help you managing a multitenant environment. And both features can help even with a single-tenant database.

Taming the PDB: Resource Management and Lockdown Profiles

Markus Flechtner

The Oracle Database with Sharding is a globally distributed multi-model (relational & document) DBMS. It is built on shared-nothing architecture in which data is horizontally partitioned across databases that share no hardware or software. It provides linear scalability, fault isolation and geographic data distribution for shard-amenable applications. This presentation was presented during Sangam18 in December 2018 - original title: "Oracle Sharding 18c for Data Sovereignty and Massive Linear Scalability"

Oracle Sharding 18c - Technical Overview

Markus Michalewicz

Kafka Streams: What it is, and how to use it?

confluent

Talk from Kafka Summit San Francisco 2019 (https://kafka-summit.org/sessions/event-driven-model-serving-stream-processing-vs-rpc-kafka-tensorflow/). Video recording will be available for free on the Summit website. Event-based stream processing is a modern paradigm to continuously process incoming data feeds, e.g. for IoT sensor analytics, payment and fraud detection, or logistics. Machine Learning / Deep Learning models can be leveraged in different ways to do predictions and improve the business processes. Either analytic models are deployed natively in the application or they are hosted in a remote model server. In the latter you combine stream processing with RPC / Request-Response paradigm instead of direct doing direct inference within the application. This talk discusses the pros and cons of both approaches and shows examples of stream processing vs. RPC model serving using Kubernetes, Apache Kafka, Kafka Streams, gRPC and TensorFlow Serving. The trade-offs of using a public cloud service like AWS or GCP for model deployment are also discussed and compared to local hosting for offline predictions directly “at the edge”. Key takeaways • Machine Learning / Deep Learning models can be used in different ways to do predictions. Scalability and loose coupling are important success factors • Stream processing vs. RPC / Request-Response for model serving has many trade-offs – learn about alternatives and best practices for your different scenarios • Understand the alternatives and trade-offs of model deployment in modern infrastructures like Kubernetes or Cloud Services like AWS or GCP • See live demos with Java, gRPC, Apache Kafka, KSQL and TensorFlow Serving to understand the trade-offs

Event-Driven Stream Processing and Model Deployment with Apache Kafka, Kafka ...

Kai Wähner

Pulsar Summit San Francisco is the event dedicated to Apache Pulsar. This one-day, action-packed event will include 5 keynotes, 12 breakout sessions, and 1 amazing happy hour. Speakers are from top companies, including Google, AWS, Databricks, Onehouse, StarTree, Intel, ScyllaDB, and more! It’s the perfect opportunity to network with Pulsar thought leaders in person. Join developers, architects, data engineers, DevOps professionals, and anyone who wants to learn about messaging and event streaming for this one-day, in-person event. Pulsar Summit San Francisco brings the Apache Pulsar Community together to share best practices and discuss the future of streaming technologies.

Beam + Pulsar: Powerful Stream Processing at Scale - Pulsar Summit SF 2022

StreamNative

Apache Kafka Fundamentals for Architects, Admins and Developers

confluent

Centralized Logging System Using ELK Stack

Rohit Sharma

What is SSO? An introduction to Single Sign On

Riddhi Sood

Azure purview

Shafqat Turza

Oracle Databaseはクラウドに移行するべきか否か全10ケースをご紹介 (Oracle Cloudウェビナーシリーズ: 2021年11月30日)

オラクルエンジニア通信

7 Ways To Crash Postgres

PostgreSQL Experts, Inc.

What's hot (20)

Deep Learning at Extreme Scale (in the Cloud)  with the Apache Kafka Open Sou...

Java for Recruiters

IBM Integration Bus High Availability Overview

Tech JAM 2016 TEC 11 実践 SAP HANA 大解剖

Garbage First Garbage Collector (G1 GC) - Migration to, Expectations and Adva...

Database Cloud Services Office Hours : Oracle sharding hyperscale globally d...

Document Imaging and the SAP Content Server 101

ssis lab

Hana sql

Taming the PDB: Resource Management and Lockdown Profiles

Oracle Sharding 18c - Technical Overview

Kafka Streams: What it is, and how to use it?

Event-Driven Stream Processing and Model Deployment with Apache Kafka, Kafka ...

Beam + Pulsar: Powerful Stream Processing at Scale - Pulsar Summit SF 2022

Apache Kafka Fundamentals for Architects, Admins and Developers

Centralized Logging System Using ELK Stack

What is SSO? An introduction to Single Sign On

Azure purview

Oracle Databaseはクラウドに移行するべきか否か全10ケースをご紹介 (Oracle Cloudウェビナーシリーズ: 2021年11月30日)

7 Ways To Crash Postgres

Similar to Introducing Kafka Connect and Implementing Custom Connectors

GDG Taipei 2020 - Cloud and On-premises Applications Integration Using Event-...

Rich Lee

Confluent Enterprise Datasheet

confluent

After a quick overview and introduction of Apache Kafka, this session cover two components which extend the core of Apache Kafka: Kafka Connect and Kafka Streams/KSQL. Kafka Connects role is to access data from the out-side-world and make it available inside Kafka by publishing it into a Kafka topic. On the other hand, Kafka Connect is also responsible to transport information from inside Kafka to the outside world, which could be a database or a file system. There are many existing connectors for different source and target systems available out-of-the-box, either provided by the community or by Confluent or other vendors. You simply configure these connectors and off you go. Kafka Streams is a light-weight component which extends Kafka with stream processing functionality. By that, Kafka can now not only reliably and scalable transport events and messages through the Kafka broker but also analyse and process these event in real-time. Interestingly Kafka Streams does not provide its own cluster infrastructure and it is also not meant to run on a Kafka cluster. The idea is to run Kafka Streams where it makes sense, which can be inside a “normal” Java application, inside a Web container or on a more modern containerized (cloud) infrastructure, such as Mesos, Kubernetes or Docker. Kafka Streams has a lot of interesting features, such as reliable state handling, queryable state and much more. KSQL is a streaming engine for Apache Kafka, providing a simple and completely interactive SQL interface for processing data in Kafka.

Kafka Connect & Streams - the ecosystem around Kafka

Guido Schmutz

In this presentation we describe the design and implementation of Kafka Connect, Kafka’s new tool for scalable, fault-tolerant data import and export. First we’ll discuss some existing tools in the space and why they fall short when applied to data integration at large scale. Next, we will explore Kafka Connect’s design and how it compares to systems with similar goals, discussing key design decisions that trade off between ease of use for connector developers, operational complexity, and reuse of existing connectors. Finally, we’ll discuss how standardizing on Kafka Connect can ultimately lead to simplifying your entire data pipeline, making ETL into your data warehouse and enabling stream processing applications as simple as adding another Kafka connector.

Data Pipelines with Kafka Connect

Kaufman Ng

Streaming the platform with Confluent (Apache Kafka)

GiuseppeBaccini

Beyond the brokers - Un tour de l'écosystème Kafka

Florent Ramiere

Kafka connect 101

Whiteklay

Kafka for Scale

Eyal Ben Ivri

Apache Kafka - Scalable Message Processing and more!

Guido Schmutz

Beyond the Brokers: A Tour of the Kafka Ecosystem

confluent

Beyond the brokers - A tour of the Kafka ecosystem

Damien Gasparina

Evolution of netflix conductor

vedu12

How to integrate your database with kafka & CDC

Abdallah Mahmoud

Large scale, distributed and reliable messaging with Kafka

Rafał Hryniewski

JHipster conf 2019 - Kafka Ecosystem

Florent Ramiere

Event-driven application architectures are becoming increasingly common as a large number of users demand more interactive, real-time, and intelligent responses. Yet it can be challenging to decide how to capture and perform real-time data analysis and deliver differentiating experiences. Join experts from Confluent and AWS to learn how to build Apache Kafka®-based streaming applications backed by machine learning models. Adopting the recommendations will help you establish repeatable patterns for high performing event-based apps.

Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...

HostedbyConfluent

Leverage Kafka to build a stream processing platform

confluent

Overview of Kafka connect

Knoldus Inc.

Overview of Kafka connect

Knoldus Inc.

Fault Tolerance with Kafka

Edureka!

Similar to Introducing Kafka Connect and Implementing Custom Connectors (20)

GDG Taipei 2020 - Cloud and On-premises Applications Integration Using Event-...

Confluent Enterprise Datasheet

Kafka Connect & Streams - the ecosystem around Kafka

Data Pipelines with Kafka Connect

Streaming the platform with Confluent (Apache Kafka)

Beyond the brokers - Un tour de l'écosystème Kafka

Kafka connect 101

Kafka for Scale

Apache Kafka - Scalable Message Processing and more!

Beyond the Brokers: A Tour of the Kafka Ecosystem

Beyond the brokers - A tour of the Kafka ecosystem

Evolution of netflix conductor

How to integrate your database with kafka & CDC

Large scale, distributed and reliable messaging with Kafka

JHipster conf 2019 - Kafka Ecosystem

Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...

Leverage Kafka to build a stream processing platform

Overview of Kafka connect

Fault Tolerance with Kafka

More from Itai Yaffe

Yulia Antonovsky (Senior Software Engineer II) @ Akamai: Our cloud-based ingest pipeline processes over 10 Gb of security events data per second, which demands high-performance processing and analysis. To achieve this, we've implemented efficient partitioning using Java and Spark applications running on AKS and leveraging Kafka. This allows us to provide real-time analytics within two minutes and heavy batch processing for deeper analysis hourly. During this talk, we will cover how we use Kafka to scale our Spark application on K8s, partitioning strategies for high-volume data processing, and how partitioning helps avoid storage throttling issues.

Mastering Partitioning for High-Volume Data Processing

Itai Yaffe

Maayan Gad (Senior Big Data Engineer) @ Wix: Who likes to maintain long SQL files? No one. Column renames, additions, deletions, changes in the KPIs your tables are based on, or the date your table should start from - all are annoying, time consuming tasks your data engineers waste their time on. In this talk we’ll uncover how we solved this problem at Wix, allowing our data engineers to easily create Data Warehouse grade tables using configuration files, and even transfer some of the business-related-only parts to the data analysts.

Solving Data Engineers Velocity - Wix's Data Warehouse Automation

Itai Yaffe

Ada Sharoni (Software Engineering Architect) @ Hunters: Imagine you had to manage thousands of Spark applications that are automatically spinning up on-demand upon every customer interaction. Our unique constraints in Hunters have led us to adopt an architecture and concepts that we believe many other companies will find useful. In this lecture we will share our solutions and insights in running many lightweight, cheap Spark applications on Kubernetes, that can easily survive frequent restarts and smartly share resources on Spot EC2 instances.

Lessons Learnt from Running Thousands of On-demand Spark Applications

Itai Yaffe

María de la Fuente (Solutions Architect Manager for IMEA) @ Databricks While most companies understand the value creation of leveraging data and are taking on board an AI strategy, only 13% of the data science projects make it to production successfully. Besides the well-known skills gap in the market, we need to level up our end-to-end approach and cover all aspects involved when working with AI. In this session, we will discuss the main obstacles to overcome and how we can avoid the major pitfalls to ensure our data science journey becomes successful.

Why do the majority of Data Science projects never make it to production?

Itai Yaffe

Eynav Mass (VP R&D) @ Oribi: When it comes to data solutions, one-size doesn't fit all. Choosing the right best-matching database, or data tools, can be a game-changer for your system. How can you take such a decision effectively? The system, the company, the product, and probably your team - all are evolving, and the best solution for today may not fit tomorrow's needs. In order to pick a data solution for longer term, you should evaluate the optional data tools according to several factors. These factors will reflect the requirements looking forward. At the session, we will discuss these factors, along with sharing some real-life stories and lessons learned, to help you properly plan & prepare your data solutions.

Planning a data solution - "By Failing to prepare, you are preparing to fail"

Itai Yaffe

Evaluating Big Data & ML Solutions - Opening Notes

Itai Yaffe

Jon Bratseth (VP Architect) @ Verizon Media: The big data world has mature technologies for offline analysis and learning from data, but have lacked options for making data-driven decisions in real time. When it is sufficient to consider a single data point model servers such as TensorFlow serving can be used but in many cases you want to consider many data points to make decisions. This is a difficult engineering problem combining state, distributed algorithms and low latency, but solving it often makes it possible to create far superior solutions when applying machine learning. This talk will explain why this is a hard problem, show the advantages of solving it, and introduce the open source Vespa.ai platform which is used to implement such solutions in some of the largest scale problems in the world including the world's third largest ad serving system.

Big data serving: Processing and inference at scale in real time

Itai Yaffe

Sharon Dashet (Sr. Data Analytics Solution Lead) @ Google Cloud: The worlds of traditional RDBMS and Data Lake Hadoop systems are converging and moving to public cloud and SaaS offerings. In this session, Sharon will share her personal journey as a data professional since the 90s weaved into the history of data management systems. The session will also cover the differences between on-premise and cloud Data Lakes.

Data Lakes on Public Cloud: Breaking Data Management Monoliths

Itai Yaffe

Orit Alul (Sr. Solutions Architect) @ AWS: As data is growing at an exponential rate, we are interested not only in being able to analyze the past or present but also in predicting the future! In this session, Orit will talk about the power of data combined with machine learning. Building a highly scalable and flexible data architecture in the cloud to collect, process, and analyze data, in order to get timely insights and react quickly to new information. In addition, Orit will present best practices, performance and optimization tips for building a Data Lake in the cloud.

Unleashing the Power of your Data

Itai Yaffe

Data Lake on Public Cloud - Opening Notes

Itai Yaffe

Roi Teveth (Data Engineer) and Itai Yaffe (Tech Lead, Big Data group) @ Nielsen: At Nielsen Identity Engine, we use Spark to process 10’s of TBs of data. Our ETLs, orchestrated by Airflow, spin-up AWS EMR clusters with thousands of nodes per day. In this talk, we’ll guide you through migrating Spark workloads to Kubernetes with minimal changes to Airflow DAGs, using the open-sourced GCP Spark-on-K8s operator and the native integration we recently contributed to the Airflow project.

Airflow Summit 2020 - Migrating airflow based spark jobs to kubernetes - the ...

Itai Yaffe

Itai Yaffe (Tech Lead, Big Data group) @ Nielsen: Every day, millions of advertising campaigns are happening around the world. As campaign owners, measuring the ongoing campaign effectiveness (e.g "how many distinct users saw my online ad VS how many distinct users saw my online ad, clicked it and purchased my product?") is super important. However, this task (often referred to as "funnel analysis") is not an easy task, especially if the chronological order of events matters. So, while the combination of Druid and ThetaSketch aggregators can answer some of these questions, it still can’t answer the question "how many distinct users viewed the brand’s homepage FIRST and THEN viewed product X page?" In this talk, we will discuss how we combine Spark, Druid and ThetaSketch aggregators to answer such questions at scale.

DevTalks Reimagined 2020 - Funnel Analysis with Spark and Druid

Itai Yaffe

Virtual Apache Druid Meetup: AIADA (Ask Itai and David Anything)

Itai Yaffe

Benjamin Hopp (Solutions Architect) @ Imply: Druid is an emerging standard in the data infrastructure world, designed for high-performance slice-and-dice analytics (“OLAP”-style) on large data sets. This talk is for you if you’re interested in learning more about pushing Druid’s analytical performance to the limit. Perhaps you’re already running Druid and are looking to speed up your deployment, or perhaps you aren’t familiar with Druid and are interested in learning the basics. Some of the tips in this talk are Druid-specific, but many of them will apply to any operational analytics technology stack. The most important contributor to a fast analytical setup is getting the data model right. The talk will center around various choices you can make to prepare your data to get best possible query performance. We’ll look at some general best practices to model your data before ingestion such as OLAP dimensional modeling (called “roll-up” in Druid), data partitioning, and tips for choosing column types and indexes. We’ll also look at how more can be less: often, storing copies of your data partitioned, sorted, or aggregated in different ways can speed up queries by reducing the amount of computation needed. We’ll also look at Druid-specific optimizations that take advantage of approximations; where you can trade accuracy for performance and reduced storage. You’ll get introduced to Druid’s features for approximate counting, set operations, ranking, quantiles, and more. And we will finish with the latest and greatest Druid news, including details about the latest roadmap and releases.

A Day in the Life of a Druid Implementor and Druid's Roadmap

Itai Yaffe

Dr. Edward (Eddie) Bortnikov (Senior Director of Research) @ Verizon Media: Ingestion and queries of real-time data in Druid are performed by a core software component named Incremental Index (I^2). I^2’s scalability is paramount to the speed of the ingested data becoming queryable as well as to the operational efficiency of the Druid cluster. The current I^2 Implementation is based on the traditional ordered JDK key-value (KV-)map. We present an experimental I^2 implementation that is based on a novel data structure named OakMap - a scalable thread-safe off-heap KV-map for Big Data applications in Java. With OakMap, I^2 can ingest data at almost 2x speed while using 30% less RAM. The project is expected to become GA in 2020.

Scalable Incremental Index for Druid

Itai Yaffe

Itai Yaffe (Tech Lead, Big Data group) @ Nielsen: Every day, millions of advertising campaigns are happening around the world. As campaign owners, measuring the ongoing campaign effectiveness (e.g “how many distinct users saw my online ad VS how many distinct users saw my online ad, clicked it and purchased my product?”) is super important. However, this task (often referred to as “funnel analysis”) is not an easy task, especially if the chronological order of events matters. So, while the combination of Druid and ThetaSketch aggregators can answer some of these questions, it still can’t answer the question "how many distinct users viewed the brand’s homepage FIRST and THEN viewed product X page?" In this talk, we will discuss how we combine Spark, Druid and ThetaSketch aggregators to answer such questions at scale.

Funnel Analysis with Spark and Druid

Itai Yaffe

Shir Bromberg (Big Data team leader) @ Yotpo: Nowadays, many of an organization’s main applications rely on Spark pipelines. As these applications become more significant to businesses, so does the need to quickly deploy, test and monitor them. The standard way of running spark jobs is to deploy it on a dedicated managed cluster. However, this solution is relatively expensive with potentially high setup time. Therefore, we developed a way to run Spark on any container orchestration platform. This allows us to run Spark in a simple, custom and testable way. In this talk, we will present our open-source dockers for running Spark on Nomad servers. We will cover: * The issues we had running spark on managed clusters and the solution we developed. * How to build a spark docker. * And finally, what you may achieve by using Spark on Nomad.

The benefits of running Spark on your own Docker

Itai Yaffe

Etti Gur (Senior Big Data developer) and Itai Yaffe (Tech Lead, Big Data group) @ Nielsen: At Nielsen Marketing Cloud, we provide our customers (marketers and publishers) real-time analytics tools to measure their ongoing campaigns' efficiency. To achieve that, we need to ingest billions of events per day into our big data stores and we need to do it in a scalable yet cost-efficient manner. In this talk, we will discuss how we significantly optimized our Spark-based in-flight analytics daily pipeline, reducing its total execution time from over 20 hours down to 2 hours, resulting in a huge cost reduction. Topics include: * Ways to identify optimization opportunities * Optimizing Spark resource allocation * Parallelizing Spark output phase with dynamic partition inserts * Running multiple Spark "jobs" in parallel within a single Spark application

Optimizing Spark-based data pipelines - are you up for it?

Itai Yaffe

Ilai Malka from Nielsen at AWS Community Day TLV, December 2019 (https://awscommunitydaytelaviv2019.splashthat.com/): Scheduling big data workloads is challenging. It's extra challenging when running on Serverless infrastructure. At Nielsen Marketing Cloud, we've built a system that uploads 250 billion events per day to partner ad platforms, running on Serverless infrastructure (AWS Lambda and OpenFaaS). Creating a 'scheduler' for this system required: 1. Rate-limiting to prevent flooding partner platforms. 2. High utilization to keep costs low 3. Careful bottleneck management to keep the system humming https://www.linkedin.com/in/ilai-malka-93b06172/ https://twitter.com/IlaiMalka #Nielsen #NielsenMarketingCloud #AWSCommunityDay #Serverless

Scheduling big data workloads on serverless infrastructure

Itai Yaffe

GraphQL API on a Serverless Environment

Itai Yaffe

More from Itai Yaffe (20)

Mastering Partitioning for High-Volume Data Processing

Solving Data Engineers Velocity - Wix's Data Warehouse Automation

Lessons Learnt from Running Thousands of On-demand Spark Applications

Why do the majority of Data Science projects never make it to production?

Planning a data solution - "By Failing to prepare, you are preparing to fail"

Evaluating Big Data & ML Solutions - Opening Notes

Big data serving: Processing and inference at scale in real time

Data Lakes on Public Cloud: Breaking Data Management Monoliths

Unleashing the Power of your Data

Data Lake on Public Cloud - Opening Notes

Airflow Summit 2020 - Migrating airflow based spark jobs to kubernetes - the ...

DevTalks Reimagined 2020 - Funnel Analysis with Spark and Druid

Virtual Apache Druid Meetup: AIADA (Ask Itai and David Anything)

A Day in the Life of a Druid Implementor and Druid's Roadmap

Scalable Incremental Index for Druid

Funnel Analysis with Spark and Druid

The benefits of running Spark on your own Docker

Optimizing Spark-based data pipelines - are you up for it?

Scheduling big data workloads on serverless infrastructure

GraphQL API on a Serverless Environment

Recently uploaded

Gartner's Data Analytics Maturity Model.pptx

chadhar227

Discover Why Less is More in B2B Research

michael115558

原版定制【微信:176555708】【圣地亚哥州立大学毕业证（SDSU毕业证书）】【微信:176555708】（留信学历认证永久存档查询）采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。【业务选择办理准则】一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信176555708】文凭即可二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信176555708】即可三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。留信网认证的作用: 1:该专业认证可证明留学生真实身份 2:同时对留学生所学专业登记给予评定 3:国家专业人才认证中心颁发入库证书 4:这个认证书并且可以归档倒地方 5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息 6:个人职称评审加20分 7:个人信誉贷款加10分 8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才留信网服务项目： 1、留学生专业人才库服务（留信分析） 2、国（境）学习人员提供就业推荐信服务 3、留学人员区块链存储服务 → 【关于价格问题（保证一手价格）】我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子我给客户的都是第一手的代理价格，因为我想坦诚对待大家不想跟大家在价格方面浪费时间对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。选择实体注册公司办理，更放心，更安全！我们的承诺：可来公司面谈，可签订合同，会陪同客户一起到教育部认证窗口递交认证材料，客户在教育部官方认证查询网站查询到认证通过结果后付款，不成功不收费！

怎样办理圣地亚哥州立大学毕业证（SDSU毕业证书）成绩单学校原版复制

vexqp

Sealdah % High Class Call Girls Kolkata - 450+ Call Girl Cash Payment 8005736733 Neha Thakur Our agency presents a selection of young, charming call girls #J11 available for bookings at Oyo Hotels. Experience high-class escort services at pocket-friendly rates, with our female escorts exuding both beauty and a delightful personality, ready to meet your desires. Whether it's Housewives,#J11 College girls, Russian girls, Muslim girls, or any other preference, we offer a diverse range of options to cater to your tastes. We provide both in-call and out-call services for your convenience. Our in-call location in Kolkata ensures cleanliness, hygiene, and 100% safety, while our out-call services offer doorstep delivery for added ease. We value your time and money, hence we kindly request pic collectors, time-passers, and bargain hunters to refrain from contacting us. Our services feature various packages at competitive rates: One shot: ₹2000/in-call, ₹5000/out-call Two shots with one girl: ₹3500/in-call, ₹6000/out-call Body to body massage with sex: ₹3000/in-call Full night for one person: ₹7000/in-call, ₹10000/out-call Full night for more than 1 person: Contact us at 🔝 8005736733 🔝. for details Operating 24/7, we serve various locations in Kolkata, including Green Park, near metro stations. For premium call girl services in Kolkata 🔝 8005736733 🔝. Thank you for considering us!

Sealdah % High Class Call Girls Kolkata - 450+ Call Girl Cash Payment 8005736...

HyderabadDolls

Lecture_2_Deep_Learning_Overview-newone1

ranjankumarbehera14

Gomti Nagar & best call girls in Lucknow | 9548273370 Independent Escorts & Dating Escorts Service CALL GIRL IN Lucknow 9548273370 ❤CALL GIRLS IN ESCORT SERVICE❤CALL GIRL IN #j11 We are Providing :- ● – Private independent collage Going girls . ● – independent Models . ● – House Wife’s . ● – Private Independent House Wife’s ● – Corporate M.N.C Working Profiles . ● – Call Center Girls . ● – Live Band Girls . ●- Foreigners & Many More . Service type: 1.In call 2.out call 3. full Lip to Lip kiss 4.69 5.b-job without Condom 6. Hard Core sex & Much More. 7 Body to Body Touch 8 Kissing 9 Sucking Boobs and More 10 Enjoy by Hand 11 Relax By Oral 12 Sex with Happy Ending • In Call and Out Call Service • 3* 5* 7* Hotels Service • 24 Hours Available • Indian, Russian, Punjabi, Kashmiri Escorts • Real Models, College Girls, House Wife, Also Available • Short Time and Full Time Service Available • Hygienic Full AC Neat and Clean Rooms Avail. In Hotel 24 hours • Daily Escorts Staff Available • Minimum to Maximu m Range Available.c

Gomti Nagar & best call girls in Lucknow | 9548273370 Independent Escorts & D...

HyderabadDolls

Top profile Call Girls In Indore [ 7014168258 ] Call Me For Genuine Models We are available 24*7 Booking Contact Details :- WhatsApp Chat :- +91-7014168258 If you're looking for India Call girls you've come to the right place. You'll find some of the most beautiful call girls in our location with. These ladies have pleasing personalities, hot figures, and a passion for physical pleasure. Call girls in India Lucknow Many men have booked them for their erotic and soul-mixing performances, which are sure to leave you with unforgettable memories. #K09 Escort Service India is available in the city for men and women of all ages. They can satisfy your sexual needs and will make your experience even more enjoyable and memorable. Whether you're looking for a blow-job, stripping, lovemaking, or other dirty acts, you'll be able to find a match for your tastes and budget. These highly trained professionals will help you have an unforgettable night. One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7014168258 We are available 24*7 all days of the year. Call us — 7014168258 Thank you for Visiting.

Top profile Call Girls In Indore [ 7014168258 ] Call Me For Genuine Models We...

gajnagarg

Digital Transformation Playbook by Graham Ware

Graham Ware

Digital advertising, or paid media, encompasses the strategic deployment of online advertisements to reach target audiences efficiently and effectively. This includes any digital platform that supports advertising to deliver unique messages for any objective. Understanding the mechanics of digital advertising platforms, along with insights into audience behaviors and preferences, allows marketers to optimize their ad spend and achieve significant engagement and conversion rates. This lecture is for Advanced Digital & Social Media Strategy (MGMTX 466.05) at UCLA Extension.

Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...

Valters Lauzums

Kings of Saudi Arabia, information about them

eitharjee

Statistics notes ,it includes mean to index numbers

suginr1

5CL-ADBA,5cladba, Chinese supplier, safety is guaranteed

amy56318795

7. Epi of Chronic respiratory diseases.ppt

ibrahimabdi22

Context 1. Housing Agent collected resale prices on HDB apartments in Singapore. Objective 2. To predict resale prices in to advise his potential clients. Strategies 3. Explore & Clean data for analysis. 4. Perform K-Means Clustering, in Orange, to find possible segments in the customer data. 5. Tune the model to improve its performance. 6. Visualise the findings, share conclusions, and give insight-driven recommendations. Author: Anthony mok Date: 18 Nov 2023 Email: xxiaohao@yahoo.com

Predicting HDB Resale Prices - Conducting Linear Regression Analysis With Orange

ThinkInnovation

High Profile Call Girls Service in Jalore { 9332606886 } VVIP NISHA Call Girls Near 5 Star Hotel Booking Now open +91- 9332606886 Pune VIP Call Girls is highly qualified professionals. #S10 They will offer only top quality services and never try to scam or drain you of money; in fact, they strive to give complete satisfaction so that each cent spent was worth its worth. Furthermore, these professionals pride themselves in keeping customer details confidential; any information gleaned will never be shared with any third party.$V15 Two shots with one girl: ₹4500/in-call, ₹7500/out-call Body to body massage with : ₹5500/in-call Full night for one person: ₹7000/in-call, ₹12000/out-call •𝐂𝐨𝐥𝐥𝐞𝐠𝐞 𝐆𝐢𝐫𝐥𝐬 •𝐇𝐨𝐮𝐬𝐞𝐰𝐢𝐯𝐞𝐬 •𝐇𝐢𝐠𝐡 𝐏𝐫𝐨𝐟𝐢𝐥𝐞 𝐆𝐢𝐫𝐥𝐬 •𝐑𝐚𝐦𝐩 𝐌𝐨𝐝𝐞𝐥𝐬 •𝐅𝐨𝐫𝐞𝐢𝐠𝐧𝐞𝐫 𝐆𝐢𝐫𝐥𝐬 𝐎𝐮𝐫 𝐞𝐬𝐜𝐨𝐫𝐭 𝐬𝐞𝐫𝐯𝐢𝐜𝐞𝐬 •𝐇𝐉 (𝐇𝐚𝐧𝐝 𝐉𝐨𝐛) •𝐁𝐉 (𝐁𝐥𝐨𝐰𝐣𝐨𝐛) # Completion # (Oral To Completion) # Special Massage # O-Level (Oral ) # Oral With A Noncondom) # COB (Come On Body) # Extraball (Have Many Times) # All Meetings 100%Safe

High Profile Call Girls Service in Jalore { 9332606886 } VVIP NISHA Call Girl...

kumargunjan9515

Discover how TrafficWave Generator can drive targeted, engaging visitors to your online assets and propel your business to new heights. Unlock the power of free, set-and-forget traffic today. Featuring a user-friendly interface, set-and-forget functionality, and the ability to monetize the traffic with your affiliate links or product offers, TrafficWave Generator is a game-changer for online entrepreneurs, bloggers, and e-commerce store owners alike. Backed by a 30-day money-back guarantee and comprehensive bonus offerings, this software has the potential to transform your traffic-generation and content-creation efforts, unlocking new levels of success in the digital marketing landscape.

TrafficWave Generator Will Instantly drive targeted and engaging traffic back...

SOFTTECHHUB

Aspirational Block Program Block Syaldey District - Almora

GovindSinghDasila

Gulbai Tekra * Cheap Call Girls In Ahmedabad Phone No 8005736733 Elite Escort Service Available 24/7 Hire Booking Contact Details :- WhatsApp Chat :- +91-6378878445 We offer all types of girls of your choice with space. Our escorts are fully cooperative and understand your needs.#K09 All types of call girls like Housewives, College girls,#K09 Russian girls, Muslim girls, Afghani girls, Bengali girls, Working girls, south Indian girls, Punjabi girls, etc. In-Call: — You Can Reach At Our Place in Bangalore Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call: — Service for Out Call You have To Come Pick The Girl From My Place We Also Provide Door-Step Services Hygienic: — Full Ac Neat And Clean Rooms Available In Hotel 24 * 7 Hrs In Bangalore Our Services and Rates: – One Shot — 2500/in call (time ½ hour), 5000/out call Two shot with one girl — 5000/in call (time 1 hour), 6000/out call Body to body massage with sex- 3000/in call (time 1 hour) full night for one person– 8000/in call, 10000/out call (shot limit 4 shot) We are available 24*7 all days of the year

Gulbai Tekra * Cheap Call Girls In Ahmedabad Phone No 8005736733 Elite Escort...

gragchanchal546

Top Call Girls in Balaghat 9332606886Call Girls Advance Cash On Delivery Service Booking Now open +91- 9332606886 Pune VIP Call Girls is highly qualified professionals. #S10 They will offer only top quality services and never try to scam or drain you of money; in fact, they strive to give complete satisfaction so that each cent spent was worth its worth. Furthermore, these professionals pride themselves in keeping customer details confidential; any information gleaned will never be shared with any third party.$V15 Two shots with one girl: ₹4500/in-call, ₹7500/out-call Body to body massage with : ₹5500/in-call Full night for one person: ₹7000/in-call, ₹12000/out-call •𝐂𝐨𝐥𝐥𝐞𝐠𝐞 𝐆𝐢𝐫𝐥𝐬 •𝐇𝐨𝐮𝐬𝐞𝐰𝐢𝐯𝐞𝐬 •𝐇𝐢𝐠𝐡 𝐏𝐫𝐨𝐟𝐢𝐥𝐞 𝐆𝐢𝐫𝐥𝐬 •𝐑𝐚𝐦𝐩 𝐌𝐨𝐝𝐞𝐥𝐬 •𝐅𝐨𝐫𝐞𝐢𝐠𝐧𝐞𝐫 𝐆𝐢𝐫𝐥𝐬 𝐎𝐮𝐫 𝐞𝐬𝐜𝐨𝐫𝐭 𝐬𝐞𝐫𝐯𝐢𝐜𝐞𝐬 •𝐇𝐉 (𝐇𝐚𝐧𝐝 𝐉𝐨𝐛) •𝐁𝐉 (𝐁𝐥𝐨𝐰𝐣𝐨𝐛) # Completion # (Oral To Completion) # Special Massage # O-Level (Oral ) # Oral With A Noncondom) # COB (Come On Body) # Extraball (Have Many Times) # All Meetings 100%Safe

Top Call Girls in Balaghat 9332606886Call Girls Advance Cash On Delivery Ser...

kumargunjan9515

Top profile Call Girls In Hapur [ 7014168258 ] Call Me For Genuine Models We are available 24*7 Booking Contact Details :- WhatsApp Chat :- +91-7014168258 If you're looking for India Call girls you've come to the right place. You'll find some of the most beautiful call girls in our location with. These ladies have pleasing personalities, hot figures, and a passion for physical pleasure. Call girls in India Lucknow Many men have booked them for their erotic and soul-mixing performances, which are sure to leave you with unforgettable memories. #K09 Escort Service India is available in the city for men and women of all ages. They can satisfy your sexual needs and will make your experience even more enjoyable and memorable. Whether you're looking for a blow-job, stripping, lovemaking, or other dirty acts, you'll be able to find a match for your tastes and budget. These highly trained professionals will help you have an unforgettable night. One Shot — 5000/in call (time 1 hour), 6000/out call Two shot with one girl — 8000/in call (time 2 hour), 10000/out call Body to body massage with sex- 8000/in call (time 1 hour) Full night Service for one person– 12000/in call, 13000/out call (shot limit 3-4 shots) Full night Service for more than 1 person — please contact Us —7014168258 We are available 24*7 all days of the year. Call us — 7014168258 Thank you for Visiting.

Top profile Call Girls In Hapur [ 7014168258 ] Call Me For Genuine Models We ...

nirzagarg

Recently uploaded (20)

Gartner's Data Analytics Maturity Model.pptx

Discover Why Less is More in B2B Research

怎样办理圣地亚哥州立大学毕业证（SDSU毕业证书）成绩单学校原版复制

Sealdah % High Class Call Girls Kolkata - 450+ Call Girl Cash Payment 8005736...

Lecture_2_Deep_Learning_Overview-newone1

Gomti Nagar & best call girls in Lucknow | 9548273370 Independent Escorts & D...

Top profile Call Girls In Indore [ 7014168258 ] Call Me For Genuine Models We...

Digital Transformation Playbook by Graham Ware

Digital Advertising Lecture for Advanced Digital & Social Media Strategy at U...

Kings of Saudi Arabia, information about them

Statistics notes ,it includes mean to index numbers

5CL-ADBA,5cladba, Chinese supplier, safety is guaranteed

7. Epi of Chronic respiratory diseases.ppt

Predicting HDB Resale Prices - Conducting Linear Regression Analysis With Orange

High Profile Call Girls Service in Jalore { 9332606886 } VVIP NISHA Call Girl...

TrafficWave Generator Will Instantly drive targeted and engaging traffic back...

Aspirational Block Program Block Syaldey District - Almora

Gulbai Tekra * Cheap Call Girls In Ahmedabad Phone No 8005736733 Elite Escort...

Top Call Girls in Balaghat 9332606886Call Girls Advance Cash On Delivery Ser...

Top profile Call Girls In Hapur [ 7014168258 ] Call Me For Genuine Models We ...

Introducing Kafka Connect and Implementing Custom Connectors

1. Introducing Kafka Connect and ImplementingCustom Connectors Kobi Hikri Kafka Connect

2. Agenda  What (is Kafka Connect)  The World of Data  Why (do we need it) Kafka Connect  When (will we consider using) Kafka Connect  How Kafka Connect  Setting up the infrastructure  Interacting with Kafka Connect  Implementing a custom source connector  Working with connectors  What will not be covered in this talk (will be covered per request)  Kafka ConnectTransformations  Working with custom schemas

3. What is Kafka Connect  A Kafka integration tool  Allows importing data into Kafka  And exporting data out of Kafka  Runs as a cluster of one or more system processes  Each such process is referred to as a Worker  Workers are grouped together in Worker Groups  Connector Instances are managed by theConnect cluster  Tasks are configured by connectors and are distributed across workers  Data in being pulled by SourceTasks (configured by Source Connectors)  Or Pushed by Sink Tasks (configured by Sink Connectors) Worker Worker Worker Worker Worker Group Worker Group

4. TheWorld of Data Dataisscatteredallover! AJigsawandplentyofglueareneeded

5. Why KafkaConnect  In one word  Integration  Of different data oriented technologies  Of a multitude of changing APIs and communication protocols  In more than one word  Fault tolerant, scalable Integration with various technologies

6. When Kafka Connect  Implementing a Kafka Producer or Consumer within the desired data source / target is impossible or expensive (e.g. the source code is unavailable)  Even when we can modify the source code of the source / target system – but prefer to have a scalable and fault tolerant integration with it (e.g. not to be limited in bandwidth by the direct connection between the technology and our Kafka cluster).

7. How KafkaConnect Demo Time Docker Compose Kafka Connect Rest API Custom Source Connector

Introducing Kafka Connect and Implementing Custom Connectors

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Introducing Kafka Connect and Implementing Custom Connectors

Similar to Introducing Kafka Connect and Implementing Custom Connectors (20)

More from Itai Yaffe

More from Itai Yaffe (20)

Recently uploaded

Recently uploaded (20)

Introducing Kafka Connect and Implementing Custom Connectors