Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Streaming Data Ingest and Processing with
Kafka
You will learn how to
• Realize the value of streaming data
ingest with Kafka
• Turn databases into live feeds for
streami...
Apache Kafka and Stream Processing
About Confluent
• Founded by the creators of Apache Kafka
• Founded September 2014
• Technology developed while at LinkedI...
What does Kafka do? Producers
Consumers
Kafka Connect
Kafka Connect
Topic
Your interfaces to the world
Connected to your s...
Kafka is much more than
a pub-sub messaging system
Before: Many Ad Hoc Pipelines
Search Security
Fraud Detection Application
User Tracking Operational Logs Operational Metri...
After: Stream Data Platform with Kafka
 Distributed  Fault Tolerant  Stores Messages
Search Security
Fraud Detection Ap...
People Using Kafka Today
Financial Services
Entertainment & Media
Consumer Tech
Travel & Leisure
Enterprise Tech
Telecom R...
Common Kafka Use Cases
Data transport and integration
• Log data
• Database changes
• Sensors and device data
• Monitoring...
What is the key challenge?
Making sure all data ends up in the right places
Kafka for Integration
1. Ad-hoc pipelines
2. Extreme processing
3. Loss of metadata
Data Integration Anti-Patterns
Tight Coupling
Agility
Because at the heart of EVERY system…
…there is a LOG,
and Kafka is a scalable and reliable system to manage LOGs
Why is K...
Basic Data Integration Patterns
Push
Pull
Kafka Connect Allows Kafka to Pull Data
Turn the Change Capture Log into a Kafka Topic
16
• Database data is available for any application
• No impact on production
• Database TABLES turned into a STREAM of event...
Confluent Platform with Attunity Connectivity
Confluent Platform
Alerting
Monitoring
Real-time
Analytics
Custom
Applicatio...
Confluent Platform: It’s Kafka ++
Feature Benefit Apache Kafka Confluent Platform 3.0 Confluent Enterprise 3.0
Apache Kafk...
Confluent Control Center
Configures Kafka Connect data pipelines
Monitors all pipelines from end-to-end
Connector Management
Attunity Replicate
Streaming databases into Kafka
About Attunity
Overview
Global operations, US HQ
2000 customers in 65 countries
NASDAQ traded, fast growing
Global Foot...
Attunity Replicate Attunity Compose Attunity Visibility
Universal Data Availability Data Warehouse Automation Data Usage P...
Stream your databases to Kafka with Attunity Replicate:
• Easily – configurable and automated solution, with a few clicks
...
Attunity Replicate architecture
Transfer
TransformFilter
Batch
CDC Incremental
In-Memory
File Channel
Batch
Hadoop
Files
R...
Demand
• Easy ingest and CDC
• Real-time processing
• Real-time monitoring
• Real-time Hadoop
• Scalable to 1000’s applica...
CDC
Attunity Replicate for Kafka - Architecture
MSG
n 2 1
MSG MSG
DataStreaming
Transaction
logs
In memory optimized metad...
"table": "table-name",
"schema": "schema-name",
"op": "operation-type",
"ts": "change-timestamp",
"data": [{"col1": "val1"...
Zero-footprint architecture
Lower impact on IT
• No software agents on
sources and targets for
mainstream databases
• Repl...
Heterogeneous – Broad support for sources and targets
RDBMS
Oracle
SQL Server
DB2 LUW
DB2 iSeries
DB2 z/OS
MySQL
Sybase AS...
Watch the recorded webinar today!
Upcoming SlideShare
Loading in …5
×

Streaming Data Ingest and Processing with Apache Kafka

3,696 views

Published on

Apache™ Kafka is a fast, scalable, durable, and fault-tolerant
publish-subscribe messaging system. It offers higher throughput, reliability and replication. To manage growing data volumes, many companies are leveraging Kafka for streaming data ingest and processing.

Join experts from Confluent, the creators of Apache™ Kafka, and the experts at Attunity, a leader in data integration software, for a live webinar where you will learn how to:

-Realize the value of streaming data ingest with Kafka
-Turn databases into live feeds for streaming ingest and processing
-Accelerate data delivery to enable real-time analytics
-Reduce skill and training requirements for data ingest

The recorded webinar on slide 32 includes a demo using automation software (Attunity Replicate) to stream live changes from a database into Kafka and also includes a Q&A with our experts.

For more information, please go to www.attunity.com/kafka.

Published in: Software
  • Dating direct: ❶❶❶ http://bit.ly/2ZDZFYj ❶❶❶
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Follow the link, new dating source: ♥♥♥ http://bit.ly/2ZDZFYj ♥♥♥
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download Full EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download Full doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THIS can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THIS is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THIS Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THIS the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THIS Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Streaming Data Ingest and Processing with Apache Kafka

  1. 1. Streaming Data Ingest and Processing with Kafka
  2. 2. You will learn how to • Realize the value of streaming data ingest with Kafka • Turn databases into live feeds for streaming ingest and processing • Accelerate data delivery to enable real- time analytics • Reduce skill and training requirements for data ingest
  3. 3. Apache Kafka and Stream Processing
  4. 4. About Confluent • Founded by the creators of Apache Kafka • Founded September 2014 • Technology developed while at LinkedIn • 73%of active Kafka committers Cheryl Dalrymple CFO Jay Kreps CEO Neha Narkhede CTO, VP Engineering Luanne Dauber CMO Leadership Todd Barnett VP WW Sales Jabari Norton VP Business Dev
  5. 5. What does Kafka do? Producers Consumers Kafka Connect Kafka Connect Topic Your interfaces to the world Connected to your systems in real time
  6. 6. Kafka is much more than a pub-sub messaging system
  7. 7. Before: Many Ad Hoc Pipelines Search Security Fraud Detection Application User Tracking Operational Logs Operational Metrics Hadoop Search Monitoring Data Warehouse Espresso Cassandra Oracle
  8. 8. After: Stream Data Platform with Kafka  Distributed  Fault Tolerant  Stores Messages Search Security Fraud Detection Application User Tracking Operational Logs Operational MetricsEspresso Cassandra Oracle Hadoop Log Search Monitoring Data Warehouse Kafka  Processes Streams
  9. 9. People Using Kafka Today Financial Services Entertainment & Media Consumer Tech Travel & Leisure Enterprise Tech Telecom Retail
  10. 10. Common Kafka Use Cases Data transport and integration • Log data • Database changes • Sensors and device data • Monitoring streams • Call data records • Stock ticker data Real-time stream processing • Monitoring • Asynchronous applications • Fraud and security
  11. 11. What is the key challenge? Making sure all data ends up in the right places Kafka for Integration
  12. 12. 1. Ad-hoc pipelines 2. Extreme processing 3. Loss of metadata Data Integration Anti-Patterns Tight Coupling Agility
  13. 13. Because at the heart of EVERY system… …there is a LOG, and Kafka is a scalable and reliable system to manage LOGs Why is Kafka such a great fit?
  14. 14. Basic Data Integration Patterns Push Pull
  15. 15. Kafka Connect Allows Kafka to Pull Data
  16. 16. Turn the Change Capture Log into a Kafka Topic 16
  17. 17. • Database data is available for any application • No impact on production • Database TABLES turned into a STREAM of events • Ready for the next challenge? Stream processing applications What’s next?
  18. 18. Confluent Platform with Attunity Connectivity Confluent Platform Alerting Monitoring Real-time Analytics Custom Application Transformations Real Time Applications Apache Kafka Core Connectors Control Center Clients & Developer Tools Hadoop ERP CRM Data Warehouse RDBMS Data Integration Connectors Database Changes Mobile DevicesloTLogs Website Events Confluent Platform Confluent Platform Enterprise External Product Support, Services and Consulting Kafka Streams Source Sink
  19. 19. Confluent Platform: It’s Kafka ++ Feature Benefit Apache Kafka Confluent Platform 3.0 Confluent Enterprise 3.0 Apache Kafka High throughput, low latency, high availability, secure distributed message system Kafka Connect Advanced framework for connecting external sources and destinations into Kafka Java Client Provides easy integration into Java applications Kafka Streams Simple library that enables streaming application development within the Kafka framework Additional Clients Supports non-Java clients; C, C++, Python, etc. Rest Proxy Provides universal access to Kafka from any network connected device via HTTP Schema Registry Central registry for the format of Kafka data – guarantees all data is always consumable Pre-Built Connectors HDFS, JDBC and other connectors fully Certified and fully supported by Confluent Confluent Control Center Includes Connector Management and Stream Monitoring Support Connection and Monitoring command center provides advanced functionality and control Community Community 24x7x365 Free Free Subscription
  20. 20. Confluent Control Center Configures Kafka Connect data pipelines Monitors all pipelines from end-to-end
  21. 21. Connector Management
  22. 22. Attunity Replicate Streaming databases into Kafka
  23. 23. About Attunity Overview Global operations, US HQ 2000 customers in 65 countries NASDAQ traded, fast growing Global Footprint Data Integration and Big Data Management 1. Accelerate data delivery and availability 2. Automate data readiness for analytics 3. Optimize data management with intelligence
  24. 24. Attunity Replicate Attunity Compose Attunity Visibility Universal Data Availability Data Warehouse Automation Data Usage Profiling & Analytics Move data to any platform Automate ETL/EDW Optimize performance and cost On Premises / Cloud Hadoop FilesRDBMS EDW SAP Mainframe Attunity Product Suite
  25. 25. Stream your databases to Kafka with Attunity Replicate: • Easily – configurable and automated solution, with a few clicks you can turn databases into live feeds for Kafka • Continuously – capture and stream data changes efficiently, in real-time, and with low impact • Heterogeneously – using the same platform for many source database systems (Oracle, SQL, DB2, Mainframe, many more…) Attunity Replicate for Kafka
  26. 26. Attunity Replicate architecture Transfer TransformFilter Batch CDC Incremental In-Memory File Channel Batch Hadoop Files RDBMS Data Warehouse Mainframe Cloud On-prem Cloud On-prem Hadoop Files RDBMS Data Warehouse Kafka Persistent Store
  27. 27. Demand • Easy ingest and CDC • Real-time processing • Real-time monitoring • Real-time Hadoop • Scalable to 1000’s applications • One publisher – multiple consumers Attunity Replicate • Direct integration using Kafka APIs • In-memory optimized data streaming • Support for multi-topic and multi- partitioned data publication • Full load and CDC • Integrated management and monitoring via GUI Kafka and real-time streaming
  28. 28. CDC Attunity Replicate for Kafka - Architecture MSG n 2 1 MSG MSG DataStreaming Transaction logs In memory optimized metadata management and data transport Message broker Message broker Bulk Load MSG n 2 1 MSG MSG DataStreaming T1/P0 T2/P1 T3/P0 Broker 1 M0 M1 M2 M3 M4 M5 M6 M7 M8 M0 M1 M2 M3 M4 M5 M0 M1 M2 M3 M4 M5 M6 M7 T1/P1 T2/P0 Broker 2 M0 M1 M2 M3 M4 M0 M1 M2 M3 M4 M5 M6
  29. 29. "table": "table-name", "schema": "schema-name", "op": "operation-type", "ts": "change-timestamp", "data": [{"col1": "val1"}, {"col2": "val2"}, …., {"colN": "valN"}] "bu_data": [{"col1": "val1"}, {"col2": "val2"}, …., {"colN": "valN"}], Easily create and manage Kafka endpoints Eliminate manual coding • Drag and drop interface for all sources and targets • Monitor and control data stream through web console • Bulk load or CDC • Multi-topic and multi- partitioned data publication Attunity Replicate Command Line
  30. 30. Zero-footprint architecture Lower impact on IT • No software agents on sources and targets for mainstream databases • Replicate data from 100’s of source systems with easy configuration • No software upgrades required at each database source or target Hadoop Files RDBMS EDW Mainframe • Log based • Source specific optimization Hadoop Files RDBMS EDW Kafka
  31. 31. Heterogeneous – Broad support for sources and targets RDBMS Oracle SQL Server DB2 LUW DB2 iSeries DB2 z/OS MySQL Sybase ASE Informix Data Warehouse Exadata Teradata Netezza Vertica Actian Vector Actian Matrix Hortonworks Cloudera MapR Pivotal Hadoop IMS/DB SQL M/P Enscribe RMS VSAM Legacy AWS RDS Salesforce Cloud RDBMS Oracle SQL Server DB2 LUW MySQL PostgreSQL Sybase ASE Informix Data Warehouse Exadata Teradata Netezza Vertica Pivotal DB (Greenplum) Pivotal HAWQ Actian Vector Actian Matrix Sybase IQ Hortonworks Cloudera MapR Pivotal Hadoop MongoDB NoSQL AWS RDS/Redshift/EC2 Google Cloud SQL Google Cloud Dataproc Azure SQL Data Warehouse Azure SQL Database Cloud Kafka Message Broker targets sources
  32. 32. Watch the recorded webinar today!

×