Real-time Data Pipelines with SAP and Apache Kafka

Real-time Data Pipelines
with SAP and Apache
Kafka

© 2017 Attunity, Accenture Analytics and Confluent
Amiya Chand
Sr. Manager, Accenture Digital | SAP Data
& Analytics Lead, Accenture
David Tucker
Director Partner Engineering, Confluent
Jordan Martz
Director of Technology Solutions, Attunity
Guest Speakers

• Accenture Analytics - Business in the moment and
imperatives
• Confluent - Apache Kafka
• Attunity - Migrating Data to a Apache Kafka
• All - Q&A
Agenda

• Real-time Data Pipelines with SAP
and Apache Kafka

News from the frontline
Significant investments have been made in traditional data integration and platforms which have
limitations in storage, data latency and performance. These complex systems inhibit ‘speed to
value’ and ‘speed to deploy’
The digitization of everything has the potential to provide organizations with “data about
everything”. Organizations that are able to effectively harness the power of this data to drive insights
actions (at speed) will have a distinct advantage over their competition.
In order to effectively address these challenges and to capitalize on “new” data sources to gain a competitive
advantage, organizations will have to ingest data at the speed of business.
To unlock value within their data, organizations must start treating data as a supply chain,
enabling it to flow more easily through the organization.

Market trends for data & analytics
Complex data-driven environment with significant opportunities to create business value
DEMOCRATIZATION
OF DATA AND DATA
DISCOVERY
NEW DATA
SOURCES
CHANGING SKILLS
REQUIREMENTS
FOCUS ON
ADVANCED
ANALYTICS
BIG DATA AND
HYBRID
ARCHITECTURES
Self-service
analytics, multi-
channel delivery
Addition of social
media, Internet of
Things, unstructured
content, etc.
Evolving patterns,
data-driven point-of-
interaction decisions
shifting to the front
line
New forms, more
complex
architectures on
Premise or Cloud
Pictorial story-
telling, customers
going digital

The changing landscape in SAP data &
analytics
SAP systems are evolving from a ‘system of record’ to ‘system of innovation’
System of Record
System of Insights
System of Innovation
SpeedandAgility
Level of Integration between People, Process and Platform
Pre 2011
Traditional SAP ERP and BI
Reporting platform on-premise
2013-15
Evolution of SAP HANA as a high
performance data base and Analytics
Engine on-premise / Cloud
SAP BW
Any
Database
Enterprise
BI
Any
Database
Any Database
Enterprise
BI
Data
Visualiza
-tion
Advanced
Analytics
SAP ERP
Any
Database
BW /
Native
HANA
HANA
SAP ERP
Any
Database
BW /
Native
HANA
S4
Innovation
Suite on
HANA
Enterprise
BI
Data
Visualiza-
tion
Advanced
Analytics
BOBJ
Cloud
HANA Hadoop/ Any database
2015+
Maturity of SAP HANA as the
Digital Platform on-premise /
Cloud / As-a-Service
Faster insights to
action
1
Bridge the digital
divide
2
Accelerate data
integration
3
BusinessImperative

Enabling the insight-driven enterprise
DataIngestion
DataAccess
Point
Analytics
&
Insights
SourceSystems
Transactional
systems
Business
Applications
Unstructured
Sources
Real time
Batch
DataMarketplace Data Warehouse/ Data
Mart
Data lake
Operational
Reporting &
Self Service
Data Science lab
Cross
Pollination
Data
Exploration
Guided
Analytics &
Advanced
Analytics
Analytics Engine
A POV and Reference Architecture for enabling the ‘insight driven enterprise’
Load to any
target
Enable real-
time analytics
Ingest/replicate
data on demand
Speed to Data
Ingest any data at
the frequency
needed by
business
Speed to Insight
Access to any
data, anywhere
and anytime
Speed to Value
Integrated
platform that is
scalable and
flexible
Data Ingestion
An abstract layer that allows
various data elements – type &
latency to flow into the storage
environment
Data Marketplace
Includes the components
that allow to store &
transform raw data into
information
Data Access
Semantic layer that allows
access to the data for end
consumption – interactive,
blending, low latency

How organizations handle data flow: a giant mess
Data
Warehouse
Hadoop
NoSQL
Oracle
SFDC
Logging
Bloomberg
…any sink/source
Web Custom Apps Microservices Monitoring Analytics
…and more
OLTP
ActiveMQ
App App
Caches
OLTP OLTPAppAppApp

Apache Kafka™: a distributed streaming
platform
Apache Kafka
Offline Batch (+1 Hour)Near-Real Time (>100s ms)Real Time (0-100 ms)
Data
Warehouse
Hadoop
NoSQL
Oracle
SFDC
Twitter
Bloomberg
…any sink/source …any sink/source
…and more
Web Custom Apps Microservices Monitoring Analytics

Over 35% of Fortune 500’s are using Apache Kafka™
6 of top 10
Travel
7 of top 10
Global banks
8 of top 10
Insurance
9 of top 10
Telecom

Confluent: a streaming platform based on Apache
Kafka™
Database
Changes
Log Events loT Data Web Events …
CRM
Data Warehouse
Database
Hadoop
Data
Integration
…
Monitoring
Analytics
Custom Apps
Transformations
Real-time
Applications
…
Apache Open Source Confluent Open Source Confluent Commercial
Confluent Platform
Confluent Platform
Apache Kafka™
Core | Connect| Streams
Data Compatibility
Schema Registry
Monitoring & Administration
Confluent Control Center
Operations
Replicator | Auto Data Balancing
Development and Connectivity
Clients | Connectors | REST Proxy

Kafka adoption spans many industries
Financial Services Enterprise Tech Consumer Tech
Entertainment & Media Telecom Retail Travel & Leisure

Apache KafkaTM Connect API – streaming data capture
JDBC
Mongo
MySQL
Elastic
Cassandra
HDFS
Kafka Connect API
Kafka Pipeline
Connector
Connector
Connector
Connector
Connector
Connector
Sources Sinks
Fault tolerant
Manage hundreds of data
sources and sinks
Preserves data schema
Part of Apache Kafka
project
Integrated within Confluent
Platform’s Control Center

Kafka Connect API -part of the Apache Kafka Project
Connect any source to any target system
Integrated
• 100% compatible with Kafka v0.9 and higher
• Integrated with Confluent’s Schema Registry
• Easy to manage with Confluent Control
Center
Flexible
• 40+ open source connectors available
• Easy to develop additional connectors
• Flexible support for data types and formats
Compatible
• Maintains critical metadata
• Preserves schema information
• Supports schema evolution
Reliable
• Automated failover
• At-least-once guaranteed
• Balances workload between nodes

Architecture of Kafka Streams API, a Part of Apache Kafka
Kafka
Streams
API
Producer
Kafka Cluster
Topic TopicTopic
Consumer Consumer
Key benefits
• No additional cluster
• Easy to run as a service
• Supports large aggregations and joins
• Security & permissions fully integrated from
Kafka
Example Use Cases
• Microservices
• Continuous queries
• Continuous transformations
• Event-triggered processes

Kafka Streams API: easiest way to process data in
Apache Kafka
Example Use Cases
• Microservices
• Large-scale continuous queries and transformations
• Event-triggered processes
• Reactive applications
• Customer 360-degree view, fraud detection, location-based
marketing, smart electrical grids, fleet management, …
Key Benefits of Apache Kafka’s Streams API
• Build Apps, Not Clusters: no additional cluster required
• Elastic, highly-performant, distributed, fault-tolerant, secure
• Equally viable for small, medium, and large-scale use cases
• “Run Everywhere”: integrates with your existing deployment
strategies such as containers, automation, cloud
Your App
Kafka
Streams API

Confluent Control Center
• Monitor all pipelines end-
to-end
• Manage and configure
Kafka connectors
• Administer Kafka cluster
and monitor overall
operational health

Why Confluent? More than just enterprise software
Confluent Platform Professional Services Enterprise Support
Complete support across the entire adoption lifecycle
Kafka Training

SAP Data Integration for Apache
Kafka

Attunity – Corporate Overview
Data Integration & Big Data Management Software
Accelerate data delivery and availability
Automate data readiness for analytics
Optimize data management with intelligence
Hadoop & Big Data
Databases & Data Warehouses
On premise & in the Cloud
Solutions for: Global OfficesOverview
2000 customers in 65 countries
250 people and growing
NASDAQ traded (ATTU)

Load to any
target
Enable real-
time analytics
Ingest/replicate
data on demand
Traditional Data Integration Requirements

UniversalReal-TimeSimpler
Attunity Replicate
Universal Platform for Data Replication/Ingestion
#1 independent provider of streaming CDC

Ingest Data Real-time and at Scale into Hadoop
• Stream data with CDC from any major source
into HDFS or Hive
 Data then available to be used in Hbase or by Spark
• Automatically generate schema in HCatalog
• Leverage same UI and process as for other
platforms
• Integrate with native APIs
 No external adapters or custom code requiredHadoop Data Lake

Attunity Replicate Architecture
Attunity
Replicate
Batch
CDC Incremental
Batch
Cloud
On-
prem
Cloud
On-
premPersistent Store
Files
RDBMS
Mainframe
EDW

Attunity Replicate Server
TransformFilter
Batch
CDC Incremental
In-Memory
File Channel
Batch
Attunity Replicate for SAP
Persistent Store
Extract relationships for Pool and Cluster Tables
RDBMS
(Oracle, DB2, etc.)
Redo/
Archive
logs
or
Journal
File
----------------
Transparent
Tables
On Premises
Hadoop RDBMS
Data
WarehouseKafka
Cloud
Attunity Replicate Agent
for SAP
SAP ECC
(Enterprise Central
Component)
Data Model Mapping
Pool/Cluster table RFC

Easier Integration with Attunity Replicate
Go Agile with Automated Processes
• No manual coding or scripting
• Automated end-to-end
• Optimized and configurable
• Target schema creation
• Heterogeneous data type
mapping
• Batch to CDC transition
• DDL change propagation
• Filtering
• Transformations
Hadoop
File
s
RDBMS
Mainframe
Hadoop
Files
RDBMS
Kafka
EDW EDW

Easier Integration with Attunity Replicate
Zero Footprint Architecture
• CDC identifies source updates by
scanning change logs
• No software agents required on
sources or targets
• Minimizes administrative
overhead
• CDC based on change logs
• No agents on source or target
Low Infrastructure Impact
Hadoop
File
s
RDBMS
Mainframe
Hadoop
Files
RDBMS
Kafka
EDW EDW

Universal Integration with Attunity Replicate
Rapidly Move Data Across Complex Hybrid Environments
• Source
 Log-based CDC
 Batch APIs
• WAN
 Encrypted multi-pathing
• Target
 Transactional/batch CDC options
 Native loader integration
TargetsSources
On Premises
Cloud Platform
HadoopRDBMS
Data Warehouse
Hadoop
RDBMS
Data Warehouse
Optimized for…

Demo

Real-time Data Pipelines with SAP and Apache Kafka

More Related Content

What's hot

Similar to Real-time Data Pipelines with SAP and Apache Kafka

Recently uploaded

Real-time Data Pipelines with SAP and Apache Kafka