Bring your SAP and Enterprise Data to Hadoop, Apache Kafka and the Cloud

Bring Your SAP and Enterprise
Data to Hadoop, Apache Kafka
and the Cloud
Lessons from Fortune 100 Companies about Data Ingestion
Ted Orme – VP Technology EMEA
Ted.Orme@Attunity.com

CONFIDENTIAL INFORMATION © 2017 Attunity
Technology Investment Areas 2016 2015 2014
BI/Analytics 1 1 1
Cloud 2 3 5
Mobile 3 5 3
Digitization/Digital Marketing 4 6 7
Infrastructure and Data Center 5 2 2
ERP 6 4 4
Security 7 7 7
Industry Specific Applications 8 10 9
CRM 9 9 10
Networking/Voice-Data Comm’s 10 8 6
Top CIO Priorities:
Sources: Gartner CIO Surveys 2013-2015
Big Data adoption
Migration & Analytics

Data Integration for Analytics and IoT

• Corporate Data Lake Initiative for Real-time Compliance
• Reduce fraudulent activity, reputational damage
Global Bank
• IoT Data Lake for New Analytics
• Predictive Maintenance
Global Auto Maker
• IoT Data Lake integrated with SAP data
• Improve food production efficiencies and fleet maintenance
Global Food Processing
• Data Lake for real-time access to online and legacy data
• Increase revenues with dynamic policy changes
Global Insurance
• Data Lake for Design Analytics with Real Time Data
• Expects 3x improvement in manufacturingAirline Manufacturer
Examples – Fortune 100 Customers
Building Data Lakes for Analytics and IoT

• 100s to 1000s of Data Sources
• Business and Machine data
Analyze
Everything
• On-premise or in the Cloud
• In DB, DW, Hadoop, In-Memory, etc.
Analyze
Anywhere
• Capture new/changing data
• Process/stream in motion
Analyze in
Real-time
• Capture new/changing data
• Process/stream in motion
Analyze in
Real-time
‘Be Prepared’ – build architecture so you
can:

Transfer
TransformFilterBatch
CDC Incremental
In-Memory
File Channel
Batch
Attunity Replicate
Hadoop
Files
RDBMS
Data Warehouse
Mainframe
Cloud
On-prem
Cloud
On-prem
Hadoop
Files
RDBMS
Data Warehouse
Kafka
Management
Automation

Needs & Challenges in SAP Analytic Environments
Real-time access to SAP
data for reporting and
analytics
Flexibility to access and
analyse SAP data across
different BI tools and
environments like Cloud,
other DW & Hadoop
Remove lock-in and
dependency on SAP BW
Reduce complex and
costly projects to
integrate SAP data
Decode Complex SAP
data for business users
for common data model
integration

Use Cases
1. Data Lakes: Live SAP Data Ingest for Hadoop Data Lakes / Kafka
2. Cloud Analytics: Live SAP Data Ingest/Migration for Cloud
Analytics
3. ODS: Create an SAP ODS (operational data store) for Real-time BI
4. Real-time data warehousing: with SAP application data

Configurable, Pre-defined
Automation
• Intuitive and easy to use web-
interface
• Simple to configure and manage
replication tasks
• Single interface for any
source to target
• Easy management with alerts
notifications
Intuitive User Experience

Replicate for SAP – App-Level Replication
• Business object level
metadata
• Real-time CDC aligned with
business object
• Unpacks pooled and
clustered tables

In Memory and File Optimised Data Transport
Enterprise-class CDC for SAP
Flexible and optimized CDC
options
• Transactions applied in real-
time and in order
• Changes applied in
optimised batches
• Integration with data
warehouse native loaders to
ingest and merge
• Message encoded
streaming of changes (for
Kafka message broker)
R1
R1
R2
R1
R2
R1
R2Batch CDC
Data Warehouse
Ingest-Merge
SQL
n 2 1
SQL SQL
Transactional CDC
Message Encoded
CDC
2017 Attunity

SAP certified agent to decode complex SAP data structures with metadata for replication
Automated SAP Data selection via UI from pool, clustered or indexed tables
Enable data transformation during replication for all SAP data sets
Support for SAP ERP (ECC), CRM, and custom SAP modules
Maintaining SAP data integrity during replication and during CDC
Minimal performance impact on SAP system by adding support for RFC calls
Key Enablers for SAP Environments

Replicate for SAP
TransformFilter
Batch
CDC Incremental
In-Memory
File Channel
Batch
Architecture
Persistent Store
Extract relationships for Pool and Cluster Tables
Navigate and
select SAP
business objects
Automated ABAP
Mapping and Change-
Data-Capture for Pool
and Cluster tables
1. Replicate for SAP UI
2. Replicate for SAP RFC Calls
RDBMS
(Oracle, DB2, etc.)
Redo/
Archive
logs
or
Journal
File
----------------
Transparent
Tables
On Premises
Hadoop RDBMS
Data
WarehouseKafka
Cloud

SAP Platforms Supported by Attunity Replicate
*Replicate 5.1.2 supports SAP as of November
Supported SAP Versions Supported DBs for SAP
• Primarily SAP ECC 6.0 + all
EhP levels
• Can also support ECC 5.0, 4.7
Enterprise and 4.6C

SAP Applications Supported by Attunity
Replicate
ERP / ECC (Enterprise Resource Planning / ERP Core Components)*
CRM (Customer Relationship Management)
SRM (Supplier Relationship Management)
GTS (Global Trade System)
MDG (Master Data Governance)
* All modules supported but HR

• Ability to realise new analytics value in native target schema aka Hive,
Kafka, Cloud
• SAP data access via BI reports running directly against the replicated
data with logical naming
• End-to-end automation from SAP data selection to loading data with
transformations via intuitive “Click to Load” interface
• Faster performance with less overhead on SAP system by limiting
RFC calls
• Real-time change data capture (CDC) applied to SAP data sets
Key Value of Attunity Replicate for SAP
Rapid Data Access for More Users and Systems

SOURCES
OLTP, ERP,
CRM Systems
Documents,
Emails
Web Logs,
Click Streams
Social
Networks
Machine
Generated
Sensor
Data
Geolocation
Data
Data Integration & Ingest
Attunity Replicate for HDP and HDF
Accelerate time-to-insights by delivering solutions
faster, with fresher data, from many sources
- Automated data ingest
- Incremental data ingest (CDC)
- Broad support for many sources
Integrating SAP Data into the Hortonworks
Connected Data Platform

• Accelerate data delivery across enterprise and cloud
• Empower rapid utilization of data by the business
• Continually optimize with intelligent insight
Attunity Corporate Overview
Over 2000 Customers in 65 Countries
Financial Services Manufacturing / Industrials GovernmentHealth Care
Technology / Telecommunications Other Industries
Enterprise Data Management
On Premise | Cloud | Across Platforms
Global Organization
USA
EMEA
APAC

Attunity – Trusted Technology and Partner
 Trusted by Microsoft with 3 OEMs, bundled inside SQL Server
 Trusted by Amazon (AWS) with technology licensing for cloud migration service
 Trusted by IBM and Oracle with respective OEMs of Attunity technology
 Trusted by Teradata as a reseller for Data warehouse and Hadoop market
 Trusted by HP as a reseller for Data warehouse and analytics market
 Trusted by Accenture, Capgemini and Cognizant as SI partners
 Trusted by Hortonworks, Cloudera and MapR for Hadoop solutions
 Trusted by over 2000 customers in over 65 countries

Data Ingest for Hadoop Data Lakes
Attunity
Replicate
Batch
CDC Incremental
Batch
Cloud
On-
prem
Cloud
On-
prem
Persistent Store
Hadoop
Files
RDBMS
Data Warehouse
Mainframe

Technical Time - Cluster Table example in
SAP
RFBLG is a table cluster for Accounting Documents. It has a BLOB
column that holds the BSEG, BSEC, BSED, BSES and BSET cluster
tables within it.
A cluster table is a special hash-keyed area of the database that
allows for more efficient storage of redundant data. You cannot have
a cluster table (like BSEG) within an OpenSQL join statement.
There used to be an Oracle limitation. Oracle could only handle a
certain number of columns in a single table, but BSEG was wider, so
SAP came up with the Cluster design

RFBLG copied Oracle to SQL
Blob Column

BSEG copied by R4SAP into SQL
SAP Table Meta Data – Field Names
BSEG is part of the Blob from RFBLG
BSEG does not exist on its own in the DB

Bring your SAP and Enterprise Data to Hadoop, Apache Kafka and the Cloud

More Related Content

What's hot

Similar to Bring your SAP and Enterprise Data to Hadoop, Apache Kafka and the Cloud

More from DataWorks Summit/Hadoop Summit

Recently uploaded

Bring your SAP and Enterprise Data to Hadoop, Apache Kafka and the Cloud