This document discusses enabling real-time analytics in the enterprise. It begins with an overview of the challenges of real-time analytics due to non-integrated systems, varied data types and volumes, and data management complexity. A case study on real-time quality analytics in automotive is presented, highlighting the need to analyze varied data sources quickly to address issues. The Hortonworks/Attunity solution is then introduced using Attunity Replicate to integrate data from various sources in real-time into Hortonworks Data Platform for analysis. A brief demonstration of data streaming from a database into Kafka and then Hortonworks Data Platform is shown.

1. Enabling the Real-Time
Analytical Enterprise
Michael Ger
General Manager, Manufacturing & Automotive,
Hortonworks
Jordan Martz,
Director, Technology Solutions
Attunity
Chris Gambino,
Solutions Engineer
Hortonworks

2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Speakers
Michael Ger
GM, Industrial
Manufacturing and
Automotive Solutions
Hortonworks
Jordan Martz
Director of
Technology Solutions
Attunity
Chris Gambino
Solutions Engineer
Hortonworks

3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
• The Real-Time Analytics Challenge
• Case Study: Real-Time Quality Analytics
• The Hortonworks/Attunity Solution
• Brief Demonstration
• Q & A

4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Connected World
Connected
World
Inventory
Factories
Devices
Vehicles
30X Connected Device growth
by 2020 (Gartner)
3X growth in connected vehicles by
2022 (PWC)
$12M to $209 BILLION growth in
RFID by 2021 (McKinsey)
26.9% CAGR growth in Manufacturing
IoT thru 2020 (Forbes)
Consumers 7X growth in connected population
within last 5 years (United Nations)

5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Driving Digital, Real-Time Business Processes
• Connected Customers,
Vehicles, Devices
• Socially crowd-
sourced requirements
• Digital design and
analysis
• Digital prototypes and
tests (simulations)
• Connected Factories,
Sensors, Devices
• Human-robotic
interaction
• 3D-printing on
demand
• Connected Trucks,
Inventory
• Location, traffic,
weather-aware
distribution
• Real-time inventory
visibility
• Dynamic rerouting
• Connected
Customers, Devices
• Omni- channel
demand sensing
• Real-Time
Recommendations
• Connected Assets
• Remote service
monitoring &
delivery
• Predictive
maintenance
• OTA Updates
Development Manufacturing Distribution Marketing/Sales Service

6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
In Connected World, Time to Insight Is Key
Leverage ALL information within
your Enterprise?
Access the latest, real-time
information?
Analyze and respond immediately to
opportunities and threats?
What if You Could…..

7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Typical Barriers to Success
Non-Integrated Systems
Data Variety
Data Volume
Data Management Complexity and Cost

8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
The Opportunity
The Real-Time Analytical Organization
 Instant visibility to trending issues
 Instant access to data for analysis
Collect and Aggregate Data in Real-Time
Enable Real-Time Insights
 “New” data sources (IOT, Social, CX, etc.)
 Enterprise Transactional Data

9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
CASE STUDY:
REAL-TIME QUALITY ANALYTICS

10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Automotive Quality In The News
RecallsCustomer Satisfaction Government Regulation
Quality Across Entire
Customer Experience
Recalls a
Growing Occurrence
The
Watchful Eye

11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Early Visibility and Resolution is Key….
Exponential increases in warranty costs by lifecycle phase*
• Initial Design $1X
• Simulation Testing $10X
• Prototype Testing $100X
• Initial Production $1,000X
The Rule of 10’s
*Source: Warranty Week
• Warranty Repair $10,000X
• Recall $100,000X
Early Visibility
Minimizes Units
in Field!

12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
What Does it Take to Respond Quickly?
What
Happened?
Why Did It
Happen?
Contain The
Problem

13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Installed
Base
Voice Of
Field
Voice Of
Supplier
Production and
Operations
Voice Of
Factory
Production and
Operations
Voice of
Vehicle
Telematics
Voice Of
Customer
Social + Web
(Forums, Surveys, etc.)
Call Center
Service & Warranty
Answers Require Wide Ranging Data
• Likes
• Dislikes
• Sentiment
• Complaints
• Claims
• Man
• Method
• Material
• Machine
• Man
• Method
• Material
• Machine
• DTC’s
• Performance
Parameters
• VINs
• Configurations
• Locations
Voice Of
Field
What
Happened?
Why Did It
Happen?
Contain The
Problem

14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
From a Variety of Data Source Types
What
Happened?
Why Did It
Happen?
Contain The
Problem
Voice Of
Customer
Social + Web
(Forums, Surveys, etc.)
Call Center
Service & Warranty
Voice of
Vehicle
Voice Of
Factory
Voice Of
Supplier
Production and
Operations
Installed
Base
Telematics
Production and
Operations
Voice Of
Field
SOCIAL, WEB
TRANSACTION
TRANSACTION
TRANSACTION,
IOT
TRANSACTION
IOT
TRANSACTION

15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
• The Real-Time Analytics Challenge
• Case Study: Real-Time Quality Analytics
• The Hortonworks/Attunity Solution
• Brief Demonstration
• Q & A

16. 16
Recommended Architecture

17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hortonworks Data Platform
Reference Architecture - Detailed
HCatalog: Shared Table & User Defined Metadata for All Workloads
Oozie: Orchestrate ProcessingIngest
Security/Governance: Full Stack Authz Policy Definition, Enforcement, Audit
Data Cleansing
Data Transformation
Data Processing
Hive LLAP
Tooling
R
° ° ° ° ° ° ° °
° ° ° ° ° ° ° °
° °
° °
° ° ° ° °
° ° ° ° °
1 ° ° ° ° ° ° ° ° ° ° ° ° ° °
HDFS (Hadoop Distributed File System)/ S3
YARN (Cluster Resource Management)
Sources
Data Science, Machine
Learning
Data
Exploration
Model
Building
Custom Applications
Dashboards
Analytics, BI, Ad-hoc
Exploration
Visualization
& Reporting
Data
Exploration
SAS
Python
Cloudbreak: Provision – via Ambari BluePrints and HDP Cluster on AWS, AutoScale nodes in
Cluster based on Job Metrics
Cubes
CDC
Vehicle
Transactional/
MessagingSystems
Iot:Telematics,Social

18. © 2016 Attunity
Attunity Replicate
2017

19. © 2016 Attunity
Replicate Product Architecture and
Demo

20. © 2016 Attunity
…Across All Major Platforms
SAP
Oracle
DB2 z/OS
DB2 LUW
SQL
Data
Warehouse
Exadata
Teradata
Netezza
Vertica
Actian Vector
Actian Matrix
Hortonworks
Cloudera
MapR
Pivotal
Hadoop
IMS/DB
SQL M/P
Enscribe
RMS
VSAM
Legacy
AWS RDS
Salesforce
Cloud
RDBMS
Oracle
SQL Server
DB2 LUW
MySQL
PostgreSQL
Sybase ASE
Informix
MariaDB
Data
Warehouse
Exadata
Teradata
Netezza
Vertica
Pivotal DB
(Greenplum)
Pivotal HAWQ
Actian Vector
Actian Matrix
Sybase IQ
Hortonworks
Cloudera
MapR
Pivotal
Hadoop
MongoDB
NoSQL
AWS
RDS/Redshift/EC2
Google Cloud SQL
Google Cloud Dataproc
Azure SQL Data
Warehouse
Azure SQL Database
Cloud
Kafka
Message
Broker
targets
Sources
RDBMS
Oracle
SQL Server
DB2 iSeries
DB2 z/OS
DB2 LUW
MySQL
Sybase ASE
Informix

21. © 2016 Attunity
Feeding the Data Lake with Attunity Replicate
Results
4500 applications
DB2 MF SQL Oracle
• Consolidating massive data
volumes for global analytics
• Hadoop Data Lake with Kafka
• Minimizing labor and cost
• Realizing faster insights and
competitive advantage
Fortune 100 auto maker

22. © 2016 Attunity
Feeding the Data Lake with Attunity Replicate
Heterogeneous applications
DB2 MF
SQL
Oracle
•Combining massive data
volumes for global analytics
• SAP HANA, Microsoft Azure
•Standardize ingestion across
the data lake
•Initial use cases include –
warranty analysis,
remanufacturing analysis
Major Construction
Equipment Manufacturer
DB2 LUW IMS
VSAM
Teradata

23. © 2016 Attunity
Data Lake Ingests with Attunity Replicate: On-Prem &
Clouds
Transfer
TransformFilter
Batch
CDC Incremental
In-Memory
File Channel
Batch
Hadoop
Files
RDBMS
Data Warehouse
Mainframe
Cloud
On-prem
Cloud
On-prem
Hadoop
Files
RDBMS
Data Warehouse
Kafka
Persistent Store

24. © 2016 Attunity
Attunity Solutions for SAP
Hadoop & Big Data
Data Warehousing
Database
Cloud
SAP
1. SAP Test Data Management
2. SAP HANA Data Integration
3. Enabling SAP Analytics

25. © 2016 Attunity
Replicate for SAP
TransformFilter
Batch
CDC Incremental
In-Memory
File Channel
Batch
Attunity Replicate for SAP – Data Flow
Persistent Store
Extract relationships for Pool and Cluster Tables
Navigate and
select SAP
business objects
Automated ABAP
Mapping and Change-
Data-Capture for Pool
and Cluster tables
1. Replicate for SAP UI
2. Replicate for SAP RFC Calls
RDBMS
(Oracle, DB2, etc.)
Redo/
Archive
logs
or
Journal
File
----------------
Transparent
Tables
On Premises
Hadoop RDBMS
Data
WarehouseKafka
Cloud

26. © 2016 Attunity
Attunity Replicate
Go agile with modern and automated integration
• No manual coding or scripting
• Automated end-to-end
• Optimized and configurable
Hadoop
Files
RDBMS
EDW
Mainframe
• Target schema creation
• Heterogeneous data type
mapping
• Batch to CDC transition
• DDL change propagation
• Filtering
• Transformations
Hadoop
Files
RDBMS
EDW
Kafka

27. © 2016 Attunity
Zero-footprint Architecture
• Lower impact on IT
• No software agents on
sources and targets for
mainstream databases
• Replicate data from 100’s
of source systems with
easy configuration
• No software upgrades
required at each
database source or target
Hadoop
Files
RDBMS
EDW
Mainframe
• Log based
• Source specific optimization
Hadoop
Files
RDBMS
EDW
Kafka

28. © 2016 Attunity
Hortonworks HDF
+ Attunity Replicate

29. © 2016 Attunity
The Connected Data Platform

30. © 2016 Attunity
The Connected Data Architecture & Attunity
SOURCES
OLTP, ERP,
CRM Systems
Documents,
Emails
Web Logs,
Click Streams
Social
Networks
Machine
Generated
Sensor
Data
Geolocation
Data
Data Integration
& Ingests
Attunity Replicate for HDP and HDF
Accelerate time-to-insights by delivering
solutions faster, with fresher data, from
many sources
- Automated data ingest
- Incremental data ingest (CDC)
- Broad support for many sources

31. © 2016 Attunity
In Memory and File Optimized Data
Transport
Enterprise-class
CDC for Data At Rest and Data In Motion
R1
R1
R2
R1
R2
R1
R2Batch
CDC
Data
Warehouse
Ingest-Merge
SQL
n 2 1
SQL SQL
Transactional
CDC
Message
Encoded
CDC
Data Sources
Attunity Replicate – Change Processing
CDC
Many Databases
and Data
Warehouses
....

32. © 2016 Attunity
Demand
• Easy ingest + CDC
• Real-time processing
• Real-time monitoring
• Real-time Hadoop
• Scalable to 1000’s applications
• One publisher – multiple
Consumers
Attunity - Replicate
• Direct integration using Kafka
APIs
• In-memory optimized data
streaming
• Support for multi-topic and multi-
partitioned data publication
• Full load and CDC
• Integrated management and
monitoring via GUI
Kafka and Real-time Streaming

33. © 2016 Attunity
T1/P0
T2/P1
T3/P0
Broker 1
Attunity Replicate for Kafka - Architecture
M0 M1 M2 M3 M4 M5 M6 M7 M8
M0 M1 M2 M3 M4 M5
M0 M1 M2 M3 M4 M5 M6 M7
T1/P1
T2/P0
Broker 2
M0 M1 M2 M3 M4
M0 M1 M2 M3 M4 M5 M6

34. 34 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
• The Real-Time Analytics Challenge
• Case Study: Real-Time Quality Analytics
• The Hortonworks/Attunity Solution
• Brief Demonstration
• Q & A

35. © 2016 Attunity
CDC
Demo: Data Streaming into Kafka  HDF 
HDP
MSG
n 2 1
MSG MSG
Data Streaming
Transaction
logs
In memory optimised metadata
management and data transport
Bulk
Load
MSG
n 2 1
MSG MSG
Data Streaming
Message
broker
Message
broker
Example
DB:

36. © 2016 Attunity
Demo: Data Streaming into Kafka  HDF 
HDP

37. 37 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

38. 38 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

39. 39 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

40. 40 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

41. 41 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Thank You!
Q/A
Hortonworks.com
Attunity.com