The document discusses the optimization of enterprise data warehouses (EDWs) using Hortonworks' Connected Data Platforms, emphasizing the transformation from traditional EDWs to more agile, real-time analytics with Apache Hadoop. It identifies challenges in current EDW implementations, including high costs, data integration difficulties, and the need for skilled practitioners, while presenting solutions that leverage open-source technology for cost-effective and efficient data processing. Additionally, it highlights case studies showcasing the benefits of using optimized EDW solutions for improved business intelligence and operational efficiency.

1. 1 © Hortonworks Inc. 2011 –2016. All Rights Reserved1 © Hortonworks Inc. 2011 –2017. All Rights Reserved

2. 2 © Hortonworks Inc. 2011 –2016. All Rights Reserved
How Customers are Optimizing their EDW
for Fast, Secure, Cost Effective Actionable Insights
Wei Wang
Sr. Director, Product Marketing
Paige Roberts
Big Data Product Manager
Speakers:

3. 3 © Hortonworks Inc. 2011 –2016. All Rights Reserved
Actionable
Insights
Human
Connections
Balanced
Supply Chains
New Products
& Services
Operational
Efficiencies
Data Is The New Currency

4. 4 © Hortonworks Inc. 2011 –2016. All Rights Reserved
The Old Way
Operational Efficiencies
Hierarchical
Historical
Highly Aggregated
One-size-fits-all
The New Way
Engagement and Innovation
Multi-structured
Predictive
Agile & Real-time
Context Sensitive
Transforming The Enterprise Data Warehouse (EDW)

5. 5 © Hortonworks Inc. 2011 –2016. All Rights Reserved
Hortonworks Connected Data Platforms and Solutions
Hortonworks
Connection
Hortonworks Solutions
Enterprise Data
Warehouse Optimization
Cyber Security and
Threat Management
Internet of Things
and Streaming Analytics
Hortonworks Connection
Subscription Support
SmartSense
Premier Support
Educational Services
Professional Services
Community Connection
Cloud
Hortonworks Data Cloud
AWS HDInsight
Data Center
Hortonworks Data Suite
HDFHDP

6. 6 © Hortonworks Inc. 2011 –2016. All Rights Reserved
The Solution for Enterprise Data Warehouse Optimization
Dramatic Cost Reductions Through the
Power of Open Source and Apache Hadoop
Reduce cost of your EDW Implementation by
augmentation or replacement of ETL processes
Deploy Business Intelligence on Hadoop
Enablement Business users (via a desktop
solution) see aggregate data, drill up, drill down
through business data models.
Enrich with Unstructured Data Support
Deliver support for photos, video, text files to
enrich your analytics

7. 7 © Hortonworks Inc. 2011 –2016. All Rights Reserved
Legacy EDW vs. EDW Optimization Solution with Connected Data Platforms

8. 910
920
930
940
950
960
970
980
990
1,000
Difficulty
transforming data
into a suitable
form for analysis.
Difficulty
integrating
Big Data with
existing
infrastructure.
Difficulty
merging
multiple,
disparate
data sources.
Lack of skilled
Big Data
practitioners.
Difficulty
maintaining
application
performance for
large volume of
concurentusers.
Implementing the Modern Data Architecture Isn’t Easy
8
Source: Wikibon Big Data Analytics Adoption Survey, 2014-2015
Syncsort Confidential andProprietary - do not copy or distribute
Of the selected technology-related
barriers to realizing the full value of your
Big Data Analytics, please rank the top 3.

9. 910
920
930
940
950
960
970
980
990
1,000
Difficulty
transforming data
into a suitable
form for analysis.
Difficulty
integrating
Big Data with
existing
infrastructure.
Difficulty
merging
multiple,
disparate
data sources.
Lack of skilled
Big Data
practitioners.
Difficulty
maintaining
application
performance for
large volume of
concurentusers.
Implementing the Modern Data Architecture Isn’t Easy
9
Additional Challenges
• Long, drawn-out development cycles
Source: Wikibon Big Data Analytics Adoption Survey, 2014-2015
9Syncsort Confidential andProprietary - do not copy or distribute
Of the selected technology-related
barriers to realizing the full value of your
Big Data Analytics, please rank the top 3.

10. 910
920
930
940
950
960
970
980
990
1,000
Difficulty
transforming data
into a suitable
form for analysis.
Difficulty
integrating
Big Data with
existing
infrastructure.
Difficulty
merging
multiple,
disparate
data sources.
Lack of skilled
Big Data
practitioners.
Difficulty
maintaining
application
performance for
large volume of
concurentusers.
Implementing the Modern Data Architecture Isn’t Easy
10
Additional Challenges
• Long, drawn-out development cycles
• Rapidly changing technology presents a
moving target, forcing constant re-design
Source: Wikibon Big Data Analytics Adoption Survey, 2014-2015
10Syncsort Confidential andProprietary - do not copy or distribute
Of the selected technology-related
barriers to realizing the full value of your
Big Data Analytics, please rank the top 3.

11. 910
920
930
940
950
960
970
980
990
1,000
Difficulty
transforming data
into a suitable
form for analysis.
Difficulty
integrating
Big Data with
existing
infrastructure.
Difficulty
merging
multiple,
disparate
data sources.
Lack of skilled
Big Data
practitioners.
Difficulty
maintaining
application
performance for
large volume of
concurentusers.
Implementing the Modern Data Architecture Isn’t Easy
11
Additional Challenges
• Long, drawn-out development cycles
• Rapidly changing technology presents a
moving target, forcing constant re-design
• Difficulty accessing and integrating legacy
and new data sources including the
mainframe
Source: Wikibon Big Data Analytics Adoption Survey, 2014-2015
11Syncsort Confidential andProprietary - do not copy or distribute
Of the selected technology-related
barriers to realizing the full value of your
Big Data Analytics, please rank the top 3.

12. 910
920
930
940
950
960
970
980
990
1,000
Difficulty
transforming data
into a suitable
form for analysis.
Difficulty
integrating
Big Data with
existing
infrastructure.
Difficulty
merging
multiple,
disparate
data sources.
Lack of skilled
Big Data
practitioners.
Difficulty
maintaining
application
performance for
large volume of
concurentusers.
Implementing the Modern Data Architecture Isn’t Easy
12
Additional Challenges
• Long, drawn-out development cycles
• Rapidly changing technology presents a
moving target, forcing constant re-design
• Difficulty accessing and integrating legacy
and new data sources including the
mainframe
• Functionality gaps in security, governance,
and compliance
Source: Wikibon Big Data Analytics Adoption Survey, 2014-2015
12Syncsort Confidential andProprietary - do not copy or distribute
Of the selected technology-related
barriers to realizing the full value of your
Big Data Analytics, please rank the top 3.

13. 910
920
930
940
950
960
970
980
990
1,000
Difficulty
transforming data
into a suitable
form for analysis.
Difficulty
integrating
Big Data with
existing
infrastructure.
Difficulty
merging
multiple,
disparate
data sources.
Lack of skilled
Big Data
practitioners.
Difficulty
maintaining
application
performance for
large volume of
concurentusers.
Implementing the Modern Data Architecture Isn’t Easy
13
Additional Challenges
• Long, drawn-out development cycles
• Rapidly changing technology presents a
moving target, forcing constant re-design
• Difficulty accessing and integrating legacy
and new data sources including the
mainframe
• Functionality gaps in security, governance,
and compliance
Source: Wikibon Big Data Analytics Adoption Survey, 2014-2015
13Syncsort Confidential andProprietary - do not copy or distribute
Big data integration
is complicated!
Of the selected technology-related
barriers to realizing the full value of your
Big Data Analytics, please rank the top 3.

14. 14 © Hortonworks Inc. 2011 –2016. All Rights Reserved
Hortonworks EDW Optimization Solution Components
Hadoop
Scalable Storage and Compute
Hive LLAP
High Performance SQL Data Mart
Fast, scalable SQL analytics
Intelligent in-memory caching

15. 15 © Hortonworks Inc. 2011 –2016. All Rights Reserved
Hortonworks EDW Optimization Solution Components
Syncsort
High-Performance
Data Movement
Hadoop
Scalable Storage and Compute
Hive LLAP
High Performance SQL Data Mart
Source Data
Systems
Fast, scalable SQL analytics
Intelligent in-memory caching
High performance data import
from all major EDW platforms

16. 16 © Hortonworks Inc. 2011 –2016. All Rights Reserved
Hortonworks EDW Optimization Solution Components
Syncsort
High-Performance
Data Movement
Hadoop
Scalable Storage and Compute
Hive LLAP
High Performance SQL Data Mart
AtScale Intelligence Platform
OLAP Cubes for Higher Performance
Source Data
Systems
Fast, scalable SQL analytics
Intelligent in-memory caching
Define OLAP cubes for 10x faster queries
Unified semantic layer for all BI tools
High performance data import
from all major EDW platforms
Pre-aggregated
data
... Or, full-fidelity
data

17. 17 © Hortonworks Inc. 2011 –2016. All Rights Reserved
EDW Optimization: Fast BI on Hadoop
à The Problem:
– Proprietary EDW systems were adopted for
Fast BI and deep slice-and-dice analytics, but
EDW prices are unsustainably high.
à The Solution:
– Interactive SQL is a reality on Hadoop today.
– AtScale Intelligence Platform adds OLAP
capabilities for deep drilldown at scale.
à The Result:
– Query terabytes of data in seconds.
– Connect your favorite BI tools like Tableau and
Excel through SQL and MDX interfaces.
– The EDW Optimization Solution is tailor-made
to deliver Fast BI on Hadoop.
ETL/ELT
DATA
MART
DATA
LANDING &
DEEP
ARCHIVE
CUBE
MART
END USER
APPLICATIONS
APPLICATIONS
APPLICATIONS
END USERS
AND APPS
EDW OPTIMIZATION SOLUTION

18. 18 © Hortonworks Inc. 2011 –2016. All Rights Reserved
EDW Optimization: ETL Offload
à The Problem:
– EDWs consume between 50% and 90% of
CPU just on ETL/ELT tasks.
– These jobs interfere with more business-
critical tasks like BI and advanced analytics.
à The Solution:
– Hive and HDP deliver ETL that scales to
petabytes.
– Syncsort DMX-h for simple drag-and-drop ETL
workflows.
– Economical scale-out processing on
commodity servers.
à The Result:
– Better SLAs for mission-critical analytics.
– Limit EDW expansion or retire old systems.
ETL/ELT
DATA
MART
DATA
LANDING &
DEEP
ARCHIVE
CUBE
MART
END USER
APPLICATIONS
APPLICATIONS
APPLICATIONS
END USERS
AND APPS
EDW OPTIMIZATION SOLUTION

19. 19 © Hortonworks Inc. 2011 –2016. All Rights Reserved
EDW Optimization: Active Archive
à The Problem:
– Increasing data volumes and cost pressure
force data to be archived to tape.
– Archived data not available for analytics, or
must be retrieved at great expense.
à The Solution:
– Adopting Hadoop delivers cost per terabyte
on par with tape backup solutions.
– Data in Hadoop can be analyzed by all major
BI tools, allowing analytics on archive data.
à The Result:
– Data always available for analytics.
– Store years of data rather than months.
ETL/ELT
DATA
MART
DATA
LANDING &
DEEP
ARCHIVE
CUBE
MART
END USER
APPLICATIONS
APPLICATIONS
APPLICATIONS
END USERS
AND APPS
EDW OPTIMIZATION SOLUTION

20. Powering the Connected Data
Platform With EDW Optimization
Paige Roberts
Big Data Product Manager
@RobertsPaige

21. Goals of the Modern Data Architecture
•Centralize all your data
•Turn raw data into insights
•Maintain governance, compliance and
security standards
•Eliminate complexities within IT
21

22. Syncsort Strategic Focus on Big Data & Hadoop
Light footprint
Self-tuning
engine
Single install. No
3rd party
dependencies
World-class data processing,
mainframe expertise
JIRA:
MAPREDUCE-2454
MAPREDUCE-4807
MAPREDUCE-4049
MAPREDUCE-5455
HIVE-8347
SQOOP-1272
PARQUET-134
Spark-packages
and more!
|
22
Syncsort Confidential andProprietary - do not copy or distribute
Ongoing Contributions to the
Open Source Community1
Leverage Syncsort Technology
Innovations & Mainframe Heritage2
Strong Partnerships with Strategic
Big Data & Hadoop Players
1
3
22

23. Ã Connect to virtual any data source, including
mainframe and MPP databases.
à Move data into and out of Hadoop up to 6x
faster without the need for manual scripts.
à Develop ETL processes without writing code.
à Automatically optimize Hadoop performance
and scalability for ETL operations.
à Fully certified and integrated with
Hortonworks Data Platform and HDF
à Secure – Kerberos, Ranger, Sentry
Syncsort Benefits
Syncsort: High Performance Import from Existing Sources

24. Bring ALL Enterprise Data Securely to the Data Lake
24
• Collect virtually any data from mainframe to
relational, cloud and NoSQL sources
• Access, re-format and load data directly into Hive &
Hadoop file formats. No staging required!
• Batch & streaming sources
• Pull hundreds of tables at once into your data hub,
whole DB schemas in one invocation
24Syncsort Confidential andProprietary - do not copy or distribute
•Load more data into Hadoop in less time

25. Get Your Database data into Hadoop, At the Press of a Button
25
• Pull multiple data sources and funnel into your data lake
• Extract and map whole DB schemas in one invocation
• Extract from multiple data sources: DB2/z, Netezza, Oracle,
Teradata,…
• One-step data movement, auto-generating jobs, auto-generating
Hive target tables, and update Hive statistics
• Process multiple funnels in parallel on your edge node or from
data nodes
‒ Leverages DMX-h high speed data engine via DTL
‒ Generated applications can be imported into GUI
• In-flight transformations
‒ Filtering, funnel dependency ordering, mixed source/target, data type
filtering, table exclusion/inclusion
25Syncsort Confidential andProprietary - do not copy or distribute
DMX
DataFunnel™

26. Intelligent
Execution
Layer
Design Once, Deploy Anywhere
26
Intelligent Execution - Insulate your people from underlying complexities of Hadoop.
One interface to design jobs to run on:
Single Node, Cluster
MapReduce, Spark, Future Platforms
Windows, Unix, Linux
On-Premise, Cloud
Batch, Streaming
• Use existing ETL skills.
• No worries about mappers, reducers, big side, small side, and so on.
• Automatic optimization for best performance, load balancing, etc.
• No changes or tuning required, even if you change execution frameworks
• Future-proof job designs for emerging compute frameworks, e.g. Spark

27. Insurance: Easy Access to ALL Data for Better Analytics
27
• Challenge: Needed hard-to-access operational data for advanced
analytics
• Solution:
• Quickly load ~1000 database tables into HDP with the click of a
button
• Access & integrate complex Mainframe VSAM files, data from
DB2/z, Oracle & SQL Server
• Track changes & keep data up to date
• Benefits:
• Insight: Better and faster analytics
• Agility: Reclaim development time; single tool to ingest, detect changes and populate the data lake
• Compliance: Build audit trails, keep HDP data lake current
• Productivity: No need for deep understanding of Hadoop

28. Hotel Chain: Ease of Use, Timely & Up-to-Date Reporting
28
• Challenge: More timely collection & reporting on room
availability, event bookings, inventory and other hotel data from
4,000+ properties globally
• Solution:
• Near real-time reporting
• DMX-h consumes property updates from Kafka every 10s
• DMX-h processes data on HDP, loading to TD every 30 min
• Deployed on Google Cloud Platform
• Benefits:
• Time to Value: DMX-h ease of use drastically cut development time
• Agility: Reports updated every 30 minutes vs every 24 hours
• Productivity: Leveraging ETL team for Hadoop (Spark), visual understanding of data pipeline
• Insight: Up-to-date data = better business decisions = happier customers

29. Leading Media Company: Accelerate New Business Initiatives
29
• Challenge: Build scalable platform to support new business
initiatives & scale for double-digit data growth, while reducing
escalating EDW & ELT Costs
• Solution:
• Shift data storage & processing out of the EDW into Hadoop
• Migrate 500+ SQL ELT workloads to DMX-h on HDP
• Benefits:
• Agility: Scalable architecture to deploy new business initiatives – analyze more set top box data, blend
website user activity data, etc.
• Cost: Millions of dollars in savings from EDW, including SQL tuning & maintenance costs
• Productivity: ETL developers can stop coding & tuning, and get up & running on Hadoop quickly

30. 30 © Hortonworks Inc. 2011 –2016. All Rights Reserved
Q / A
Learn More:
à EDW Optimization with HDP
– http://hortonworks.com/solutions/edw-optimization/
– EDW Optimization 7 min video
à Try Syncsort DMX-h: syncsort.com/try
– Your existing cluster or use our fully functional Hortonworks Sandbox
– Get a jump-start with our library of pre-built jobs, including Use Case Accelerators for Hadoop
ETL and HDFS Extract & Load
à Syncsort DMX-h
– http://hortonworks.com/partner/syncsort/

31. 31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Thank You