What is data gravity? how solve troubles caused by it?

1. 1 © Hortonworks Inc. 2011–2018. All rights reserved.
Hortonworks & IBM Cognitive
The Future of Data Science
Hortonworks & IBM Better Together

2. 2 © Hortonworks Inc. 2011–2018. All rights reserved.
Hortonworks & IBM Cloud
Data has gravity.
Manage it right with a hybrid data
architecture for BigData
Thiago Santiago
Solution Engineer – Latam

3. 3 © Hortonworks Inc. 2011–2018. All rights reserved.
Gravity:
...the force that attracts a body toward any other physical body
having mass.

4. 4 © Hortonworks Inc. 2011–2018. All rights reserved.

5. 5 © Hortonworks Inc. 2011–2018. All rights reserved.

6. 6 © Hortonworks Inc. 2011–2018. All rights reserved.

7. 7 © Hortonworks Inc. 2011–2018. All rights reserved.
...the ability of bodies of data to attract applications, services and
other data.
Data gravity:

8. 8 © Hortonworks Inc. 2011–2018. All rights reserved.
Do These Problems Sound Familiar?
Data Analytics
Governance
- Don’t have the right data
- Data resides in silos
- High volumes of data
- Lack of unstructured data
- Difficult to access
- Data is hard to find and access
- Self-Service is limited
- No centrally managed security system
Data source not trusted -
Impossible to audit or track data lineage -
Lack of skills and tools -
Departmental silos -
Technology has limitations -

9. 9 © Hortonworks Inc. 2011–2018. All rights reserved.
Technology Trends: Shifting the Data Paradigm
Artificial IntelligenceInternet of Things Cloud Computing Streaming Data
Industrial Internet
Connected Business
Consumer Devices
Smart Devices
Autonomy
Prescriptive Analytics
SaaS/PaaS Applications
Ephemeral Use Cases
Operational Efficiency
Collaboration
Real-time Applications
Targeted Retail
Recommendations
Industrial Applications

10. 10 © Hortonworks Inc. 2011–2018. All rights reserved.
From the edge, through movement, to rest
Hortonworks DataPlane Service
a foundational platform for the delivery of data
solutions that will:
• Support enterprise hybrid deployment strategy
and adoption of cloud
• Common Metadata, Security and Governance
across all tiers and types of data
• Simplified enterprise data asset management
• Extensible to new services: Services enablement
layer for rapidly bringing new solutions to market
• Brings all data under management
HORTONWORKS
DATAPLANE
SERVICE
MULTIPLE CLUSTERS AND SOURCES
MULTIHYBRID
Manage, Secure, Govern
DATA AT REST
Hortonworks
Data Platform
DATA IN MOTION
Hortonworks
Data Flow
What is Hortonworks DataPlane Service?

11. 11 © Hortonworks Inc. 2011–2018. All rights reserved.
MULTIPLE CLUSTERS AND SOURCES
MULTIHYBRID
MODERN DATA USE CASES
EDW
OPTIMIZATION
CYBER SECURITY DATA SCIENCE
ADVANCED
ANALYTICS
PARTNER
SOLUTIONS
IOT/ STREAMING
ANALYTICS
HORTONWORKS
CONNECTION
ENTERPRISE SUPPORT
PREMIER SUPPORT
EDUCATIONAL SERVICES
PROFESSIONAL SERVICES
COMMUNITY CONNECTION
HORTONWORKS
PLATFORM SERVICES
OPERATIONAL SERVICES
SMARTSENSE™
Global Data Management With Hortonworks
DATAPLANE SERVICE (DPS)
MANAGE, GOVERN, SECURE
DATA
LIFECYCLE
MANAGER
DATA
STEWARD
STUDIO
IBM DSX
EXTENSIBLE SERVICES
CLOUD
BREAK
STREAMS
MESSAGING
MANAGER
DATA
ANALYTICS
STUDIO
CONNECTED DATA PLATFORMS
HORTONWORKS
DATA PLATFORM (HDP®)
DATA-AT-REST
HORTONWORKS
DATAFLOW (HDF™)
DATA-IN-MOTION

12. 12 © Hortonworks Inc. 2011–2018. All rights reserved.
DPS Console: One place to visualize Virtual Data Lakes
DPS PlatformData Plane Service Console

13. 13 © Hortonworks Inc. 2011–2018. All rights reserved.
DATAPLANE SERVICE (DPS)
MANAGE, GOVERN, SECURE
DATA
LIFECYCLE
MANAGER
- DLM -
DATA
STEWARD
STUDIO
- DSS -
IBM DSX
EXTENSIBLE SERVICES
CLOUDBREAK
- CB -
STREAMS
MESSAGING
MANAGER
- SMM -
DATA
ANALYTICS
STUDIO
- DAS -
CONNECTED DATA PLATFORMS
HORTONWORKS
DATA PLATFORM (HDP®)
DATA-AT-REST
HORTONWORKS
DATAFLOW (HDF™)
DATA-IN-MOTION
Global Data Management With Hortonworks

14. 14 © Hortonworks Inc. 2011–2018. All rights reserved.
• First Extensible Service Delivered on DPS Platform –
GA Oct, 2017
• Manage the Data Lifecycle:
– Replication/failback to another cloud/on-prem
site for Disaster Recovery
– Auto Tiering of hot/warm/cold data to cloud
object storage/on-prem for TCO reduction
– Backup & Recover Critical Business Data
• Maintain Common Security and Governance Policies
Across Multi Data Sources/ Environments
Data Lifecycle Manager (DLM)
SERVICE: DATA LIFECYCLE MANAGER
REPLICATION &
DISASTER
RECOVERY
Cluster Cluster ClusterMOVE MOVE
AUTO TIERING
BACKUP &
RESTORE
P(use): high
Cost: $$$
P(use): medium
Cost: $$
P(use): low
Cost: $
Full
backup
day 1 day 2 day 3
Cumulative incremental
backups
Accident
delete
X
FAILBACK
REPLICATION
RESTORE
Prod
Cluster
Backup
Cluster
Generally Available
Coming Soon
Coming Soon

15. 15 © Hortonworks Inc. 2011–2018. All rights reserved.
DLM: Pair clusters and manage data replication flows
Data Lifecycle Manager (DLM)

16. 16 © Hortonworks Inc. 2011–2018. All rights reserved.
DLM: Replicate data between on-prem HDP and Cloud.
Data Lifecycle Manager (DLM)

17. 17 © Hortonworks Inc. 2011–2018. All rights reserved.
DLM: Replication policies and instances
Data Lifecycle Manager (DLM)

18. 18 © Hortonworks Inc. 2011–2018. All rights reserved.
DATAPLANE SERVICE (DPS)
MANAGE, GOVERN, SECURE
DATA
LIFECYCLE
MANAGER
- DLM -
DATA
STEWARD
STUDIO
- DSS -
IBM DSX
EXTENSIBLE SERVICES
CLOUDBREAK
- CB -
STREAMS
MESSAGING
MANAGER
- SMM -
DATA
ANALYTICS
STUDIO
- DAS -
CONNECTED DATA PLATFORMS
HORTONWORKS
DATA PLATFORM (HDP®)
DATA-AT-REST
HORTONWORKS
DATAFLOW (HDF™)
DATA-IN-MOTION
Global Data Management With Hortonworks

19. 19 © Hortonworks Inc. 2011–2018. All rights reserved.
Data Steward Studio
Understand, secure & govern data across enterprise data lakes

20. 20 © Hortonworks Inc. 2011–2018. All rights reserved.
DSS: Understand shape of Hive column data with
statistical profiler, example: Profile shows box plot and
histogram for distribution of column values
Data Steward Studio (DSS)

21. 21 © Hortonworks Inc. 2011–2018. All rights reserved.
DSS: Data lineage shows complete chain of custody
and downstream dependencies for an asset!
Data Steward Studio (DSS)

22. 22 © Hortonworks Inc. 2011–2018. All rights reserved.
DSS: Audit Profiler shows both summarized views &
patterns of access for a data asset.
Data Steward Studio (DSS)

23. 23 © Hortonworks Inc. 2011–2018. All rights reserved.
DATAPLANE SERVICE (DPS)
MANAGE, GOVERN, SECURE
DATA
LIFECYCLE
MANAGER
- DLM -
DATA
STEWARD
STUDIO
- DSS -
IBM DSX
EXTENSIBLE SERVICES
CLOUDBREAK
- CB -
STREAMS
MESSAGING
MANAGER
- SMM -
DATA
ANALYTICS
STUDIO
- DAS -
CONNECTED DATA PLATFORMS
HORTONWORKS
DATA PLATFORM (HDP®)
DATA-AT-REST
HORTONWORKS
DATAFLOW (HDF™)
DATA-IN-MOTION
Global Data Management With Hortonworks

24. 24 © Hortonworks Inc. 2011–2018. All rights reserved.
DAS: Data Analytics Studio gives database heatmap,
quickly discover and see what part of your cluster is
being utilized more
Data Analytics Studio (DAS)

25. 25 © Hortonworks Inc. 2011–2018. All rights reserved.
DAS: Built-in batch operations
No more scripting needed for day-to-day operations
Data Analytics Studio (DAS)

26. 26 © Hortonworks Inc. 2011–2018. All rights reserved.
DAS: Full featured Auto-complete, results direct
download, quick-data preview and many other
quality-of-life improvements.
Data Analytics Studio (DAS)

27. 27 © Hortonworks Inc. 2011–2018. All rights reserved.
DAS: Pre-defined searches to quickly narrow down
problematic queries in a large cluster
Data Analytics Studio (DAS)

28. 28 © Hortonworks Inc. 2011–2018. All rights reserved.
DAS: Heuristic recommendation engine
Fully self-serviced query and storage optimization
Data Analytics Studio (DAS)

29. 29 © Hortonworks Inc. 2011–2018. All rights reserved.
DATAPLANE SERVICE (DPS)
MANAGE, GOVERN, SECURE
DATA
LIFECYCLE
MANAGER
- DLM -
DATA
STEWARD
STUDIO
- DSS -
IBM DSX
EXTENSIBLE SERVICES
CLOUDBREAK
- CB -
STREAMS
MESSAGING
MANAGER
- SMM -
DATA
ANALYTICS
STUDIO
- DAS -
CONNECTED DATA PLATFORMS
HORTONWORKS
DATA PLATFORM (HDP®)
DATA-AT-REST
HORTONWORKS
DATAFLOW (HDF™)
DATA-IN-MOTION
Global Data Management With Hortonworks

30. 30 © Hortonworks Inc. 2011–2018. All rights reserved.
Operations: SMM

31. 31 © Hortonworks Inc. 2011–2018. All rights reserved.
SMM
(Topic Profile / Detail Page)

32. 32 © Hortonworks Inc. 2011–2018. All rights reserved.
DATAPLANE SERVICE (DPS)
MANAGE, GOVERN, SECURE
DATA
LIFECYCLE
MANAGER
- DLM -
DATA
STEWARD
STUDIO
- DSS -
IBM DSX
EXTENSIBLE SERVICES
CLOUDBREAK
- CB -
STREAMS
MESSAGING
MANAGER
- SMM -
DATA
ANALYTICS
STUDIO
- DAS -
CONNECTED DATA PLATFORMS
HORTONWORKS
DATA PLATFORM (HDP®)
DATA-AT-REST
HORTONWORKS
DATAFLOW (HDF™)
DATA-IN-MOTION
Global Data Management With Hortonworks

33. 33 © Hortonworks Inc. 2011–2018. All rights reserved.
CB: Deploy clusters on the Cloud
DPS PlatformCloudbreak (CB)

34. 34 © Hortonworks Inc. 2011–2018. All rights reserved.
CB: Workload specific prescriptive clusters
DPS PlatformCloudbreak (CB)

35. 35 © Hortonworks Inc. 2011–2018. All rights reserved.
MULTIPLE CLUSTERS AND SOURCES
MULTIHYBRID
MODERN DATA USE CASES
EDW
OPTIMIZATION
CYBER SECURITY DATA SCIENCE
ADVANCED
ANALYTICS
PARTNER
SOLUTIONS
IOT/ STREAMING
ANALYTICS
HORTONWORKS
CONNECTION
ENTERPRISE SUPPORT
PREMIER SUPPORT
EDUCATIONAL SERVICES
PROFESSIONAL SERVICES
COMMUNITY CONNECTION
HORTONWORKS
PLATFORM SERVICES
OPERATIONAL SERVICES
SMARTSENSE™
Global Data Management With Hortonworks
DATAPLANE SERVICE (DPS)
MANAGE, GOVERN, SECURE
DATA
LIFECYCLE
MANAGER
DATA
STEWARD
STUDIO
IBM DSX
EXTENSIBLE SERVICES
CLOUD
BREAK
STREAMS
MESSAGING
MANAGER
DATA
ANALYTICS
STUDIO
CONNECTED DATA PLATFORMS
HORTONWORKS
DATA PLATFORM (HDP®)
DATA-AT-REST
HORTONWORKS
DATAFLOW (HDF™)
DATA-IN-MOTION

36. 36 © Hortonworks Inc. 2011–2018. All rights reserved.
Why Customers Choose Hortonworks
Global Data Management
• Hybrid
• Multi-cloud
• End-to-end security and governance
100% Open Source –
“We are the Linux of Big Data”
• Innovation
• Interoperability
• No vendor lock-in
• Rapid community innovation
Proven Business Model:
• 1,300 enterprise customers
• First to IPO
• Fastest to $100M
• First to profitability
Most Comprehensive Platform
• Data at Rest and Data in Motion
• Any style of workload
• Centralized management, security,
governance

37. 37 © Hortonworks Inc. 2011–2018. All rights reserved.
Why Do Customers Migrate to Hortonworks?
Open source model aligned to
customer success with no vendor
lock-in
Cost advantage
Strong references from customers
Best practices captured from 1,300
customer engagements
Dozens of successful large-scale
migrations to HDP
Latest and greatest Hadoop stack

38. 38 © Hortonworks Inc. 2011–2018. All rights reserved.38 © Hortonworks Inc. 2011–2018. All rights reserved.
A Continuous Track Record
of Leading Innovation
DATA-AT-REST
HADOOP 1.0
100% Open
YARN HADOOP 2.0
Enable multiple
workloads
DATA-IN-MOTION
HDP & HDF
Out to the edge
CONNECT DATA
PLATFORMS
Cloud/On-Prem
Performance and
cost control
Hortonworks 3.0
HDP IPO
Dec 2014
First OSS IPO
in 10 years
Innovation
Hortonworks 1.0
Hadoop as an
enterprise
viable data platform
Fastest Company
to $100M in Revenue
August 2015
Hortonworks 2.0
Bring this to the edge
with connected
platforms
Ecosystem consolidation
IBM
First to Cash Flow
Breakeven
2011 2013 2015
2016
2017
2018
First to Multi-Cloud
DataPlane Service

39. 39 © Hortonworks Inc. 2011–2018. All rights reserved.
Thank you