A modern, flexible approach to Hadoop implementation incorporating innovations from HP Haven Jeff Veis Vice President HP Software Big Data Gilles Noisette Master Solution Architect HP EMEA Big Data CoE

1. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 4/24/20151 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
A modern, flexible approach to Hadoop implementation
incorporatinginnovations from HP Haven
Jeff Veis Gilles Noisette
Vice President Master Solution Architect
HP Software Big Data HP EMEA Big Data CoE
Hadoop Summit Europe – Brussels
April 15th, 2015

2. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Agenda
• HP Haven
• HP Haven & Hadoop
• HP Vertica  Fast analytics on Hadoop
• HP IDOL  Smart Hadoop Data Lake
• HP Platforms for Hadoop
• HP Reference Architectures for Hadoop
• HP Big Data Reference Architecture
• HP Big Data Services

3. Data Accessibility today
Infrastructure that becomes
unaffordable at scale
Analytics power that is
accessible to only the few
A trade-off between quality of
insight & the speed of decisions
Typical Compromises
Data is often past its effective
expiration date to add value

4. IT
• Static Reporting
• Uniformity & Traceability
• Resource Rationing
• Cost focus
• Governance through denial
Efficiency of the Answer
Business
• Interactive Exploration
• Unfettered access
• Always on anywhere access
• Results focus
• Governance through enablement
Importance of the Question
Over 50% of all analytics related buying is now coming from the
business and increasingly from individuals – Gartner ‘15SHIFT >

5. Empty
• Loss of Control & Budget
• IT’s future viability
• Risk of Duplication
• Unintentional Siloed Data
• Tie IT results to IT operations
Full
• Opportunity to collaborate
• Refocus on innovation
• Enable data-driven risk taking
• Spur business agility
• Tie IT results to business outcomes
Changing Role of the CIO
Emergence of Decentralized Analytics

6. 6
OLD
NEW
Management &
Governance
Data lake
Business Aligned Insight in Action
Enabling ubiquitous data flows for business-driven composite applications & services
Data-driven Composite Apps
& OnDemand Services
Business as a passive
consumer of data
Business as an active,
collaborative data-
driven partner with IT
EDW
Big Data Analytics
Descriptive Analytics (Data Discovery, Embedded
Analytics, Analytic Applications)

7. Management & Governance
A connected intelligence platform designed to harness 100% of the data
EDW
App
DB
App
DB
App
DB
Next gen data
services
Composite analytic
apps
Next gen predictive
analytics
Data lake
HP Haven Big Data platform
Reporting
Other data
New Style of IT
Data Tone
HP Haven Big Data Platform

8. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
HP Haven

9. Haven
Big Data Platform
Turn 100% of your
data into action.
Powering Big Data Analytics to Applications
Insight
Haven OnDemand
• Open APIs
• Rapid POCs & deployment
• Elastic / Multi-tenant
• Private Cloud-ready
• Pay-as-you-go
Haven Enterprise
• SQL / BI / Reporting
• Predictive Analytics
• Machine Learning
• Log Analytics
• Search
• Image / Audio / Video
The HP Haven Big Data Platform
Haven OnHadoop
• Secure Data Lake
• Exploration
• Open Data Format
• YARN-ready
• Governance
• Native support for MapR,
Hortonworks & Cloudera
Human Data
Business Data
Machine Data
HP Vertica, HP IDOL, KeyView,
HP Distributed R Predictive Analytics
HP Vertica SQL on Hadoop
HP IDOL for Hadoop
HP Vertica OnDemand &
HP IDOL OnDemand

10. Gain insights into your data in near-real time by running queries 50x-1,000x faster than legacy products
Blazing Fast Analytics
Speed, Scalability, and Openness at Lower TCO
HP Vertica
High-Performance Data Analytics Platform Purpose Built for Big Data
HP Vertica Analytics Platform
Infinitely scale your solution by adding an unlimited number of industry-standard servers
Massive Scalability
Protect and embrace your investment in hardware and software, with built-in support for
Hadoop, R, and a range of ETL and BI tools
Open Architecture
Store 10x-30x more data per server than row databases with patented columnar compression
Optimized Data Storage

11. HP Vertica – Built for Speed
We boost performance
Use to take Now takes
1 hour 3.6 Seconds
8 hours (overnight) Under 30 seconds
What Vertrica Performance Advantage means:
"When we did the first queries, they were done so
fast, we thought they were broken.“
- Michael Relich, Guess?

12. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
HP Vertica SQL on Hadoop
Fast analytics on Hadoop

13. Haven OnHadoop – Delivering a Smarter Data Lake
Vertica Optimized Storage Hadoop
Enterprise-class discovery analytics on ANY Hadoop node
HP Vertica SQL on Hadoop
HP Vertica SQL on Hadoop :
- Best-in-class ANSI SQL Analytics
- Hadoop Distribution Agnostic
- Query data in place in Hadoop Formats
- Co-Locate and leverage existing Hadoop infrastructure
- HP Vertica performance on lower-cost infrastructure
-Single query engine across diverse formats and infrastructure

14. Apache YARN : The resource manager for Hadoop 2.0
HP Vertica on Hadoop YARN
HP Software works on porting Vertica on YARN
Data Processing Engines Run Natively IN Hadoop
INTERACTIVE
Tez
STREAMING
Storm
GRAPH
Giraph
ANALYTICS
hp Vertica
ONLINE
HBase
OTHERS
…
HDFS: Redundant, Reliable Storage
YARN: Cluster Resource Management
BATCH
MapReduce
F U T U R E
ANALYTICS
hp Vertica

15. HP Haven Predictive Analytics
Delivering scale and performance with Distributed R breakthrough technology
Build models
Evaluate models
Deploy
models
(In-database
scoring)
BI integration
1 2
3
Build and evaluate
predictive models on large
data sets using Distributed
R
2
1 Ingest and prepare data by
leveraging HP Vertica
3 Deploy models to Vertica and
use in-database scoring to
produce prediction results for
BI and applications
5XPerformance
improvement
A scalable, high-performance engine for the R language developed by HP Labs
•Natively integration to HP Vertica
•Compatible with popular tools like R Studio and existing R libraries
•Open source supported by HP with enterprise-class support
HP powered
clustered
computing
New

16. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
HP IDOL for Hadoop
To Build a Smarter data Lake

17. © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.19
The OS for human information
HP Intelligent Data Operating Layer (IDOL)
Single processing layer to handle the continuum of
human information
Connect
Understand
Over 500 functions to derive actionable
insights
Act &
Automate
Form an understanding of information,
including docs, emails, databases, social
media, rich media, etc.
Access virtually any source of information
aka: HP Autonomy IDOL

18. © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.20
A Smarter Data Lake Needs…
Automatically analyse rich media
Connectors & Policies
HP IDOL Features
Integration points with Hadoop
Understand myriad file formats and types
Breakdown information silos across enterprise
Improved, intuitive visibility to contents
KeyView + IDOL to Vertica
IDOL Server (incl HDFS Sync)
Image Server & Video Server
Advanced Speech-to-Text
Knowledge Graph

19. Haven OnDemand Big Data services powered by IDOL
+ 50 easy-to-use web services to power the next generation of apps
Now includes
Speech to Text powered by
Deep Neural Network
technology that is 75% more
accurate as well as advanced
Knowledge Graph search
technology

20. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
HP Reference Architectures for Hadoop

21. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.23
HP Reference Architecture(s) for Hadoop
• Scaling from 4 to thousands of HP ProLiant Servers
• Sized to customer’s workload and storage needs
• Impressive Processor and Storage density
A set of pre-tested hardware components
• Processor, Drives, Network, 1TB/6TB disk size etc.
Full rack capacity
27 servers (9 chassis)
540 cores
2.4 PB disk space
10G/1G NIC & Infiniband
Breakthrough economics, density, simplicity
2.4 PB raw storage
607TB Hadoop usable
for a full rack
Flexible, pre-approved & optimized configurations
HP Apollo 4000
Scalable System
example
24 x HP ProLiant
SL4540 3x15
Worker Nodes
HP 5900 10GbE x 2
HP 5830 1GbE
Network Switches
DL360 Gen9
Head Nodes
Apollo 4530
UID
ProLiant
DL380e
Gen8
SATA
7.2K
2.0 TB
SATA
7.2K
2.0 TB
SATA
7.2K
2.0 TB
SATA
7.2K
2.0 TB
SATA
7.2K
2.0 TB
SATA
7.2K
2.0 TB
SATA
7.2K
2.0 TB
SATA
7.2K
2.0 TB
SATA
7.2K
2.0 TB
SATA
7.2K
2.0 TB
SATA
7.2K
2.0 TB
SATA
7.2K
2.0 TB
DL380
+
UID
28
30
29
31
33
21
34
36
35
37
39
38
40
42
41
43
45
44
1
3
2
4
6
5
7
9
8
10
12
11
13
15
14
16
18
17
19
21
20
22
24
23
25
27
26
BA
Moonshot
1500
Moonshot 1500
180 Xeon E3 cores
360 linux CPUs
in 4Us
UID
10 134 71
11 145 82
12 156 93
UID
10 134 71
11 145 82
12 156 93
UID
10 134 71
11 145 82
12 156 93
UID UID UID
ProLiant
SL4540
Gen8
SATA
7.2K
500GB
SATA
7.2K
500GB
SATA
7.2K
500GB
SATA
7.2K
500GB
SATA
7.2K
500GB
SATA
7.2K
500GB
HP Switch
1GbE, 10GbE or 40GbE

22. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.25
HP Networking
You need more than good servers to get a good cluster
HP Insight Cluster Management utility
• HP CMU is Designed to operate top500 clusters
• Provision thousand of nodes in minutes
• Monitor clusters of any size (2D instant view, 3D time view)
• Control thousand of servers like one
• Perfectly fits Hadoop cluster operation needs
+
It’s also about Networking and Cluster operation
Hadoop cluster behavior real time analysis
HP Switch
1GbE, 10GbE or 40GbE
• Network matters for Hadoop clusters
• Help to avoid bringing the load to the backbone
• HP’s perfect Top of Rack and Aggregation switch offer
• Hadoop likes the HP deep buffer caching feature
• HP IRF simplifies architecture of server access
networks and enables massive scalability
• HP FlexFabric 5930 Switch Series : 32 x 40GbE + 6 x 40G uplink ports
• family of high-density, ultra-low-latency Aggregation switches
• HP FlexFabric 5900 Switch Series : 48 x 10GbE + 4 x 40GbE ports
• Family of low-latency Top of Racks (ToR) switches

23. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.26
What’s coming downstream ?
HP keeps working on designing servers for Big data
Our goal is to increase the compute and storage density !
You should make sure you don’t miss the HP announcement on May the 5th 2015

24. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.27
HP Big Data Reference Architecture

25. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.29
New approach to address Big Data demands
Current traditional Big Data approach
• Compute and storage are always collocated
• All servers are identical
• Data is partitioned across servers on direct-attached storage
(DAS)
New HP Big Data approach
• Separate compute and storage tiers connected by Ethernet
networking
• Standard Hadoop installed asymmetrically with storage
components on the storage servers and yarn applications on
the compute servers
Two Socket, 2U Servers
YARN Applications,
HDFS, ORC Files,
Parquet, Hbase,
Cassandra
Compute Optimized Servers
Storage Optimized Servers
YARN Applications
HDFS, ORC Files,
Parquet, Hbase,
Cassandra

26. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.30
Benefits of HP Big Data Reference Architecture
HP Moonshot and SL4540 addresses a variety of enterprise big data needs
Ethernet (RoCE)
Cluster consolidation
Multiple big data environments can
directly access a shared pool of data
Flexibility to scale
Scale compute and storage independently
Maximum elasticity
Rapidly provision compute without
affecting storage
Breakthrough economics
Significantly better density, cost and power
through workload optimized components

27. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.31
Maximum Elasticity for Big Data workloads
Hadoop Labels feature (jira YARN-796)
• HP contributed IP into the Hadoop trunk, working with Hortonworks
• Specifying labels on nodes allows for scheduling of YARN containers to specific pools of nodes
• Admins able to target workloads at optimized platforms
• Combined with the HP Big Data Reference Architecture, compute nodes can be dynamically assigned
• No data repartitioning
Hadoop Cluster 1 Vertica Analytics Spark
12am – 6am
6am – 12am
Hadoop Cluster 2
Hadoop Cluster 1 Hadoop Cluster 2
Storage Node Storage Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node
Node

28. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.32
Evolve to support multiple compute and storage blocks
Big Data long term view
Low Cost Nodes
SSD Nodes Disk Nodes Archive Nodes
Multi-temperate Storage using HDFS Tiering, NoSQLs and Objectstores
GPU Nodes FPGA Nodes Big Memory Nodes
Workload Optimized compute nodes to accelerate various big data software

29. © Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
HP Trafodion v1.0.0
Forrester - Mike Gualtieri (October 22nd, 2013)
‘The Future of Hadoop is real time and transactional’
Doug Cutting (October 30th, 2013)
‘We're in the middle of a revolution in data processing’
‘… it is inevitable that we will see just about every kind of
workload be moved to this platform – even OnLine Transaction
Processing’ (OLTP)
(OpenSourcesinceJune2014)

30. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.36
Trafodion
Trafodion is a joint HP Labs and HP-IT research project to develop
operational SQL on Hadoop database capabilities
Complete : Full-function SQL
• Reuse existing SQL skills and improve developer productivity
Protected : Distributed ACID transactions
• Guarantees data consistency across multiple rows, tables, SQL statements
Efficient : Optimized for low-latency read and write transactions
• Supports real-time, high concurrency, transaction processing applications
Interoperable : Standard ODBC/JDBC access
• Works with existing tools and applications
Open : Hadoop and Linux distribution neutral
• Easy to add to your existing infrastructure and no vendor lock-in
+
Operational SQL
Hadoop
Open source project sponsorship and investment from HP
Production ready version 1.0 release available at www.trafodion.org

31. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
HP Big Data Services

32. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.39
Advisory and Discovery Services for Big Data
Big Data TW
Discovery Workshop
Discovery Experience
Discovery Lab
Advisory
• Used to define Big Data strategy
• Transformation Workshop format
• Our industry and technical experts can support people in
technology assessments and strategy development.
• Used to identify/prioritize use-cases
• Validate functional and technical viability
• Time boxed engagement to run a pilot
• Based on use-cases from workshop
• Run on Haven cloud environment
• Insert a Haven lab in the customer ecosystem
• Platform, platform management and lab function
management (on-premise or cloud)

33. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.40
Support/Management Services
HP Services for Hadoop
Bringing value to the customer
Cluster Support Managed Services As-a-Service
Technical Services Analytics Services
Hadoop Roadmap
Service
Enterprise Design
Services
Hadoop Proof of
Concept
Cluster
Implementation
Services
Data Science
Services
Information
Management
Services
Hadoop Solutions
& Applications
Development
Advisory &
Discovery Services

34. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Summary
+

35. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.42
Most vendors handle only 15% of the problem
Make Data Matter
Only HP handles 100% of data

36. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Thank You

37. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.44
www.hp.com/go/haven
hortonworks.com/partner/hp/
Solution brochure
Technical white paper
HP Vertica SQL on Hadoop
FAQ
Customer analytics use case
Learn more about HP Haven

38. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.45
External Collateral
HP Big data Reference Architecture
White papers:
HP Big Data Reference Architecture: A Modern Approach
http://h20195.www2.hp.com/V2/GetDocument.aspx?docname=4AA5-6141ENW&cc=us&lc=en
HP Big Data Reference Architecture: Cloudera Enterprise reference architecture implementation
http://h20195.www2.hp.com/V2/GetDocument.aspx?docname=4AA5-6137ENW&cc=us&lc=en
HP Big Data Reference Architecture: Hortonworks Data Platform reference architecture implementation
http://h20195.www2.hp.com/V2/GetDocument.aspx?docname=4AA5-6136ENW&cc=us&lc=en
Blog posts:
HP Blog post (from Greg Battas)
http://h30507.www3.hp.com/t5/Hyperscale-Computing-Blog/The-Future-of-Big-Data-Platforms-Bringing-order-to-chaos-and/ba-p/178209#.VH91WKPna9I
Hortonworks’ blog post
http://hortonworks.com/blog/want-new-ways-optimize-big-data-workloads/
Joseph George’s blog post (The HP Big Data Reference Architecture: It’s Worth Taking a Closer Look…)
http://hp.nu/I20Rn
Silicon Angle Blog post
http://siliconangle.com/blog/2014/12/23/hp-thinks-its-got-a-better-way-to-run-hadoop-hpdiscover/
Forrester Blog Post
http://blogs.forrester.com/richard_fichera/15-01-28-rethinking_analytics_infrastructure
Videos:
Steve Tramack interview on The Cube at Discover
https://www.youtube.com/watch?v=X2ymmUHzXAs&list=PLenh213llmcbDrKaiHfw9Ue9ZKXdYgKxS