Uploaded byDataWorks Summit
A modern, flexible approach to Hadoop implementation incorporating innovations from HP Haven
The document outlines HP's innovative approach to Hadoop implementation and big data analytics, showcasing their platforms like HP Haven and HP Vertica, which offer fast and scalable analytics capabilities. It emphasizes the shift towards decentralized analytics, enabling businesses to leverage data actively for decision-making, while also detailing HP's big data services and reference architectures designed for efficient data management. Furthermore, it highlights upcoming developments in big data technology, indicating HP's commitment to addressing diverse enterprise needs.
More Related Content
Similar to A modern, flexible approach to Hadoop implementation incorporating innovations from HP Haven
![[Keynote HP] Guido Pezzin - Big Data - from theory to practice with the simpl...](https://cdn.slidesharecdn.com/ss_thumbnails/codemotion21112015-hpbigdatasw-guidopezzin-final-151124153003-lva1-app6892-thumbnail.jpg?width=640&height=640&fit=bounds)
A modern, flexible approach to Hadoop implementation incorporating innovations from HP Haven
- 1. © Copyright 2014Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 4/24/20151 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. A modern, flexible approach to Hadoop implementation incorporatinginnovations from HP Haven Jeff Veis Gilles Noisette Vice President Master Solution Architect HP Software Big Data HP EMEA Big Data CoE Hadoop Summit Europe – Brussels April 15th, 2015
- 2. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Agenda • HP Haven • HP Haven & Hadoop • HP Vertica Fast analytics on Hadoop • HP IDOL Smart Hadoop Data Lake • HP Platforms for Hadoop • HP Reference Architectures for Hadoop • HP Big Data Reference Architecture • HP Big Data Services
- 3. Data Accessibility today Infrastructurethat becomes unaffordable at scale Analytics power that is accessible to only the few A trade-off between quality of insight & the speed of decisions Typical Compromises Data is often past its effective expiration date to add value
- 4. IT • Static Reporting •Uniformity & Traceability • Resource Rationing • Cost focus • Governance through denial Efficiency of the Answer Business • Interactive Exploration • Unfettered access • Always on anywhere access • Results focus • Governance through enablement Importance of the Question Over 50% of all analytics related buying is now coming from the business and increasingly from individuals – Gartner ‘15SHIFT >
- 5. Empty • Loss ofControl & Budget • IT’s future viability • Risk of Duplication • Unintentional Siloed Data • Tie IT results to IT operations Full • Opportunity to collaborate • Refocus on innovation • Enable data-driven risk taking • Spur business agility • Tie IT results to business outcomes Changing Role of the CIO Emergence of Decentralized Analytics
- 6. 6 OLD NEW Management & Governance Data lake BusinessAligned Insight in Action Enabling ubiquitous data flows for business-driven composite applications & services Data-driven Composite Apps & OnDemand Services Business as a passive consumer of data Business as an active, collaborative data- driven partner with IT EDW Big Data Analytics Descriptive Analytics (Data Discovery, Embedded Analytics, Analytic Applications)
- 7. Management & Governance Aconnected intelligence platform designed to harness 100% of the data EDW App DB App DB App DB Next gen data services Composite analytic apps Next gen predictive analytics Data lake HP Haven Big Data platform Reporting Other data New Style of IT Data Tone HP Haven Big Data Platform
- 8. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Haven
- 9. Haven Big Data Platform Turn100% of your data into action. Powering Big Data Analytics to Applications Insight Haven OnDemand • Open APIs • Rapid POCs & deployment • Elastic / Multi-tenant • Private Cloud-ready • Pay-as-you-go Haven Enterprise • SQL / BI / Reporting • Predictive Analytics • Machine Learning • Log Analytics • Search • Image / Audio / Video The HP Haven Big Data Platform Haven OnHadoop • Secure Data Lake • Exploration • Open Data Format • YARN-ready • Governance • Native support for MapR, Hortonworks & Cloudera Human Data Business Data Machine Data HP Vertica, HP IDOL, KeyView, HP Distributed R Predictive Analytics HP Vertica SQL on Hadoop HP IDOL for Hadoop HP Vertica OnDemand & HP IDOL OnDemand
- 10. Gain insights intoyour data in near-real time by running queries 50x-1,000x faster than legacy products Blazing Fast Analytics Speed, Scalability, and Openness at Lower TCO HP Vertica High-Performance Data Analytics Platform Purpose Built for Big Data HP Vertica Analytics Platform Infinitely scale your solution by adding an unlimited number of industry-standard servers Massive Scalability Protect and embrace your investment in hardware and software, with built-in support for Hadoop, R, and a range of ETL and BI tools Open Architecture Store 10x-30x more data per server than row databases with patented columnar compression Optimized Data Storage
- 11. HP Vertica –Built for Speed We boost performance Use to take Now takes 1 hour 3.6 Seconds 8 hours (overnight) Under 30 seconds What Vertrica Performance Advantage means: "When we did the first queries, they were done so fast, we thought they were broken.“ - Michael Relich, Guess?
- 12. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Vertica SQL on Hadoop Fast analytics on Hadoop
- 13. Haven OnHadoop –Delivering a Smarter Data Lake Vertica Optimized Storage Hadoop Enterprise-class discovery analytics on ANY Hadoop node HP Vertica SQL on Hadoop HP Vertica SQL on Hadoop : - Best-in-class ANSI SQL Analytics - Hadoop Distribution Agnostic - Query data in place in Hadoop Formats - Co-Locate and leverage existing Hadoop infrastructure - HP Vertica performance on lower-cost infrastructure -Single query engine across diverse formats and infrastructure
- 14. Apache YARN :The resource manager for Hadoop 2.0 HP Vertica on Hadoop YARN HP Software works on porting Vertica on YARN Data Processing Engines Run Natively IN Hadoop INTERACTIVE Tez STREAMING Storm GRAPH Giraph ANALYTICS hp Vertica ONLINE HBase OTHERS … HDFS: Redundant, Reliable Storage YARN: Cluster Resource Management BATCH MapReduce F U T U R E ANALYTICS hp Vertica
- 15. HP Haven PredictiveAnalytics Delivering scale and performance with Distributed R breakthrough technology Build models Evaluate models Deploy models (In-database scoring) BI integration 1 2 3 Build and evaluate predictive models on large data sets using Distributed R 2 1 Ingest and prepare data by leveraging HP Vertica 3 Deploy models to Vertica and use in-database scoring to produce prediction results for BI and applications 5XPerformance improvement A scalable, high-performance engine for the R language developed by HP Labs •Natively integration to HP Vertica •Compatible with popular tools like R Studio and existing R libraries •Open source supported by HP with enterprise-class support HP powered clustered computing New
- 16. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP IDOL for Hadoop To Build a Smarter data Lake
- 17. © Copyright 2013Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.19 The OS for human information HP Intelligent Data Operating Layer (IDOL) Single processing layer to handle the continuum of human information Connect Understand Over 500 functions to derive actionable insights Act & Automate Form an understanding of information, including docs, emails, databases, social media, rich media, etc. Access virtually any source of information aka: HP Autonomy IDOL
- 18. © Copyright 2013Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.20 A Smarter Data Lake Needs… Automatically analyse rich media Connectors & Policies HP IDOL Features Integration points with Hadoop Understand myriad file formats and types Breakdown information silos across enterprise Improved, intuitive visibility to contents KeyView + IDOL to Vertica IDOL Server (incl HDFS Sync) Image Server & Video Server Advanced Speech-to-Text Knowledge Graph
- 19. Haven OnDemand BigData services powered by IDOL + 50 easy-to-use web services to power the next generation of apps Now includes Speech to Text powered by Deep Neural Network technology that is 75% more accurate as well as advanced Knowledge Graph search technology
- 20. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Reference Architectures for Hadoop
- 21. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.23 HP Reference Architecture(s) for Hadoop • Scaling from 4 to thousands of HP ProLiant Servers • Sized to customer’s workload and storage needs • Impressive Processor and Storage density A set of pre-tested hardware components • Processor, Drives, Network, 1TB/6TB disk size etc. Full rack capacity 27 servers (9 chassis) 540 cores 2.4 PB disk space 10G/1G NIC & Infiniband Breakthrough economics, density, simplicity 2.4 PB raw storage 607TB Hadoop usable for a full rack Flexible, pre-approved & optimized configurations HP Apollo 4000 Scalable System example 24 x HP ProLiant SL4540 3x15 Worker Nodes HP 5900 10GbE x 2 HP 5830 1GbE Network Switches DL360 Gen9 Head Nodes Apollo 4530 UID ProLiant DL380e Gen8 SATA 7.2K 2.0 TB SATA 7.2K 2.0 TB SATA 7.2K 2.0 TB SATA 7.2K 2.0 TB SATA 7.2K 2.0 TB SATA 7.2K 2.0 TB SATA 7.2K 2.0 TB SATA 7.2K 2.0 TB SATA 7.2K 2.0 TB SATA 7.2K 2.0 TB SATA 7.2K 2.0 TB SATA 7.2K 2.0 TB DL380 + UID 28 30 29 31 33 21 34 36 35 37 39 38 40 42 41 43 45 44 1 3 2 4 6 5 7 9 8 10 12 11 13 15 14 16 18 17 19 21 20 22 24 23 25 27 26 BA Moonshot 1500 Moonshot 1500 180 Xeon E3 cores 360 linux CPUs in 4Us UID 10 134 71 11 145 82 12 156 93 UID 10 134 71 11 145 82 12 156 93 UID 10 134 71 11 145 82 12 156 93 UID UID UID ProLiant SL4540 Gen8 SATA 7.2K 500GB SATA 7.2K 500GB SATA 7.2K 500GB SATA 7.2K 500GB SATA 7.2K 500GB SATA 7.2K 500GB HP Switch 1GbE, 10GbE or 40GbE
- 22. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.25 HP Networking You need more than good servers to get a good cluster HP Insight Cluster Management utility • HP CMU is Designed to operate top500 clusters • Provision thousand of nodes in minutes • Monitor clusters of any size (2D instant view, 3D time view) • Control thousand of servers like one • Perfectly fits Hadoop cluster operation needs + It’s also about Networking and Cluster operation Hadoop cluster behavior real time analysis HP Switch 1GbE, 10GbE or 40GbE • Network matters for Hadoop clusters • Help to avoid bringing the load to the backbone • HP’s perfect Top of Rack and Aggregation switch offer • Hadoop likes the HP deep buffer caching feature • HP IRF simplifies architecture of server access networks and enables massive scalability • HP FlexFabric 5930 Switch Series : 32 x 40GbE + 6 x 40G uplink ports • family of high-density, ultra-low-latency Aggregation switches • HP FlexFabric 5900 Switch Series : 48 x 10GbE + 4 x 40GbE ports • Family of low-latency Top of Racks (ToR) switches
- 23. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.26 What’s coming downstream ? HP keeps working on designing servers for Big data Our goal is to increase the compute and storage density ! You should make sure you don’t miss the HP announcement on May the 5th 2015
- 24. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.27 HP Big Data Reference Architecture
- 25. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.29 New approach to address Big Data demands Current traditional Big Data approach • Compute and storage are always collocated • All servers are identical • Data is partitioned across servers on direct-attached storage (DAS) New HP Big Data approach • Separate compute and storage tiers connected by Ethernet networking • Standard Hadoop installed asymmetrically with storage components on the storage servers and yarn applications on the compute servers Two Socket, 2U Servers YARN Applications, HDFS, ORC Files, Parquet, Hbase, Cassandra Compute Optimized Servers Storage Optimized Servers YARN Applications HDFS, ORC Files, Parquet, Hbase, Cassandra
- 26. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.30 Benefits of HP Big Data Reference Architecture HP Moonshot and SL4540 addresses a variety of enterprise big data needs Ethernet (RoCE) Cluster consolidation Multiple big data environments can directly access a shared pool of data Flexibility to scale Scale compute and storage independently Maximum elasticity Rapidly provision compute without affecting storage Breakthrough economics Significantly better density, cost and power through workload optimized components
- 27. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.31 Maximum Elasticity for Big Data workloads Hadoop Labels feature (jira YARN-796) • HP contributed IP into the Hadoop trunk, working with Hortonworks • Specifying labels on nodes allows for scheduling of YARN containers to specific pools of nodes • Admins able to target workloads at optimized platforms • Combined with the HP Big Data Reference Architecture, compute nodes can be dynamically assigned • No data repartitioning Hadoop Cluster 1 Vertica Analytics Spark 12am – 6am 6am – 12am Hadoop Cluster 2 Hadoop Cluster 1 Hadoop Cluster 2 Storage Node Storage Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node
- 28. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.32 Evolve to support multiple compute and storage blocks Big Data long term view Low Cost Nodes SSD Nodes Disk Nodes Archive Nodes Multi-temperate Storage using HDFS Tiering, NoSQLs and Objectstores GPU Nodes FPGA Nodes Big Memory Nodes Workload Optimized compute nodes to accelerate various big data software
- 29. © Copyright 2013Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Trafodion v1.0.0 Forrester - Mike Gualtieri (October 22nd, 2013) ‘The Future of Hadoop is real time and transactional’ Doug Cutting (October 30th, 2013) ‘We're in the middle of a revolution in data processing’ ‘… it is inevitable that we will see just about every kind of workload be moved to this platform – even OnLine Transaction Processing’ (OLTP) (OpenSourcesinceJune2014)
- 30. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.36 Trafodion Trafodion is a joint HP Labs and HP-IT research project to develop operational SQL on Hadoop database capabilities Complete : Full-function SQL • Reuse existing SQL skills and improve developer productivity Protected : Distributed ACID transactions • Guarantees data consistency across multiple rows, tables, SQL statements Efficient : Optimized for low-latency read and write transactions • Supports real-time, high concurrency, transaction processing applications Interoperable : Standard ODBC/JDBC access • Works with existing tools and applications Open : Hadoop and Linux distribution neutral • Easy to add to your existing infrastructure and no vendor lock-in + Operational SQL Hadoop Open source project sponsorship and investment from HP Production ready version 1.0 release available at www.trafodion.org
- 31. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Big Data Services
- 32. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.39 Advisory and Discovery Services for Big Data Big Data TW Discovery Workshop Discovery Experience Discovery Lab Advisory • Used to define Big Data strategy • Transformation Workshop format • Our industry and technical experts can support people in technology assessments and strategy development. • Used to identify/prioritize use-cases • Validate functional and technical viability • Time boxed engagement to run a pilot • Based on use-cases from workshop • Run on Haven cloud environment • Insert a Haven lab in the customer ecosystem • Platform, platform management and lab function management (on-premise or cloud)
- 33. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.40 Support/Management Services HP Services for Hadoop Bringing value to the customer Cluster Support Managed Services As-a-Service Technical Services Analytics Services Hadoop Roadmap Service Enterprise Design Services Hadoop Proof of Concept Cluster Implementation Services Data Science Services Information Management Services Hadoop Solutions & Applications Development Advisory & Discovery Services
- 34. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Summary +
- 35. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.42 Most vendors handle only 15% of the problem Make Data Matter Only HP handles 100% of data
- 36. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Thank You
- 37. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.44 www.hp.com/go/haven hortonworks.com/partner/hp/ Solution brochure Technical white paper HP Vertica SQL on Hadoop FAQ Customer analytics use case Learn more about HP Haven
- 38. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.45 External Collateral HP Big data Reference Architecture White papers: HP Big Data Reference Architecture: A Modern Approach http://h20195.www2.hp.com/V2/GetDocument.aspx?docname=4AA5-6141ENW&cc=us&lc=en HP Big Data Reference Architecture: Cloudera Enterprise reference architecture implementation http://h20195.www2.hp.com/V2/GetDocument.aspx?docname=4AA5-6137ENW&cc=us&lc=en HP Big Data Reference Architecture: Hortonworks Data Platform reference architecture implementation http://h20195.www2.hp.com/V2/GetDocument.aspx?docname=4AA5-6136ENW&cc=us&lc=en Blog posts: HP Blog post (from Greg Battas) http://h30507.www3.hp.com/t5/Hyperscale-Computing-Blog/The-Future-of-Big-Data-Platforms-Bringing-order-to-chaos-and/ba-p/178209#.VH91WKPna9I Hortonworks’ blog post http://hortonworks.com/blog/want-new-ways-optimize-big-data-workloads/ Joseph George’s blog post (The HP Big Data Reference Architecture: It’s Worth Taking a Closer Look…) http://hp.nu/I20Rn Silicon Angle Blog post http://siliconangle.com/blog/2014/12/23/hp-thinks-its-got-a-better-way-to-run-hadoop-hpdiscover/ Forrester Blog Post http://blogs.forrester.com/richard_fichera/15-01-28-rethinking_analytics_infrastructure Videos: Steve Tramack interview on The Cube at Discover https://www.youtube.com/watch?v=X2ymmUHzXAs&list=PLenh213llmcbDrKaiHfw9Ue9ZKXdYgKxS