SNIA 2012 - Creating an Enterprise Hadoop Platform

604 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
604
On SlideShare
0
From Embeds
0
Number of Embeds
12
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

SNIA 2012 - Creating an Enterprise Hadoop Platform

  1. 1. Creating an Enterprise-class Hadoop Platform Joey Jablonski Practice Director, Analytic Services DataDirect Networks, Inc. (DDN)2012 SNIA Analytics and Big Data Summit. © DataDirect Networks, Inc. (DDN). All RightsReserved.
  2. 2. Who am I?  Practice Director, Analytic Services at DataDirect Networks, Inc.  3+ years with Hadoop, 12+ with HPC  Contact Details  @jrjablo  jjablonski@ddn.com/jrjablo@gmail.com  www.linkedin.com/in/joeyjablonski2012 SNIA Analytics and Big Data Summit. © DataDirect Networks, Inc. (DDN). All RightsReserved. 2
  3. 3. Why Hadoop?  Scalable – Performance & Capacity  Growing Ecosystem (Flexibility)  Established APIs & Interfaces  Location on the adoption curve  Proven base to create Analytical Platforms2012 SNIA Analytics and Big Data Summit. © DataDirect Networks, Inc. (DDN). All RightsReserved. 3
  4. 4. What is Enterprise Class?  Scalable – OPEX & CAPEX  Manageable  Integration with existing tools  Flexible Workflow – Process Integration  No Rip & Replace  Metrics to manage towards  Business Driven, Technological Capabilities 2012 SNIA Analytics and Big Data Summit. © DataDirect Networks, Inc. (DDN). All Rights Reserved. 4
  5. 5. The Big Data ChallengeThe Big Data Equation:Volume Velocity Variety + + Petabytes of Data GB/s  TB/s Structured Trillions of Objects Millions of IO/s Unstructured Object Operations Streams & Batches 2012 SNIA Analytics and Big Data Summit. © DataDirect Networks, Inc. (DDN). All Rights Reserved.
  6. 6. Analytics | Looking for ActionableInformationBillions of DataPoints toConsider lts• Consumer purchasing trends e su bl eR• na Product perception Ac tio• Drug Discovery• Genomics• Surveillance• Financial Analysis 2012 SNIA Analytics and Big Data Summit. © DataDirect Networks, Inc. (DDN). All Rights Reserved.
  7. 7. Data Gravity Applications DATA Services 2012 SNIA Analytics and Big Data Summit. © DataDirect Networks, Inc. (DDN). All Rights Reserved. 7
  8. 8. Why is data Analytics so hard? Technical Business Hacking Skills Business Acumen Data Science Analytics Math & g n o s i c e D Substantive Statistics Communications r o o P Expertise Curiosity knowledge R h c a s e n o d T r a r t l i 2012 SNIA Analytics and Big Data Summit. © DataDirect Networks, Inc. (DDN). All Rights Reserved.
  9. 9. What is Hadoop missing today?  Active-Active high-availability  Established management tools  Enterprise integration mindset  Enterprise class hardware  Consistent version-compatibility & deployment  Efficient CAPEX & OPEX scaling  Resource management/SLAs/QoS  Security. 2012 SNIA Analytics and Big Data Summit. © DataDirect Networks, Inc. (DDN). All Rights Reserved. 9
  10. 10. Hadoop Operational Considerations y ge plo na De Ma d e r gra n ito Up Mo nd s po Re Software Platform Hardware Platform 2012 SNIA Analytics and Big Data Summit. © DataDirect Networks, Inc. (DDN). All Rights Reserved.
  11. 11. Todays Enterprise Picture The Cloud 2012 SNIA Analytics and Big Data Summit. © DataDirect Networks, Inc. (DDN). All Rights Reserved. 11
  12. 12. Getting there…. 2012 SNIA Analytics and Big Data Summit. © DataDirect Networks, Inc. (DDN). All Rights Reserved.
  13. 13. Hadoop Architectural Considerations2012 SNIA Analytics and Big Data Summit. © DataDirect Networks, Inc. (DDN). All RightsReserved. 13
  14. 14. Planning for Growth sts t s os Co C al an on um sts Co ati H n itio er or is Op lf ua AcquHigher is Better t Ac Adoption Goal for Human Costs Capacity Performance Scalability User Growth 2012 SNIA Analytics and Big Data Summit. © DataDirect Networks, Inc. (DDN). All Rights Reserved. 14
  15. 15. Shared v. Commodity Shared Component Approach •Lower Operational Costs •Efficient operational resource scaling •Shared resources with other IT platforms •Efficiency in computing, connectivity & service placement Commodity Server Approach •Lower Entry Costs •Shorter MTBF •Inefficient scaling of tools and processes •Mis-match with traditional IT operations models 2012 SNIA Analytics and Big Data Summit. © DataDirect Networks, Inc. (DDN). All Rights Reserved. 15
  16. 16. Ethernet v. Infiniband Infiniband•100% Storage Management Offload•End-End InfiniBand Networking with RDMAAcceleration•Real-Time Data Delivery to Provide MapReduceProcess Consistency•Smaller Compute, Compact Storage to MinimizeData Center Impact Ethernet •Compatibility, ensured connectivity •Limitations in traffic types and bandwidth availability •High CPU/Overhead cost •Minimal options for offloading with Linux environments 2012 SNIA Analytics and Big Data Summit. © DataDirect Networks, Inc. (DDN). All Rights Reserved. 16
  17. 17. Analytic User Types Empowered Users Aware Users Enabled Users 2012 SNIA Analytics and Big Data Summit. © DataDirect Networks, Inc. (DDN). All Rights Reserved. 17
  18. 18. Hadoop Enterprise Integration Monitoring Extract Transform Load & Response APIs Integration Data Information Insight Results 2012 SNIA Analytics and Big Data Summit. © DataDirect Networks, Inc. (DDN). All Rights Reserved. 18
  19. 19. And finally, Hadoop is…  …more then just hardware,  It is about an ecosystem of hardware & software.  …about integrating with existing systems.  …a toolkit to build Analytical Platforms.  …a component of the larger corporate processes and mandates.  …a component of the wider business KPIs. 2012 SNIA Analytics and Big Data Summit. © DataDirect Networks, Inc. (DDN). All Rights Reserved. 19
  20. 20. Q&A2012 SNIA Analytics and Big Data Summit. © DataDirect Networks, Inc. (DDN). All RightsReserved. 20

×