Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Hadoop-as-a-Service for Lifecycle Management S...
© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Operational Approach to Virtualizing Hadoop
2
...
© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Benefits of Virtualizing Hadoop
© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Reference Architecture
4
ESXi
Tier-0 Flash
OS
...
© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Adobe Use-Cases
5
Two unique use-cases
Experim...
© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
3rd Party Integrated Deployment
6
© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Big Questions
7
What are the two questions op ...
© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Solution
8
• Integration with Adobe DMBU Priva...
© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
 VMware vSphere BDE web site
 http://www.vmw...
© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Hadoop Virtual Extensions
 Topology Extension...
© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
HVE Adds a New Layer in Hadoop Network Topolog...
© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
vSphere Big Data Extensions Architecture
vClou...
© 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
State, stats
(Slots used,
Pending work)
Comman...
Upcoming SlideShare
Loading in …5
×

Hadoop-as-a-Service for Lifecycle Management Simplicity

3,238 views

Published on

Published in: Technology
  • Be the first to comment

Hadoop-as-a-Service for Lifecycle Management Simplicity

  1. 1. © 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Hadoop-as-a-Service for Lifecycle Management Simplicity Chris Mutchler | Adobe Compute Platform Engineer | @chrismutchler Andrew Nelson | VMware Staff Systems Engineer | @vmwnelson | virtual-hiking.blogspot.com 1
  2. 2. © 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Operational Approach to Virtualizing Hadoop 2 Why even bother? These are the four reasons why I wanted to tackle this problem. » Excited about the idea of developing internal Platform-as-a-Service offering. » Solves a common “shadow IT” problem in infrastructure organizations and save $$$. » Adobe is a Big Data company, it makes sense for us to have a Hadoop offering. » It’s bleeding edge. Innovate quickly and scale.
  3. 3. © 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Benefits of Virtualizing Hadoop
  4. 4. © 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Reference Architecture 4 ESXi Tier-0 Flash OS App OS App OS App Private Cloud Resource Pools VMware vCenter VMware vCloud Automation Center VMware Big Data Extensions
  5. 5. © 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Adobe Use-Cases 5 Two unique use-cases Experimentation Production A Production X Test/Dev Production Test Production Test Experimentation Service A… …Service X Experimentation Engineering Pre-Production Environment: Multiple teams with zero Hadoop experience with a desire to investigate Hadoop. Production Environment: Digital Marketing products looking to take advantage of existing data managed by Ops.
  6. 6. © 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 3rd Party Integrated Deployment 6
  7. 7. © 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Big Questions 7 What are the two questions op teams must answer? »Where is my data? »How do I access it? Local StorageShared Storage
  8. 8. © 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Solution 8 • Integration with Adobe DMBU Private Cloud • HDFS Storage Integration • Service Blueprints in vCAC Data Layer – Hadoop on Isilon Elastic Virtual Compute Layer
  9. 9. © 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
  10. 10. © 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.  VMware vSphere BDE web site  http://www.vmware.com/bde  Virtualized Hadoop Performance with VMware vSphere 5.1  http://www.vmware.com/resources/techresources/10220  Benchmarking Case Study of Virtualized Hadoop Performance on vSphere 5  http://vmware.com/files/pdf/VMW-Hadoop-Performance-vSphere5.pdf  Hadoop Virtualization Extensions (HVE) :  http://www.vmware.com/files/pdf/Hadoop-Virtualization-Extensions-on-VMware-vSphere-5.pdf  Apache Hadoop High Availability Solution on VMware vSphere 5.1 http://vmware.com/files/pdf/Apache-Hadoop-VMware-HA-solution.pdf  Hadoop-as-a-Service workflows for vCloud Automation Center https://solutionexchange.vmware.com/store/products/hadoop-as-a-service-vmware-vcloud-automation-center-and-big-data-extension  Project Serengeti website http://www.projectserengeti.org https://github.com/vmware-serengeti VMware vSphere BDE and Hadoop Resources
  11. 11. © 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. Hadoop Virtual Extensions  Topology Extensions: • Enable Hadoop to recognize additional virtualization layer for read/write/balancing for proper replica placement • Enable compute/data node separation without losing locality  Elasticity Extensions: • Ability to dynamically adjust resources allocated (CPU, memory, map/reduce slots) to compute nodes • Enables runtime elasticity of Hadoop nodes
  12. 12. © 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. HVE Adds a New Layer in Hadoop Network Topology • D = data center • R = rack • NG = node group • HG = node N13N1 N2 N3 N4 N5 N6 N7 N8 N9 N10 N11 N12 R1 R2 R3 R4 D1 D2 / NG1 NG2 NG3 NG4 NG5 NG6 NG7 NG8
  13. 13. © 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. vSphere Big Data Extensions Architecture vCloud Automation Center Big Data Extensions vCenterOperationsManager
  14. 14. © 2013 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. State, stats (Slots used, Pending work) Commands (Decommission, Recommission) Stats and VM configuration Serengeti Job Tracker vCenter DB Manual/Auto Power on/off Virtual Hadoop Manager (VHM) Job Tracker Task Tracker Task Tracker Task Tracker vCenter Server Serengeti Configuration VC state and stats Hadoop state and stats VC actions Hadoop actions Algorithms Cluster Configuration Resource Management Module

×