CC-4009, "Optimizing Hadoop Deployments with SeaMicro SM15000" by Satheesh Nanniyur and Anil Rao


Published on

Presentation, "Optimizing Hadoop Deployments with SeaMicro SM15000" at the AMD Developer Summit (APU13) Nov. 11-13, 2013.

Published in: Technology, Business
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

CC-4009, "Optimizing Hadoop Deployments with SeaMicro SM15000" by Satheesh Nanniyur and Anil Rao

  2. 2. BIG  DATA  IS  A  STRATEGIC  DECISION:  CAPEX  AND  OPEX   Devices   2   |      RIGHT  SIZING  HADOOP  DEPLOYMENTS     Apps   Cloud  
  3. 3. HIGHER  PERFORMANCE,  LESS  POWER  AND  SPACE   Hadoop  Technology  Stack   Data   Warehouse   Data  AnalyRcs   Management   Data  Access   Data  Processing   Data  Storage   3   |      RIGHT  SIZING  HADOOP  DEPLOYMENTS    
  4. 4. SEAMICRO  SM15000™  ACCELERATES  APACHE™  HADOOP™   DEPLOYMENTS   !  Superior  high  availability   ‒  AcRve/standby  NameNode   ‒  AcRve/standby  JobTracker   ‒  Highly  resilient  fabric  for  inter-­‐node   east-­‐west  traffic   !  Reduced  down  Rme   ‒  Remap  or  rezone  disks  to  recover   data   ‒  Hot-­‐swappable  upgrades  or   component  replacements   !  Hardware  redundancy   ‒  Power  supplies   ‒  Network  I/O   4   |      RIGHT  SIZING  HADOOP  DEPLOYMENTS    
  5. 5. SM15000  OVERVIEW   64  HDDs/SDDs   •  Share  drives  across  all  servers   •  Assign  one  server  to  one  or  more  drives  as  needed   •  In  service  upgrades  as  needed   64  Industry  standard  x86  servers   •  AMD  Opteron™,  Intel  Xeon®,  Atom™   •  Energy  efficient  processor   •  20  Gbps  per  socket,  16X  tradiRonal  servers   960  terabytes  Fabric  Storage   •  Extends  supercompute  fabric  to  external  storage   •  Up  to  3.84  PB  storage  capacity;  up  to  960  3.5”  SAS/ SATA  drives   •  Map  to  any  CPU—same  as  internal  drives   5   |      RIGHT  SIZING  HADOOP  DEPLOYMENTS     160  Gbps  Network  I/O   •  Share  network  I/O  across  all  servers   •  Eliminate  TOR  switch   •  Minimize  cabling   •  In  service  upgrades  as  needed  
  6. 6. SEAMICRO  FREEDOM™  FABRIC  ASIC  PROVIDES  MASSIVE   PERFORMANCE,  REDUCES  POWER  AND  SPACE     B E N E F I T S   Freedom™   SeaMicro IOVT TIO Freedom Supercompute Fabric 6   |      RIGHT  SIZING  HADOOP  DEPLOYMENTS     Eliminates 90% of the components on a motherboard shrinking power used, cost and space Reduces the power used by any CPU by consolidating and shutting off unused functionality Provides massive bandwidth while eliminating power hungry top of rack switches
  7. 7. FS  4060-­‐L  FABRIC  STORAGE  ENCLOSURE  WITH  ZONING   CAPABILITY   !  High  density,  power  opRmized  4U  enclosure  with  60  3.5”  drives   !  Up  to  16  enclosures  per  SM15000,  960  drives,  and  3.84  PB   storage  capacity   !  Redundant  controllers,  ports,  fans,  and  PSUs   !  Support  cost  opRmized  24x7  operaRons  SATA  HDD  for  high   density  Big  Data  and  Object  Storage  deployments   !  OpRonal  configuraRon  to  logically  parRRon  an  enclosure  into   two  30  3.5”  drive  enclosures   !  Balanced  disk  to  core  raRo  (1:1)  for  opRmizing  Hadoop   performance   !  Field  configurable  to  provide  utmost    flexibility  to  balance   density  and  performance   7   |      RIGHT  SIZING  HADOOP  DEPLOYMENTS    
  8. 8. FREEDOM  FABRIC  DISAGGREGATES  SERVER  RESOURCES   PROVIDES  FLEXIBILITY  FOR  EXPANSION  AND  INFRASTRUCTURE  OPTIMIZATION   !  SM15000  provides  independent  scaling  of  Compute,  Storage,  and  Network   !  Centrally  managed  provisioning  of  storage  and  network  resources  to  compute   nodes  enabled  by  CLI  and  API  interfaces   SeaMicro  SM15000  Server   Hadoop  OpRmizaRon   Compute  and  Memory   Pool   CPU   CPU   CPU   CPU   Deploy   CPU   CPU   Fabric  Interconnect   Shared  Storage   Pool   8   |      RIGHT  SIZING  HADOOP  DEPLOYMENTS     Network   Pool   Cost  and   Performance   Tune  and   OpWmize   Run  and   Analyze  
  9. 9. SM15000  FLEXIBLE  STORAGE  ALLOWS  ITERATIVELY   OPTIMIZING  APACHE®  HADOOP™  DEPLOYMENT   !  Flexible  shared  storage  with  commodity  hardware  enabled  by  SeaMicro  fabric   technology   !  Decoupled  from  Compute  and  Network  to  grow  storage  independently   !  IteraRvely  opRmize  Hadoop  disk  to  core  raRo  as  applicaRon  needs  evolve   Captive DAS with Rigid Storage to Compute Ratio Flexible scale-out Fabric Storage up to 5PB Freedom  Fabric   Traditional Rackmount 9   |      RIGHT  SIZING  HADOOP  DEPLOYMENTS     Intel  /AMD  X86   servers  
  10. 10. APACHE®  HADOOP™  DEPLOYMENT  ON  THE  SEAMICRO   SM15000™   SM15000   ZooKeeper   HDFS   NameNode   ZooKeeper   MapReduce   JobTracker   MapReduce   JobTracker   Up  to  160  Gb/s   Redundant  NameNode  and   JobTracker   bandwidth  for  data   ingesWon   10   |      RIGHT  SIZING  HADOOP  DEPLOYMENTS     DataNode/   TaskTracker   DataNode/   TaskTracker   DataNode/   TaskTracker   DataNode/   TaskTracker   HDFS   NameNode   DataNode/   TaskTracker   DataNode/   TaskTracker   10Gb/s  network   bandwidth/node   DataNode/   TaskTracker   DataNode/   TaskTracker   60  DataNode/  TaskTracker   Nodes       •  8GB/s  storage  bandwidth   •  Flexible  storage  capacity  
  11. 11. SEAMICRO  REFERENCE  ARCHITECTURE  FOR  APACHE®   HADOOP™   !  Hadoop  HA  with  NameNode  and  JobTracker  AcRve/Standby  ConfiguraRon   !  Up  to  60  DataNode/TaskTracker  nodes,  512  x86  cores,  960  TB  raw  capacity,  10   Gb/s  Internode  bandwidth  and  160  Gb/s  uplink  bandwidth  in  28  RU  and  5.8    kW   SoluWon  Components   •  •  •  •  •  •  Highly  Available  AcRve/Standby  NameNode   Highly  Available  AcRve/Standby  JobTracker   DataNode/TaskTracker   SM15000  Internal  Drives  for  NameNode  and  JobTracker   11   |      RIGHT  SIZING  HADOOP  DEPLOYMENTS       •  •  SeaMicro  SM15000  AMD  Opteron  or   Intel  Xeon  (Ivy  Bridge)  CPU  chassis   32  GB  memory  per  node   2  10GbE  Network  cards   8  Storage  Controller  cards   4  FS  4060-­‐L  enclosures  with  SAS   zoning  enabled   60  4TB  3.5”  SAS  or  SATA  drives  per   enclosure   Any  Hadoop  distribuRon  (CDH,  HDP,   MapR,  Apache  Hadoop  etc.)   ZooKeeper  for  NameNode  and   JobTracker  HA  
  12. 12. SM15000  HADOOP  DEPLOYMENT   SoluWon  Components   SM15000  Intel   Xeon   SM15000   AMD  Opteron   Performance   OpRmized   Cost  OpRmized   Racks   <1   <1   Servers   64   64   Cores   256   512   DRAM   2TB   4  TB   240/960  TB   240/960  TB   Cable  Management   0  RU   0  RU   ToR  SwRches   None   None   Downlink  Network  Cables   None   None   60   60   10  Gb/s   10  Gb/s   Uplink  bandwidth   Up  to  160  Gb/s   Up  to  160  Gb/s   Storage  bandwidth   8  GigaBytes/s   8  GigaBytes/s   Use  case   Hard  Drives   Data  Nodes   Bandwidth  per  node   12   |      RIGHT  SIZING  HADOOP  DEPLOYMENTS    
  13. 13. SM15000  –  INDUSTRY’S  ONLY  FLEXIBLE  SYSTEM  FOR   OPTIMIZING  HADOOP  CLUSTERS   Storage   Intensive   Compute   Intensive   Network   Intensive   Compute   Intensive   Storage   Intensive   Map   Reduce   Map   Reduce   Map   HDFS   Input   Up  to  512  x86   cores  with  4TB   DRAM  per  Fabric   Server  in  10RU   13   |      RIGHT  SIZING  HADOOP  DEPLOYMENTS     Map  and   Intermediate   Data  Write   Flexible  scale-­‐out   storage  with  over   1400  spindles  and   5  Petabytes  of   capacity   Shuffle   Reduce   10  Gpbs  Inter-­‐ Node  Bandwidth   per  server     HDFS   Output   160  Gbps    shared     uplink  for  Inter-­‐ Rack  traffic  
  14. 14. SM15000  HADOOP  PERFORMANCE  BETTER  THAN   COMPETITIVE  OFFERINGS   !  77%  less  power  per  node   !  30%  less  power  per  core   !  63%  more  data  sorted  per  second  per  Wat  than  Large  Vendor   Large  Vendor   Terasort   CompleRon   7  min   13  seconds   8  min   33  seconds   Nodes   62  incl.  HA   18   248   216   5800  W   7200  W   MB/s  per  Wat   0.4   0.24   MB/s  per  CPU   core   9.3   9.0   Wats/Node   94  W   400  W   Wats/Core   23  W   33  W   SeaMicro  SM15000  with  64  Nodes  based  on     Intel  Xeon®  (Ivy  Bridge)  1265  L-­‐v2  CPU,  32  GB   memory,  and  2  3.5”  3TB  SAS  drives  per  node   CPU  Cores   Power   CompeRRve  soluRon  consists  of  dual  socket  2U  rack-­‐ mount  servers  with  Intel  Xeon  E5-­‐2667  2.9  GHz  octal   core  CPUs  with  64  GB  memory,  16  disks,  and  4  GbE   network  links  per  node   14   |      RIGHT  SIZING  HADOOP  DEPLOYMENTS     Terasort  -­‐  Sort  rate  per  Wa`   MB/s  per  Wa`   SM15000   0.4   0.35   0.3   0.25   0.2   0.15   0.1   0.05   0   SM15000   Large  Vendor  
  15. 15. WAYFAIR.COM:  PERSONALIZED  SHOPPING  EXPERIENCE   FOR  “A  ZILLION  THINGS”   Applica'ons:  Apache®  Hadoop™,  SQL  server,   PHP   !  Challenge   ‒  Space  and  power  constraints  hindered  availability  of  “shared   nothing”  servers  for  development   ‒  Too  costly  in  space  and  power  to  use  tradiRonal  servers  for  the   number  of  servers  required  and  accurately  test  applicaRon   performance   !  SoluRon   ‒  SeaMicro  SM  high  density  server     ‒  256  Intel®  Xeon®  cores  in  10  RU  system   ‒  64  servers,  1.28  Tbps  SeaMicro  Freedom™  Supercompute  Fabric   !  Results   ‒  Reduced  development  cycles  and  shortened  Rme  to  market  for   new  products   ‒  Increased  producRvity  of  development  engineers  by  providing   abundant  access  to  “shared  nothing”  servers  versus  developing   on  virtualized  server  farms   ‒  Eliminated  unnecessary  equipment  such  as  top  of  rack  switches   and  terminal  servers;  simplified  network  and  power  cabling   15   |      RIGHT  SIZING  HADOOP  DEPLOYMENTS     “The  SeaMicro  SM  server  is  helping  us  operate  at  a  large  scale   and  fast  pace.    The  key  benefits  are  reduced  operaRng  costs  and   increased  efficiency  for  our  big  data  development  infrastructure.     It  provides  the  highest  density  and  flexibility  while  slashing   energy  consumpRon:  256  Intel  Xeon  cores,  64  hosts.    It   consumes  50  percent  less  power  and  doubled  our  compuRng   capacity...”     Ben  Clark,  Director  of  So@ware  Engineering  
  16. 16. EHARMONY:  INCREASE  COMPUTING  WHILE     REDUCING  TOTAL  COST  OF  OWNERSHIP   Applica'ons:  Apache®  Hadoop™   !  Challenge   ‒  Provide  cost  effecRve  compuRng     plaxorm  for  Apache  Hadoop   ‒  Reduce  costs  incurred  from  external     cloud  compuRng   !  SoluRon   ‒  SeaMicro  SM10000-­‐64  high  density  server     ‒  512  Intel®  Atom™  cores  in  10  RU  system   !  Results   ‒  Reduce  TCO  by  more  than  74  percent   ‒  Save  thousands  per  month  spent  on     cloud  compuRng  service   ‒  URlize  compuRng  resources  7  x  24   16   |      RIGHT  SIZING  HADOOP  DEPLOYMENTS     “We  purchased  SeaMicro  servers  and  immediately   reduced  our  operaRng  expenses…The  system  has   been  in  place  for  over  two  years,  and  we  have  had   zero  down  Rme.”     Cormac  Twomey,  Data  Center  Opera'ons  
  17. 17. AMD  SEAMICRO  PARTNERS  WITH  INDUSTRY  LEADERS   Hadoop   Partner 17   |      RIGHT  SIZING  HADOOP  DEPLOYMENTS     OS/Hypervisor   OpenStack  
  18. 18. Discussion  
  19. 19. FS  4060-­‐L  SAS  ZONING   !  SAS  zoning  allows  the  logical  parRRoning  of  FS  4060-­‐L  enclosure  into  two  30   disk  enclosures  in  4U   ‒  Provides  fully  independent    end  to  end  path  to  all  30  drives  in  each  zone   ‒  Up  to  2  S-­‐cards  connected  to  a  storage  enclosure   SM15000  S-­‐card   SM15000  S-­‐card   Zone  1   FS  4060-­‐L   Zone  2   FS  4060-­‐L  with  SAS  zoning   19   |      RIGHT  SIZING  HADOOP  DEPLOYMENTS    
  20. 20. DISCLAIMER  &  ATTRIBUTION   The  informaRon  presented  in  this  document  is  for  informaRonal  purposes  only  and  may  contain  technical  inaccuracies,  omissions  and   typographical  errors.     The  informaRon  contained  herein  is  subject  to  change  and  may  be  rendered  inaccurate  for  many  reasons,  including  but  not  limited  to   product  and  roadmap  changes,  component  and  motherboard  version  changes,  new  model  and/or  product  releases,  product  differences   between  differing  manufacturers,  so{ware  changes,  BIOS  flashes,  firmware  upgrades,  or  the  like.  AMD  assumes  no  obligaRon  to  update  or   otherwise  correct  or  revise  this  informaRon.  However,  AMD  reserves  the  right  to  revise  this  informaRon  and  to  make  changes  from  Rme  to   Rme  to  the  content  hereof  without  obligaRon  of  AMD  to  noRfy  any  person  of  such  revisions  or  changes.     AMD  MAKES  NO  REPRESENTATIONS  OR  WARRANTIES  WITH  RESPECT  TO  THE  CONTENTS  HEREOF  AND  ASSUMES  NO  RESPONSIBILITY  FOR   ANY  INACCURACIES,  ERRORS  OR  OMISSIONS  THAT  MAY  APPEAR  IN  THIS  INFORMATION.     AMD  SPECIFICALLY  DISCLAIMS  ANY  IMPLIED  WARRANTIES  OF  MERCHANTABILITY  OR  FITNESS  FOR  ANY  PARTICULAR  PURPOSE.  IN  NO   EVENT  WILL  AMD  BE  LIABLE  TO  ANY  PERSON  FOR  ANY  DIRECT,  INDIRECT,  SPECIAL  OR  OTHER  CONSEQUENTIAL  DAMAGES  ARISING  FROM   THE  USE  OF  ANY  INFORMATION  CONTAINED  HEREIN,  EVEN  IF  AMD  IS  EXPRESSLY  ADVISED  OF  THE  POSSIBILITY  OF  SUCH  DAMAGES.     ATTRIBUTION   ©  2013  Advanced  Micro  Devices,  Inc.  All  rights  reserved.  AMD,  the  AMD  Arrow  logo,  AMD  Opteron,  Freedom  and  combinaRons  thereof   are  trademarks  of  Advanced  Micro  Devices,  Inc.  in  the  United  States  and/or  other  jurisdicRons..  Other  names  are  for  informaRonal   purposes  only  and  may  be  trademarks  of  their  respecRve  owners.   20   |      RIGHT  SIZING  HADOOP  DEPLOYMENTS