HP Insight CMUCluster Management Utility TourSébastien Cabaniols, CMU WW Team lead / EMEA HPC Presales consultant7th HPC D...
Agenda   HP Insight CMU Introduction & Review               Introduction, History / Customers               Product min...
Insight CMUIntroduction & History© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained here...
Insight CMU introduction                                                         CMU = Cluster Management Utility         ...
Insight CMU major milestones2000: initial implementation for Tru64 Unix (Alphaserver)‫‏‬2001: port to Alpha Linux, 1600 se...
Worldwide CMU Deployments                             HP ships >2 CMU clusters per week WW          UNIVERSITIES          ...
Insight CMU project mindset© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is ...
Insight CMU project mindset                     CMU provides the core functionalities for a farm/cloud   runs any HP* ser...
Insight CMU v7.0 tour      CMU functionalities / typical CMU implementation      CMU Provisioning      CMU Monitoring ...
CMU starts here: typical { farm / HPC cluster } implementation                                                            ...
Insight Cluster Management Utility Basics CMU is a single package running on the head node (upgrade is trivial)CMU mgt no...
The three pillars of HP Insight CMU              Provision                                                                ...
Insight CMU v7.0 Tour       CMU Provisioning                   Scalable Cloning                   Legacy/Compatible Aut...
CMU provisioning engine backup/cloning (up to 4k nodes)‫‏‬ RHEL (& clones) / SLES ( & clones) / Debian / Ubuntu (only co...
CMU firmware / bare metal netboot tools firmware hooks: firmware version checks (HP conrep based currently / HP rcu soon...
Insight CMU v7.0 Tour   CMU Monitoring               scalable / ‘HPC aware’ monitoring engine (collectl, GPGPUS)        ...
CMU monitoring   Backend: ‚HPC aware‛ monitoring since years       Scalable monitoring ( proven on 4k nodes system )    ...
‚Instant View*‛ CMU Display* renamed « Instant View » since CMU v7.022   © Copyright 2012 Hewlett-Packard Development Comp...
23   © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change with...
TimeView (Live History)© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subj...
Existing ‘well known’ CMU Display since 200425   © Copyright 2012 Hewlett-Packard Development Company, L.P. The informatio...
3D Display of Sensor Historiesreadability, efficiency, precision26     2     © Copyright 2012 Hewlett-Packard Development ...
3D Display of Sensor Historiesglobal job overview27   © Copyright 2012 Hewlett-Packard Development Company, L.P. The infor...
28   © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change with...
29   © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change with...
30   © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change with...
31   © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change with...
GPGPUs monitoring© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to...
CMU GPGPU SupportCMU provides a tool for extracting GPU metric datafrom GPU driver‚cmu_get_nvidia_gpu‛ monitors:load, mem_...
37   © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change with...
Extended monitoring© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject ...
CMU Extended MonitoringInject monitoring data from another source into CMUExtended metrics will be used for:Server hardw...
CMU alerts & reactions© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subje...
CMU alerts & CMU reactionsCMU monitoring engine can trigger alertsCMU alerts can trigger scripts as reactions to alertsrea...
Insight CMU v7.0 Tour     Scalable / Frictionless monitoring                 Interactive command broadcast (ssh, BMC int...
Insight CMU GUI basicsCluster mgmtpaneldisplays allnodes inselectedgroupings: byswitchlocation; byimage; or bycustomgroupi...
CMU GUI Basics–   Right-click to select sensors to    display–   CMU pre-configured with standard    sensors: CPU and memo...
Friction-less remote control of target nodes                Selected                                       Power          ...
CMU remote management commands•    Multi-window broadcast command (access OS or console)51      type here…     © Copyright...
cmudiffscaling the command line.© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herei...
Compare node outputs with Scalable Text Analyser (cmudiff)                                                                ...
dshbak vs cmudiff: round #1…. ‘date’ on five hosts                                                                        ...
dshbak vs cmudiff: round #2..‘ifconfig’ on 3 hosts                                                                        ...
Partners software integration& Custom menu GUI© Copyright 2012 Hewlett-Packard Development Company, L.P. The information c...
62   © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change with...
63   © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change with...
CMU Custom Menu Support                                               /opt/cmu/etc/cmu_custom_menu64   © Copyright 2012 He...
cmu_dynamic_user_groups© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subj...
Insight CMU as a (job) power monitor70   © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contai...
Insight CMU Partner Integrations   Moab – Dynamic Provisioning   PBS Pro – Green Scheduling & OS Provisioning   LSF – P...
Thank you for your interest inHP Insight CMU © Copyright 2012 Hewlett-Packard Development Company, L.P. The information co...
Upcoming SlideShare
Loading in...5
×

Hp cmu – easy to use cluster management utility @ hpcday 2012 kiev

1,584

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,584
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Hp cmu – easy to use cluster management utility @ hpcday 2012 kiev

  1. 1. HP Insight CMUCluster Management Utility TourSébastien Cabaniols, CMU WW Team lead / EMEA HPC Presales consultant7th HPC Day conference, Kiev, UkraineOctober 2012© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  2. 2. Agenda HP Insight CMU Introduction & Review  Introduction, History / Customers  Product mindset Insight CMU v7.0 tour  Provisioning (Cloning / Autoinstall / Diskless)  Monitoring ( TimeView / Collectl / GPGPUs…)  Scalable/Frictionless administration ( cmudiff…)  Custom GUI & partners integration© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  3. 3. Insight CMUIntroduction & History© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  4. 4. Insight CMU introduction CMU = Cluster Management Utility ‚CMU optimizes the TCO of compute farms‛ CMU scaling specification: 4k nodes CMU has lots of industrial clusters in production with 2k/3k+ nodes CMU has a strong presence in the TOP500 (www.top500.org) CMU at customer site since 2000 4 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  5. 5. Insight CMU major milestones2000: initial implementation for Tru64 Unix (Alphaserver)‫‏‬2001: port to Alpha Linux, 1600 servers commercial cluster2002: port to x86 & IA64 Linux / HPUX Itanium2004: port to x86_64 Linux. (only port maintained*)2007: Swedish gov, 6th @ TOP5002010: Tsubame 2, HP first public 1+ PFlop cluster, 5th @ TOP500*ARM port in progress... 5 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  6. 6. Worldwide CMU Deployments HP ships >2 CMU clusters per week WW UNIVERSITIES ENGINEERING GOVERNMENT and RESEARCH LABS ENERGY6 6 April 2009 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  7. 7. Insight CMU project mindset© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  8. 8. Insight CMU project mindset CMU provides the core functionalities for a farm/cloud runs any HP* server (even mix) / any Linux distribution (even mix)‫‏‬ independent of many architectural aspects of the system: interconnects / GPGPUs / IO accelerators... network topology (open cluster, guarded cluster, WAN…) batch/job schedulers, MPI stacks, math libraries, compilers.... CMU is not a ‘predefined’ SW supercomputer appliance >90% systems delivered as ‚turn-key solutions‛ CMU can also be purchased standalone with support and manuals 8 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  9. 9. Insight CMU v7.0 tour  CMU functionalities / typical CMU implementation  CMU Provisioning  CMU Monitoring  CMU Scalable / Frictionless administration  CMU Custom GUI & partners integration© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  10. 10. CMU starts here: typical { farm / HPC cluster } implementation { high speed Interconnect } { Highly Avail.}Head node© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  11. 11. Insight Cluster Management Utility Basics CMU is a single package running on the head node (upgrade is trivial)CMU mgt node can be an HA cluster (HP service guard,Redhat Cluster, SLES HA) Provides a full fledged interactive CLI Provides cmu_* commands as an API (for scripting)For integration with other software or command-line activity (see partner’s integration) Provides GUI client for single dashboard controllaunch from a web page served from head node (JAVA© webstart technology)run on a local laptop/desktopuser mode for monitoringadmin mode for remote administrationcontained herein is subject to change without notice. 11 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information
  12. 12. The three pillars of HP Insight CMU Provision Monitor Control • Simplified discovery, • ‘At a glance’ view of • GUI & CLI & API firmware audits entire system / • Easy GUI, friction- • Fast & scalable partition less control of cloning • Customizable remote servers • Legacy support of • Lightweight • Scalable pdsh with Kickstart/Autoyast/D • Instant 2D view cmudiff analyser ebian Preseed • TimeView, 3D live • Diskless support history© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  13. 13. Insight CMU v7.0 Tour  CMU Provisioning  Scalable Cloning  Legacy/Compatible Autoinstall (Kickstart/Autoyast/Preseed)  Diskless  Firmware audit  Bare metal netboot low level tools (hpacucli / hponcf / ipmitool)© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  14. 14. CMU provisioning engine backup/cloning (up to 4k nodes)‫‏‬ RHEL (& clones) / SLES ( & clones) / Debian / Ubuntu (only compute nodes)‫‏‬ performance only depends on the image size & harddrive speed  no architectural dependence on trunked/IB/10gig networks, 1Gig ethernet is sufficient. 22 minutes to clone 1000 nodes with SAS drives and 10 GB image  continuous cloning for reprovisionning from batch schedulers autoinstall ‫ ‏‬CMU bridge to legacy/standard tools : Redhat Kickstart / SLES Autoyast / Debian Preseed  do not use above 100 nodes diskless (advised if improving the density of the solution and/or data security) statefull diskless engine (hybrid NFS ro + rw personalities) 14 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  15. 15. CMU firmware / bare metal netboot tools firmware hooks: firmware version checks (HP conrep based currently / HP rcu soon) firmware settings audit (HP conrep + cmudiff) firmware flashing engine (to feed with SCEXE HP files) bare metal netboot tools (available from ‚pre_reconf ‚/ ‚reconf‛ ) hpacucli : configure HP smartarray controllers locfg.pl/ hponcfg: configure HP ILO from the CMU netboot environment ipmitools: configure IPMI capable BMC from the CMU netboot environment 19 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  16. 16. Insight CMU v7.0 Tour CMU Monitoring  scalable / ‘HPC aware’ monitoring engine (collectl, GPGPUS)  2D Instant View / 3D Time View (Live History) © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  17. 17. CMU monitoring Backend: ‚HPC aware‛ monitoring since years  Scalable monitoring ( proven on 4k nodes system )  Non intrusive (leverage collectl + ‚HPC synchro‛ mode)  Programmable (monitor anything you can script )  Nvidia & AMD GPGPUs monitoring tool  Extended Monitoring to inject arbitrary monitoring data  Alerting system & CMU Reactions Frontend GUI (JAVA client/server) / CLI  GUI: Instant view 2D / TimeView 3D (Live History)  cmu_dynamic_user groups (see later in presentation) 21 CLI/API: cmu_monstat and flat human readable files © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  18. 18. ‚Instant View*‛ CMU Display* renamed « Instant View » since CMU v7.022 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  19. 19. 23 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  20. 20. TimeView (Live History)© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  21. 21. Existing ‘well known’ CMU Display since 200425 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  22. 22. 3D Display of Sensor Historiesreadability, efficiency, precision26 2 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  23. 23. 3D Display of Sensor Historiesglobal job overview27 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  24. 24. 28 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  25. 25. 29 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  26. 26. 30 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  27. 27. 31 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  28. 28. GPGPUs monitoring© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  29. 29. CMU GPGPU SupportCMU provides a tool for extracting GPU metric datafrom GPU driver‚cmu_get_nvidia_gpu‛ monitors:load, mem_util, mem_alloc, power_state, and ECC_double_bitalerts by defaultPower_usage, various clock speeds, fan speeds, andtemperature also configured but commented out by default 33 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  30. 30. 37 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.F3ooter goes here
  31. 31. Extended monitoring© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  32. 32. CMU Extended MonitoringInject monitoring data from another source into CMUExtended metrics will be used for:Server hardware metrics (ILO4 out-of-band & agentless monitoring)• Temperatures, fan speeds, power usage• Gathered out-of-band• OS-neutralCluster peripherals• MCS temperatures, switch statusWorkload schedulers 39 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  33. 33. CMU alerts & reactions© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  34. 34. CMU alerts & CMU reactionsCMU monitoring engine can trigger alertsCMU alerts can trigger scripts as reactions to alertsreaction examples:• SNMP traps (send all alerts to an SNMP sink such as HPSIM)• Send an email• Remove a compute node from a batch scheduler…44 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  35. 35. Insight CMU v7.0 Tour Scalable / Frictionless monitoring  Interactive command broadcast (ssh, BMC interfaces)  cmudiff non interactive scalable command output analyzer  GUI accelerators (power off / UID leds/ three clicks ‘en masse’ cloning….)© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  36. 36. Insight CMU GUI basicsCluster mgmtpaneldisplays allnodes inselectedgroupings: byswitchlocation; byimage; or bycustomgrouping node states display current state of each node CMU Main Alerts displayed47 Display Panel © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. along the bottom
  37. 37. CMU GUI Basics– Right-click to select sensors to display– CMU pre-configured with standard sensors: CPU and memory usage, and disk and network I/O– Simple to add any sensor or alert– CMU provides simple support for monitoring GPU temp and ECC errors– Three clicks to clone compute nodes ! 48 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  38. 38. Friction-less remote control of target nodes Selected Power nodes Broadcast commands commands Provisioning commands User-defined commands49 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  39. 39. CMU remote management commands• Multi-window broadcast command (access OS or console)51 type here… © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. ...and see it there
  40. 40. cmudiffscaling the command line.© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  41. 41. Compare node outputs with Scalable Text Analyser (cmudiff) Single-window pdsh with cmu_diff example One command executed across a set of selected nodes… …finds one node running with an old BIOS version! 53 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  42. 42. dshbak vs cmudiff: round #1…. ‘date’ on five hosts cmudiffdshbak 57 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  43. 43. dshbak vs cmudiff: round #2..‘ifconfig’ on 3 hosts cmudiffdshbak 58 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 58 HP Confidential
  44. 44. Partners software integration& Custom menu GUI© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  45. 45. 62 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  46. 46. 63 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  47. 47. CMU Custom Menu Support /opt/cmu/etc/cmu_custom_menu64 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  48. 48. cmu_dynamic_user_groups© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  49. 49. Insight CMU as a (job) power monitor70 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  50. 50. Insight CMU Partner Integrations  Moab – Dynamic Provisioning  PBS Pro – Green Scheduling & OS Provisioning  LSF – Platform HPC  ScaleMP – create large virtual SMP nodes  StackIQ – CMU part of HP ‚roll‛  HP Matrix CMU CloudMap© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  51. 51. Thank you for your interest inHP Insight CMU © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

×