• Save
Hp cmu – easy to use cluster management utility @ hpcday 2012 kiev
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Hp cmu – easy to use cluster management utility @ hpcday 2012 kiev

  • 1,691 views
Uploaded on

 

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
1,691
On Slideshare
1,331
From Embeds
360
Number of Embeds
1

Actions

Shares
Downloads
0
Comments
0
Likes
0

Embeds 360

http://supercomputers.kiev.ua 360

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. HP Insight CMUCluster Management Utility TourSébastien Cabaniols, CMU WW Team lead / EMEA HPC Presales consultant7th HPC Day conference, Kiev, UkraineOctober 2012© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 2. Agenda HP Insight CMU Introduction & Review  Introduction, History / Customers  Product mindset Insight CMU v7.0 tour  Provisioning (Cloning / Autoinstall / Diskless)  Monitoring ( TimeView / Collectl / GPGPUs…)  Scalable/Frictionless administration ( cmudiff…)  Custom GUI & partners integration© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 3. Insight CMUIntroduction & History© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 4. Insight CMU introduction CMU = Cluster Management Utility ‚CMU optimizes the TCO of compute farms‛ CMU scaling specification: 4k nodes CMU has lots of industrial clusters in production with 2k/3k+ nodes CMU has a strong presence in the TOP500 (www.top500.org) CMU at customer site since 2000 4 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 5. Insight CMU major milestones2000: initial implementation for Tru64 Unix (Alphaserver)‫‏‬2001: port to Alpha Linux, 1600 servers commercial cluster2002: port to x86 & IA64 Linux / HPUX Itanium2004: port to x86_64 Linux. (only port maintained*)2007: Swedish gov, 6th @ TOP5002010: Tsubame 2, HP first public 1+ PFlop cluster, 5th @ TOP500*ARM port in progress... 5 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 6. Worldwide CMU Deployments HP ships >2 CMU clusters per week WW UNIVERSITIES ENGINEERING GOVERNMENT and RESEARCH LABS ENERGY6 6 April 2009 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 7. Insight CMU project mindset© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 8. Insight CMU project mindset CMU provides the core functionalities for a farm/cloud runs any HP* server (even mix) / any Linux distribution (even mix)‫‏‬ independent of many architectural aspects of the system: interconnects / GPGPUs / IO accelerators... network topology (open cluster, guarded cluster, WAN…) batch/job schedulers, MPI stacks, math libraries, compilers.... CMU is not a ‘predefined’ SW supercomputer appliance >90% systems delivered as ‚turn-key solutions‛ CMU can also be purchased standalone with support and manuals 8 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 9. Insight CMU v7.0 tour  CMU functionalities / typical CMU implementation  CMU Provisioning  CMU Monitoring  CMU Scalable / Frictionless administration  CMU Custom GUI & partners integration© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 10. CMU starts here: typical { farm / HPC cluster } implementation { high speed Interconnect } { Highly Avail.}Head node© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 11. Insight Cluster Management Utility Basics CMU is a single package running on the head node (upgrade is trivial)CMU mgt node can be an HA cluster (HP service guard,Redhat Cluster, SLES HA) Provides a full fledged interactive CLI Provides cmu_* commands as an API (for scripting)For integration with other software or command-line activity (see partner’s integration) Provides GUI client for single dashboard controllaunch from a web page served from head node (JAVA© webstart technology)run on a local laptop/desktopuser mode for monitoringadmin mode for remote administrationcontained herein is subject to change without notice. 11 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information
  • 12. The three pillars of HP Insight CMU Provision Monitor Control • Simplified discovery, • ‘At a glance’ view of • GUI & CLI & API firmware audits entire system / • Easy GUI, friction- • Fast & scalable partition less control of cloning • Customizable remote servers • Legacy support of • Lightweight • Scalable pdsh with Kickstart/Autoyast/D • Instant 2D view cmudiff analyser ebian Preseed • TimeView, 3D live • Diskless support history© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 13. Insight CMU v7.0 Tour  CMU Provisioning  Scalable Cloning  Legacy/Compatible Autoinstall (Kickstart/Autoyast/Preseed)  Diskless  Firmware audit  Bare metal netboot low level tools (hpacucli / hponcf / ipmitool)© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 14. CMU provisioning engine backup/cloning (up to 4k nodes)‫‏‬ RHEL (& clones) / SLES ( & clones) / Debian / Ubuntu (only compute nodes)‫‏‬ performance only depends on the image size & harddrive speed  no architectural dependence on trunked/IB/10gig networks, 1Gig ethernet is sufficient. 22 minutes to clone 1000 nodes with SAS drives and 10 GB image  continuous cloning for reprovisionning from batch schedulers autoinstall ‫ ‏‬CMU bridge to legacy/standard tools : Redhat Kickstart / SLES Autoyast / Debian Preseed  do not use above 100 nodes diskless (advised if improving the density of the solution and/or data security) statefull diskless engine (hybrid NFS ro + rw personalities) 14 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 15. CMU firmware / bare metal netboot tools firmware hooks: firmware version checks (HP conrep based currently / HP rcu soon) firmware settings audit (HP conrep + cmudiff) firmware flashing engine (to feed with SCEXE HP files) bare metal netboot tools (available from ‚pre_reconf ‚/ ‚reconf‛ ) hpacucli : configure HP smartarray controllers locfg.pl/ hponcfg: configure HP ILO from the CMU netboot environment ipmitools: configure IPMI capable BMC from the CMU netboot environment 19 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 16. Insight CMU v7.0 Tour CMU Monitoring  scalable / ‘HPC aware’ monitoring engine (collectl, GPGPUS)  2D Instant View / 3D Time View (Live History) © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 17. CMU monitoring Backend: ‚HPC aware‛ monitoring since years  Scalable monitoring ( proven on 4k nodes system )  Non intrusive (leverage collectl + ‚HPC synchro‛ mode)  Programmable (monitor anything you can script )  Nvidia & AMD GPGPUs monitoring tool  Extended Monitoring to inject arbitrary monitoring data  Alerting system & CMU Reactions Frontend GUI (JAVA client/server) / CLI  GUI: Instant view 2D / TimeView 3D (Live History)  cmu_dynamic_user groups (see later in presentation) 21 CLI/API: cmu_monstat and flat human readable files © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 18. ‚Instant View*‛ CMU Display* renamed « Instant View » since CMU v7.022 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 19. 23 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 20. TimeView (Live History)© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 21. Existing ‘well known’ CMU Display since 200425 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 22. 3D Display of Sensor Historiesreadability, efficiency, precision26 2 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 23. 3D Display of Sensor Historiesglobal job overview27 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 24. 28 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 25. 29 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 26. 30 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 27. 31 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 28. GPGPUs monitoring© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 29. CMU GPGPU SupportCMU provides a tool for extracting GPU metric datafrom GPU driver‚cmu_get_nvidia_gpu‛ monitors:load, mem_util, mem_alloc, power_state, and ECC_double_bitalerts by defaultPower_usage, various clock speeds, fan speeds, andtemperature also configured but commented out by default 33 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 30. 37 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.F3ooter goes here
  • 31. Extended monitoring© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 32. CMU Extended MonitoringInject monitoring data from another source into CMUExtended metrics will be used for:Server hardware metrics (ILO4 out-of-band & agentless monitoring)• Temperatures, fan speeds, power usage• Gathered out-of-band• OS-neutralCluster peripherals• MCS temperatures, switch statusWorkload schedulers 39 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 33. CMU alerts & reactions© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 34. CMU alerts & CMU reactionsCMU monitoring engine can trigger alertsCMU alerts can trigger scripts as reactions to alertsreaction examples:• SNMP traps (send all alerts to an SNMP sink such as HPSIM)• Send an email• Remove a compute node from a batch scheduler…44 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 35. Insight CMU v7.0 Tour Scalable / Frictionless monitoring  Interactive command broadcast (ssh, BMC interfaces)  cmudiff non interactive scalable command output analyzer  GUI accelerators (power off / UID leds/ three clicks ‘en masse’ cloning….)© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 36. Insight CMU GUI basicsCluster mgmtpaneldisplays allnodes inselectedgroupings: byswitchlocation; byimage; or bycustomgrouping node states display current state of each node CMU Main Alerts displayed47 Display Panel © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. along the bottom
  • 37. CMU GUI Basics– Right-click to select sensors to display– CMU pre-configured with standard sensors: CPU and memory usage, and disk and network I/O– Simple to add any sensor or alert– CMU provides simple support for monitoring GPU temp and ECC errors– Three clicks to clone compute nodes ! 48 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 38. Friction-less remote control of target nodes Selected Power nodes Broadcast commands commands Provisioning commands User-defined commands49 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 39. CMU remote management commands• Multi-window broadcast command (access OS or console)51 type here… © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. ...and see it there
  • 40. cmudiffscaling the command line.© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 41. Compare node outputs with Scalable Text Analyser (cmudiff) Single-window pdsh with cmu_diff example One command executed across a set of selected nodes… …finds one node running with an old BIOS version! 53 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 42. dshbak vs cmudiff: round #1…. ‘date’ on five hosts cmudiffdshbak 57 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 43. dshbak vs cmudiff: round #2..‘ifconfig’ on 3 hosts cmudiffdshbak 58 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 58 HP Confidential
  • 44. Partners software integration& Custom menu GUI© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 45. 62 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 46. 63 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 47. CMU Custom Menu Support /opt/cmu/etc/cmu_custom_menu64 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 48. cmu_dynamic_user_groups© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 49. Insight CMU as a (job) power monitor70 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 50. Insight CMU Partner Integrations  Moab – Dynamic Provisioning  PBS Pro – Green Scheduling & OS Provisioning  LSF – Platform HPC  ScaleMP – create large virtual SMP nodes  StackIQ – CMU part of HP ‚roll‛  HP Matrix CMU CloudMap© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
  • 51. Thank you for your interest inHP Insight CMU © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.