• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Hp cmu – easy to use cluster management utility @ hpcday 2012 kiev
 

Hp cmu – easy to use cluster management utility @ hpcday 2012 kiev

on

  • 1,423 views

 

Statistics

Views

Total Views
1,423
Views on SlideShare
1,063
Embed Views
360

Actions

Likes
0
Downloads
0
Comments
0

1 Embed 360

http://supercomputers.kiev.ua 360

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Hp cmu – easy to use cluster management utility @ hpcday 2012 kiev Hp cmu – easy to use cluster management utility @ hpcday 2012 kiev Presentation Transcript

    • HP Insight CMUCluster Management Utility TourSébastien Cabaniols, CMU WW Team lead / EMEA HPC Presales consultant7th HPC Day conference, Kiev, UkraineOctober 2012© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • Agenda HP Insight CMU Introduction & Review  Introduction, History / Customers  Product mindset Insight CMU v7.0 tour  Provisioning (Cloning / Autoinstall / Diskless)  Monitoring ( TimeView / Collectl / GPGPUs…)  Scalable/Frictionless administration ( cmudiff…)  Custom GUI & partners integration© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • Insight CMUIntroduction & History© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • Insight CMU introduction CMU = Cluster Management Utility ‚CMU optimizes the TCO of compute farms‛ CMU scaling specification: 4k nodes CMU has lots of industrial clusters in production with 2k/3k+ nodes CMU has a strong presence in the TOP500 (www.top500.org) CMU at customer site since 2000 4 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • Insight CMU major milestones2000: initial implementation for Tru64 Unix (Alphaserver)‫‏‬2001: port to Alpha Linux, 1600 servers commercial cluster2002: port to x86 & IA64 Linux / HPUX Itanium2004: port to x86_64 Linux. (only port maintained*)2007: Swedish gov, 6th @ TOP5002010: Tsubame 2, HP first public 1+ PFlop cluster, 5th @ TOP500*ARM port in progress... 5 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • Worldwide CMU Deployments HP ships >2 CMU clusters per week WW UNIVERSITIES ENGINEERING GOVERNMENT and RESEARCH LABS ENERGY6 6 April 2009 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • Insight CMU project mindset© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • Insight CMU project mindset CMU provides the core functionalities for a farm/cloud runs any HP* server (even mix) / any Linux distribution (even mix)‫‏‬ independent of many architectural aspects of the system: interconnects / GPGPUs / IO accelerators... network topology (open cluster, guarded cluster, WAN…) batch/job schedulers, MPI stacks, math libraries, compilers.... CMU is not a ‘predefined’ SW supercomputer appliance >90% systems delivered as ‚turn-key solutions‛ CMU can also be purchased standalone with support and manuals 8 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • Insight CMU v7.0 tour  CMU functionalities / typical CMU implementation  CMU Provisioning  CMU Monitoring  CMU Scalable / Frictionless administration  CMU Custom GUI & partners integration© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • CMU starts here: typical { farm / HPC cluster } implementation { high speed Interconnect } { Highly Avail.}Head node© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • Insight Cluster Management Utility Basics CMU is a single package running on the head node (upgrade is trivial)CMU mgt node can be an HA cluster (HP service guard,Redhat Cluster, SLES HA) Provides a full fledged interactive CLI Provides cmu_* commands as an API (for scripting)For integration with other software or command-line activity (see partner’s integration) Provides GUI client for single dashboard controllaunch from a web page served from head node (JAVA© webstart technology)run on a local laptop/desktopuser mode for monitoringadmin mode for remote administrationcontained herein is subject to change without notice. 11 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information
    • The three pillars of HP Insight CMU Provision Monitor Control • Simplified discovery, • ‘At a glance’ view of • GUI & CLI & API firmware audits entire system / • Easy GUI, friction- • Fast & scalable partition less control of cloning • Customizable remote servers • Legacy support of • Lightweight • Scalable pdsh with Kickstart/Autoyast/D • Instant 2D view cmudiff analyser ebian Preseed • TimeView, 3D live • Diskless support history© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • Insight CMU v7.0 Tour  CMU Provisioning  Scalable Cloning  Legacy/Compatible Autoinstall (Kickstart/Autoyast/Preseed)  Diskless  Firmware audit  Bare metal netboot low level tools (hpacucli / hponcf / ipmitool)© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • CMU provisioning engine backup/cloning (up to 4k nodes)‫‏‬ RHEL (& clones) / SLES ( & clones) / Debian / Ubuntu (only compute nodes)‫‏‬ performance only depends on the image size & harddrive speed  no architectural dependence on trunked/IB/10gig networks, 1Gig ethernet is sufficient. 22 minutes to clone 1000 nodes with SAS drives and 10 GB image  continuous cloning for reprovisionning from batch schedulers autoinstall ‫ ‏‬CMU bridge to legacy/standard tools : Redhat Kickstart / SLES Autoyast / Debian Preseed  do not use above 100 nodes diskless (advised if improving the density of the solution and/or data security) statefull diskless engine (hybrid NFS ro + rw personalities) 14 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • CMU firmware / bare metal netboot tools firmware hooks: firmware version checks (HP conrep based currently / HP rcu soon) firmware settings audit (HP conrep + cmudiff) firmware flashing engine (to feed with SCEXE HP files) bare metal netboot tools (available from ‚pre_reconf ‚/ ‚reconf‛ ) hpacucli : configure HP smartarray controllers locfg.pl/ hponcfg: configure HP ILO from the CMU netboot environment ipmitools: configure IPMI capable BMC from the CMU netboot environment 19 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • Insight CMU v7.0 Tour CMU Monitoring  scalable / ‘HPC aware’ monitoring engine (collectl, GPGPUS)  2D Instant View / 3D Time View (Live History) © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • CMU monitoring Backend: ‚HPC aware‛ monitoring since years  Scalable monitoring ( proven on 4k nodes system )  Non intrusive (leverage collectl + ‚HPC synchro‛ mode)  Programmable (monitor anything you can script )  Nvidia & AMD GPGPUs monitoring tool  Extended Monitoring to inject arbitrary monitoring data  Alerting system & CMU Reactions Frontend GUI (JAVA client/server) / CLI  GUI: Instant view 2D / TimeView 3D (Live History)  cmu_dynamic_user groups (see later in presentation) 21 CLI/API: cmu_monstat and flat human readable files © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • ‚Instant View*‛ CMU Display* renamed « Instant View » since CMU v7.022 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • 23 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • TimeView (Live History)© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • Existing ‘well known’ CMU Display since 200425 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • 3D Display of Sensor Historiesreadability, efficiency, precision26 2 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • 3D Display of Sensor Historiesglobal job overview27 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • 28 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • 29 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • 30 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • 31 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • GPGPUs monitoring© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • CMU GPGPU SupportCMU provides a tool for extracting GPU metric datafrom GPU driver‚cmu_get_nvidia_gpu‛ monitors:load, mem_util, mem_alloc, power_state, and ECC_double_bitalerts by defaultPower_usage, various clock speeds, fan speeds, andtemperature also configured but commented out by default 33 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • 37 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.F3ooter goes here
    • Extended monitoring© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • CMU Extended MonitoringInject monitoring data from another source into CMUExtended metrics will be used for:Server hardware metrics (ILO4 out-of-band & agentless monitoring)• Temperatures, fan speeds, power usage• Gathered out-of-band• OS-neutralCluster peripherals• MCS temperatures, switch statusWorkload schedulers 39 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • CMU alerts & reactions© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • CMU alerts & CMU reactionsCMU monitoring engine can trigger alertsCMU alerts can trigger scripts as reactions to alertsreaction examples:• SNMP traps (send all alerts to an SNMP sink such as HPSIM)• Send an email• Remove a compute node from a batch scheduler…44 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • Insight CMU v7.0 Tour Scalable / Frictionless monitoring  Interactive command broadcast (ssh, BMC interfaces)  cmudiff non interactive scalable command output analyzer  GUI accelerators (power off / UID leds/ three clicks ‘en masse’ cloning….)© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • Insight CMU GUI basicsCluster mgmtpaneldisplays allnodes inselectedgroupings: byswitchlocation; byimage; or bycustomgrouping node states display current state of each node CMU Main Alerts displayed47 Display Panel © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. along the bottom
    • CMU GUI Basics– Right-click to select sensors to display– CMU pre-configured with standard sensors: CPU and memory usage, and disk and network I/O– Simple to add any sensor or alert– CMU provides simple support for monitoring GPU temp and ECC errors– Three clicks to clone compute nodes ! 48 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • Friction-less remote control of target nodes Selected Power nodes Broadcast commands commands Provisioning commands User-defined commands49 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • CMU remote management commands• Multi-window broadcast command (access OS or console)51 type here… © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. ...and see it there
    • cmudiffscaling the command line.© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • Compare node outputs with Scalable Text Analyser (cmudiff) Single-window pdsh with cmu_diff example One command executed across a set of selected nodes… …finds one node running with an old BIOS version! 53 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • dshbak vs cmudiff: round #1…. ‘date’ on five hosts cmudiffdshbak 57 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • dshbak vs cmudiff: round #2..‘ifconfig’ on 3 hosts cmudiffdshbak 58 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 58 HP Confidential
    • Partners software integration& Custom menu GUI© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • 62 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • 63 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • CMU Custom Menu Support /opt/cmu/etc/cmu_custom_menu64 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • cmu_dynamic_user_groups© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • Insight CMU as a (job) power monitor70 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • Insight CMU Partner Integrations  Moab – Dynamic Provisioning  PBS Pro – Green Scheduling & OS Provisioning  LSF – Platform HPC  ScaleMP – create large virtual SMP nodes  StackIQ – CMU part of HP ‚roll‛  HP Matrix CMU CloudMap© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
    • Thank you for your interest inHP Insight CMU © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.