APOLLO GROUPAutomated Configuration of FirmwareMichael ArnoldPrincipal Systems Engineer27 September 2012
Agenda  Who  Tools  Example Workflow  Example Code  Links  QuestionsAPOLLO GROUP          © 2012 Apollo Group        2
APOLLO GROUP  WhoAPOLLO GROUP Apollo Group          © 2012            3
Who is Apollo?        Apollo Group is a leading provider of higher          education programs for working adults.APOLLO G...
Who is Michael Arnold?  Systems Administrator  Automation geek  13 years in IT  Hadoop Cluster Admin  I deal with:   ...
How does firmware tie in with Puppet?  It has nothing to do with Puppet.      –Unless someone writes a module to program ...
APOLLO GROUP  ToolsAPOLLO GROUP Apollo Group          © 2012            7
Hardware Toolkits  HP SmartStart Scripting Toolkit  Dell OpenManage Deployment Toolkit  IBM ServerGuide Scripting Toolk...
What do the tools provide?  Configuration of      –BIOS      –BMC (DRAC/iLO/RSA/etc.)      –Hardware RAID      –FC HBAAPO...
How does the toolkit work?  Manually configure reference hardware  Capture the firmware configuration  Clone the config...
Toolkit Components    Kernel &   Firmware                                                Toolkit     Initrd       ToolsAPO...
Toolkit Requirements                TFTP         DHCP                          NFS                ToolkitAPOLLO GROUP    ©...
APOLLO GROUP  Example WorkflowAPOLLO GROUP Apollo Group          © 2012            13
Assumptions  Hardware is racked and cabled  Switch ports are up  Server is manually powered on  Firmware defaults are ...
Constraints  All server hardware is from the same vendor (HP)APOLLO GROUP           © 2012 Apollo Group             15
The Process     Firmware   Hardware                  Firmware      Buildsystem      Update    Discovery               Conf...
ScreencastAPOLLO GROUP   © 2012 Apollo Group            17
Workflow  Unknown system netbootsAPOLLO GROUP         © 2012 Apollo Group          18
Workflow  Unknown system netboots  Boot into HP SmartStart Scripting ToolkitAPOLLO GROUP            © 2012 Apollo Group ...
Workflow  Unknown system netboots  Boot into HP SmartStart Scripting Toolkit  Gather hardware information (CPU/RAM/disk...
Workflow  Unknown system netboots  Boot into HP SmartStart Scripting Toolkit  Gather hardware information (CPU/RAM/disk...
Workflow  Unknown system netboots  Boot into HP SmartStart Scripting Toolkit  Gather hardware information (CPU/RAM/disk...
Workflow  Unknown system netboots  Boot into HP SmartStart Scripting Toolkit  Gather hardware information (CPU/RAM/disk...
Workflow  Modify Cobbler system profile, giving it an identity   (hostname, IP, OS metadata)APOLLO GROUP             © 20...
Workflow  Modify Cobbler system profile, giving it an identity   (hostname, IP, OS metadata)  Known system netbootsAPOLL...
Workflow  Modify Cobbler system profile, giving it an identity   (hostname, IP, OS metadata)  Known system netboots  Bo...
Workflow  Modify Cobbler system profile, giving it an identity   (hostname, IP, OS metadata)  Known system netboots  Bo...
Workflow  Modify Cobbler system profile, giving it an identity   (hostname, IP, OS metadata)  Known system netboots  Bo...
Workflow  Modify Cobbler system profile, giving it an identity   (hostname, IP, OS metadata)  Known system netboots  Bo...
Workflow  Modify Cobbler system profile, giving it an identity   (hostname, IP, OS metadata)  Known system netboots  Bo...
APOLLO GROUP  Example CodeAPOLLO GROUP Apollo Group          © 2012            31
Dell TFTP Config  LABEL dtk-install_hadoop    kernel /images/dtk-3.5.1/SA.1    append initrd=/images/dtk-3.5.1/SA.2       ...
HP TFTP Config  LABEL sstk-install_hadoop    kernel /images/sstk-8.50/vmlinuz    append initrd=/images/sstk-8.50/initrd.im...
Dell Config Capture  cd /opt/dell/toolkit/template/scripts  /opt/dell/toolkit/template/scripts/syscap.sh            /tmp/s...
HP Config Capture  cd /TOOLKIT  ./conrep -s -f /tmp/conrep.dat                            # BIOS  ./hpacuscripting -c /tmp...
Platform Detection  *** Detecting system type ***      System Information:          Name      : ProLiant DL360 G5         ...
Some Available Variables   HP SSSTK has a function library that sets many variables.      – $PRODUCT_NAME =          • "P...
Inside install_hadoop.sh   Set Variables Based on Hardware Type  export "`./hwquery ${HWDISC_FILE} allboards.xml SERVERNA...
Inside install_hadoop.sh  #...  "ProLiant DL360 G5"* )       CONREPDAT=${TOOLKIT_MNTPNT}/data_files/dl360g5_conrep.dat    ...
Inside install_hadoop.sh  if [ -n "$CONREPDAT" ] ; then    echo "** Applying BIOS Configuration"    ./conrep -l -f ${CONRE...
Register with Buildsystem  echo "** Registering with cobbler"  # Koan does not come with SSSTK, so "install" it.  ${TOOLKI...
Clear the HP RAID Controller  $ cat dl360g5_saE200i_cpqacuxe.dat_clear  ; Version: 8.50.6.0  ; http://h30499.www3.hp.com/t...
Configure the HP RAID Controller  $ cat dl360g5_saE200i_cpqacuxe.dat         ; Array Specifications  ; Version: 8.50.6.0  ...
Error Handling  You may have to test for any failures when   applying configurations.  When reusing HP hardware, adding ...
Summary     Firmware   Hardware                  Firmware      Buildsystem      Update    Discovery               Configur...
Links   HP SmartStart Scripting Toolkit      http://h18013.www1.hp.com/products/servers/management/toolkit/index.html   ...
Contact   https://intelligentsysadmin.wordpress.com/   https://github.com/razorsedge   http://forge.puppetlabs.com/razo...
APOLLO GROUP  Questions?APOLLO GROUP Apollo Group          © 2012            48
Upcoming SlideShare
Loading in …5
×

Automated Configuration of Firmware

2,788 views

Published on

PuppetConf 2012 - Everyone hears about automation of OS installs and configuration of services on top of the OS yet no one talks about automating the configuration of firmware. There are a number of venders who provide tools that allow the intrepid systems administrator to never again have to arrow through BIOS menus to toggle some less known but needed setting. Gone are the days of manually entering disk configurations into a RAID controller. This session is for SysAdmins or hardware geeks that want to free up their time to enjoy the finer things in life. You will be presented with the knowledge gained from two years of experience managing firmware configuration as a part of hardware discovery and provisioning.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Automated Configuration of Firmware

  1. 1. APOLLO GROUPAutomated Configuration of FirmwareMichael ArnoldPrincipal Systems Engineer27 September 2012
  2. 2. Agenda Who Tools Example Workflow Example Code Links QuestionsAPOLLO GROUP © 2012 Apollo Group 2
  3. 3. APOLLO GROUP WhoAPOLLO GROUP Apollo Group © 2012 3
  4. 4. Who is Apollo? Apollo Group is a leading provider of higher education programs for working adults.APOLLO GROUP © 2012 Apollo Group 4
  5. 5. Who is Michael Arnold? Systems Administrator Automation geek 13 years in IT Hadoop Cluster Admin I deal with: –Server hardware specification/configuration –Server firmware –Server operating system –Hadoop Cluster performance and availabilityAPOLLO GROUP © 2012 Apollo Group 5
  6. 6. How does firmware tie in with Puppet? It has nothing to do with Puppet. –Unless someone writes a module to program firmware from the running OS. • Which could work for RAID controllers.APOLLO GROUP © 2012 Apollo Group 6
  7. 7. APOLLO GROUP ToolsAPOLLO GROUP Apollo Group © 2012 7
  8. 8. Hardware Toolkits HP SmartStart Scripting Toolkit Dell OpenManage Deployment Toolkit IBM ServerGuide Scripting Toolkit Fujitsu ServerView Scripting ToolkitAPOLLO GROUP © 2012 Apollo Group 8
  9. 9. What do the tools provide? Configuration of –BIOS –BMC (DRAC/iLO/RSA/etc.) –Hardware RAID –FC HBAAPOLLO GROUP © 2012 Apollo Group 9
  10. 10. How does the toolkit work? Manually configure reference hardware Capture the firmware configuration Clone the configuration to multiple machinesAPOLLO GROUP © 2012 Apollo Group 10
  11. 11. Toolkit Components Kernel & Firmware Toolkit Initrd ToolsAPOLLO GROUP © 2012 Apollo Group 11
  12. 12. Toolkit Requirements TFTP DHCP NFS ToolkitAPOLLO GROUP © 2012 Apollo Group 12
  13. 13. APOLLO GROUP Example WorkflowAPOLLO GROUP Apollo Group © 2012 13
  14. 14. Assumptions Hardware is racked and cabled Switch ports are up Server is manually powered on Firmware defaults are to boot from network if no other boot device is found. –No other boot devices are foundAPOLLO GROUP © 2012 Apollo Group 14
  15. 15. Constraints All server hardware is from the same vendor (HP)APOLLO GROUP © 2012 Apollo Group 15
  16. 16. The Process Firmware Hardware Firmware Buildsystem Update Discovery Configuration RegistrationAPOLLO GROUP © 2012 Apollo Group 16
  17. 17. ScreencastAPOLLO GROUP © 2012 Apollo Group 17
  18. 18. Workflow Unknown system netbootsAPOLLO GROUP © 2012 Apollo Group 18
  19. 19. Workflow Unknown system netboots Boot into HP SmartStart Scripting ToolkitAPOLLO GROUP © 2012 Apollo Group 19
  20. 20. Workflow Unknown system netboots Boot into HP SmartStart Scripting Toolkit Gather hardware information (CPU/RAM/disk/NIC) via hpdiscoveryAPOLLO GROUP © 2012 Apollo Group 20
  21. 21. Workflow Unknown system netboots Boot into HP SmartStart Scripting Toolkit Gather hardware information (CPU/RAM/disk/NIC) via hpdiscovery If this is HP hardware, then –Update firmware –Configure BIOS –Configure iLO –Configure RAIDAPOLLO GROUP © 2012 Apollo Group 21
  22. 22. Workflow Unknown system netboots Boot into HP SmartStart Scripting Toolkit Gather hardware information (CPU/RAM/disk/NIC) via hpdiscovery If this is HP hardware, then –Update firmware –Configure BIOS –Configure iLO –Configure RAID cobbler-register to the hardware profileAPOLLO GROUP © 2012 Apollo Group 22
  23. 23. Workflow Unknown system netboots Boot into HP SmartStart Scripting Toolkit Gather hardware information (CPU/RAM/disk/NIC) via hpdiscovery If this is HP hardware, then –Update firmware –Configure BIOS –Configure iLO –Configure RAID cobbler-register to the hardware profile System rebootsAPOLLO GROUP © 2012 Apollo Group 23
  24. 24. Workflow Modify Cobbler system profile, giving it an identity (hostname, IP, OS metadata)APOLLO GROUP © 2012 Apollo Group 24
  25. 25. Workflow Modify Cobbler system profile, giving it an identity (hostname, IP, OS metadata) Known system netbootsAPOLLO GROUP © 2012 Apollo Group 25
  26. 26. Workflow Modify Cobbler system profile, giving it an identity (hostname, IP, OS metadata) Known system netboots Boot into OS installerAPOLLO GROUP © 2012 Apollo Group 26
  27. 27. Workflow Modify Cobbler system profile, giving it an identity (hostname, IP, OS metadata) Known system netboots Boot into OS installer Install OSAPOLLO GROUP © 2012 Apollo Group 27
  28. 28. Workflow Modify Cobbler system profile, giving it an identity (hostname, IP, OS metadata) Known system netboots Boot into OS installer Install OS Install PuppetAPOLLO GROUP © 2012 Apollo Group 28
  29. 29. Workflow Modify Cobbler system profile, giving it an identity (hostname, IP, OS metadata) Known system netboots Boot into OS installer Install OS Install Puppet …APOLLO GROUP © 2012 Apollo Group 29
  30. 30. Workflow Modify Cobbler system profile, giving it an identity (hostname, IP, OS metadata) Known system netboots Boot into OS installer Install OS Install Puppet … Profit!APOLLO GROUP © 2012 Apollo Group 30
  31. 31. APOLLO GROUP Example CodeAPOLLO GROUP Apollo Group © 2012 31
  32. 32. Dell TFTP Config LABEL dtk-install_hadoop kernel /images/dtk-3.5.1/SA.1 append initrd=/images/dtk-3.5.1/SA.2 ramdisk_size=156482 Stage3_type=cdrom quiet DEBUG=0 share_location=192.168.1.1:/srv/nfs/ro/dtk-3.5.1 share_type=nfs selinux=0 share_opts=ro,nolock share_script=install_hadoop.sh cprofile=Hadoop ipappend 2APOLLO GROUP © 2012 Apollo Group 32
  33. 33. HP TFTP Config LABEL sstk-install_hadoop kernel /images/sstk-8.50/vmlinuz append initrd=/images/sstk-8.50/initrd.img root=/dev/ram0 rw ramdisk_size=184248 ide=nodma ide=noraid pnpbios=off network=1 sstk_mount=192.168.1.1:/srv/nfs/ro/sstk-8.50 sstk_mount_type=nfs sstk_mount_options=ro,nolock sstk_script=install_hadoop.sh cprofile=Hadoop ipappend 2APOLLO GROUP © 2012 Apollo Group 33
  34. 34. Dell Config Capture cd /opt/dell/toolkit/template/scripts /opt/dell/toolkit/template/scripts/syscap.sh /tmp/syscap /opt/dell/toolkit/template/scripts/raidcap.sh /tmp/raidcap /opt/dell/toolkit/template/scripts/raccap.sh /tmp/raccap # MegaCLI does not come with the DTK so "install" it. cp -a /opt/dell/toolkit/systems/MegaCLI/opt/MegaRAID /opt/MegaRAID /opt/MegaRAID/MegaCli/MegaCli -CfgSave -a0 -f /tmp/mega.0APOLLO GROUP © 2012 Apollo Group 34
  35. 35. HP Config Capture cd /TOOLKIT ./conrep -s -f /tmp/conrep.dat # BIOS ./hpacuscripting -c /tmp/cpqacuxe.dat # RAID ./hplpcfg /s /tmp/hplpcfg.dat # Emulex HBA ./hpqlarep /S /tmp/hpqlarep.dat # Qlogic HBA if ./ifhw /tmp/hw_discovery.dat allboards.xml "PCI:Integrated Lights-Out" 2> /dev/null ; then ./hponcfg -w /tmp/hponcfg.dat # iLO 2/3 else ./lo100cfg -o /tmp/lo100cfg.dat # LO100i fiAPOLLO GROUP © 2012 Apollo Group 35
  36. 36. Platform Detection *** Detecting system type *** System Information: Name : ProLiant DL360 G5 Serial : USE123D456 Arch : x86_64 LNAME : ProLiantDL360G5 SNAME : DL360 G5 Asset Tag : 01234567APOLLO GROUP © 2012 Apollo Group 36
  37. 37. Some Available Variables  HP SSSTK has a function library that sets many variables. – $PRODUCT_NAME = • "ProLiant SL170z G6" • "ProLiant DL360 G6" – $SERIAL_ID = hardware serial number – $PRODUCT_LNAME = • "ProLiantSL170zG6" • "ProLiantDL360G6" – $PRODUCT_SNAME = • "SL170z G6" • "DL360 G6" – $TOOLKIT_MNTPNT = /mnt/main – $TOOLKIT_SERVER_IP = NFS server IP – $HWDISC_FILE = /TOOLKIT/hpdiscovery.xmlAPOLLO GROUP © 2012 Apollo Group 37
  38. 38. Inside install_hadoop.sh  Set Variables Based on Hardware Type export "`./hwquery ${HWDISC_FILE} allboards.xml SERVERNAME=SystemName`" case "${SERVERNAME}" in "ProLiant DL160 G6"* ) CONREPDAT=${TOOLKIT_MNTPNT}/data_files/dl160_conrep.dat LO100CFGDAT=${TOOLKIT_MNTPNT}/data_files/dl160_lo100cfg.dat ;; "ProLiant SL170z G6"* ) CONREPDAT="${TOOLKIT_MNTPNT}/data_files/sl170z_conrep.dat -x ${TOOLKIT_MNTPNT}/conrep_xml/conrep_SL170zg6_20100401.xml" LO100CFGDAT=${TOOLKIT_MNTPNT}/data_files/sl170z_lo100cfg.dat ;; #...APOLLO GROUP © 2012 Apollo Group 38
  39. 39. Inside install_hadoop.sh #... "ProLiant DL360 G5"* ) CONREPDAT=${TOOLKIT_MNTPNT}/data_files/dl360g5_conrep.dat if ./ifhw ${HWDISC_FILE} allboards.xml "PCI:Smart Array E200i Controller" 2> /dev/null ; then HPACUDAT=${TOOLKIT_MNTPNT}/data_files/dl360g5_saE200i_cpqacuxe.dat fi if ./ifhw ${HWDISC_FILE} allboards.xml "PCI:Smart Array P400i Controller" 2> /dev/null ; then HPACUDAT=${TOOLKIT_MNTPNT}/data_files/dl360g5_saP400i_cpqacuxe.dat fi HPONCFGDAT=${TOOLKIT_MNTPNT}/data_files/dl360g5_hponcfg.dat ;; esacAPOLLO GROUP © 2012 Apollo Group 39
  40. 40. Inside install_hadoop.sh if [ -n "$CONREPDAT" ] ; then echo "** Applying BIOS Configuration" ./conrep -l -f ${CONREPDAT} fi if [ -n "$HPACUDAT" ] ; then echo "** Clearing RAID Configuration" ./hpacucli/hpacuscripting -i ${HPACUDAT}_clear; sleep 5 echo "** Applying RAID Configuration" ./hpacucli/hpacuscripting -i ${HPACUDAT} fi if [ -n "$LO100CFGDAT" ] ; then echo "** Applying BMC Configuration"; ./lo100cfg -i ${LO100CFGDAT} elif [ -n "$HPONCFGDAT" ] ; then echo "** Applying iLO Configuration"; ./hponcfg -f ${HPONCFGDAT} fiAPOLLO GROUP © 2012 Apollo Group 40
  41. 41. Register with Buildsystem echo "** Registering with cobbler" # Koan does not come with SSSTK, so "install" it. ${TOOLKIT_MNTPNT}/koan/install # cprofile is set via kernel boot parameter and points at part of a # Cobbler profile name. # SERIAL_ID and PRODUCT_LNAME are set via SSSTK. # PRODUCT_LNAME = ProLiantSL170zG6, ProLiantDL360G6, etc. # SERIAL_ID = hardware serial number echo "** Registering to profile ${cprofile}-${PRODUCT_LNAME}" cobbler-register --server=${TOOLKIT_SERVER_IP} --fqdn=${SERIAL_ID} --profile=${cprofile}-${PRODUCT_LNAME} --batchAPOLLO GROUP © 2012 Apollo Group 41
  42. 42. Clear the HP RAID Controller $ cat dl360g5_saE200i_cpqacuxe.dat_clear ; Version: 8.50.6.0 ; http://h30499.www3.hp.com/t5/ProLiant-Servers-ML-DL- SL/CPQACUXE-reset-utility-within-WinPE/td-p/3939475 Action= Reconfigure Method= Custom Controller= SLOT 0 ClearConfigurationWithDataLoss= YesAPOLLO GROUP © 2012 Apollo Group 42
  43. 43. Configure the HP RAID Controller $ cat dl360g5_saE200i_cpqacuxe.dat ; Array Specifications ; Version: 8.50.6.0 Array= A Action= Configure ; Array Drive Type is SAS Method= Custom ; 1I:1:1 (72 GB), 1I:1:2 (72 GB) ; Controller Specifications Drive= 1I:1:1, 1I:1:2 ; Controller HP Smart Array E200i OnlineSpare= No ; Firmware Version 1.82 ; Logical Drive Specifications Controller= SLOT 0 LogicalDrive= 1 ReadCache= 100 RAID= 1 WriteCache= 0 Size= 69973 RebuildPriority= Medium Sectors= 32 ExpandPriority= Medium StripeSize= 128 SurfaceScanDelay= 15 ArrayAccelerator= Enabled DriveWriteCache= DisabledAPOLLO GROUP © 2012 Apollo Group 43
  44. 44. Error Handling You may have to test for any failures when applying configurations. When reusing HP hardware, adding a user to the BMC that already exists will cause a failure to apply all of the configuration. Newer firmware can have different config options that will cause an older capture to fail.APOLLO GROUP © 2012 Apollo Group 44
  45. 45. Summary Firmware Hardware Firmware Buildsystem Update Discovery Configuration RegistrationAPOLLO GROUP © 2012 Apollo Group 45
  46. 46. Links  HP SmartStart Scripting Toolkit http://h18013.www1.hp.com/products/servers/management/toolkit/index.html  Dell OpenManage Deployment Toolkit http://en.community.dell.com/techcenter/systems-management/w/wiki/1772.dell- openmanage-deployment-toolkit.aspx  IBM ServerGuide Scripting Toolkit http://www-03.ibm.com/systems/be/management/sgstk/  Fujitsu ServerView Scripting Toolkit http://www.fujitsu.com/fts/products/computing/servers/primergy/management/de ploy/  Cobbler - Linux Installation Server http://cobbler.github.com/  LSI MegaRAID MegaCLI http://www.lsi.com/downloads/Public/MegaRAID%20Common%20Files/8.02.24_ MegaCLI.zipAPOLLO GROUP © 2012 Apollo Group 46
  47. 47. Contact  https://intelligentsysadmin.wordpress.com/  https://github.com/razorsedge  http://forge.puppetlabs.com/razorsedgeAPOLLO GROUP © 2012 Apollo Group 47
  48. 48. APOLLO GROUP Questions?APOLLO GROUP Apollo Group © 2012 48

×