Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Isn't it ironic - managing a bare metal cloud (OSL TES 2015)

2,515 views

Published on

Published in: Technology

Isn't it ironic - managing a bare metal cloud (OSL TES 2015)

  1. 1. Isn't it Ironic? Managing a bare metal cloud Devananda van der Veen http://github.com/devananda/talks/isnt­it­ironic.html twitter: @devananda
  2. 2. Who am I Master Engineer at HP OpenStack Technical Committee OpenStack Ironic PTL
  3. 3. What we're going to talk about Virtualization & OpenStack Ironic's architecture Configuration choices you need to make Operators <­> Developers Walk through a deploy Multitenancy Some other cool stuff
  4. 4. Why is virtualization important?
  5. 5. "OpenStack is not a virtualization layer. It is an abstraction layer." ­ Daniel Sabbah, CTO @ IBM
  6. 6. Compute Virtualization Drivers (KVM, Xen, HyperV ...) flexibility: dev controls their whole environment isolation: guests don't share resources ... but noisy neighbors
  7. 7. Compute Containerization Drivers (lxc, docker ...) specificity: just your app, nothing more density: far more instances per server ... but shared kernel, memory, and other resources
  8. 8. Compute Physicalization Drivers (Ironic) raw performance non­virtualizable workloads (eg, GPUs) ... but no hypervisor
  9. 9. Ironic is: An OpenStack service A Nova driver also capable of being used stand­ alone
  10. 10. Design Goals Be resilient to hardware and service failures Predictable and horizontal scale­out Common API across hardware vendors Common API for both virtual and physical resources
  11. 11. Key Components ironic­api: RESTful API service ironic­conductor: interacts directly with hardware; asynchronous handling of both requested and periodic actions. deploy ramdisk: temporary agent, booted on machines, to provide remote access to hardware for provisioning and management. Nova driver: interface for Nova; enables OpenStack to provide common abstraction for virtual and physical machines. discoverd ramdisk: optional tool for hardware inventory management
  12. 12. Key Technologies PXE: pre­boot execution environment, allows host to boot from network DHCP: dynamic host configuration protocol, used to locate the NBP on the network, and provide the host OS with IP address during init IPMI: intelligent platform management interface, for remote control of machine power state, boot device, serial console, etc. TFTP: trivial file transfer protocol, copies the NBP over the network iSCSI: used to remotely attach to HDD and copy the machine image
  13. 13. But you have options... ! PXE: use virtual media channel instead of PXE. Requires Swift; available in Kilo; limited to "ilo" drivers. ! DHCP: static IP injection is possible, but not really suitable for larger or dynamic environments ! IPMI: many other out­of­band power management tools supported ! TFTP: iPXE / HTTP(S) supported, too ! iSCSI: image can be fetched directly by "agent" drivers
  14. 14. ... and options ... Homogeneous hardware?  Easy! Heterogeneous hardware?  Use nova­scheduler to match flavors to node.properties Single tenant / small deployment?  Flat network. Maybe use Ironic stand­alone Service provider for multiple tenants?  Use keystone for auth, nova for quota management, ...  Basically, use OpenStack Untrusted tenants?  Network isolation is possible (but not trivial) via Neutron  Secure­erase disks, flash firmware between each use
  15. 15. Operators <­> Developers
  16. 16. Examples
  17. 17. Enroll Hardware $ironicnode-create-dagent_ipmitool -iipmi_username=admin-iipmi_password=fake-iipmi_address=10.1.2.3 -pcpus=4-pmemory_mb=8192-plocal_gb=500 -enote='spareserver'-nmytest +--------------+-------------------------------------------------------------+ |Property |Value | +--------------+-------------------------------------------------------------+ |chassis_uuid|None | |driver |agent_ipmitool | |driver_info |{u'ipmi_address':u'10.1.2.3',u'ipmi_username':u'admin', | | |u'ipmi_password':u'******'} | |extra |{u'note':u'spareserver'} | |properties |{u'memory_mb':u'8192',u'local_gb':u'500',u'cpus':u'4'}| |uuid |7a1ce8d0-9679-4d87-8f54-b11f6e8adb8f | |name |mytest | +--------------+-------------------------------------------------------------+ $ironicport-create-n7a1ce8d0-9679-4d87-8f54-b11f6e8adb8f-a00:11:22:00:11:22 +-----------+--------------------------------------+ |Property |Value | +-----------+--------------------------------------+ |node_uuid|7a1ce8d0-9679-4d87-8f54-b11f6e8adb8f| |extra |{} | |uuid |024e52b2-6ae4-483b-a039-d6afae7f6a22| |address |00:11:22:00:11:22 | +-----------+--------------------------------------+
  18. 18. Validate provided info $ironicnode-validate7a1ce8d0-9679-4d87-8f54-b11f6e8adb8f +------------+--------+-------------------------------------------------------------- |Interface |Result|Reason +------------+--------+-------------------------------------------------------------- |console |False |Missing'ipmi_terminal_port'parameterinnode'sdriver_info. |deploy |False |Node7a1ce8d0-9679-4d87-8f54-b11f6e8adb8ffailedtovalidate deployimageinfo.Someparametersweremissing.Missingare: ['driver_info.deploy_kernel','driver_info.deploy_ramdisk','instance_info.image_source']| |inspect |None |notsupported |management|True | |power |True | +------------+--------+--------------------------------------------------------------
  19. 19. (I forgot a few options) Oops
  20. 20. Add or change options $ironicnode-updatemytestadd instance_info/image_source=http://192.168.1.1/myimage.qcow2 instance_info/image_checksum=e1d99d6d0ef2144a8d672b0420c547b5 $ironicnode-updatemytestadd driver_info/deploy_ramdisk=http://192.168.1.1/deploy.initrd driver_info/deploy_kernel=http://192.168.1.1/deploy.vmlinuz $ironicnode-updatemytestreplaceextra/note='database'name=db01.example +------------------------+------------------------------------------------- |Property |Value +------------------------+------------------------------------------------- |extra |{u'note':u'database'} |name |db01.example
  21. 21. Validate info (again) $ironicnode-validatedb01.example +------------+--------+---------------------------------------------------------------+ |Interface |Result|Reason | +------------+--------+---------------------------------------------------------------+ |console |False |Missing'ipmi_terminal_port'parameterinnode'sdriver_info.| |deploy |True | | |inspect |None |notsupported | |management|True | | |power |True | | +------------+--------+---------------------------------------------------------------+
  22. 22. Show details $ironicnode-showdb01.example +------------------------+------------------------------------------------------------ |Property |Value +------------------------+------------------------------------------------------------ |target_power_state |None |last_error | |maintenance_reason | |provision_state |available |console_enabled |False |target_provision_state|None |maintenance |False |power_state |poweroff |driver |agent_ipmitool |reservation |None |instance_uuid |None |driver_internal_info |{} |chassis_uuid |
  23. 23. Maintenance Mode $ironicnode-set-maintenance--reason'replacingdisks'db01.exampletrue $ironicnode-showdb01.example +------------------------+------------------------------------------------------------ |Property |Value +------------------------+------------------------------------------------------------ |target_power_state |None |last_error | |maintenance_reason |replacingdisks |provision_state |available |console_enabled |False |target_provision_state|None |maintenance |True |power_state |poweroff |instance_uuid |None |driver_internal_info |{}
  24. 24. Power Status Loop $ironicnode-showmy.broken.node +-----------------+----------------------------------------------------------------------+ |Property |Value | +-----------------+----------------------------------------------------------------------+ |last_error |Duringsync_power_state,maxretriesexceededfornode | | |9729f0b2-b270-4d06-aa87-40f2b2cad6ee,nodestateNonedoesnotmatch| | |expectedstate'off'.UpdatingDBstateto'None'Switchingnodeto | | |maintenancemode. | $cat/var/log/upstart/ironic-conductor.log 2015-03-2404:29:19.34926317WARNINGironic.conductor.manager[-] Duringsync_power_state,couldnotgetpowerstatefornode9729f0b2-b270-4d06-aa87-40f2b2cad6ee. Error:IPMIcallfailed:powerstatus.
  25. 25. Deployment (via Ironic) $ironicnode-set-provision-statedb01.exampleactive Theprovisioningoperationcan'tbeperformedonnode 7a1ce8d0-9679-4d87-8f54-b11f6e8adb8fbecauseit'sinmaintenancemode. $ironicnode-set-maintenancedb01.examplefalse $ironicnode-set-provision-statedb01.exampleactive $ #...timegoeson...
  26. 26. Deployment (via Ironic) $ironicnode-showdb01.example +------------------------+------------------------------------- |Property |Value +------------------------+------------------------------------- |target_power_state |None |last_error | |maintenance_reason |None |provision_state |active |console_enabled |False |target_provision_state|None |maintenance |False |power_state |poweron |instance_uuid |None |driver_internal_info |{}
  27. 27. Deployment (via Nova) $novaboot–flavorbaremetal-imagemyimage-key-namemy_ssh_key... $tail-f/var/log/upsart/nova-compute.log ... 2014-05-0103:47:05.878AUDITnova.compute.resource_tracker[-]Freeram(MB):8192 2014-05-0103:47:05.878AUDITnova.compute.resource_tracker[-]Freedisk(GB):500 2014-05-0103:47:05.878AUDITnova.compute.resource_tracker[-]FreeVCPUS:4 ... 2014-05-0103:47:05.878AUDITnova.compute.resource_tracker[-]Freeram(MB):0 2014-05-0103:47:05.878AUDITnova.compute.resource_tracker[-]Freedisk(GB):0 2014-05-0103:47:05.878AUDITnova.compute.resource_tracker[-]FreeVCPUS:0
  28. 28. Two methods for image deployment Direct from source || Cache on conductor agent_ipmitool agent_pyghmi agent_ilo pxe_ipmitool pxe_ipminative pxe_seamicro pxe_iboot pxe_ilo pxe_snmp pxe_drac pxe_irmc pxe_amt iscsi_ilo
  29. 29. PXE Deploy Process
  30. 30. PXE Deploy Process (cont)
  31. 31. Agent Deploy Process
  32. 32. Agent Deploy Process (cont)
  33. 33. (there's a hash ring) Fault Tolerance Each node is mapped across the set of conductors which have enabled that node's driver, using a consistent hash ring. Starting or stopping a conductor will initiate a rebalance. Periodic task ensures consistent local state for mapped nodes. No impact to running instances, but may disrupt reboots (if nodes netboot)
  34. 34. Example $ironicdriver-list +---------------------+----------------+ |Supporteddriver(s)|Activehost(s)| +---------------------+----------------+ |agent_ilo |cdr-A | <<alltheilonodesgohere |agent_ipmitool |cdr-A,cdr-B | <<ipminodesaresplit50/50 +---------------------+----------------+
  35. 35. What happens when I stop cdr­B? $ironicdriver-list +---------------------+----------------+ |Supporteddriver(s)|Activehost(s)| +---------------------+----------------+ |agent_ilo |cdr-A | <<conductor"A"isnowmanaging |agent_ipmitool |cdr-A | <<100%ofallmynodes +---------------------+----------------+ Actually, I was just migrating to cdr­C $ironicdriver-list +---------------------+----------------+ |Supporteddriver(s)|Activehost(s)| +---------------------+----------------+ |agent_ilo |cdr-A,cdr-C | <<bothilo&ipminodesare |agent_ipmitool |cdr-A,cdr-C | <<sharedevenlyacrossconductors +---------------------+----------------+
  36. 36. Is there a command to view the mapping? Nope. It's dynamically generated as the cluster changes.
  37. 37. Thanks!

×