Building cloud stack at scale


Published on

Pauls presentation at cloudstack collab 2012, Las Vegas

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Building cloud stack at scale

  1. 1. Building CloudStack at Scale Paul Angus Cloud Architect ShapeBlue Twitter: @ShapeBlue
  2. 2. Building CloudStack at ScaleOverview The design phase Redundancy & resilience Automation Documentation
  3. 3. The Design Phase Tips for the design
  4. 4. The Design PhaseGet everyone in the room
  5. 5. The Design PhaseKnow the fine print
  6. 6. The Design PhaseUnderstand your work loads
  7. 7. Design PhaseUnderstand your bottlenecks
  8. 8. Redundancy & Resilience Designing for Redundancy and Resiliency
  9. 9. Redundancy & ResilienceCloudStack Management ServerMySQL ServersVirtualised Management Farm
  10. 10. Redundancy & ResilienceCloudStack Management Server Very easy to set up additional management servers Load balancing required to give high availabilty
  11. 11. Redundancy & ResilienceMySQL Master / Slave is ‘standard’ Alternatives include MySQL Proxy (Mirroring) Galera Cluster MMM
  12. 12. Redundancy & Resilience Load Load Balancer Balancer MySQL MySQL MySQLCS Man CS Man Galera
  13. 13. Redundancy & ResilienceDC1 DC2 F5 F5 CS Man CS Man CS Man CS Man F5 F5MySQL MySQL MySQL MySQL MySQL MySQL Galera Galera
  14. 14. Redundancy & ResilienceServer ‘pairs’ MySQL masters and slaves CS Man & MySQL master vCenter & MS SQL server Any other redundant servers i.e. DNS
  15. 15. Redundancy & ResilienceLaws of probability…. Same failure probability as raid 0 If management server OR MySQL master fails – downtime for the whole management system.
  16. 16. Automation Automation of the infrastructure
  17. 17. AutomationUses: Why automate Automation tools Examples Configuring management VMs Build a CloudStack 4.0 management server Deployment of hosts Configuration of CloudStack (through API)
  18. 18. AutomationWork smarter, not harder
  19. 19. AutomationAutomation gives you: Speed Efficiency Repeatability Accuracy
  20. 20. AutomationTools: Chef, Puppet KickStart, Python Shell scripting
  21. 21. AutomationExample: Base Build of Management Servers using Shell Scripting
  22. 22. Automation – Management VM Configuration Create Deployment VM Download CSV & Make it executable, CentOS (Minimal) + hostconfig script run it. wget
  23. 23. Automation – Management VM Configuration The CSV: PrimaryNIC PrimaryNIC PrimaryNIC SecondaryNIC SecondaryNIC NTPSERVERIPVMhostname Searchdomain PrimaryNICIP Prefix Gateway Network SecondaryNICIP Prefix Network DNS1 1 CSMan1 27 Tools 21 Mgmt CSMan2 27 Tools 21 Mgmt MySQL1 27 Database 26 Mgmt MySQL2 27 Database 26 Mgmt
  24. 24. Automation – Management VM Configuration#!/bin/bashecho "Please enter hostname [without domain]: ie CSMan1"read VMHOSTNAMESearchDomain=`grep $VMHOSTNAME hostconfig.csv | awk -F , {print $2}`PrimaryNICIP=`grep $VMHOSTNAME hostconfig.csv | awk -F , {print $3}`PrimaryNICIP=`grep $VMHOSTNAME hostconfig.csv | awk -F , {print $3}`PrimaryNICPrefix=`grep $VMHOSTNAME hostconfig.csv | awk -F , {print $4}`PrimaryNICGateway=`grep $VMHOSTNAME hostconfig.csv | awk -F , {print $5}`PrimaryNICNetwork=`grep $VMHOSTNAME hostconfig.csv | awk -F , {print $6}`
  25. 25. Automation – Management VM Configurationecho "DEVICE="eth0" nameserver $DNS2BOOTPROTO=none " > /etc/resolv.confNM_CONTROLLED="no"ONBOOT=yesTYPE="Ethernet" echo "# ShapeBlue Script created fileIPADDR=$PrimaryNICIP NETWORKING=yesPREFIX=$PrimaryNICPrefix HOSTNAME=$VMHOSTNAMEGATEWAY=$PrimaryNICGateway " > /etc/sysconfig/networkDEFROUTE=yesIPV4_FAILURE_FATAL=yesIPV6INIT=no echo "$NTPSERVERIP1NAME="$PrimaryNICNetwork" $NTPSERVERIP2" > /etc/ntp.conf"> /etc/sysconfig/network-scripts/ifcfg-eth0 Rebootecho "# ShapeBlue Script created filesearch $SearchDomainnameserver $DNS1
  26. 26. Automation – Management VM Configuration# if vm management use this one NETMASK4= [ "$SecondaryNICNetwork" == “Mgmt" ]; then GATEWAY4= echo "ADDRESS0= ADDRESS5= NETMASK0= NETMASK5= GATEWAY0= GATEWAY5=" > /etc/sysconfig/network- ADDRESS1= scripts/route-eth1 NETMASK1= fi GATEWAY1= ADDRESS2= NETMASK2= GATEWAY2= ADDRESS3= NETMASK3= GATEWAY3= ADDRESS4=
  27. 27. AutomationExample: Build a CS4 management server
  28. 28. Automation – CS4 Management ServerCreate Deployment VM Set static IP address Download the script, CentOS (Minimal) + and ensure it has an make it executable, wget FQDN run it.
  29. 29. Automation – CS4 Management Server#!/bin/bash chkconfig ntpd on chkconfig mysqld onsetenforce permissive chkconfig nfs onsed -i "/SELINUX=/ cSELINUX=permissive" /etc/selinux/config chkconfig rpcbind onecho " service ntpd restart[cloudstack] service mysqld restartname=cloudstack service rpcbind startbaseurl= service nfs startenabled=1gpgcheck=0" > /etc/yum.repos.d/cloudstack.repo /usr/bin/mysqladmin -u root password passwordyum update -y cloud-setup-databases cloud:cloud@localhost --deploy-yum install ntp cloud-client mysql-server -y as=root:passwordsed -i -e /datadir/ ainnodb_rollback_on_timeout=1 -e /datadir/ cloud-setup-managementainnodb_lock_wait_timeout=600 -e /datadir/amax_connections=350 -e /datadir/ alog-bin=mysql-bin -e"/datadir/ abinlog-format = ROW" /etc/my.cnf
  30. 30. AutomationExample: Host deployment server build using shell scipting
  31. 31. Automation – Host DeploymentCreate Deployment VM Download to VM Hypervisor Download build script, installation media (inc. make it executable,CentOS (Minimal) + wget XenServer Updates) run it.
  32. 32. Automation – Host DeploymentDeployment server (VM) The script downloads, builds and configures: DHCP PXE (TFTP) HTTP server Script writes the scripts needed to for PXE boot of XenServer & ESXi hosts Script also writes the answer files and post installation scripts to configure XenServer and ESXi hosts
  33. 33. Automation – Host Deployment# write DHCP config file allow booting;echo "# DHCP for PXE allow bootp; use-host-decl-names true;ddns-update-style interim; option option-128 code 128 = string;authoritative; option option-129 code 129 = text;log-facility local7; option space gpxe;default-lease-time 600; option gpxe-encap-opts code 175 = encapsulate gpxe;max-lease-time 7200; option gpxe.bus-id code 177 = string;option subnet-mask $DHCPNETMASK;option broadcast-address $DHCPBROADCAST; next-server $PXESERVERIP;option routers $DHCPGATEWAY; filename "/gpxelinux.0";option domain-name-servers $DNS1;option domain-name "$DOMAIN"; " > /etc/dhcp/dhcpd.confsubnet $DHCPSUBNET netmask $DHCPMASK {range $DHCPSTART $DHCPFINISH; }
  34. 34. Automation – Host Deployment# create pxelinux.cfg/default fileecho "# script built pxeconfigDEFAULT menu.c32menu title ShapeBlue Host Deploymenttimeout 0LABEL ESXi5.0 KERNEL http://$PXESERVERIP/esxi/mboot.c32 APPEND -c http://$PXESERVERIP/esxipxeboot/boot.cfg pxebooting +++ IPAPPEND 1MENU LABEL ESXi-5.0.0" > /tftpboot/pxelinux.cfg/default
  35. 35. Automation – Host Deployment# Create xenserver answerfileecho "<?xml version="1.0"?><installation srtype="ext"><primary-disk gueststorage="yes">sda</primary-disk><keymap>uk</keymap><root-password>P@ssword</root-password><source type="url">http://$PXESERVERIP/xenserver/</source><script stage="filesystem-populated" type="url">http://$PXESERVERIP/xenpxeboot/</script><admin-interface name="eth0" proto="dhcp" /><timezone>Europe/Dublin</timezone><time-config-method>ntp</time-config-method><ntpservers>$NTPSERVER1</ntpservers><nameserver>$DNS1</nameserver></installation>" > /var/www/html/xenpxeboot/xenanswerfile
  36. 36. Automation – Host Deployment MAC IP HOSTNAME STORAGEIP80:c1:6e:6a:b5:9c XenHost1 XenHost2 XenHost3 ESXiHost1 ESXiHost2
  37. 37. Automation – Host Deploymentecho “# sample XenServer post install scriptwget http://$PXESERVERIP/hostconfig.csvMACADD=`ifconfig | grep eth0 | awk {print $NF}`“IPADDR=’grep -i $MACADD /tmp/arp_table.csv | awk -F , {print $2}‘’XENHOSTNAME =/`grep -i $MACADD /tmp/arp_table.csv | awk -F , {print $3}‘/`STORAGEIP=/`grep -i $MACADD /tmp/arp_table.csv | awk -F , {print $4}‘/`STORAGENICUUID=/’xe pif-list | awk $0 ~ str{print b}{b=$0} str="eth3" | awk {print $NF}‘/`xe host-list params=uuid --minimal > /tmp/tmphostuuidxe pif-list management=true params=uuid --minimal > /tmp/tmpnicuuidread NICUUID < /tmp/tmpnicuuidread HOSTUUID < /tmp/tmphostuuidxe host-set-hostname-live host-uuid=$HOSTUUID host-name=$XENHOSTNAMExe host-param-set name-label=$XENHOSTNAME uuid=$HOSTUUIDxe pif-reconfigure-ip uuid=$NICUUID mode=static IP=$IPADDR netmask=$HOSTNETMASK gateway=$HOSTGATEWAYDNS=$DNS1,$DNS2xe pif-param-set uuid=$NICUUID other-config:domain=$DOMAIN" > /var/www/html/xenpxeboot/
  38. 38. Automation – Host Deploymentecho "#!/bin/bash if [ "$PATCHACTION" == "restartXAPI" ]; then# Install XenServer Updates xe-toolstack-restart elsexe host-list params=uuid --minimal > /tmp/tmphostuuid rebootfiles=$(ls /tmp/*.xsupdate 2> /dev/null | wc -l) exitif [ "$files" != "0" ]; then fi read HOSTUUID < /tmp/tmphostuuid done cd /tmp fi for updatefile in /tmp/*.xsupdate; do PATCHUUID=$(xe patch-upload file- # Disable boot script for subsequent rebootsname=$updatefile) rm -f /etc/rc3.d/S99postinstall xe patch-apply host-uuid=$HOSTUUIDuuid=$PATCHUUID # Final Reboot rm -f $updatefile reboot PATCHACTION=$(xe patch-list uuid=$PATCHUUID " > /var/www/html/xenpxeboot/xenupdatescript.shparams=after-apply-guidance --minimal) echo "Patch Action:" $PATCHACTION
  39. 39. Automation Utilising the API when building the
  40. 40. AutomationThe API What it is How can you use it Examples
  41. 41. Automation – The APIWhat is the API The API is the real engine of CloudStack The web GUI is simply making API calls Port 8096 by default
  42. 42. Automation – The APIUsing the API CloudStack GUI Browser Word, Excel Using Firebug/IE Developer Tools with CloudStack
  43. 43. Automation – The APIAPI commands directly through a browserImmediate response
  44. 44. Automation – The APIAPI calls from a Word document or Excel spread sheet
  45. 45. Automation – The APIUsing Firebug / IE Developer Tools IE press F12 or ‘view developer tools’ Firefox install Firebug add-on
  46. 46.
  47. 47. Automation – The APIGlobal Settingshttp://csman:8096/client/api?command=updateConfiguration&
  48. 48. Automation – The APIGetting information http://csman1:8096/client/api?command=listServiceOfferings http://csman1:8096/client/api?command=listTemplates&templatefilter=featur ed
  49. 49. Automation – The APIDeploying an instanceBase command:http://csman1:8096/client/api?command=deployVirtualMachineThe required options:The Service Offering and Template IDs &serviceofferingid=XXX &templateid=XXX &zoneid=XXX &domainid=XXX &account=XXXOptional options: &displayname=xxx
  50. 50. Automation – The APIDeploying an instancehttp://csman1:8096/client/api?command=deployVirtualMachine&serviceofferingid=XXX&templateid=XXXCan be used to create a large number of instances very quickly
  51. 51. Documentation A word on
  52. 52. DocumentationDull, boring, tedious, slow – Crucial. Write what you’re going to do Follow what you wrote Update it With redundant servers; follow it again With scripts ‘snapshot’ and start again
  53. 53. Any Questions ? Paul Angus Twitter: @ShapeBlue
  54. 54. Thank you