Building cloud stack at scale

3,128 views

Published on

Pauls presentation at cloudstack collab 2012, Las Vegas

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Building cloud stack at scale

  1. 1. Building CloudStack at Scale Paul Angus Cloud Architect ShapeBlue paul.angus@shapeblue.com Twitter: @ShapeBlue www.shapeblue.com
  2. 2. Building CloudStack at ScaleOverview The design phase Redundancy & resilience Automation Documentation www.shapeblue.com
  3. 3. The Design Phase Tips for the design phasewww.shapeblue.com
  4. 4. The Design PhaseGet everyone in the room www.shapeblue.com
  5. 5. The Design PhaseKnow the fine print www.shapeblue.com
  6. 6. The Design PhaseUnderstand your work loads www.shapeblue.com
  7. 7. Design PhaseUnderstand your bottlenecks www.shapeblue.com
  8. 8. Redundancy & Resilience Designing for Redundancy and Resiliency www.shapeblue.com
  9. 9. Redundancy & ResilienceCloudStack Management ServerMySQL ServersVirtualised Management Farm www.shapeblue.com
  10. 10. Redundancy & ResilienceCloudStack Management Server Very easy to set up additional management servers Load balancing required to give high availabilty www.shapeblue.com
  11. 11. Redundancy & ResilienceMySQL Master / Slave is ‘standard’ Alternatives include MySQL Proxy (Mirroring) Galera Cluster MMM www.shapeblue.com
  12. 12. Redundancy & Resilience Load Load Balancer Balancer MySQL MySQL MySQLCS Man CS Man Galera www.shapeblue.com
  13. 13. Redundancy & ResilienceDC1 DC2 F5 F5 CS Man CS Man CS Man CS Man F5 F5MySQL MySQL MySQL MySQL MySQL MySQL Galera Galera www.shapeblue.com
  14. 14. Redundancy & ResilienceServer ‘pairs’ MySQL masters and slaves CS Man & MySQL master vCenter & MS SQL server Any other redundant servers i.e. DNS www.shapeblue.com
  15. 15. Redundancy & ResilienceLaws of probability…. Same failure probability as raid 0 If management server OR MySQL master fails – downtime for the whole management system. www.shapeblue.com
  16. 16. Automation Automation of the infrastructure buildwww.shapeblue.com
  17. 17. AutomationUses: Why automate Automation tools Examples Configuring management VMs Build a CloudStack 4.0 management server Deployment of hosts Configuration of CloudStack (through API) www.shapeblue.com
  18. 18. AutomationWork smarter, not harder www.shapeblue.com
  19. 19. AutomationAutomation gives you: Speed Efficiency Repeatability Accuracy www.shapeblue.com
  20. 20. AutomationTools: Chef, Puppet KickStart, Python Shell scripting www.shapeblue.com
  21. 21. AutomationExample: Base Build of Management Servers using Shell Scripting www.shapeblue.com
  22. 22. Automation – Management VM Configuration Create Deployment VM Download CSV & Make it executable, CentOS (Minimal) + hostconfig script run it. wget www.shapeblue.com
  23. 23. Automation – Management VM Configuration The CSV: PrimaryNIC PrimaryNIC PrimaryNIC SecondaryNIC SecondaryNIC NTPSERVERIPVMhostname Searchdomain PrimaryNICIP Prefix Gateway Network SecondaryNICIP Prefix Network DNS1 1 CSMan1 cs4.net 10.14.16.67 27 10.14.16.65 Tools 10.144.72.21 21 Mgmt 10.128.9.9 64.238.19.20 CSMan2 cs4.net 10.14.16.68 27 10.14.16.65 Tools 10.144.72.22 21 Mgmt 10.128.9.9 64.238.19.20 cs4.net MySQL1 10.14.1.35 27 10.14.1.33 Database 10.141.163.4 26 Mgmt 10.128.9.9 64.238.19.20 MySQL2 cs4.net 10.14.1.36 27 10.14.1.33 Database 10.141.163.5 26 Mgmt 10.128.9.9 64.238.19.20 www.shapeblue.com
  24. 24. Automation – Management VM Configuration#!/bin/bashecho "Please enter hostname [without domain]: ie CSMan1"read VMHOSTNAMESearchDomain=`grep $VMHOSTNAME hostconfig.csv | awk -F , {print $2}`PrimaryNICIP=`grep $VMHOSTNAME hostconfig.csv | awk -F , {print $3}`PrimaryNICIP=`grep $VMHOSTNAME hostconfig.csv | awk -F , {print $3}`PrimaryNICPrefix=`grep $VMHOSTNAME hostconfig.csv | awk -F , {print $4}`PrimaryNICGateway=`grep $VMHOSTNAME hostconfig.csv | awk -F , {print $5}`PrimaryNICNetwork=`grep $VMHOSTNAME hostconfig.csv | awk -F , {print $6}` www.shapeblue.com
  25. 25. Automation – Management VM Configurationecho "DEVICE="eth0" nameserver $DNS2BOOTPROTO=none " > /etc/resolv.confNM_CONTROLLED="no"ONBOOT=yesTYPE="Ethernet" echo "# ShapeBlue Script created fileIPADDR=$PrimaryNICIP NETWORKING=yesPREFIX=$PrimaryNICPrefix HOSTNAME=$VMHOSTNAMEGATEWAY=$PrimaryNICGateway " > /etc/sysconfig/networkDEFROUTE=yesIPV4_FAILURE_FATAL=yesIPV6INIT=no echo "$NTPSERVERIP1NAME="$PrimaryNICNetwork" $NTPSERVERIP2" > /etc/ntp.conf"> /etc/sysconfig/network-scripts/ifcfg-eth0 Rebootecho "# ShapeBlue Script created filesearch $SearchDomainnameserver $DNS1 www.shapeblue.com
  26. 26. Automation – Management VM Configuration# if vm management use this one NETMASK4=255.255.255.248if [ "$SecondaryNICNetwork" == “Mgmt" ]; then GATEWAY4=10.141.163.1 echo "ADDRESS0=10.128.9.9 ADDRESS5=213.212.69.0 NETMASK0=255.255.255.255 NETMASK5=255.255.255.0 GATEWAY0=10.14.16.1 GATEWAY5=10.141.163.1" > /etc/sysconfig/network- ADDRESS1=10.128.3.13 scripts/route-eth1 NETMASK1=255.255.255.255 fi GATEWAY1=10.14.16.1 ADDRESS2=64.238.199.202 NETMASK2=255.255.255.255 GATEWAY2=10.141.163.1 ADDRESS3=213.212.65.202 NETMASK3=255.255.255.255 GATEWAY3=10.141.163.1 ADDRESS4=213.212.68.168 www.shapeblue.com
  27. 27. AutomationExample: Build a CS4 management server www.shapeblue.com
  28. 28. Automation – CS4 Management ServerCreate Deployment VM Set static IP address Download the script, CentOS (Minimal) + and ensure it has an make it executable, wget FQDN run it. www.shapeblue.com
  29. 29. Automation – CS4 Management Server#!/bin/bash chkconfig ntpd on chkconfig mysqld onsetenforce permissive chkconfig nfs onsed -i "/SELINUX=/ cSELINUX=permissive" /etc/selinux/config chkconfig rpcbind onecho " service ntpd restart[cloudstack] service mysqld restartname=cloudstack service rpcbind startbaseurl=http://cloudstack.apt-get.eu/rhel/4.0/ service nfs startenabled=1gpgcheck=0" > /etc/yum.repos.d/cloudstack.repo /usr/bin/mysqladmin -u root password passwordyum update -y cloud-setup-databases cloud:cloud@localhost --deploy-yum install ntp cloud-client mysql-server -y as=root:passwordsed -i -e /datadir/ ainnodb_rollback_on_timeout=1 -e /datadir/ cloud-setup-managementainnodb_lock_wait_timeout=600 -e /datadir/amax_connections=350 -e /datadir/ alog-bin=mysql-bin -e"/datadir/ abinlog-format = ROW" /etc/my.cnf www.shapeblue.com
  30. 30. AutomationExample: Host deployment server build using shell scipting www.shapeblue.com
  31. 31. Automation – Host DeploymentCreate Deployment VM Download to VM Hypervisor Download build script, installation media (inc. make it executable,CentOS (Minimal) + wget XenServer Updates) run it. www.shapeblue.com
  32. 32. Automation – Host DeploymentDeployment server (VM) The script downloads, builds and configures: DHCP PXE (TFTP) HTTP server Script writes the scripts needed to for PXE boot of XenServer & ESXi hosts Script also writes the answer files and post installation scripts to configure XenServer and ESXi hosts www.shapeblue.com
  33. 33. Automation – Host Deployment# write DHCP config file allow booting;echo "# DHCP for PXE allow bootp; use-host-decl-names true;ddns-update-style interim; option option-128 code 128 = string;authoritative; option option-129 code 129 = text;log-facility local7; option space gpxe;default-lease-time 600; option gpxe-encap-opts code 175 = encapsulate gpxe;max-lease-time 7200; option gpxe.bus-id code 177 = string;option subnet-mask $DHCPNETMASK;option broadcast-address $DHCPBROADCAST; next-server $PXESERVERIP;option routers $DHCPGATEWAY; filename "/gpxelinux.0";option domain-name-servers $DNS1;option domain-name "$DOMAIN"; " > /etc/dhcp/dhcpd.confsubnet $DHCPSUBNET netmask $DHCPMASK {range $DHCPSTART $DHCPFINISH; } www.shapeblue.com
  34. 34. Automation – Host Deployment# create pxelinux.cfg/default fileecho "# script built pxeconfigDEFAULT menu.c32menu title ShapeBlue Host Deploymenttimeout 0LABEL ESXi5.0 KERNEL http://$PXESERVERIP/esxi/mboot.c32 APPEND -c http://$PXESERVERIP/esxipxeboot/boot.cfg pxebooting +++ IPAPPEND 1MENU LABEL ESXi-5.0.0" > /tftpboot/pxelinux.cfg/default www.shapeblue.com
  35. 35. Automation – Host Deployment# Create xenserver answerfileecho "<?xml version="1.0"?><installation srtype="ext"><primary-disk gueststorage="yes">sda</primary-disk><keymap>uk</keymap><root-password>P@ssword</root-password><source type="url">http://$PXESERVERIP/xenserver/</source><script stage="filesystem-populated" type="url">http://$PXESERVERIP/xenpxeboot/postxeninstall.sh</script><admin-interface name="eth0" proto="dhcp" /><timezone>Europe/Dublin</timezone><time-config-method>ntp</time-config-method><ntpservers>$NTPSERVER1</ntpservers><nameserver>$DNS1</nameserver></installation>" > /var/www/html/xenpxeboot/xenanswerfile www.shapeblue.com
  36. 36. Automation – Host Deployment MAC IP HOSTNAME STORAGEIP80:c1:6e:6a:b5:9c 10.14.73.9 XenHost1 10.14.97.980:c1:6e:6a:b4:0c 10.14.73.10 XenHost2 10.14.97.1080:c1:6e:6a:f3:fc 10.14.73.11 XenHost3 10.14.97.1180:c1:6e:6a:f0:ec 10.14.73.12 ESXiHost1 10.14.97.1280:c1:6e:6a:b5:a4 10.14.73.13 ESXiHost2 10.14.97.13 www.shapeblue.com
  37. 37. Automation – Host Deploymentecho “# sample XenServer post install scriptwget http://$PXESERVERIP/hostconfig.csvMACADD=`ifconfig | grep eth0 | awk {print $NF}`“IPADDR=’grep -i $MACADD /tmp/arp_table.csv | awk -F , {print $2}‘’XENHOSTNAME =/`grep -i $MACADD /tmp/arp_table.csv | awk -F , {print $3}‘/`STORAGEIP=/`grep -i $MACADD /tmp/arp_table.csv | awk -F , {print $4}‘/`STORAGENICUUID=/’xe pif-list | awk $0 ~ str{print b}{b=$0} str="eth3" | awk {print $NF}‘/`xe host-list params=uuid --minimal > /tmp/tmphostuuidxe pif-list management=true params=uuid --minimal > /tmp/tmpnicuuidread NICUUID < /tmp/tmpnicuuidread HOSTUUID < /tmp/tmphostuuidxe host-set-hostname-live host-uuid=$HOSTUUID host-name=$XENHOSTNAMExe host-param-set name-label=$XENHOSTNAME uuid=$HOSTUUIDxe pif-reconfigure-ip uuid=$NICUUID mode=static IP=$IPADDR netmask=$HOSTNETMASK gateway=$HOSTGATEWAYDNS=$DNS1,$DNS2xe pif-param-set uuid=$NICUUID other-config:domain=$DOMAIN" > /var/www/html/xenpxeboot/firstfullbootxenscript.sh www.shapeblue.com
  38. 38. Automation – Host Deploymentecho "#!/bin/bash if [ "$PATCHACTION" == "restartXAPI" ]; then# Install XenServer Updates xe-toolstack-restart elsexe host-list params=uuid --minimal > /tmp/tmphostuuid rebootfiles=$(ls /tmp/*.xsupdate 2> /dev/null | wc -l) exitif [ "$files" != "0" ]; then fi read HOSTUUID < /tmp/tmphostuuid done cd /tmp fi for updatefile in /tmp/*.xsupdate; do PATCHUUID=$(xe patch-upload file- # Disable boot script for subsequent rebootsname=$updatefile) rm -f /etc/rc3.d/S99postinstall xe patch-apply host-uuid=$HOSTUUIDuuid=$PATCHUUID # Final Reboot rm -f $updatefile reboot PATCHACTION=$(xe patch-list uuid=$PATCHUUID " > /var/www/html/xenpxeboot/xenupdatescript.shparams=after-apply-guidance --minimal) echo "Patch Action:" $PATCHACTION www.shapeblue.com
  39. 39. Automation Utilising the API when building the infrastructurewww.shapeblue.com
  40. 40. AutomationThe API What it is How can you use it Examples www.shapeblue.com
  41. 41. Automation – The APIWhat is the API The API is the real engine of CloudStack The web GUI is simply making API calls Port 8096 by default www.shapeblue.com
  42. 42. Automation – The APIUsing the API CloudStack GUI Browser Word, Excel Using Firebug/IE Developer Tools with CloudStack www.shapeblue.com
  43. 43. Automation – The APIAPI commands directly through a browserImmediate response www.shapeblue.com
  44. 44. Automation – The APIAPI calls from a Word document or Excel spread sheet www.shapeblue.com
  45. 45. Automation – The APIUsing Firebug / IE Developer Tools IE press F12 or ‘view developer tools’ Firefox install Firebug add-on www.shapeblue.com
  46. 46. www.shapeblue.com
  47. 47. Automation – The APIGlobal Settingshttp://csman:8096/client/api?command=updateConfiguration&name=vmware.management.portgroup&value=svc-consolehttp://csman:8096/client/api?command=updateConfiguration&name=allow.user.create.projects&value=falsehttp://csman:8096/client/api?command=updateConfiguration&name=allow.public.user.templates&value=falsehttp://csman:8096/client/api?command=updateConfiguration&name=apply.allocation.algorithm.to.pods&value=truehttp://csman:8096/client/api?command=updateConfiguration&name=cpu.overprovisioning.factor&value=2http://csman:8096/client/api?command=updateConfiguration&name=vm.allocation.algorithm&value=random www.shapeblue.com
  48. 48. Automation – The APIGetting information http://csman1:8096/client/api?command=listServiceOfferings http://csman1:8096/client/api?command=listTemplates&templatefilter=featur ed www.shapeblue.com
  49. 49. Automation – The APIDeploying an instanceBase command:http://csman1:8096/client/api?command=deployVirtualMachineThe required options:The Service Offering and Template IDs &serviceofferingid=XXX &templateid=XXX &zoneid=XXX &domainid=XXX &account=XXXOptional options: &displayname=xxx www.shapeblue.com
  50. 50. Automation – The APIDeploying an instancehttp://csman1:8096/client/api?command=deployVirtualMachine&serviceofferingid=XXX&templateid=XXXCan be used to create a large number of instances very quickly www.shapeblue.com
  51. 51. Documentation A word on documentationwww.shapeblue.com
  52. 52. DocumentationDull, boring, tedious, slow – Crucial. Write what you’re going to do Follow what you wrote Update it With redundant servers; follow it again With scripts ‘snapshot’ and start again www.shapeblue.com
  53. 53. Any Questions ? Paul Angus paul.angus@shapeblue.com Twitter: @ShapeBlue www.shapeblue.comwww.shapeblue.com
  54. 54. Thank you Paulwww.shapeblue.com

×