This document discusses using Chef with Apache CloudStack and Citrix CloudPlatform for automation and configuration management. It provides an overview of CloudStack and CloudPlatform, and explains two Chef knife plugins - knife-cloudstack and knife-cloudstack-fog. knife-cloudstack-fog provides comprehensive API coverage for provisioning CloudStack servers using Chef. The document also covers options for getting started with CloudStack and discusses future plans, like testing and merging the plugins.

1. Chef & Apache
CloudStack™
Cloud Systems automation and configuration
management using Chef with
Apache CloudStack™ and/or
Citrix CloudPlatform™
Createdby /Jeff Moody @fifthecho

2. #> whoami
Jeff Moody
Cloud Engineer at Datapipe
Managed Service Provider based in Jersey City, NJ
Managed AWS Services (World's largest AWS reseller)
Infrastructure-as-a-Service Platform built using Citrix CloudPlatform
(Stratosphere)
Managed Services on Stratosphere
Primary author/maintainer of knife-cloudstack-fog, kitchen-cloudstack,
CloudStack-PowerShell

3. Disclaimer
My opinions are my own and not necessarily representative of my
employer
YMMV (Your Milage May Vary)
No code is perfect, there may be bugs (please sent pull requests/submit
issues if you find them)

4. What is Apache CloudStack™?
Virtualization Orchestration Platform
Private, Public, VPC Style Clouds
Hypervisor Agnostic
Xen
KVM
VMware ESXi (with VCenter)
LXC
Hyper-V
Oracle VM (OVM)
Bare Metal
Docker support in development (announced last week)

5. What is Apache CloudStack™?
Project founded in 2008
Started by Cloud.com (First OSS release May 2010)
Bought by Citrix (July 2011)
Donated to ASF (April 2012)
Accepted as an ASF Top-Level Project (March 2013)
Written in Java/Tomcat
Robust Native API
Baked-in AWS EC2 Compatibility with SOAP and REST API

6. Then what is Citrix CloudPlatform™?
Commercial release of CloudStack
Deeper 3rd-Party Vendor integrations
Non-Apache Licence Compatible Libraries/Tools
Includes licensing for Citrix XenServer

7. CloudStack and Chef
Why are there two knife plugins for CloudStack?
What makes the two plugins different?
What benefits does a knife plugin have over native API + knife bootstrap?

8. What is knife-cloudstack?
Developed initially by Edmunds.
Less API coverage than knife-cloudstack-fog
Provides "stacks" for building multiple servers at once.
David Nalley of Apache/Citrix looking at making this part of knife, not just
knife-cloudstack
Uses its own CloudStack API library.

9. What, then, is knife-cloudstack-fog?
Forked from an earlier, pre-Edmunds knife-cloudstack which worked
sometimes.
Developed initially by me. Several contributors, but still mostly my project.
Provides coverage for almost every CloudStack API call needed to manage
infrastructure.
Uses FOG for its CloudStack API calls.

10. What is FOG?
Cloud-independent/agnostic toolkit for Ruby
Initial CloudStack support added by Brian Dorry (also of Datapipe)
Uniform-ish access to all Cloud resources (independent of provider) using
common-ish language
Used by Chef first-party knife plugins

11. What is knife?
(Hopefully you know this already)
CLI interface for Chef
Built to be extensible and support plug-ins
Many of these plug-ins are for VM provisioning
Plugins include support for:
AWS, CloudStack, OpenStack, Google Compute Engine, VMware,
XenServer, Docker...
Used to interact with Chef server and nodes

12. What does knife-cloudstack-fog
provide?
One-stop-shop for all information to manage CloudStack instances
**CLOUDSTACKCOMMANDS**
knifecloudstackdiskofferinglist
knifecloudstackkeypaircreate-kNAME(options)
knifecloudstackkeypairdeleteNAME
knifecloudstackkeypairlist
knifecloudstacknetworkcreate-nNAME-oNETWORKOFFERINGID-zZONE(options)
knifecloudstacknetworkdeleteID
knifecloudstacknetworklist
knifecloudstacknetworkofferinglist
knifecloudstackportforwardingrulelist
knifecloudstackpublicipcreate(options)
knifecloudstackpubliciplist
knifecloudstacksecuritygrouplist(options)
knifecloudstackservercreate-sSERVICEID-tTEMPLATEID-zZONEID(options)
knifecloudstackserverdeleteINSTANCE_ID
knifecloudstackserverdestroyINSTANCE_ID
knifecloudstackserverlist(options)
knifecloudstackserverstartINSTANCE_ID
knifecloudstackserverstopINSTANCE_ID(options)
knifecloudstackserviceofferinglist
knifecloudstacktemplatelist(options)
knifecloudstackvolumelist
knifecloudstackzonelist

13. With all these options, what is the
main thing we use this plugin for?
Provisioning servers!

14. knife cloudstack server create --help
knifecloudstackservercreate-sSERVICEID-tTEMPLATEID-zZONEID(options)
--server-urlURL ChefServerURL
--chef-zero-portPORT Porttostartchef-zeroon
--keyKEY APIClientKey
-AKEY, YourCloudstackAccessKeyID
--cloudstack-access-key-id
--cloudstack-api-endpointENDPOINT
YourCloudstackAPIendpoint
-g,--groupidsSECURITYGROUPIDS CommaseparatedlistofCloudStackSecurityGroupIDs.
-GSECURITYGROUPNAMES, CommaseparatedlistofCloudStackSecurityGroupnames.
Eachgroupnamemustbeencapuslatedinquotesifit
containswhitespace.
--groupnames
-w,--networkidsNETWORKIDS CommaseparatedlistofCloudStacknetworkIDs.
-KSECRET, YourCloudstackAPISecretAccessKey
--cloudstack-secret-access-key
-s,--serviceidSERVICEID TheCloudStackserviceofferingID.
-t,--templateidTEMPLATEID TheCloudStacktemplateIDfortheserver.
-Z,--zoneidZONE TheCloudStackzoneIDfortheserver.
--[no-]color Usecoloredoutput,defaultstofalseonWindows,true
otherwise
-c,--configCONFIG Theconfigurationfiletouse
--defaults Acceptdefaultvaluesforallquestions
--disable-editing DonotopenEDITOR,justacceptthedataasis
-DDISKOFFERINGID, SpecifieseithertheDiskOfferingIDforthe
ROOTdiskforanISOtemplate,oraDATAdisk.
--diskoffering
-d,--distroDISTRO Bootstrapadistrousingatemplate;defaultis'chef-full'
-e,--editorEDITOR Settheeditortouseforinteractivecommands
-E,--environmentENVIRONMENT SettheChefenvironment
-F,--formatFORMAT Whichformattouseforoutput
-H,--hostnameNAME Theinstancehostname
-iPRIVATE_KEY_FILE, ThePrivatekeyfileforauthenticatingSSHsession.
--keypairoptionisalsoneeded.
--identity-file
-k,--keypairKEYPAIR TheCloudStackKeyPairtouseforSSHkeyauthentication.
-z,--local-mode Pointknifecommandsatlocalrepositoryinsteadofserver

15. With all those options, how do we
start?
1. Pick a Zone
2. Pick a Template
3. Pick a Service Offering
4. Pick a Network/Security Group
5. Add any additional options
Then, launch the VM!

16. knife cloudstack zone list
No configurable options
knifecloudstackzonelist
ID Name NetworkType
3 NewYorkMetroEPN Advanced
5 NewYorkMetro Advanced
6 HongKong Advanced
7 SiliconValley Advanced
8 HongKongEPN Advanced
9 London Advanced
11 Shanghai Advanced
12 LondonEPN Advanced
13 SiliconValleyEPN Advanced
14 ShanghaiEPN Advanced
08e8fdf1-f64a-4ab6-bd21-7bffd578be9e IcelandEPN Advanced
e20be308-482b-4d10-8885-cea306e87de9 SiliconValley2 Advanced
cb765f1b-2638-465e-9c3d-9013e04116da SiliconValley2EPN Advanced
51345d53-bf3b-4280-b5dd-8541703eff9a SingaporeEPN Advanced
4baf6857-c7db-437a-96e6-f420761e128c Singapore Advanced

17. knife cloudstack template list --help
knifecloudstacktemplatelist(options)
-s,--server-urlURL ChefServerURL
--chef-zero-portPORT Porttostartchef-zeroon
-k,--keyKEY APIClientKey
-AKEY, YourCloudstackAccessKeyID
--cloudstack-access-key-id
--cloudstack-api-endpointENDPOINT
YourCloudstackAPIendpoint
-KSECRET, YourCloudstackAPISecretAccessKey
--cloudstack-secret-access-key
--[no-]color Usecoloredoutput,defaultstofalseon
Windows,trueotherwise
-c,--configCONFIG Theconfigurationfiletouse
--defaults Acceptdefaultvaluesforallquestions
-d,--disable-editing DonotopenEDITOR,justacceptthedataasis
-e,--editorEDITOR Settheeditortouseforinteractivecommands
-E,--environmentENVIRONMENT SettheChefenvironment
-L,--filterFILTER Thetemplatesearchfilter.Defaultis'featured.'
Otheroptionsare'self,''self-executable,'
'executable,'and'community.'
-F,--formatFORMAT Whichformattouseforoutput
-H,--hypervisorHYPERVISOR Limitresponsestotemplatesonlyrunningona
specifichypervisor.Defaultprovidestemplates
fromallhypervisors.
--local-mode Pointknifecommandsatlocalrepositoryinstead
ofserver
-u,--userUSER APIClientUsername
--print-after Showthedataafteradestructiveoperation
-T,--templateidTEMPLATEID LimitresponsestoasingletemplateID.
Defaultprovidesalltemplates.
-V,--verbose Moreverboseoutput.Usetwiceformaxverbosity
-v,--version Showchefversion
-y,--yes Sayyestoallpromptsforconfirmation
-Z,--zoneZONE Limitresponsestotemplatesonlylocatedina
specificzone.Defaultprovidestemplatesfromallzones.
-z,--zoneidZONEID Limitresponsestotemplatesonlyrunningina
specificzone(specifiedbyID#).Defaultprovides

18. knife cloudstack template list
knifecloudstacktemplatelist
ID Hypervisor Size(inGB) ZoneName ZoneID
086ce1e1-3e92-4a9e-b416-2dd5445c1e7f VMware 20 IcelandEPN 08e8fdf1-f64a-4ab6-b
1573395c-7489-4a93-9ca3-11a51a285457 VMware 20 IcelandEPN 08e8fdf1-f64a-4ab6-b
d16563da-64c6-4358-b6e6-0c313c43223c VMware 20 IcelandEPN 08e8fdf1-f64a-4ab6-b
187dc53e-8f4b-405b-9067-7ee056c75fcc VMware 20 IcelandEPN 08e8fdf1-f64a-4ab6-b
20e58c82-2b7b-4552-adb2-59d4f28021cd VMware 40 IcelandEPN 08e8fdf1-f64a-4ab6-b
27b96e7e-7315-48f2-8ac0-4a4aa7cc7e14 VMware 20 IcelandEPN 08e8fdf1-f64a-4ab6-b
27fa2e69-ad13-43c2-b000-56cce241f769 VMware 20 IcelandEPN 08e8fdf1-f64a-4ab6-b
88c62685-b6f5-4acc-9ca4-9b937e5112b5 XenServer 20 IcelandEPN 08e8fdf1-f64a-4ab6-b
0f755a47-92bf-4bbf-beb0-bb3fc2ae6c91 VMware 20 IcelandEPN 08e8fdf1-f64a-4ab6-b
a494ea35-e64f-40d4-a30c-b5c9dc1971bd VMware 20 IcelandEPN 08e8fdf1-f64a-4ab6-b
...

19. knife cloudstack serviceoffering list
No configurable options
knifecloudstackserviceofferinglist
ID Name Description
f26eb96f-98d5-4f34-81b0-d817c81227a2 kilo-1-40-hpc 1CPUcore,4Gmemory,highperformancerootvolume
2b9a7b24-c977-467e-b0c8-eb0c97512f22 nano-h-5-hpc 0.5CPUcore,512MBmemory,highperformancerootvo
9d865b43-296e-4ea6-83a2-122bc9baf6ac nano-h-10-hpc 0.5CPUcore,1Gmemory,highperformancerootvolum
39 mega-2-160 2CPUcore,16Gmemory,standardrootvolume
29ef5f70-8346-472a-9f84-f00a1fd727a7 kilo-4-80-hpc 4CPUcore,8Gmemory,highperformancerootvolume
38 mega-4-160 4CPUcore,16Gmemory,standardrootvolume
4414bc19-7ee8-443f-80cc-0b19ff274ad9 nano-1-10-hpc 1CPUcore,1Gmemory,highperformancerootvolume
37 mega-8-160 8CPUcore,16Gmemory,standardrootvolume
10f888d6-f6aa-4ebb-8f40-9acc5c4f0fa4 kilo-1-20-hpc 1CPUcore,2Gmemory,highperformancerootvolume
...

20. knife cloudstack network list
Used for Advanced Isolated or Advanced VPC Zones
knifecloudstacknetworklist
ID Name ZoneID
1a036de9-dd01-4f9a-af80-f5deac2d6131 Demo1 3
4cbc9ef7-d276-4858-beb0-ce6643a247b8 TestingNetwork 3
91b06fd1-bf74-4d02-a28a-0d9aca3d5fd5 SiliconValleyEPNTest 13
623 DefaultNetwork 12
535 DefaultNetwork 3
3d4af98c-7787-417a-8e26-8f265129e7c3 defaultGuestNetwork 4baf6857-c7db-437a-96e6-f42076
1124 SharedSGenablednetwork 5
...

21. knife cloudstack securitygroup list
Used for Basic or Advanced with Security Groups Zones
knifecloudstacksecuritygrouplist
ID Name Description
129 default DefaultSecurityGroup
295 NetworkTesting
320 ChefServer
324 TestSG
432 PuppetServer
f2bf049f-8c6b-4b8f-87d1-4846da688dbf AppTesting
...

22. knife cloudstack server create
knifecloudstackservercreate-Z4baf6857-c7db-437a-96e6-f420761e128c-tb5870d22-e586-4da9-add5-100dc1c1
Waitingforserver....
Name:0bdb2cd0-d991-4d3c-8c4e-809597dfaf05
PrimaryIP:64.106.186.210
Username:root
Password:jA2rszsie
WaitingforSSH.

23. Other information is available from
knife-cloudstack-fog for:
disk offerings
Creating data-disks with instances
Creating VMs from ISO templates for the ROOT disk
keypairs
Assumes that the cloud-set-guest-sshkey is in place.
Supports creation of and deletion of keypairs (end user feature request)
networks
Supports creation and deletion of networks (for Advanced Zones)
portforwardingrules
For Advanced Zone networks, NAT rules can be created with knife
Server stop/start
SSH Gateway
Random port forwarding (for Advanced Zones)

24. Future
Move code to knife-cloud?
One plugin for all Clouds
Common semantics
LOTS of work to do
Merge with knife-cloudstack?
Unlikely as some core knife-cloudstack devs are anti-FOG
knife-cloudstack-fog more feature rich, minus stacks and system VMs
Tests
There currently are no tests.
I don't know how to write tests (as of March 2014)
If someone wants to start writing tests, please do!

25. Bonus content!
There is a Test Kitchen driver for CloudStack/CloudPlatform! (...and I wrote it!)
The setup for the driver is on the GitHub page
(https://github.com/test-kitchen/kitchen-cloudstack)
There's also a Vagrant driver for CloudStack/CloudPlatform (...and I didn't
write it!)

26. So, how do I get in on this?
Datapipe Stratosphere
Global IaaS Platform running on Citrix CloudPlatform
Install Apache CloudStack™
Available from YUM and apt repositories for EL-6 and Ubuntu 12.04
I've built a Docker build script for CloudStack!
Still under heavy work.
Definitely not recommended for production.
(Currently coordinating with CloudStack peoples on how we want this in
the Docker Index)
http://www.datapipe.com/cloud/stratosphere/
http://cloudstack.apache.org/
https://github.com/fifthecho/docker-cloudstack-management

27. Questions?
Jeff Moody
Datapipe (http://www.datapipe.com)
E-Mail: jmoody@datapipe.com / fifthecho@gmail.com
GitHub: https://github.com/fifthecho
Twitter: @fifthecho
YouTube: https://www.youtube.com/user/fifthecho

28. Thank you!
Also many thanks to the Chef
community, FOG developers, my
contributors, and people who submit
issues.
Speaker Office Hours: Today 3:15-
3:55 in Marina