- Extend CloudStack with plugins and include a ServerResource for device access - Lessons learned include using HTTPRequest to escape Java, serializing JSON objects instead of ported Java classes, writing code against tests, automating with CloudMonkey, adapting existing plugins, using QuickCloud instead of system VMs, and planning to avoid proprietary tools and libraries.

1. How to add a new hypervisor to CloudStack:
Lessons learned from Hyper-V effort
Donal Lafferty
Friday, 15 November 2013

2. Summary
• Extend CloudStack with Plug-ins
• Include ServerResource for device access
• Lessons:
• HTTPRequest lets you escape Java
• Serialise JSON objects, not ported Java classes
• TDD: write you code against tests
• Automate with CloudMonkey
• Adapt existing plug-ins
• QuickCloud instead of System VMs
• Plan to avoid proprietary tools & libs

3. Background: Extend CloudStack with Plug-ins
• Java centric
• Plug-ins are distributed
• .jar
• Modules are loaded
• Spring config + class loader
• Extensions implement
• .class file implementing
interface
(e.g. Discoverer)
(e.g. Compute)
(e.g. cloud-plugin-hypervisor-hyperv-4.3.0.jar)

4. • ServerResource
• Steps around Javas limits
• Two Agent types
• implemented directly by
the ServerResource
• E.g. XAPI calls
• running remotely,
connected to mgmt server
• E.g. KVM Agent
Discoverer Module
Background: ServerResource for Device Access
Remote CloudStack Agent
CloudStack Message Bus
Connected Agent Direct Connect Agent
Extension: Discoverer
- Brings resource under CloudStack control
ServerResource
- Provides a communication layer in the form of an Agent
Device-specific
Connection
Extensible API

5. Problem: Create Plug-in for Hyper-V Support
• VM lifecycle
• Avoid intermediaries
• CIFS for primary & secondary storage
• Analogous to NFS
• Hyper-V is SMB centric
• Advanced networking (ideally)
• Esp. VLANs for tenant isolation
• Console access (ideally)

6. Solution: Remote agent
AgentShell
(Windows Service)
Message Bus Agent
(Java - NIO)
Hyper-V API
(Python)
Server Resource
(Java)
Mgmt Server
(AgentManager)
Phase 1 – Connected Agent
Custom
TCP/JSON

7. Lesson: HTTPRequest lets you escape Java
AgentShell
(C# - Windows Service)
Web Server
(C# - not IIS)
Hyper-V API
(WMI)
Server Resource
(C# - ASP.NET MVC4)
Mgmt Server
(Direct Connect Agent)
Phase 2 – Direct Connect Agent
JSON over
HTTP

8. import com.cloud.agent.api.Answer;
import com.cloud.agent.api.AttachIsoCommand;
import com.cloud.agent.api.AttachVolumeAnswer;
import com.cloud.agent.api.AttachVolumeCommand;
import com.cloud.agent.api.BackupSnapshotAnswer;
import com.cloud.agent.api.BackupSnapshotCommand;
import com.cloud.agent.api.BumpUpPriorityCommand;
import com.cloud.agent.api.CheckHealthAnswer;
import com.cloud.agent.api.CheckHealthCommand;
import com.cloud.agent.api.CheckNetworkAnswer;
import com.cloud.agent.api.CheckNetworkCommand;
import com.cloud.agent.api.CheckOnHostAnswer;
import com.cloud.agent.api.CheckOnHostCommand;
import com.cloud.agent.api.CheckRouterAnswer;
import com.cloud.agent.api.CheckRouterCommand;
import com.cloud.agent.api.CheckS2SVpnConnectionsAnswer;
import com.cloud.agent.api.CheckS2SVpnConnectionsCommand;
import com.cloud.agent.api.CheckVirtualMachineAnswer;
import com.cloud.agent.api.CheckVirtualMachineCommand;
import com.cloud.agent.api.CleanupNetworkRulesCmd;
import com.cloud.agent.api.ClusterSyncAnswer;
import com.cloud.agent.api.ClusterSyncCommand;
import com.cloud.agent.api.Command;
import com.cloud.agent.api.CreatePrivateTemplateFromSnapshotCommand;
import com.cloud.agent.api.CreatePrivateTemplateFromVolumeCommand;
import com.cloud.agent.api.CreateStoragePoolCommand;
import com.cloud.agent.api.CreateVMSnapshotAnswer;
import com.cloud.agent.api.CreateVMSnapshotCommand;
import com.cloud.agent.api.CreateVolumeFromSnapshotAnswer;
import com.cloud.agent.api.CreateVolumeFromSnapshotCommand;
import com.cloud.agent.api.DeleteStoragePoolCommand;
import com.cloud.agent.api.DeleteVMSnapshotAnswer;
import com.cloud.agent.api.DeleteVMSnapshotCommand;
import com.cloud.agent.api.GetDomRVersionAnswer;
import com.cloud.agent.api.GetDomRVersionCmd;
import com.cloud.agent.api.GetHostStatsAnswer;
import com.cloud.agent.api.GetHostStatsCommand;
import com.cloud.agent.api.GetStorageStatsAnswer;
import com.cloud.agent.api.GetStorageStatsCommand;
import com.cloud.agent.api.GetVmStatsAnswer;
import com.cloud.agent.api.GetVmStatsCommand;
import com.cloud.agent.api.GetVncPortAnswer;
import com.cloud.agent.api.GetVncPortCommand;
import com.cloud.agent.api.HostStatsEntry;
import com.cloud.agent.api.MaintainAnswer;
import com.cloud.agent.api.MaintainCommand;
import com.cloud.agent.api.ManageSnapshotAnswer;
import com.cloud.agent.api.ManageSnapshotCommand;
import com.cloud.agent.api.MigrateAnswer;
import com.cloud.agent.api.MigrateCommand;
import com.cloud.agent.api.ModifySshKeysCommand;
import com.cloud.agent.api.ModifyStoragePoolAnswer;
import com.cloud.agent.api.ModifyStoragePoolCommand;
import com.cloud.agent.api.NetworkRulesSystemVmCommand;
import com.cloud.agent.api.NetworkRulesVmSecondaryIpCommand;
import com.cloud.agent.api.PingCommand;
import com.cloud.agent.api.PingRoutingCommand;
import com.cloud.agent.api.PingRoutingWithNwGroupsCommand;
import com.cloud.agent.api.PingRoutingWithOvsCommand;
import com.cloud.agent.api.PingTestCommand;
import com.cloud.agent.api.PlugNicAnswer;
import com.cloud.agent.api.PlugNicCommand;
import com.cloud.agent.api.PoolEjectCommand;
import com.cloud.agent.api.PrepareForMigrationAnswer;
import com.cloud.agent.api.PrepareForMigrationCommand;
import com.cloud.agent.api.PvlanSetupCommand;
import com.cloud.agent.api.ReadyAnswer;
import com.cloud.agent.api.ReadyCommand;
import com.cloud.agent.api.RebootAnswer;
import com.cloud.agent.api.RebootCommand;
import com.cloud.agent.api.RebootRouterCommand;
import com.cloud.agent.api.RevertToVMSnapshotAnswer;
import com.cloud.agent.api.RevertToVMSnapshotCommand;
import com.cloud.agent.api.ScaleVmAnswer;
import com.cloud.agent.api.ScaleVmCommand;
import com.cloud.agent.api.SecurityGroupRuleAnswer;
import com.cloud.agent.api.SecurityGroupRulesCmd;
import com.cloud.agent.api.SetupAnswer;
import com.cloud.agent.api.SetupCommand;
import com.cloud.agent.api.SetupGuestNetworkAnswer;
import com.cloud.agent.api.SetupGuestNetworkCommand;
import com.cloud.agent.api.StartAnswer;
import com.cloud.agent.api.StartCommand;
import com.cloud.agent.api.StartupCommand;
import com.cloud.agent.api.StartupRoutingCommand;
import com.cloud.agent.api.StartupStorageCommand;
import com.cloud.agent.api.StopAnswer;
[HttpPost]
[ActionName(CloudStackTypes.StopCommand)]
public JContainer StopCommand([FromBody]dynamic cmd) {
string details = null; bool result = false;
try {
wmiCallsV2.DestroyVm(cmd.vmName);
result = true;
}
catch (Exception wmiEx) {
details = wmiEx.Message;
}
object ansContent = new {
result = result,
details = details,
vm = cmd.vm
};
return ReturnCloudStackTypedJArray(ansContent,
"com.cloud.agent.api.StopAnswer");
}
}
Lesson: Serialise JSON objects, not ported Java classes

9. public void TestDestroyCommand() {
// Arrange
String sampleVolume = getSampleVolumeObjectTO();
String destoryCmd = //"{"volume":" + getSampleVolumeObjectTO() + "}";
"{"volume":{"name":"" + testSampleVolumeTempUUIDNoExt
+ "","storagePoolType":"Filesystem",“
+ ""mountPoint":" + testLocalStorePathJSON
+ ","path":" + testSampleVolumeTempURIJSON
+ ","storagePoolUuid":"" + testLocalStoreUUID + "","
+ ""type":"ROOT","id":9,"size":0}}";
HypervResourceController rsrcServer = new HypervResourceController();
dynamic jsonDestoryCmd = JsonConvert.DeserializeObject(destoryCmd);
// Act
dynamic destoryAns = rsrcServer.DestroyCommand(jsonDestoryCmd);
// Assert
JObject ansAsProperty2 = destoryAns[0];
dynamic ans = ansAsProperty2.GetValue(CloudStackTypes.Answer);
String path = jsonDestoryCmd.volume.path;
Assert.True((bool)ans.result, "DestroyCommand did not succeed " + ans.details);
Assert.True(!File.Exists(path), "Failed to delete file " + path);
}
Lesson: TDD: write your code against tests

10. Lesson: Automate with CloudMonkey
cloudmonkey api createZone networktype="Advanced" securitygroupenabled="false" guestcidraddress="10.1.1.0/24“
name="HybridZone" localstorageenabled="true" dns1="4.4.4.4" internaldns1="10.70.176.118“
internaldns2="10.70.160.66"
…
apirequest=cloudmonkey api addSecondaryStorage zoneid=$zone
url="cifs://10.70.176.4/secondary?user=administrator&password=1pass%40word1"
cacheid=echo $apiresult | sed -e s/^.*"id": //; s/,.*$//
…
apiresult=cloudmonkey api addHost zoneid=$zone podid=$pod url="http://10.70.176.4" password="1pass@word1“
username="root" hypervisor="Hyperv" clusterid=$cluster
hostid=echo $apiresult | sed -e s/^.*"id": //; s/,.*$//
…
apiresult=cloudmonkey api listNetworkOfferings name="QuickCloudNoServices"
qcNetOffId=echo $apiresult | sed -e s/^.*"id": //; s/,.*$//
cloudmonkey api createNetwork zoneid=$zone networkofferingid=$qcNetOffId physicalnetworkid=$physnetid
name="QuickCloudNetName" displaytext="QuickCloudNetDesc" vlan=untagged acltype=domain gateway="10.70.176.1"
netmask="255.255.240.0" startip="10.70.176.124" endip="10.70.176.144"

11. Lesson: Adapt existing plug-ins
• Add CIFS support to NFS plug-in
• Similar workflow
• Mount, use local file system operations
• Can be specified in similar format
• cifs://192.168.1.128/CSHV3?user=root+password=1pass%40word1
• nfs://192.168.1.128/CSHV3
• Avoid scope creep…
“While you’re at Ikea for some towels, could you pick me up a couch?”
– unnamed flatmate

12. Lesson: QuickCloud instead of System VMs
Virtual
Router VM
Console VM
Secondary
Storage VM
CloudStack
Mgmt Server
• System VMs
• VM application
• Offload cloud services
• 3 kinds
• Hypervisor specific
• Esp boot args!
• QuickCloud alternative:
• Daemon for Secondary Storage
Service
• QuickCloudNoServices network
• No console VM

13. Lesson: Plan to avoid proprietary tools & libs
• Provide opensource build
• Mono
• Donation process takes time
• Proposal
• Vote
• Post source on Review Board
• SGA

14. Summary
• Extend CloudStack with Plug-ins
• Include ServerResource for device access
• Lessons:
• HTTPRequest lets you escape Java
• Serialise JSON objects, not ported Java classes
• TDD: write your code against tests
• Automate with CloudMonkey
• Adapt existing plug-ins
• QuickCloud instead of System VMs
• Plan to avoid proprietary tools & libs

15. References
• CloudStack Plugins / Modules / Extensions
• https://cwiki.apache.org/confluence/display/CLOUDSTACK/Plug-ins%2C+Modules%2C+and+Extensions
• Dynamic types in C#
• http://stackoverflow.com/questions/14071715/passing-dynamic-json-object-to-web-api-newtonsoft-example
• http://stackoverflow.com/questions/3142495/deserialize-json-into-c-sharp-dynamic-object
• CloudMonkey
• http://pythonhosted.org/cloudmonkey/
• http://dlafferty.blogspot.com/2013/07/using-cloudmonkey-to-automate.html
• QuickCloud
• https://cwiki.apache.org/confluence/display/CLOUDSTACK/QuickCloud
• CIFS Support
• https://cwiki.apache.org/confluence/display/CLOUDSTACK/CIFS+Support
• Build with Mono
• http://dlafferty.blogspot.com/2013/08/building-your-microsoft-solution-with.html
• Apache SGA
• http://www.apache.org/licenses/software-grant.txt