This document discusses lessons learned from developing a plugin to add Hyper-V hypervisor support to CloudStack. Key lessons include using HTTPRequest to escape Java's limits, serializing JSON objects instead of porting Java classes, writing code with test-driven development, automating tasks with CloudMonkey, adapting existing plugins where possible, using QuickCloud instead of system VMs, and avoiding proprietary tools and libraries. The plugin extends CloudStack using plugins, includes a ServerResource for device access, and implements the hypervisor API via a remote or direct connect agent.

1. How to add a new hypervisor to
CloudStack:
Lessons learned from Hyper-V
effort
Donal Lafferty
Friday, 15 November 2013

2. Summary
• Extend CloudStack with Plug-ins
• Include ServerResource for device access
• Lessons:
•
•
•
•
•
•
•
HTTPRequest lets you escape Java
Serialise JSON objects, not ported Java classes
TDD: write you code against tests
Automate with CloudMonkey
Adapt existing plug-ins
QuickCloud instead of System VMs
Plan to avoid proprietary tools & libs

3. Background: Extend CloudStack with
Plug-ins
• Java centric
• Plug-ins are distributed
(e.g. cloud-plugin-hypervisor-hyperv-4.3.0.jar)
(e.g. Discoverer)
• .jar
• Modules are loaded
• Spring config + class loader
• Extensions implement
• .class file implementing
interface
(e.g. Compute)

4. Background: ServerResource for Device
Access
• ServerResource
• Steps around Javas limits
Discoverer Module
Extension: Discoverer
- Brings resource under CloudStack control
• Two Agent types
• implemented directly by
the ServerResource
• E.g. XAPI calls
• running remotely,
connected to mgmt server
• E.g. KVM Agent
ServerResource
- Provides a communication layer in the form of an Agent
Connected Agent
Direct Connect Agent
CloudStack Message Bus
Device-specific
Connection
Remote CloudStack Agent
Extensible API

5. Problem: Create Plug-in for Hyper-V
Support
• VM lifecycle
• Avoid intermediaries
• CIFS for primary & secondary storage
• Analogous to NFS
• Hyper-V is SMB centric
• Advanced networking (ideally)
• Esp. VLANs for tenant isolation
• Console access (ideally)

6. Solution: Remote agent
Phase 1 – Connected Agent
Hyper-V API
(Python)
Server Resource
(Java)
Message Bus Agent
(Java - NIO)
AgentShell
(Windows Service)
Mgmt Server
(AgentManager)
Custom
TCP/JSON

7. Lesson: HTTPRequest lets you escape Java
Phase 2 – Direct Connect Agent
Hyper-V API
(WMI)
Server Resource
(C# - ASP.NET MVC4)
Web Server
(C# - not IIS)
AgentShell
(C# - Windows Service)
Mgmt Server
(Direct Connect Agent)
JSON over
HTTP

8. Lesson: Serialise JSON objects, not ported Java
classes
import com.cloud.agent.api.Answer;
import com.cloud.agent.api.AttachIsoCommand;
import com.cloud.agent.api.AttachVolumeAnswer;
import com.cloud.agent.api.AttachVolumeCommand;
import com.cloud.agent.api.BackupSnapshotAnswer;
import com.cloud.agent.api.BackupSnapshotCommand;
import com.cloud.agent.api.BumpUpPriorityCommand;
import com.cloud.agent.api.CheckHealthAnswer;
[HttpPost]
[ActionName(CloudStackTypes.StopCommand)]
public JContainer StopCommand([FromBody]dynamic cmd) {
string details = null; bool result = false;
import com.cloud.agent.api.CheckHealthCommand;
import com.cloud.agent.api.CheckNetworkAnswer;
try {
wmiCallsV2.DestroyVm(cmd.vmName);
result = true;
}
catch (Exception wmiEx) {
details = wmiEx.Message;
}
object ansContent = new {
result = result,
details = details,
vm = cmd.vm
};
return ReturnCloudStackTypedJArray(ansContent,
"com.cloud.agent.api.StopAnswer");
}
import com.cloud.agent.api.CheckNetworkCommand;
import com.cloud.agent.api.CheckOnHostAnswer;
import com.cloud.agent.api.CheckOnHostCommand;
import com.cloud.agent.api.CheckRouterAnswer;
import com.cloud.agent.api.CheckRouterCommand;
import com.cloud.agent.api.CheckS2SVpnConnectionsAnswer;
import com.cloud.agent.api.CheckS2SVpnConnectionsCommand;
import com.cloud.agent.api.CheckVirtualMachineAnswer;
import com.cloud.agent.api.CheckVirtualMachineCommand;
import com.cloud.agent.api.CleanupNetworkRulesCmd;
import com.cloud.agent.api.ClusterSyncAnswer;
import com.cloud.agent.api.ClusterSyncCommand;
import com.cloud.agent.api.Command;
import com.cloud.agent.api.CreatePrivateTemplateFromSnapshotCommand;
import com.cloud.agent.api.CreatePrivateTemplateFromVolumeCommand;
import com.cloud.agent.api.CreateStoragePoolCommand;
import com.cloud.agent.api.CreateVMSnapshotAnswer;
import com.cloud.agent.api.CreateVMSnapshotCommand;
import com.cloud.agent.api.CreateVolumeFromSnapshotAnswer;
import com.cloud.agent.api.CreateVolumeFromSnapshotCommand;
import com.cloud.agent.api.DeleteStoragePoolCommand;
import com.cloud.agent.api.DeleteVMSnapshotAnswer;
import com.cloud.agent.api.DeleteVMSnapshotCommand;
import com.cloud.agent.api.GetDomRVersionAnswer;
import com.cloud.agent.api.GetDomRVersionCmd;
import com.cloud.agent.api.GetHostStatsAnswer;
import com.cloud.agent.api.GetHostStatsCommand;
import com.cloud.agent.api.GetStorageStatsAnswer;
import com.cloud.agent.api.GetStorageStatsCommand;
import com.cloud.agent.api.GetVmStatsAnswer;
import com.cloud.agent.api.GetVmStatsCommand;
import com.cloud.agent.api.GetVncPortAnswer;
import com.cloud.agent.api.GetVncPortCommand;
import com.cloud.agent.api.HostStatsEntry;
import com.cloud.agent.api.MaintainAnswer;
import com.cloud.agent.api.MaintainCommand;
import com.cloud.agent.api.ManageSnapshotAnswer;
import com.cloud.agent.api.ManageSnapshotCommand;
import com.cloud.agent.api.MigrateAnswer;
import com.cloud.agent.api.MigrateCommand;
import com.cloud.agent.api.ModifySshKeysCommand;
import com.cloud.agent.api.ModifyStoragePoolAnswer;
import com.cloud.agent.api.ModifyStoragePoolCommand;
import com.cloud.agent.api.NetworkRulesSystemVmCommand;
import com.cloud.agent.api.NetworkRulesVmSecondaryIpCommand;
import com.cloud.agent.api.PingCommand;
import com.cloud.agent.api.PingRoutingCommand;
import com.cloud.agent.api.PingRoutingWithNwGroupsCommand;
import com.cloud.agent.api.PingRoutingWithOvsCommand;
import com.cloud.agent.api.PingTestCommand;
import com.cloud.agent.api.PlugNicAnswer;
import com.cloud.agent.api.PlugNicCommand;
import com.cloud.agent.api.PoolEjectCommand;
import com.cloud.agent.api.PrepareForMigrationAnswer;
import com.cloud.agent.api.PrepareForMigrationCommand;
import com.cloud.agent.api.PvlanSetupCommand;
import com.cloud.agent.api.ReadyAnswer;
import com.cloud.agent.api.ReadyCommand;
import com.cloud.agent.api.RebootAnswer;
import com.cloud.agent.api.RebootCommand;
import com.cloud.agent.api.RebootRouterCommand;
import com.cloud.agent.api.RevertToVMSnapshotAnswer;
import com.cloud.agent.api.RevertToVMSnapshotCommand;
import com.cloud.agent.api.ScaleVmAnswer;
import com.cloud.agent.api.ScaleVmCommand;
import com.cloud.agent.api.SecurityGroupRuleAnswer;
import com.cloud.agent.api.SecurityGroupRulesCmd;
import com.cloud.agent.api.SetupAnswer;
import com.cloud.agent.api.SetupCommand;
import com.cloud.agent.api.SetupGuestNetworkAnswer;
import com.cloud.agent.api.SetupGuestNetworkCommand;
import com.cloud.agent.api.StartAnswer;
import com.cloud.agent.api.StartCommand;
import com.cloud.agent.api.StartupCommand;
import com.cloud.agent.api.StartupRoutingCommand;
import com.cloud.agent.api.StartupStorageCommand;
import com.cloud.agent.api.StopAnswer;
}

9. Lesson: TDD: write your code against tests
public void TestDestroyCommand() {
// Arrange
String sampleVolume = getSampleVolumeObjectTO();
String destoryCmd = //"{"volume":" + getSampleVolumeObjectTO() + "}";
"{"volume":{"name":"" + testSampleVolumeTempUUIDNoExt
+ "","storagePoolType":"Filesystem",“
+ ""mountPoint":" + testLocalStorePathJSON
+ ","path":" + testSampleVolumeTempURIJSON
+ ","storagePoolUuid":"" + testLocalStoreUUID + "","
+ ""type":"ROOT","id":9,"size":0}}";
HypervResourceController rsrcServer = new HypervResourceController();
dynamic jsonDestoryCmd = JsonConvert.DeserializeObject(destoryCmd);
// Act
dynamic destoryAns = rsrcServer.DestroyCommand(jsonDestoryCmd);
// Assert
JObject ansAsProperty2 = destoryAns[0];
dynamic ans = ansAsProperty2.GetValue(CloudStackTypes.Answer);
String path = jsonDestoryCmd.volume.path;
Assert.True((bool)ans.result, "DestroyCommand did not succeed " + ans.details);
Assert.True(!File.Exists(path), "Failed to delete file " + path);
}

10. Lesson: Automate with CloudMonkey
cloudmonkey api createZone networktype="Advanced" securitygroupenabled="false" guestcidraddress="10.1.1.0/24“
name="HybridZone" localstorageenabled="true" dns1="4.4.4.4" internaldns1="10.70.176.118“
internaldns2="10.70.160.66"
…
apirequest=cloudmonkey api addSecondaryStorage zoneid=$zone
url="cifs://10.70.176.4/secondary?user=administrator&password=1pass%40word1"
cacheid=echo $apiresult | sed -e s/^.*"id": //; s/,.*$//
…
apiresult=cloudmonkey api addHost zoneid=$zone podid=$pod url="http://10.70.176.4" password="1pass@word1“
username="root" hypervisor="Hyperv" clusterid=$cluster
hostid=echo $apiresult | sed -e s/^.*"id": //; s/,.*$//
…
apiresult=cloudmonkey api listNetworkOfferings name="QuickCloudNoServices"
qcNetOffId=echo $apiresult | sed -e s/^.*"id": //; s/,.*$//
cloudmonkey api createNetwork zoneid=$zone networkofferingid=$qcNetOffId physicalnetworkid=$physnetid
name="QuickCloudNetName" displaytext="QuickCloudNetDesc" vlan=untagged acltype=domain gateway="10.70.176.1"
netmask="255.255.240.0" startip="10.70.176.124" endip="10.70.176.144"

11. Lesson: Adapt existing plug-ins
• Add CIFS support to NFS plug-in
• Similar workflow
• Mount, use local file system operations
• Can be specified in similar format
• cifs://192.168.1.128/CSHV3?user=root+password=1pass%40word1
• nfs://192.168.1.128/CSHV3
• Avoid scope creep…
“While you’re at Ikea for some towels, could you pick me up a couch?”
– unnamed flatmate

12. Lesson: QuickCloud instead of System
VMs
• System VMs
Virtual
Router VM
CloudStack
Mgmt Server
Console VM
• VM application
• Offload cloud services
• 3 kinds
• Hypervisor specific
• Esp boot args!
• QuickCloud alternative:
Secondary
Storage VM
• Daemon for Secondary Storage
Service
• QuickCloudNoServices network
• No console VM

13. Lesson: Plan to avoid proprietary tools &
libs
• Provide opensource build
• Mono
• Donation process takes time
• Proposal
• Vote
• Post source on Review Board
• SGA

14. Summary
• Extend CloudStack with Plug-ins
• Include ServerResource for device access
• Lessons:
•
•
•
•
•
•
•
HTTPRequest lets you escape Java
Serialise JSON objects, not ported Java classes
TDD: write your code against tests
Automate with CloudMonkey
Adapt existing plug-ins
QuickCloud instead of System VMs
Plan to avoid proprietary tools & libs

15. References
• CloudStack Plugins / Modules / Extensions
• https://cwiki.apache.org/confluence/display/CLOUDSTACK/Plug-ins%2C+Modules%2C+and+Extensions
• Dynamic types in C#
• http://stackoverflow.com/questions/14071715/passing-dynamic-json-object-to-web-api-newtonsoft-example
• http://stackoverflow.com/questions/3142495/deserialize-json-into-c-sharp-dynamic-object
• CloudMonkey
• http://pythonhosted.org/cloudmonkey/
• http://dlafferty.blogspot.com/2013/07/using-cloudmonkey-to-automate.html
• QuickCloud
• https://cwiki.apache.org/confluence/display/CLOUDSTACK/QuickCloud
• CIFS Support
• https://cwiki.apache.org/confluence/display/CLOUDSTACK/CIFS+Support
• Build with Mono
• http://dlafferty.blogspot.com/2013/08/building-your-microsoft-solution-with.html
• Apache SGA
• http://www.apache.org/licenses/software-grant.txt