How to add a new hypervisor to
CloudStack:

Lessons learned from Hyper-V
effort
Donal Lafferty
Friday, 15 November 2013
Summary
• Extend CloudStack with Plug-ins
• Include ServerResource for device access

• Lessons:
•
•
•
•
•
•
•

HTTPReques...
Background: Extend CloudStack with
Plug-ins
• Java centric
• Plug-ins are distributed

(e.g. cloud-plugin-hypervisor-hyper...
Background: ServerResource for Device
Access
• ServerResource
• Steps around Javas limits

Discoverer Module

Extension: D...
Problem: Create Plug-in for Hyper-V
Support
• VM lifecycle
• Avoid intermediaries

• CIFS for primary & secondary storage
...
Solution: Remote agent
Phase 1 – Connected Agent

Hyper-V API
(Python)

Server Resource
(Java)

Message Bus Agent
(Java - ...
Lesson: HTTPRequest lets you escape Java
Phase 2 – Direct Connect Agent

Hyper-V API
(WMI)

Server Resource
(C# - ASP.NET ...
Lesson: Serialise JSON objects, not ported Java
classes
import com.cloud.agent.api.Answer;
import com.cloud.agent.api.Atta...
Lesson: TDD: write your code against tests
public void TestDestroyCommand() {
// Arrange
String sampleVolume = getSampleVo...
Lesson: Automate with CloudMonkey
cloudmonkey api createZone networktype="Advanced" securitygroupenabled="false" guestcidr...
Lesson: Adapt existing plug-ins
• Add CIFS support to NFS plug-in
• Similar workflow
• Mount, use local file system operat...
Lesson: QuickCloud instead of System
VMs
• System VMs
Virtual
Router VM
CloudStack
Mgmt Server

Console VM

• VM applicati...
Lesson: Plan to avoid proprietary tools &
libs
• Provide opensource build
• Mono

• Donation process takes time
• Proposal...
Summary
• Extend CloudStack with Plug-ins
• Include ServerResource for device access

• Lessons:
•
•
•
•
•
•
•

HTTPReques...
References
• CloudStack Plugins / Modules / Extensions
• https://cwiki.apache.org/confluence/display/CLOUDSTACK/Plug-ins%2...
Upcoming SlideShare
Loading in …5
×

How to add a new hypervisor to CloudStack - Lessons learned from Hyper-V effort

1,212 views

Published on

CloudStack European User Group Meetup in London Jan 2014. Presentation by Donal Lafferty

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,212
On SlideShare
0
From Embeds
0
Number of Embeds
248
Actions
Shares
0
Downloads
9
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

How to add a new hypervisor to CloudStack - Lessons learned from Hyper-V effort

  1. 1. How to add a new hypervisor to CloudStack: Lessons learned from Hyper-V effort Donal Lafferty Friday, 15 November 2013
  2. 2. Summary • Extend CloudStack with Plug-ins • Include ServerResource for device access • Lessons: • • • • • • • HTTPRequest lets you escape Java Serialise JSON objects, not ported Java classes TDD: write you code against tests Automate with CloudMonkey Adapt existing plug-ins QuickCloud instead of System VMs Plan to avoid proprietary tools & libs
  3. 3. Background: Extend CloudStack with Plug-ins • Java centric • Plug-ins are distributed (e.g. cloud-plugin-hypervisor-hyperv-4.3.0.jar) (e.g. Discoverer) • .jar • Modules are loaded • Spring config + class loader • Extensions implement • .class file implementing interface (e.g. Compute)
  4. 4. Background: ServerResource for Device Access • ServerResource • Steps around Javas limits Discoverer Module Extension: Discoverer - Brings resource under CloudStack control • Two Agent types • implemented directly by the ServerResource • E.g. XAPI calls • running remotely, connected to mgmt server • E.g. KVM Agent ServerResource - Provides a communication layer in the form of an Agent Connected Agent Direct Connect Agent CloudStack Message Bus Device-specific Connection Remote CloudStack Agent Extensible API
  5. 5. Problem: Create Plug-in for Hyper-V Support • VM lifecycle • Avoid intermediaries • CIFS for primary & secondary storage • Analogous to NFS • Hyper-V is SMB centric • Advanced networking (ideally) • Esp. VLANs for tenant isolation • Console access (ideally)
  6. 6. Solution: Remote agent Phase 1 – Connected Agent Hyper-V API (Python) Server Resource (Java) Message Bus Agent (Java - NIO) AgentShell (Windows Service) Mgmt Server (AgentManager) Custom TCP/JSON
  7. 7. Lesson: HTTPRequest lets you escape Java Phase 2 – Direct Connect Agent Hyper-V API (WMI) Server Resource (C# - ASP.NET MVC4) Web Server (C# - not IIS) AgentShell (C# - Windows Service) Mgmt Server (Direct Connect Agent) JSON over HTTP
  8. 8. Lesson: Serialise JSON objects, not ported Java classes import com.cloud.agent.api.Answer; import com.cloud.agent.api.AttachIsoCommand; import com.cloud.agent.api.AttachVolumeAnswer; import com.cloud.agent.api.AttachVolumeCommand; import com.cloud.agent.api.BackupSnapshotAnswer; import com.cloud.agent.api.BackupSnapshotCommand; import com.cloud.agent.api.BumpUpPriorityCommand; import com.cloud.agent.api.CheckHealthAnswer; [HttpPost] [ActionName(CloudStackTypes.StopCommand)] public JContainer StopCommand([FromBody]dynamic cmd) { string details = null; bool result = false; import com.cloud.agent.api.CheckHealthCommand; import com.cloud.agent.api.CheckNetworkAnswer; try { wmiCallsV2.DestroyVm(cmd.vmName); result = true; } catch (Exception wmiEx) { details = wmiEx.Message; } object ansContent = new { result = result, details = details, vm = cmd.vm }; return ReturnCloudStackTypedJArray(ansContent, "com.cloud.agent.api.StopAnswer"); } import com.cloud.agent.api.CheckNetworkCommand; import com.cloud.agent.api.CheckOnHostAnswer; import com.cloud.agent.api.CheckOnHostCommand; import com.cloud.agent.api.CheckRouterAnswer; import com.cloud.agent.api.CheckRouterCommand; import com.cloud.agent.api.CheckS2SVpnConnectionsAnswer; import com.cloud.agent.api.CheckS2SVpnConnectionsCommand; import com.cloud.agent.api.CheckVirtualMachineAnswer; import com.cloud.agent.api.CheckVirtualMachineCommand; import com.cloud.agent.api.CleanupNetworkRulesCmd; import com.cloud.agent.api.ClusterSyncAnswer; import com.cloud.agent.api.ClusterSyncCommand; import com.cloud.agent.api.Command; import com.cloud.agent.api.CreatePrivateTemplateFromSnapshotCommand; import com.cloud.agent.api.CreatePrivateTemplateFromVolumeCommand; import com.cloud.agent.api.CreateStoragePoolCommand; import com.cloud.agent.api.CreateVMSnapshotAnswer; import com.cloud.agent.api.CreateVMSnapshotCommand; import com.cloud.agent.api.CreateVolumeFromSnapshotAnswer; import com.cloud.agent.api.CreateVolumeFromSnapshotCommand; import com.cloud.agent.api.DeleteStoragePoolCommand; import com.cloud.agent.api.DeleteVMSnapshotAnswer; import com.cloud.agent.api.DeleteVMSnapshotCommand; import com.cloud.agent.api.GetDomRVersionAnswer; import com.cloud.agent.api.GetDomRVersionCmd; import com.cloud.agent.api.GetHostStatsAnswer; import com.cloud.agent.api.GetHostStatsCommand; import com.cloud.agent.api.GetStorageStatsAnswer; import com.cloud.agent.api.GetStorageStatsCommand; import com.cloud.agent.api.GetVmStatsAnswer; import com.cloud.agent.api.GetVmStatsCommand; import com.cloud.agent.api.GetVncPortAnswer; import com.cloud.agent.api.GetVncPortCommand; import com.cloud.agent.api.HostStatsEntry; import com.cloud.agent.api.MaintainAnswer; import com.cloud.agent.api.MaintainCommand; import com.cloud.agent.api.ManageSnapshotAnswer; import com.cloud.agent.api.ManageSnapshotCommand; import com.cloud.agent.api.MigrateAnswer; import com.cloud.agent.api.MigrateCommand; import com.cloud.agent.api.ModifySshKeysCommand; import com.cloud.agent.api.ModifyStoragePoolAnswer; import com.cloud.agent.api.ModifyStoragePoolCommand; import com.cloud.agent.api.NetworkRulesSystemVmCommand; import com.cloud.agent.api.NetworkRulesVmSecondaryIpCommand; import com.cloud.agent.api.PingCommand; import com.cloud.agent.api.PingRoutingCommand; import com.cloud.agent.api.PingRoutingWithNwGroupsCommand; import com.cloud.agent.api.PingRoutingWithOvsCommand; import com.cloud.agent.api.PingTestCommand; import com.cloud.agent.api.PlugNicAnswer; import com.cloud.agent.api.PlugNicCommand; import com.cloud.agent.api.PoolEjectCommand; import com.cloud.agent.api.PrepareForMigrationAnswer; import com.cloud.agent.api.PrepareForMigrationCommand; import com.cloud.agent.api.PvlanSetupCommand; import com.cloud.agent.api.ReadyAnswer; import com.cloud.agent.api.ReadyCommand; import com.cloud.agent.api.RebootAnswer; import com.cloud.agent.api.RebootCommand; import com.cloud.agent.api.RebootRouterCommand; import com.cloud.agent.api.RevertToVMSnapshotAnswer; import com.cloud.agent.api.RevertToVMSnapshotCommand; import com.cloud.agent.api.ScaleVmAnswer; import com.cloud.agent.api.ScaleVmCommand; import com.cloud.agent.api.SecurityGroupRuleAnswer; import com.cloud.agent.api.SecurityGroupRulesCmd; import com.cloud.agent.api.SetupAnswer; import com.cloud.agent.api.SetupCommand; import com.cloud.agent.api.SetupGuestNetworkAnswer; import com.cloud.agent.api.SetupGuestNetworkCommand; import com.cloud.agent.api.StartAnswer; import com.cloud.agent.api.StartCommand; import com.cloud.agent.api.StartupCommand; import com.cloud.agent.api.StartupRoutingCommand; import com.cloud.agent.api.StartupStorageCommand; import com.cloud.agent.api.StopAnswer; }
  9. 9. Lesson: TDD: write your code against tests public void TestDestroyCommand() { // Arrange String sampleVolume = getSampleVolumeObjectTO(); String destoryCmd = //"{"volume":" + getSampleVolumeObjectTO() + "}"; "{"volume":{"name":"" + testSampleVolumeTempUUIDNoExt + "","storagePoolType":"Filesystem",“ + ""mountPoint":" + testLocalStorePathJSON + ","path":" + testSampleVolumeTempURIJSON + ","storagePoolUuid":"" + testLocalStoreUUID + ""," + ""type":"ROOT","id":9,"size":0}}"; HypervResourceController rsrcServer = new HypervResourceController(); dynamic jsonDestoryCmd = JsonConvert.DeserializeObject(destoryCmd); // Act dynamic destoryAns = rsrcServer.DestroyCommand(jsonDestoryCmd); // Assert JObject ansAsProperty2 = destoryAns[0]; dynamic ans = ansAsProperty2.GetValue(CloudStackTypes.Answer); String path = jsonDestoryCmd.volume.path; Assert.True((bool)ans.result, "DestroyCommand did not succeed " + ans.details); Assert.True(!File.Exists(path), "Failed to delete file " + path); }
  10. 10. Lesson: Automate with CloudMonkey cloudmonkey api createZone networktype="Advanced" securitygroupenabled="false" guestcidraddress="10.1.1.0/24“ name="HybridZone" localstorageenabled="true" dns1="4.4.4.4" internaldns1="10.70.176.118“ internaldns2="10.70.160.66" … apirequest=cloudmonkey api addSecondaryStorage zoneid=$zone url="cifs://10.70.176.4/secondary?user=administrator&password=1pass%40word1" cacheid=echo $apiresult | sed -e s/^.*"id": //; s/,.*$// … apiresult=cloudmonkey api addHost zoneid=$zone podid=$pod url="http://10.70.176.4" password="1pass@word1“ username="root" hypervisor="Hyperv" clusterid=$cluster hostid=echo $apiresult | sed -e s/^.*"id": //; s/,.*$// … apiresult=cloudmonkey api listNetworkOfferings name="QuickCloudNoServices" qcNetOffId=echo $apiresult | sed -e s/^.*"id": //; s/,.*$// cloudmonkey api createNetwork zoneid=$zone networkofferingid=$qcNetOffId physicalnetworkid=$physnetid name="QuickCloudNetName" displaytext="QuickCloudNetDesc" vlan=untagged acltype=domain gateway="10.70.176.1" netmask="255.255.240.0" startip="10.70.176.124" endip="10.70.176.144"
  11. 11. Lesson: Adapt existing plug-ins • Add CIFS support to NFS plug-in • Similar workflow • Mount, use local file system operations • Can be specified in similar format • cifs://192.168.1.128/CSHV3?user=root+password=1pass%40word1 • nfs://192.168.1.128/CSHV3 • Avoid scope creep… “While you’re at Ikea for some towels, could you pick me up a couch?” – unnamed flatmate
  12. 12. Lesson: QuickCloud instead of System VMs • System VMs Virtual Router VM CloudStack Mgmt Server Console VM • VM application • Offload cloud services • 3 kinds • Hypervisor specific • Esp boot args! • QuickCloud alternative: Secondary Storage VM • Daemon for Secondary Storage Service • QuickCloudNoServices network • No console VM
  13. 13. Lesson: Plan to avoid proprietary tools & libs • Provide opensource build • Mono • Donation process takes time • Proposal • Vote • Post source on Review Board • SGA
  14. 14. Summary • Extend CloudStack with Plug-ins • Include ServerResource for device access • Lessons: • • • • • • • HTTPRequest lets you escape Java Serialise JSON objects, not ported Java classes TDD: write your code against tests Automate with CloudMonkey Adapt existing plug-ins QuickCloud instead of System VMs Plan to avoid proprietary tools & libs
  15. 15. References • CloudStack Plugins / Modules / Extensions • https://cwiki.apache.org/confluence/display/CLOUDSTACK/Plug-ins%2C+Modules%2C+and+Extensions • Dynamic types in C# • http://stackoverflow.com/questions/14071715/passing-dynamic-json-object-to-web-api-newtonsoft-example • http://stackoverflow.com/questions/3142495/deserialize-json-into-c-sharp-dynamic-object • CloudMonkey • http://pythonhosted.org/cloudmonkey/ • http://dlafferty.blogspot.com/2013/07/using-cloudmonkey-to-automate.html • QuickCloud • https://cwiki.apache.org/confluence/display/CLOUDSTACK/QuickCloud • CIFS Support • https://cwiki.apache.org/confluence/display/CLOUDSTACK/CIFS+Support • Build with Mono • http://dlafferty.blogspot.com/2013/08/building-your-microsoft-solution-with.html • Apache SGA • http://www.apache.org/licenses/software-grant.txt

×