Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

OpenNebulaConf2019 - How We Use GOCA to Manage our OpenNebula Cloud - Jean-Philippe Fourès - Iguane Solutions

24 views

Published on

At Iguane Solutions, a lot of our "DevOps" tools are developed in Golang, and we have a good amount of experience in contributing to the Goca. I'll review just what contributions we make, as well as how we use Goca with different tools, on a daily basis, to manage and monitor our OpenNebula cloud.

I will delve into the concept of Infrastructure as Code - deployment of VM instances on cloud, as well as, also address the metrics collection of deployed VMs. Finally, I will present how we can abstract VM management with automation tools thanks to GOCA.

Published in: Software
  • Be the first to comment

  • Be the first to like this

OpenNebulaConf2019 - How We Use GOCA to Manage our OpenNebula Cloud - Jean-Philippe Fourès - Iguane Solutions

  1. 1. Iguane Solutions | Public Iguane Solutions ©2019
  2. 2. Iguane Solutions Iguane Solutions ©2019 - all rights reserved 2 Our contribution to “Goca” Jean-Philippe Fourès
  3. 3. Iguane Solutions Agenda 3 Iguane Solutions ©2019 - all rights reserved ● Iguane Solutions small introduction ● Goca contribution ○ Goca introduction ○ Details of our contribution ○ Roadmap ● Example of applications using Goca
  4. 4. 1. Company Introduction 4 Iguane Solutions Iguane Solutions ©2019 - all rights reserved
  5. 5. Iguane Solutions 5 •Network & CDN •>200 racks in 3 DCs •OpenNebula clusters • Infrastructure Expert (Network, Hardware, Storage) • “DevOps” culture: everything is automated • Our tools are Open Source based: CI/CD, Infra as code, metrology, Cloud • 1 OpenNebula cluster (2 very soon ☺) in Paris: ○ 10 KVM nodes (580 vCPU, 3.6 TB RAM) ○ Storpool for performance Storage • Cloud Hybridation specialist: Public / Private or Physical / Virtual •Network & CDN •>5 racks •Network & CDN •>10 racks •Network & CDN •>5 racks •Network & CDN •>10 racks 1.1 Who are we ? Founded in 2000, HQ in Paris, France Iguane Solutions ©2019 - all rights reserved
  6. 6. 66 Iguane Solutions ● For 7 years, IG1 cloud has been using VMware but: ○ Licence are expensive ○ Troubleshooting VSAN is a nightmare (and it is a black box) ○ VMware has lots of features => complexity 1.2 Iguane and One Iguane Solutions ©2019 - all rights reserved } ● In 2018, development of a new cloud service ○ Simpler than VMware ○ API friendly ○ Easy to deploy, to extend, to manage, to maintain and to hack around ○ Able to manage: VMware clusters and Public clouds => End of 2018: deployment of our first cluster based on OpenNebula In parallel to the deployment: Contribution to Goca and development tools using it
  7. 7. 7 Iguane Solutions 2. Goca Iguane Solutions ©2019 - all rights reserved
  8. 8. 88 Iguane Solutions 2.1 Introduction to Goca Iguane Solutions ©2019 - all rights reserved OCA (OpenNebula Cloud API) (ruby, java, python, golang)
  9. 9. 99 Iguane Solutions 2.1 Introduction to Goca Iguane Solutions ©2019 - all rights reserved What is Goca ? Why Goca instead of Ruby or Python API ? Golang Wrapper on top of XML RPC API Written in Golang Go + OCA = GOCA Add helpers to ease XML RPC instructions Python API not maintained Not Ruby friendly: Sorry Guys Golang: growing language in area of backend development https://talks.golang.org/2014/gocon-tokyo.slide#16
  10. 10. 1010 Iguane Solutions 2.2 Our contribution Iguane Solutions ©2019 - all rights reserved Rewrite XML mapping to parse and build entities data ● Change the XML handling package and add structures to map XML content ● Add dynamic management of template parts Rework Architecture: adding a controller layer ● Better split of concerns, client and controller ● Modify the API to access the resources in a controller centric way Rework of the XML-RPC client ● Give the control of the client to the user: ○ Client externalization, ○ Possible to extend it ● Leverage the use of the initial XML-RPC: ○ Manage only XML-RPC in XML-RPC package ○ Manage HTTP in the client directly Rewrite the error management ● Enable fine-grained management of errors with types and codes ● Better error handling: Don’t kill everyone on a package error
  11. 11. 1111 Iguane Solutions 2.3 Goca Organization Iguane Solutions ©2019 - all rights reserved Goca subpackages OpenNebula cluster “errors”: Error management package “schema”: Package containing all XML structures defined in different packages. Each package matches an OpenNebula object “goca”: It contains a file per object family api call. client.go manages HTTP connection controller.go manages resources
  12. 12. 1212 Iguane Solutions 2.4 Goca workflow Iguane Solutions ©2019 - all rights reserved How it works ? OpenNebula /RPC2 client Model VM Controller group ... ask client to call one.vm.info with value ‘42’ Controller eases objects manipulation and parsing. An application only needs to “play” with the controller to perform queries to OpenNebula Model, here is the client which performs HTTP requests to OpenNebula. It builds XML requests (session, XML formatting) and HTTP layer. # application code to get # info for VM 42 cli = goca.NewClient(...) c = goca.NewController(cli) vm = c.VM(42).Info() HTTP request to OpenNebula XML RPC endpoint Goca
  13. 13. 1313 Iguane Solutions 2.5 Roadmap Ideas Iguane Solutions ©2019 - all rights reserved Improve Tests - CI/CD ● Add more test cases to improve reliability and code coverage on missing entities ● Rewrite existing tests according to Go coding style, and consistently across entities ● Add a complete CI to test some scenarios with HTTP request to a miniOne in a container Increase the coverage of the XML-RPC API ● Highlights non-covered methods (missing tests case) ● Highlights unimplemented methods (dead code) ● Detect breaking changes, implement fallback methods ● Bug fixes Increase the user friendliness of the code ● Make the code more readable, improve code comments ● Try to minimize boilerplate code on developer side. For instance: type conversions ● Minimize the user need to read the documentation. For instance: remove variable parameters handling New functionalities ● Allow to filter entities on various criterias as template elements for instance ● Contribute to the development of OpenNebula new functionalities Suggestions are welcomed
  14. 14. 3. Applications Iguane Solutions 14 Iguane Solutions ©2019 - all rights reserved
  15. 15. 1515 Iguane Solutions 3.1 Terraform Provider Iguane Solutions ©2019 - all rights reserved Important note: Runtastic and Blackberry developed an OpenNebula provider using XML RPC calls directly before goca merge into One repository. ● Software developed by Hashicorp: Commonly used to deploy Infrastructure on Public clouds thanks to a large ecosystem of providers: AWS, GCP, Azure, … ● Written in Golang, it offers APIs to write custom providers easily Thanks to the Goca, we initiated the development of the OpenNebula Provider. Tested with ● Terraform version: 0.12+ ● OpenNebula 5.8 + Official OpenNebula Add-on since June 2019!
  16. 16. 1616 Iguane Solutions 3.1 Terraform Provider Iguane Solutions ©2019 - all rights reserved Group Image Virtual Network Security Group VDC VM Template VM For more details, issues, roadmap, contribution are welcomed: https://github.com/OpenNebula/terraform-provider-opennebula October 2018 June 2019 September 2019 Major steps history Internal beta release Added as OpenNebula Add-on Submission to Terraform Contribution to Goca Validation of first resources Resources currently supported: Official in Terraform Hopefully: December 2019
  17. 17. 1717 Iguane Solutions 3.1 Terraform Provider Iguane Solutions ©2019 - all rights reserved Goca used subpackages OpenNebula cluster schema: Terraform provider uses schema packages linked to goca files and permissions.go to manage object permissions quota.goto manage group quotas lock.goto manage lock on images “goca”: Each terraform resources matches with an OpenNebula object
  18. 18. 1818 Iguane Solutions Sismograf: ● Metrology agent, developed internally, deployed on each monitored host or VM ● Metrics are then collected by our Metrology tool: Sismology ● Several plugins available to monitor system and common application metrics: Sismograf plugin for OpenNebula has been developed to get usage metrics of our OpenNebula clusters 3.2 Sismograf Plugin Iguane Solutions ©2019 - all rights reserved OpenNebula Metrics into Iguane Solutions Metrology OpenNebula Fronts Sismograf agents to collect OpenNebula metrics VMs on OpenNebula Sismograf agents to collect VM metrics Iguane Sismology: Metrology system
  19. 19. 1919 Iguane Solutions 3.2 Sismograf Plugin Iguane Solutions ©2019 - all rights reserved Goca used subpackages OpenNebula cluster “goca”: It is used in the Sismograf plugin dedicated to OpenNebula. It gets all metrics of our cloud. It mainly use Info API calls. schema: Sismograf OpenNebula plugin uses schema packages linked to goca to get metrics on the various objects it also uses quota.goand snapshot.go because we also have metrics on quotas and size of a snapshot.
  20. 20. Questions ? Iguane Solutions 20 Iguane Solutions ©2019 - all rights reserved
  21. 21. Iguane Solutions ©2019

×