Deep Dive into the Microsoft OpenStack CI Infrastructure (Alessandro Pilotti)

1,009 views

Published on

OpenStack is not only the leading open source cloud computing project, it also features one of the most advanced code review and continuous integration (CI) frameworks.

Since Hyper-V is one of the main compute options available in OpenStack, it required the development of a dedicated infrastructure for CI testing, which is hosted by Microsoft in Cambridge, MA and mantained by Microsoft and Cloudbase Solutions.

Building this infrastructure was no easy task: it features full Puppet based automation with almost 200 physical hosts, Jenkins, a KVM based OpenStack cloud that deploys nested OpenStack Hyper-V clouds, more than 2000 integration tests executed for each run, hundreds of times a day, LogStash and ElasticSearch logging running on Azure and more.

During this session we’ll be able to show how it works and to dive into all the technical and architectural choices we made.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,009
On SlideShare
0
From Embeds
0
Number of Embeds
52
Actions
Shares
0
Downloads
23
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Deep Dive into the Microsoft OpenStack CI Infrastructure (Alessandro Pilotti)

  1. 1. Premium community conference on Microsoft technologies itcampro@ itcamp14# Deep dive into the Microsoft OpenStack CI infrastructure Alessandro Pilotti Cloudbase Solutions Hyper-V MVP
  2. 2. Premium community conference on Microsoft technologies itcampro@ itcamp14# Huge thanks to our sponsors & partners!
  3. 3. Premium community conference on Microsoft technologies itcampro@ itcamp14# • OpenStack - what’s this? • Hyper-V integration • OpenStack continuous integration • The Hyper-V CI infrastructure • Undercloud and Overclouds Agenda
  4. 4. Premium community conference on Microsoft technologies itcampro@ itcamp14# An open source IaaS project –Apache 2 license Managed by the OpenStack Foundation –More than 150 companies joined it including: –AMD, Intel, Canonical, SUSE Linux, Red Hat, Cisco, Dell, HP, IBM, NEC, Vmware Portable –Mostly used on Linux Written in Python
  5. 5. Premium community conference on Microsoft technologies itcampro@ itcamp14# • Essex – Apr 2012 – Hyper-V removed • Folsom – October 2012 – Cloudbase Solutions involvement - Hyper-V re-included  • Grizzly – Apr 2013 • Havana – October 2013 • Icehouse – Apr 2014 Releases
  6. 6. Premium community conference on Microsoft technologies itcampro@ itcamp14# • Distributed Components – Can be deployed on a single server or multiple servers • Public RESTful API – Bindings available for various languages (including .Net) • Queues (AMQP) for private APIs – Great scalability • Relational DBs for storing the configuration – MySQL, PostgreSQL, SQLite (and SQL server) • UI – CLI tools – Web dashboard OpenStack Architecture
  7. 7. Premium community conference on Microsoft technologies itcampro@ itcamp14# Compute (Nova) Object Storage (Swift) Block Storage (Cinder) Image Service (Glance) Networking (Quantum / Neutron) Dashboard (Horizon) Identity (Keystone) Metering (Ceilometer) Orchestration (Heat) Components
  8. 8. Premium community conference on Microsoft technologies itcampro@ itcamp14# Architecture 1
  9. 9. Premium community conference on Microsoft technologies itcampro@ itcamp14# Architecture 2
  10. 10. Premium community conference on Microsoft technologies itcampro@ itcamp14# • AMQP –RabbitMQ –Apache Qpid • API –RESTful services • Database –Any Python SQLAlchemy provider –Mostly MySQL –SQL Server required various bug fixes Component interaction
  11. 11. Premium community conference on Microsoft technologies itcampro@ itcamp14# • FREE edition –Full Hypervisor –Minimum OS support • Minimum impact on security updates, etc • Windows Server 2012 R2 –Just enable the related role • Windows 8.1 –For workstation / testing / development usage Hyper-V Server availability
  12. 12. Premium community conference on Microsoft technologies itcampro@ itcamp14# • Nova Compute driver • Neutron plugin • Cinder Volume driver • Windows Cloud-Init • Ceilometer Agent • Heat templates Hyper-V / Windows Openstack components
  13. 13. Premium community conference on Microsoft technologies itcampro@ itcamp14# • Python Application installed as an application/service on the Hyper-V node. • OpenStack compute utilizes key features baked into the Hyper-V Virtualization Platform • Does not require windows clustering services • Does not require shared storage Hyper-V Nova Compute
  14. 14. Premium community conference on Microsoft technologies itcampro@ itcamp14# • Hyper-V plugin is part of Quantum since Jan 2013 – Project renamed in Neutron since Havana • Supported network types: – VLAN – Flat – Local • Plugin / agent model – Plugin runs in quantum-server (controller) – Agent runs on each Hyper-V compute node Neutron
  15. 15. Premium community conference on Microsoft technologies itcampro@ itcamp14# Typical Neutron setup
  16. 16. Premium community conference on Microsoft technologies itcampro@ itcamp14# • Quantum plugin / agent AMQP RPC protocol is compatible with OpenVSwitch • You can use the OVS plugin with Hyper-V agents (or vice versa) – Limited to compatible L2 protocols: Flat / VLAN • You can use the L3 and DHCP agents with the Hyper-V plugin – Using the OVS L2 agent on the networking nodes • Supports ML2 plugin! Neutron OVS interop
  17. 17. Premium community conference on Microsoft technologies itcampro@ itcamp14# • We ported OVS to Hyper-V Open vSwitch Windows porting
  18. 18. Premium community conference on Microsoft technologies itcampro@ itcamp14# • De facto standard for SDN • Great interoperability – Hyper-V / KVM / etc • OpenFlow • Tunnelling –VXLAN –GRE • Neutron OVS plugin –No need for a different one Why OpenVSwitch?
  19. 19. Premium community conference on Microsoft technologies itcampro@ itcamp14# • Great code review platform –Gerrit • Strong emphasis on unit tests –Commits do not pass w/o tests –Python is a dynamic language • Syntactical and semantic validation needed • Great continuous integration effort –Lots of moving parts How does OpenStack handle such complexity?
  20. 20. Premium community conference on Microsoft technologies itcampro@ itcamp14# • Originally developed by Google for Android • Stand-alone patch review system • Manages: –Git Repositories –Code Review –Event Stream –Privileges (propose / merge code, manage git repositories) Gerrit
  21. 21. Premium community conference on Microsoft technologies itcampro@ itcamp14# Gerrit workflow
  22. 22. Premium community conference on Microsoft technologies itcampro@ itcamp14# Gerrit UI
  23. 23. Premium community conference on Microsoft technologies itcampro@ itcamp14# • Except special cases every contribution must have either: –Blueprint • for new features • to be approved by core team –Bug report • for, ahem, bugs • http://launchpad.net Code contribution workflow
  24. 24. Premium community conference on Microsoft technologies itcampro@ itcamp14# • Clone repository, e.g.: – git clone https://github.com/openstack/nova.git • Create a short lived branch: – git checkout -b bug/12345678 – Do changes – git commit • Commit message must contain referefence to bug / bp – git review • Install with pip install git-review Code contribution workflow
  25. 25. Premium community conference on Microsoft technologies itcampro@ itcamp14# • Gerrit triggers: – Pep8 / Flake8 tests • Code conformity über alles! • Very important with > 850 contributors – Unit tests – Integration tests with multiple env (e.g. Nova): • KVM • Hyper-V • VMWare • XenServer • etc Code contribution workflow
  26. 26. Premium community conference on Microsoft technologies itcampro@ itcamp14# • Each CI results can vote +1 / -1 or be neutral –At the moment both errors and failures account for -1 • Hard to distinguish between them • In case of failures users can issue a “recheck” Code contribution workflow
  27. 27. Premium community conference on Microsoft technologies itcampro@ itcamp14# Code contribution workflow
  28. 28. Premium community conference on Microsoft technologies itcampro@ itcamp14# • At this stage code can be peer reviewed • Every user can review the code – Leave comments – Vote +1 / -1 • Each project (e.g Nova) has a peer elected team with “+2 rights” – Vote +2 / -2 • A commit requires 2x +2 and approval to be merged Code contribution workflow
  29. 29. Premium community conference on Microsoft technologies itcampro@ itcamp14# • At this point the code is ready to be merged –Another CI run is required –Code base is changed since code submission • If successful, the code will finally merge into the git repository! Code contribution workflow
  30. 30. Premium community conference on Microsoft technologies itcampro@ itcamp14# Queue monitoring
  31. 31. Premium community conference on Microsoft technologies itcampro@ itcamp14# • Microsoft contributes hardware for CI –Around 200 servers ATM –Currently based in Cambridge, MA • CI support team provided by: –Microsoft –Cloudbase Solutions Hyper-V CI
  32. 32. Premium community conference on Microsoft technologies itcampro@ itcamp14# • Consistency – All projects should act the same way during code reviews and offer the same interface to the devs • Repeatability – system should run 100 or 500 times per day without failing • Availability – complete automation means that if it stops, the whole project stops OpenStack Requirements of automation
  33. 33. Premium community conference on Microsoft technologies itcampro@ itcamp14# • Zuul • Jenkins • Undercloud – KVM compute nodes • Overcloud – Devstack running on KVM VMs – Hyper-V compute nodes • Logs HTTP server • Logstash Hyper-V CI components
  34. 34. Premium community conference on Microsoft technologies itcampro@ itcamp14# • Open source project • monitors in-progress jobs for a set of related patches • will pre-emptively cancel any dependent test jobs that would not succeed due to a failure in a dependent patch Zuul
  35. 35. Premium community conference on Microsoft technologies itcampro@ itcamp14# Zuul workflow Gerrit registers commit Zuul listens Gerrit stream and records commit Zuul clones referenced git repo locally Predefined Jenkins job(s) started by Zuul Zuul collects and pushes upstream Jenkins job(s) results
  36. 36. Premium community conference on Microsoft technologies itcampro@ itcamp14# pipelines: - name: check description: Newly uploaded patchsets enter this pipeline to receive an initial +/-1 Verified vote from Jenkins. failure-message: Build failed. For rechecking only on the Hyper-V CI, add a review comment with recheck hyper-v manager: IndependentPipelineManager precedence: low trigger: gerrit: - event: patchset-created - event: change-restored - event: comment-added comment_filter: (?i)^(Patch Set [0-9]+:nn)?s*recheck(( (?:bug|lp)[s#:]*(d+))|( no bug)|( hyper-v))s*$ success: gerrit: verified: 0 failure: gerrit: verified: 0 Zuul yaml config 1/2
  37. 37. Premium community conference on Microsoft technologies itcampro@ itcamp14# jobs: - name: ^.*$ parameter-function: set_log_url - name: 01_Reserve_Hyper-V_Node_01 voting: true success-pattern: http://64.119.130.115/{build.parameters[ZUUL_CHANGE]}/{build.parameters[ZUUL_PATCHSET ]}/ failure-pattern: http://64.119.130.115/{build.parameters[ZUUL_CHANGE]}/{build.parameters[ZUUL_PATCHSET ]}/ success-message: Looks good to me failure-message: Test run failed. projects: - name: openstack/nova check: - 01_Reserve_Hyper-V_Node_01 - name: openstack/neutron check: - 01_Reserve_Hyper-V_Node_01 Zuul yaml config 2/2
  38. 38. Premium community conference on Microsoft technologies itcampro@ itcamp14# • The de facto CI framework –Open Source, written in Java –Provides a well done UI (CLI as well) • Lots of available plugins • Supports git and other SCMs • Executes jobs: –on demand –scheduled –based on repository changes Jenkins
  39. 39. Premium community conference on Microsoft technologies itcampro@ itcamp14# • Jobs can be executed on – Jenkins host – Slaves (Linux, Windows) • Jobs can be interdependent – The CI executes multiple ones for each patchset • Client / Server architecture – Scales poorly • Zuul integrates with Jenkins – Executes jobs and waits for outcome jenkins
  40. 40. Premium community conference on Microsoft technologies itcampro@ itcamp14# • 5 jobs defined 1. Reserve Hyper-V Node 01 2. Reserve Hyper-V Node 02 3. Initialize Test Environment 4. Run Tempest Test 5. Clean Test Environment • Return result is the result of executing jobs 1 to 4, result of job 5 is ignored Jenkins CI jobs
  41. 41. Premium community conference on Microsoft technologies itcampro@ itcamp14# Jenkins UI
  42. 42. Premium community conference on Microsoft technologies itcampro@ itcamp14# • OpenStack’s integration tests suite • Test types – Smoke tests – Positive tests – Negative tests – White box tests • Roughly 2.200 tests executed by the Hyper-V CI Tempest
  43. 43. Premium community conference on Microsoft technologies itcampro@ itcamp14# Tempest
  44. 44. Premium community conference on Microsoft technologies itcampro@ itcamp14# class MyTest(tempest.test.BaseTestCase): def setUp(self): … def tearDown(self): … def do_whatever_action_test(self): … def do_whatever_other_action_test(self): … Tempest example - regular testcase
  45. 45. Premium community conference on Microsoft technologies itcampro@ itcamp14# • Each time that a patchset is tested, an entire OpenStack cloud needs to be spin up • To achieve this, we use OpenStack – Dogfooding FTW  • Undercloud – An OpenStack infrastructure – Spins up IaaS instances for each patchset • Overcloud – The OpenStack cloud being tests – runs the tempest tests Under and Over clouds
  46. 46. Premium community conference on Microsoft technologies itcampro@ itcamp14# • CentOS 6.5 servers –Deployed with RDO • Puppet based –Controller • APIs and most OpenStack services –Network • networking node(s) • running Neutron + Open vSwitch –KVM compute nodes Undercloud
  47. 47. Premium community conference on Microsoft technologies itcampro@ itcamp14# • Ubuntu 12.04 VMs –on undercloud’s VMs –Running DevStack –OpenStack development environment –Code dependencies pulled via git • Hyper-V compute nodes –Bare metal –Two per job, to test live migration –Attached by Jenkins to the DevStack overcloud Overcloud
  48. 48. Premium community conference on Microsoft technologies itcampro@ itcamp14# Undercloud UI - showing overcloud instances
  49. 49. Premium community conference on Microsoft technologies itcampro@ itcamp14# • Jenkins and tempests logs are copied to a file server • Served to the user from an HTTP server –Linked in the Zuul job results • Logs are quite tricky to parse –they’re fairly big! • Enter Logstash! Logs
  50. 50. Premium community conference on Microsoft technologies itcampro@ itcamp14# • LogStash –A tool for managing events and logs • Elasticsearch –Distributed search and analytics engine • Kibana –Web UI • The Hyper-V CI logstash instance is hosted on Azure LogStash
  51. 51. Premium community conference on Microsoft technologies itcampro@ itcamp14# LogStash
  52. 52. Premium community conference on Microsoft technologies itcampro@ itcamp14# Q & A

×