Deep Dive into the Microsoft OpenStack CI Infrastructure (Alessandro Pilotti)

Premium community conference on Microsoft technologies itcampro@ itcamp14#
Deep dive into the Microsoft
OpenStack CI infrastructure
Alessandro Pilotti
Cloudbase Solutions
Hyper-V MVP

Huge thanks to our sponsors & partners!

• OpenStack - what’s this?
• Hyper-V integration
• OpenStack continuous integration
• The Hyper-V CI infrastructure
• Undercloud and Overclouds
Agenda

An open source IaaS project
–Apache 2 license
Managed by the OpenStack Foundation
–More than 150 companies joined it including:
–AMD, Intel, Canonical, SUSE Linux, Red Hat,
Cisco, Dell, HP, IBM, NEC, Vmware
Portable
–Mostly used on Linux
Written in Python

• Essex
– Apr 2012
– Hyper-V removed
• Folsom
– October 2012
– Cloudbase Solutions involvement - Hyper-V re-included 
• Grizzly
– Apr 2013
• Havana
– October 2013
• Icehouse
– Apr 2014
Releases

• Distributed Components
– Can be deployed on a single server or multiple servers
• Public RESTful API
– Bindings available for various languages (including .Net)
• Queues (AMQP) for private APIs
– Great scalability
• Relational DBs for storing the configuration
– MySQL, PostgreSQL, SQLite (and SQL server)
• UI
– CLI tools
– Web dashboard
OpenStack Architecture

Compute (Nova)
Object Storage (Swift)
Block Storage (Cinder)
Image Service (Glance)
Networking (Quantum / Neutron)
Dashboard (Horizon)
Identity (Keystone)
Metering (Ceilometer)
Orchestration (Heat)
Components

Architecture 1

Architecture 2

• AMQP
–RabbitMQ
–Apache Qpid
• API
–RESTful services
• Database
–Any Python SQLAlchemy provider
–Mostly MySQL
–SQL Server required various bug fixes
Component interaction

• FREE edition
–Full Hypervisor
–Minimum OS support
• Minimum impact on security updates, etc
• Windows Server 2012 R2
–Just enable the related role
• Windows 8.1
–For workstation / testing / development usage
Hyper-V Server availability

• Nova Compute driver
• Neutron plugin
• Cinder Volume driver
• Windows Cloud-Init
• Ceilometer Agent
• Heat templates
Hyper-V / Windows Openstack components

• Python Application installed as an
application/service on the Hyper-V node.
• OpenStack compute utilizes key features
baked into the Hyper-V Virtualization
Platform
• Does not require windows clustering
services
• Does not require shared storage
Hyper-V Nova Compute

• Hyper-V plugin is part of Quantum since Jan
2013
– Project renamed in Neutron since Havana
• Supported network types:
– VLAN
– Flat
– Local
• Plugin / agent model
– Plugin runs in quantum-server (controller)
– Agent runs on each Hyper-V compute node
Neutron

Typical Neutron setup

• Quantum plugin / agent AMQP RPC protocol is
compatible with OpenVSwitch
• You can use the OVS plugin with Hyper-V agents (or
vice versa)
– Limited to compatible L2 protocols: Flat / VLAN
• You can use the L3 and DHCP agents with the Hyper-V
plugin
– Using the OVS L2 agent on the networking nodes
• Supports ML2 plugin!
Neutron OVS interop

• We ported OVS to Hyper-V
Open vSwitch Windows porting

• De facto standard for SDN
• Great interoperability
– Hyper-V / KVM / etc
• OpenFlow
• Tunnelling
–VXLAN
–GRE
• Neutron OVS plugin
–No need for a different one
Why OpenVSwitch?

• Great code review platform
–Gerrit
• Strong emphasis on unit tests
–Commits do not pass w/o tests
–Python is a dynamic language
• Syntactical and semantic validation needed
• Great continuous integration effort
–Lots of moving parts
How does OpenStack handle such complexity?

• Originally developed by Google for Android
• Stand-alone patch review system
• Manages:
–Git Repositories
–Code Review
–Event Stream
–Privileges (propose / merge code, manage git
repositories)
Gerrit

Gerrit workflow

Gerrit UI

• Except special cases every contribution
must have either:
–Blueprint
• for new features
• to be approved by core team
–Bug report
• for, ahem, bugs
• http://launchpad.net
Code contribution workflow

• Clone repository, e.g.:
– git clone https://github.com/openstack/nova.git
• Create a short lived branch:
– git checkout -b bug/12345678
– Do changes
– git commit
• Commit message must contain referefence to bug / bp
– git review
• Install with pip install git-review

• Gerrit triggers:
– Pep8 / Flake8 tests
• Code conformity über alles!
• Very important with > 850 contributors
– Unit tests
– Integration tests with multiple env (e.g. Nova):
• KVM
• Hyper-V
• VMWare
• XenServer
• etc

• Each CI results can vote +1 / -1 or be
neutral
–At the moment both errors and failures
account for -1
• Hard to distinguish between them
• In case of failures users can issue a
“recheck”

• At this stage code can be peer reviewed
• Every user can review the code
– Leave comments
– Vote +1 / -1
• Each project (e.g Nova) has a peer elected team
with “+2 rights”
– Vote +2 / -2
• A commit requires 2x +2 and approval to be
merged

• At this point the code is ready to be merged
–Another CI run is required
–Code base is changed since code submission
• If successful, the code will finally merge into
the git repository!

Queue monitoring

• Microsoft contributes hardware for CI
–Around 200 servers ATM
–Currently based in Cambridge, MA
• CI support team provided by:
–Microsoft
–Cloudbase Solutions
Hyper-V CI

• Consistency
– All projects should act the same way during code
reviews and offer the same interface to the devs
• Repeatability
– system should run 100 or 500 times per day
without failing
• Availability
– complete automation means that if it stops, the
whole project stops
OpenStack Requirements of automation

• Zuul
• Jenkins
• Undercloud
– KVM compute nodes
• Overcloud
– Devstack running on KVM VMs
– Hyper-V compute nodes
• Logs HTTP server
• Logstash
Hyper-V CI components

• Open source project
• monitors in-progress jobs for a set of
related patches
• will pre-emptively cancel any dependent
test jobs that would not succeed due to a
failure in a dependent patch
Zuul

Zuul workflow
Gerrit
registers
commit
Zuul listens
Gerrit
stream and
records
commit
Zuul clones
referenced
git repo
locally
Predefined
Jenkins
job(s)
started by
Zuul
Zuul collects
and pushes
upstream
Jenkins
job(s)
results

pipelines:
- name: check
description: Newly uploaded patchsets enter this pipeline to receive an initial +/-1 Verified vote from Jenkins.
failure-message: Build failed. For rechecking only on the Hyper-V CI, add a review comment with recheck hyper-v
manager: IndependentPipelineManager
precedence: low
trigger:
gerrit:
- event: patchset-created
- event: change-restored
- event: comment-added
comment_filter: (?i)^(Patch Set [0-9]+:nn)?s*recheck(( (?:bug|lp)[s#:]*(d+))|( no bug)|( hyper-v))s*$
success:
gerrit:
verified: 0
failure:
gerrit:
verified: 0
Zuul yaml config 1/2

jobs:
- name: ^.*$
parameter-function: set_log_url
- name: 01_Reserve_Hyper-V_Node_01
voting: true
success-pattern:
http://64.119.130.115/{build.parameters[ZUUL_CHANGE]}/{build.parameters[ZUUL_PATCHSET
]}/
failure-pattern:
http://64.119.130.115/{build.parameters[ZUUL_CHANGE]}/{build.parameters[ZUUL_PATCHSET
]}/
success-message: Looks good to me
failure-message: Test run failed.
projects:
- name: openstack/nova
check:
- 01_Reserve_Hyper-V_Node_01
- name: openstack/neutron
check:
- 01_Reserve_Hyper-V_Node_01
Zuul yaml config 2/2

• The de facto CI framework
–Open Source, written in Java
–Provides a well done UI (CLI as well)
• Lots of available plugins
• Supports git and other SCMs
• Executes jobs:
–on demand
–scheduled
–based on repository changes
Jenkins

• Jobs can be executed on
– Jenkins host
– Slaves (Linux, Windows)
• Jobs can be interdependent
– The CI executes multiple ones for each patchset
• Client / Server architecture
– Scales poorly
• Zuul integrates with Jenkins
– Executes jobs and waits for outcome
jenkins

• 5 jobs defined
1. Reserve Hyper-V Node 01
2. Reserve Hyper-V Node 02
3. Initialize Test Environment
4. Run Tempest Test
5. Clean Test Environment
• Return result is the result of executing jobs
1 to 4, result of job 5 is ignored
Jenkins CI jobs

Jenkins UI

• OpenStack’s integration tests suite
• Test types
– Smoke tests
– Positive tests
– Negative tests
– White box tests
• Roughly 2.200 tests executed by the Hyper-V CI
Tempest

Tempest

class MyTest(tempest.test.BaseTestCase):
def setUp(self):
…
def tearDown(self):
…
def do_whatever_action_test(self):
…
def do_whatever_other_action_test(self):
…
Tempest example - regular testcase

• Each time that a patchset is tested, an entire
OpenStack cloud needs to be spin up
• To achieve this, we use OpenStack
– Dogfooding FTW 
• Undercloud
– An OpenStack infrastructure
– Spins up IaaS instances for each patchset
• Overcloud
– The OpenStack cloud being tests
– runs the tempest tests
Under and Over clouds

• CentOS 6.5 servers
–Deployed with RDO
• Puppet based
–Controller
• APIs and most OpenStack services
–Network
• networking node(s)
• running Neutron + Open vSwitch
–KVM compute nodes
Undercloud

• Ubuntu 12.04 VMs
–on undercloud’s VMs
–Running DevStack
–OpenStack development environment
–Code dependencies pulled via git
• Hyper-V compute nodes
–Bare metal
–Two per job, to test live migration
–Attached by Jenkins to the DevStack overcloud
Overcloud

Undercloud UI - showing overcloud instances

• Jenkins and tempests logs are copied to a
file server
• Served to the user from an HTTP server
–Linked in the Zuul job results
• Logs are quite tricky to parse
–they’re fairly big!
• Enter Logstash!
Logs

• LogStash
–A tool for managing events and logs
• Elasticsearch
–Distributed search and analytics engine
• Kibana
–Web UI
• The Hyper-V CI logstash instance is hosted
on Azure
LogStash

LogStash

Q & A

Deep Dive into the Microsoft OpenStack CI Infrastructure (Alessandro Pilotti)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Deep Dive into the Microsoft OpenStack CI Infrastructure (Alessandro Pilotti)

Similar to Deep Dive into the Microsoft OpenStack CI Infrastructure (Alessandro Pilotti) (20)

More from ITCamp

More from ITCamp (20)

Recently uploaded

Recently uploaded (20)

Deep Dive into the Microsoft OpenStack CI Infrastructure (Alessandro Pilotti)