Openstack is one of the largest OSS projects today with hundreds of commits flowing in daily. This high rate of change requires an advanced CI infrastructure. The purpose of the talk is to provide an overview of this infrastructure, explaining the role of each tool and the pipelines along which changes have to travel before they find their way into the approved Openstack codebase. Talk delivered at Openstack Day Israel 2016 : http://www.openstack-israel.org/#!agenda/cjg9

1. How Openstack
is Built
Ant(on) Weiss - Otomato
http://otomato.link
1

2. How Big is OpenStack?
265,850 commits
5,744 developers
54 project teams
585 git repos
2

3. OpenStack CI System
3
• Developed and managed by the OpenStack Infra team

4. Ansible+Puppet - two invisible hands.
■ All configuration of CI infra nodes is managed by Puppet manifests
■ The ‘Puppetmaster’ is actually an Ansible control machine
■ Puppetmaster holds playbook + hiera data
■ Ansible playbooks trigger ‘puppet apply’ on the nodes
4
+

5. Git - the mighty archivist
■ the most widely used distributed
open-source VCS
■ Served by ‘cgit’
■ Changes submitted with the help
of ‘git-review’ tool
5

6. Gerrit – the gatekeeper
■ Originally developed by Shawn
Pearce at Google for Android
development.
■ Manages git repositories
■ Provides code review and
approval rules
■ Collaborates with human and non-
human gatekeepers
6

7. Zuul – the dispatcher
■ Written by James E. Blair
■ Defines and enforces the gating
process
■ Listens to Gerrit events
■ Manages the queue and
dependencies
■ Dispatches the verification
process to Jenkins (through
Gearman)
7

8. Zuul – the beautiful dispatcher
■ Written by James E. Blair
■ Defines and enforces the gating
process
■ Listens to Gerrit events
■ Manages the queue and
dependencies
■ Dispatches the verification
process to Jenkins (through
Gearman)
8

9. Gearman - the outsourcer
■ a framework to farm out work to
other machines or processes
9
■ sends jobs to Jenkins
through jenkins gearman
plugin (by Khai Do & James
E. Blair)
■ manages the job queues

10. Jenkins - the butler :)
■ Continuous Integration Server
■ Executes testing jobs, records executions results
■ Jobs performed on slaves
■ “Our jenkins servers, at peak load, runs 20,000+ jobs
per day. At that load we require many jenkins slaves
(900+)…” (From Jenkins Gearman plugin page)
■ Jobs created and managed by Jenkins Job Builder
■ devstack slaves provisioned by Nodepool
10

11. Jenkins Job Builder
■ Configures Jenkins jobs using YAML files stored in git
■ Defaults, macros and job configs
- job:
name: example-docs
node: node-label
triggers:
- zuul
builders:
- git-prep
- docs
publishers:
- scp:
site: 'scp-server'
files:
- target: 'dir/ectory'
source: 'build/html/foo'
keep-hierarchy: true
- console-log 11

12. Nodepool - the provisioner
■ Manages a pool of Devstack images to use in project testing.
■ Provisions nodes on HP and Rackspace clouds. (using novaclient)
■ Makes sure there are always available nodes for each provider type.
■ Listens to events from Jenkins ZeroMQ publisher plugin. (by C.Boylan)
■ Tears down old instances when tests are completed.
12

13. DevStack Gate
■ The integration test that verifies that all projects still work together after the
changes
■ Provides a full OpenStack installation on a single machine with Devstack
■ Installs Devstack and runs Tempest (the OpenStack integration test suite)
13

14. Logstash
■ All jenkins job logs are indexed
and stored by Logstash
■ Logs pushing also handled by
Gearman jobs
14
■ Elastic Search helps analyze the
test results

15. Wrap Up
■ Git
■ Gerrit
■ Zuul
■ Gearman
■ Jenkins Job Builder
■ Jenkins
■ Devstack
■ Nodepool
■ Logstash
15
Only 9 out of 32 projects/tools
managed by the
Openstack Infra
Team

16. Some References:
■ Scaling the OpenStack Test Environment - http://amo-probos.org/post/15
■ Zuul: a Pipelining Trunk Gating System - http://amo-probos.org/post/14
■ Multiple Jenkins Masters with Jenkins Gearman Plugin - http://
docs.openstack.org/infra/publications/gearman-plugin
■ Understanding the OpenStack CI System - www.joinfu.com/2014/01/
understanding-the-openstack-ci-system/
■ cgit - https://git.zx2c4.com/cgit/about/
16