These slides describe Nodepool, the Openstack Infra tool used to spin up virtual machines for testing, according to the realtime demand for tests. They explain how to setup it, and how to integrate with Gearman and Jenkins, to allow to run tests on real-time and scaling that automatically. Link to project source code: https://github.com/openstack-infra/nodepool Additional OpenStack Infra documentation: http://docs.openstack.org/infra/system-config/nodepool.html

1. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Accelerate and scale your
tests with nodepool
Yolanda Robla <yolanda.robla-mota@hp.com> / Sep, 2015

2. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
What is nodepool?
- Guess what? It's a “node pooler”
- Service used by OpenStack CI to deploy and manage a pool of
instances
- Using OpenStack clouds as target
- Used to execute all tests in OpenStack CI workflow
- Allows to grow or shrink on demand

3. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Nodepool workflow

4. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Nodepool main advantages
- Ability to use multiple OpenStack providers at same time
- Distribute load between several jenkins masters
- Allows to construct different image types depending on
your needs
- Run any job you need, in any language
- Based on open-source protocol gearman
- OpenStack Infra standard for running jobs

5. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
How to configure nodepool
- Based on /etc/nodepool/nodepool.yaml file
gearman-servers:
- host: your_gearman_server_host
zmq-publishers:
- tcp://jenkins1_host:8888
- tcp://jenkins2_host:8888
providers:
- name: hpcloud-region-b
username: 'openstack_provider_name'
password: 'openstack_provider_pass'
project-id: 'openstack_project_Name'
auth-url: 'https://url_of_keystone:35357/v2.0'

6. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
How to configure your base images
- Based on glance images, or generate your own using dib
- Steps for a nodepool node using snapshots:
1. Choose a base image available on your provider
2. Apply them the scripts you need to construct the final image
3. Nodepool takes an snapshot and uses as base image for Vms
4. When launching a VM, apply a runtime script as well
5. This node will be ready and attached to jenkins, to apply jobs
6. Images can be refreshed on demand or scheduled

7. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Configure base images using diskimage-builder
https://github.com/openstack/diskimage-builder
Allows generation of 100% customized images
Can use base images not available in your openstack provider
- Steps for a nodepool node using diskimage-builder:
1. Use a local image from your system
2. Apply them elements to conform your final image
3. Upload image to glance, nodepool uses as base image for Vms
4. When launching a VM, apply a runtime script as well
5. This node will be ready and attached to jenkins, to apply jobs

8. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Image configuration on nodepool and jenkins
- Nodepool uses labels:
labels:
- name: test-precise
image: test-precise
ready-script: ready_node.sh
min-ready: 4
providers:
- name: provider_1
Providers:
- name: provider_1
Images:
- name: test-precise
base-image: 'Ubuntu Precise xxx'
setup: prepare_base_image.sh
- Matching job in jenkins need to specify the label to use

9. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
What will you need
- You will need nodepool:
http://docs.openstack.org/infra/nodepool/installation.html
- Based on gearman protocol, python implementation:
https://pypi.python.org/pypi/gear
- You need to spin up a gear server:
import gear
gear.Server(4730)
- You need a gear client that sends jobs ...

10. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Gear client sample
import gear
gearman = gear.Client()
gearman.addServer('127.0.0.1', 4730)
gearman.waitForServer()
# send demand to nodepool
import json
from uuid import uuid4
gearman_job = gear.Job("build:%s:%s" % ('job_name', 'node_type'),
json.dumps({"OFFLINE_NODE_WHEN_COMPLETE": True}),
unique=str(uuid4().hex))
gear.submitJob(gearman_job)
Based on: https://github.com/yrobla/nodepool_job_launcher

11. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Integration with Jenkins
- Gearman worker plugin: https://wiki.jenkins-ci.org/display/JENKINS/Gearman+Plugin
- Zmq to publish job state: http://git.openstack.org/cgit/openstack-infra/zmq-event-publisher/

12. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Want to know more?
- Nodepool documentation:
http://docs.openstack.org/infra/system-config/nodepool.html
- Openstack-infra: http://docs.openstack.org/infra/
- Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
- IRC meetings: https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting
- Reach me on IRC (yolanda) or email (yolanda.robla-mota@hp.com)

13. © Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Thank you
Yolanda Robla Mota
<yolanda.robla-mota@hp.com>
Openstack CI resources
Docs: http://ci.openstack.org
Freenode: #openstack-infra