This document discusses transforming Ceph tests to use OpenStack. It describes running unit tests locally but running integration tests on OpenStack instances. Developers can now run integration tests on their own OpenStack tenant without waiting for resources. Specifying resources for the OpenStack machines makes the tests more self-service. Future improvements include better multi-cloud support and making archival and setup more convenient.

1. Transforming the Ceph tests
with OpenStack
Loic Dachary
Artisan Logiciel Libre
2015/10/27

2. Ceph developers population
Elite:
Smart people : Sage, Sam, Josh...
Large companies : Red Hat, SUSE, Intel...
Normal:
You and me
Non-profit : Tetaneutral, FSF...

3. Normal developer loop 2014
Setup Ceph $1
Hit a bug and report it $10
Fix the bug $1000
Rebuild the packages $10
Run tests N/A
Upgrade the package N/A

4. Normal developer loop 2015
Setup Ceph $1
Hit a bug and report it $10
Fix the bug $1000
Rebuild the packages $10
Run tests $100
Upgrade the package $1

5. Ceph development lifecycle
commit => test => review => merge
=> test => release

6. Tests
commit => test => review => merge
=> test => release
test == unit tests => packages =>
integration tests

7. Unit tests
Unit tests and functional tests : not too
long, single machine, no network, not
root
Make check local
Make check bot
Stable and under surveillance

8. Debugging unit tests
make check bot fails => run locally =>
fix => push --force

9. Integration tests
Requires more than one machine or
Takes more than a few minutes
Each job taks ~2 machines and takes ~2
hours
Before a release ~2,000 jobs are run and
failures analyzed

10. Running integration tests 2014
Push a branch on the official repository
Wait for gitbuilder to create a package
teuthology-suite to add the test in the sepia
lab

11. Integration tests 2015
teuthology-openstack with an OpenStack
tenant

12. Debugging integration tests
2015
teuthology fails => debug from log or
pause the test => fix => push --force

13. Commit review 2014
Developer: Pull request
Automatic: Unit test pass
Reviewer: Review the commit
Reviewer: Add needs-qa label
Reviewer: Merge and push in a testing
branch
Reviewer: Wait for gitbuilders
Reviewer: Run teuthology-suite
Reviewer: Sort the teuthology-suite results

14. Commit review 2015
Developer: Pull request
Automatic: Unit test pass
Reviewer: Review the commit
Developer: Run teuthology-openstack
Developer: Sort the teuthology-suite results
Developer: Fix and re-push

15. OpenStack prerequisites
Flavors with 16 cpus and 32GB RAM
Floating IPs (optional)
Volumes (optional)
Security groups (optional)
Tested with Havana, Juno & Kilo clusters

16. Adding OpenStack workers
add openstack: to ~/.teuthology.yaml
source openrc.sh
teuthology-worker –tube openstack
teuthology-suite –machine-type openstack

17. self-service teuthology-
openstack
use a tenant with no pets
source openrc.sh
create a passwordless keypair
~/myself.pem
teuthology-openstack –suite rados
watch http://120.43.25.39:8081/
teuthology-openstack --teardown

18. --key-name –key-filename
openstack keypair create myself
teuthology-openstack --key-name myself
--key-filename ~/myself.pem

19. --throttle --simultaneous-jobs
teuthology-openstack --throttle 30
--simultaneous-jobs 200

20. --name
teuthology-openstack –name other

21. --archive-upload --upload
teuthology-openstack –archive-upload ubuntu@teuthology-
logs.public.ceph.com:./ --upload

22. Resources description
openstack:
- machine:
disk: 40 # GB
ram: 8000 # MB
openstack:
- machine:
ram: 32000 # MB

23. Resources description
openstack:
- machine:
disk: 40 # GB
ram: 32000 # MB

24. Resources description defaults
openstack:
machine:
disk: 20 # GB
ram: 8000 # MB
cpus: 1
volumes:
count: 0
size: 1 # GB

25. Caveats
mulitple OpenStack provider support
--verbose to see feedback
archival with –upload would be more
convenient as a standalone action
setting up teuthology should be done when
compiling Ceph locally

26. Commit review 20XX
Developer: Pull request
Automatic: Unit test and integration tests
pass, display coverage of the commit
modifications and cross reference the
results in the corresponding issue
Reviewer: Review the commit

27. THANK YOU
plus.google.com/+RedHat
linkedin.com/company/red-hat
youtube.com/user/RedHatVideos
facebook.com/redhatinc
twitter.com/RedHatNews