This talk was presented at FOSS ASIA 2018 summit at Singapore on 25th Mar, 2018. https://2018.fossasia.org/event/schedule.html#

1. Delivering a bleeding edge
community-led OpenStack distribution:
RDO
Chandan Kumar at FOSSASIA 2018

2. About me:
● RDO/OpenStack Contributor
● Software Engineer at Red Hat

3. About RDO
● The rpm based distribution of upstream OpenStack
● But also a group of Ridiculously Dedicated OpenStackers..
● Who are here to help you Rapidly Deploy OpenStack
● 350 + packages maintained, and growing

4. What’s in a package?
● <Name>-<Version>-<Release>.rpm
● Source + spec file = package
● Source -> upstream code
● List of included files
● What is in a spec file:
○ Building Steps
○ Installation/Removal steps
○ Dependencies
○ Patches

5. Packager == Spec Developer
● Version control (dist-git), on a public platform, community oriented
● Peer review
● Automated testing & validation of contributions (ie continuous integration), on a controlled testing
environment
● Smart merge management
● Responsible with system integration like: systemd unit or specific path etc.
● Follow upstream changes in order to validate spec files against them
● Release as closely as possible to upstream (esp. vulnerability or security fixes)

6. Packaging OpenStack

7. Current State till Queens release.
● 700+ projects
● 1600 + contributors
● 250 commits per day
● Releases every 6 months

8. Extra constraints for RDO Packagers
● Spec file validation on each commit merged
● Strong dependencies management between packages
● Test system should properly test and validate the package.
● Package upgrade should works smoothly.
● Making sure Package works on Fedora, Centos/RHEL

9. RDO releases status
Release Upstream OpenStack RDO releases Duration (in days)
Newton 06th Oct, 2016 07th Oct, 2017 1
Ocata 22nd Feb, 2017 22nd Feb, 2017 In 12 hours
Pike 30th Aug, 2017 01st Sept, 2017 In 2 days *
Queens 01st Mar, 2018 01st Mar, 2018 In 7 hours
* Production push ready on 28th Aug, 2017, Infra issues leads few days delay.

10. RDO got released on the same
day as upstream released? But
how it got possible?

11. How we made it possible?
● Building RPM Factory from Software Factory, RDO and community tools
● Started chasing the trunk
● Testing each commit as it merged upstream through puppet and Tripleo CI
● More scenario jobs leads to more robust openstack
● Catching issues early once a patch landed upstream through RDO third party CI.
● Improving RDO Production chain workflow

12. Community Tools: Koji and Centos Cloud Sig
● Koji is a software collection to build and store RPM packages in an isolated environment.
● Centos Build System: http://cbs.centos.org/koji/
● Centos Cloud Sig providing Centos Infra and CI to ship packages for stable releases.
● CICO <ci.centos.org> cloud for providing additional CI capacity.
● Dogfooding our own openstack through RDO Cloud

13. RDO Tools: DLRN
● Continuous Delivery Platform that takes every upstream commit and build it as a consumable package
for CI purpose
New commit current Consistent Current Passed CI
RDO CI
Upstream CI
(Tripleo, Kolla,
Puppet)every merged commit
or almost is built.
set of packages without
build failures
snapshot that passed CI

14. RDO Tools: rdopkg, rdoinfo, config and graffiti
● RPM packaging automation tool providing more power to packagers
● Automates most of the packaging tasks like requirements sync or flattening patch chains for
inclusion in specs
● rdoinfo is a git repo containing all the information about openstack packages and its dependencies
including maintainers, version, dist-git, upstream and a lot more.
● Graffiti is a utility tool in order to interact with koji to retrieve releases and tags
● Config: Treating each config changes for jobs and infra as code using git

15. RDO originated tools: WeiRDO
● Ansible-based framework using gate jobs and make them usable outside upstream gates
● Goal is to improve coverage upon what upstream provides.
provision
test
node
Setup
trunk
repositor
ies
Install
project
test
dependen
cies
execute
run_test.
sh
Deploy/c
onfigure
opensta
ck
Run
tempest
tests
recover
logs and
results
destroy
node

16. RDO originated tools: ARA
● Non-intrusive plugin to Ansible.
● Tool to records Ansible Playbook runs and provides intuitive interfaces to browse them.
● Used heavily in upstream CI and openstack-ansible project
● Look for ‘ara’ folder in CI logs
● Making it easier to end-user to trace errors
● Try out through $ pip install ara

17. Software Factory
● SF is a CI/CD platform based on OpenStack's own CI ("Infra")
● Code hosting and review (Gerrit)
● Jobs orchestration (Zuul & Jenkins)
● Project dependencies management when building testing environments (Zuul)
● Jobs executors spawned & terminated on demand on an OpenStack cloud (Nodepool)
● Smart commit gating (Zuul)
● config: everything as code
● Flexible workflow, based on reviewing and testing code prior to merging it
● Etherpad, pastebin, RepoXplorer and much more.
● Visualizing all the stuff through Kibana dashboard using opstools-ansible

18. Combining all to form RPM Factory
● rpmfactory : https://review.rdoproject.org
● The community platform for building RDO
● A Software Factory deployment branded for RDO
● Hosts all the distgit repositories + extra patches repositories
● All upstream changes acted upon with DLRN
● All distgit changes reviewed via Gerrit
● All distgit changes tested via Zuul + Jenkins + Nodepool + CBS
● New builds in CBS are automatically proposed when new releases are tagged upstream.
● Updating new versions of package for stable releases using bots
● Promoting new dependencies periodically in CBS through rdoinfo

19. RDO workflow overview

20. Resulted in
● Master/ stable-branches/promoted yum repositories served https://trunk.rdoproject.org
● Consuming each bits by tripleo, kolla, packstack and puppet CI which run against different
project’s submitted patches to find issues early
● Making it easy for installers to test against latest upstream code so that they can get released
shortly after the services.
● Testing different stacks in different fashion through periodic and scenario jobs.
● Testing upstream gates jobs out of the gate using Weirdo.
● Leads to delivering more robust RDO cloud on time.

21. RDO stats for Ocata
● ~919 commits
● ~86 contributors
● RDO Ocata packages were available less than 12h after upstream release
● ~230 build failures caught by DLRN

22. RDO stats and learnings from Pike
● 1000+ commits
● 172 contributors
● Released in 2 days due to infra issues providing lessons to improve our infra
● Learnings:
○ Enable more automation
○ Better planning to cover PTO
○ Pipeline release speed matter

23. RDO Stats/Upcoming challenges for Queens
● 22 new contributors
● We are back on Track by releasing RDO same day.
● Auto Version updates for packages and dependencies
● Testing each spec changes on Fedora and Centos Builder
● Introduced Test cloud to test RDO Queens Milestone releases
● Upcoming challenges:
○ Moving to Zuul v3
○ Python3 support

24. RDO Success Stories
● Cumulus Networks
● CERN
● Thoughtworks
● RDO Cloud on which develop OpenStack as well as running complete RDO infrastructure
● CentOS
● VEXXHOST

25. How to become a part of RDO
community?

26. Contributing to RDO also leads to
contributing to OpenStack.

27. How to contribute:
● Checkout https://www.rdoproject.org/easyfix/
● Participate in #rdo meeting every wednesday at 08:00 P.M. (IST) on #rdo IRC channel
● Help on migrating stuff to zuulv3 and Python3
● Improve RDO by participating in test days.
● Help us in improving RDO project documentation
● Look forward to improve Tripleo, Puppet-openstack, Kolla and OpenStack itself by taking a baby
step
● Join #RDO{user, dev} mailing list

28. Thanks
Questions?
Tweet at @chkumar246

29. References:
● https://www.rdoproject.org/
● https://softwarefactory-project.io/sf/welcome.html
● http://ara.readthedocs.io/en/latest/
● https://review.rdoproject.org/sf/welcome.html
● http://cistatus.tripleo.org/
● https://docs.openstack.org/infra/
● https://www.openstack.org/videos/boston-2017/how-rdo-uses-software-factory-and-ansible
● https://www.rdoproject.org/user-stories/
● https://blogs.rdoproject.org/
● http://openstack.redhat.com/contribute/mailing-lists/
● http://weirdo.readthedocs.io/en/latest/