RDO is a community-led distribution of OpenStack that aims to rapidly deploy OpenStack by maintaining over 300 RPM packages. It utilizes several tools like DLRN, rdopkg, and ARA to continuously test packages against upstream changes to catch issues early. RDO has adopted a "software factory" model using tools like Gerrit, Zuul, Jenkins and Nodepool for code review, testing and continuous integration. These efforts have helped RDO release packages within hours or days of upstream, improving the user experience for deploying OpenStack.

1. Delivering a bleeding
edge community-led
OpenStack distribution:
RDO
Chandan Kumar

2. About me:
● RDO/OpenStack Contributor
● Currently serving at Red Hat

3. About RDO
● The rpm distribution of upstream OpenStack
● But also a group of Ridiculously Dedicated OpenStackers..
● Who are here to help you Rapidly Deploy OpenStack
● 300 + packages maintained, and growing

4. What’s in a package?
● <Name>-<Version>-<Release>.rpm
● Source + spec file = package
● Source -> upstream code
● List of included files

5. Spec file
● Building Steps
● Installation/Removal steps
● Dependencies
● Patches

6. Packager == Spec Developer
● Version control (distgit), on a public platform, community oriented
● Peer review
● Automated testing & validation of contributions (ie continuous integration), on a
controlled testing environment
● Smart merge management
● Responsible with system integration like: systemd unit or specific path etc.
● Follow upstream changes in order to validate spec files against them
● Release as closely as possible to upstream (esp. vulnerability or security fixes)

7. Packaging OpenStack

8. Current State till Pike release.
● 600+ projects
● 1800 + contributors
● 250 commits per day
● Releases every 6 months

9. Extra constraints for RDO Packagers
● Spec file validation on each commit merged
● Strong dependencies management between packages
● Test system should properly test and validate the package.

10. RDO release status till Mitaka
● Using review.gerrithub.io for peer-review
● Using github as a dist-git
● Packstack to validate the Changes
● Final release manually

11. RDO release status till Mitaka
Release Upstream OpenStack RDO Release Duration (in days)
Icehouse 17th Apr, 2014 29th Apr, 2014 13
Juno 16th Oct, 2014 27th Oct, 2014 12
Kilo 30th Apr, 2015 21st May, 2015 22
Liberty 15th Oct, 2015 22nd Oct, 2015 8
Mitaka 07th Apr, 2016 12th Apr, 2016 6

12. RDO releases status after Mitaka
Release Upstream OpenStack RDO releases Duration (in days)
Newton 06th Oct, 2016 07th Oct, 2017 1
Ocata 22nd Feb, 2017 22nd Feb, 2017 In 12 hours
Pike 30th Aug, 2017 01st Sept, 2017 In 2 days *
* Production push ready on 28th Aug, 2017, Infra issues leads few days delay.

13. RDO got released on the same day as
upstream released? But how it got
possible?

14. How we made it possible?
● Building RPM Factory from software factory, RDO and community tools
● Started chasing the trunk
● Testing each commit as it merged upstream through puppet and Tripleo CI
● More scenario jobs leads to more robust openstack
● Catching issues early once a patch landed upstream through RDO third party CI.
● Improving RDO Production chain workflow

15. Community Tools: Koji and Centos Cloud Sig
● Koji is a software collection to build and store RPM packages in an isolated environment.
● Centos Build System: http://cbs.centos.org/koji/
● Centos Cloud Sig providing Centos Infra and CI to ship packages for stable releases.
● RDO Cloud and CICO <ci.centos.org> cloud for providing additional CI capacity.

16. RDO Tools: DLRN
● DLRN is a tool to continuously test the packaging recipes against OpenStack's upstream
master to detect packaging issues early
● It can generate a full test package repository to test interdependencies, and generally
acts as the "master RPM" repository
● Would you like to know more?
https://blogs.rdoproject.org/7834/delorean-openstack-packages-from-the-future

17. RDO Tools: rdopkg, rdoinfo and graffiti
● RPM packaging automation tool providing more power to packagers
● Automates most of the packaging tasks like requirements sync or flattening patch chains
for inclusion in specs
● rdoinfo is a git repo containing all the information about openstack packages and its
dependencies including maintainers, version, dist-git, upstream and a lot more.
● Graffiti is a utility tool in order to interact with koji to retrieve releases and tags

18. RDO originated tools: WeiRDO
● An idea that is about leveraging the effort the community puts towards their own CI
initiative to improve the coverage that upstream RDO provides.
● This is about testing as many ways of installing, configuring and integrating OpenStack as
possible without re-inventing the wheel.
● WeIRDO takes the gate jobs outside of the gate to test what RDO ships before it’s
packages even make it to the upstream OpenStack CI.
● Resulted in making RDO better.

19. RDO originated tools: ARA
● Non-intrusive plugin to Ansible.
● Tool to records Ansible Playbook runs and provides intuitive interfaces to browse them.
● Used heavily in upstream CI and openstack-ansible project
● Look for ‘ara’ folder in CI logs
● Making it easier to end-user to trace errors
● Try out through $ pip install ara

20. Software Factory
● SF is a CI/CD platform based on OpenStack's own CI ("Infra")
● Code hosting and review (Gerrit)
● Jobs orchestration (Zuul & Jenkins)
● Project dependencies management when building testing environments (Zuul)
● Jobs executors spawned & terminated on demand on an OpenStack cloud (Nodepool)
● Smart commit gating (Zuul)
● config: everything as code
● Flexible workflow, based on reviewing and testing code prior to merging it
● Etherpad, pastebin, RepoXplorer and much more.

21. Combining all to form RPM Factory
● rpmfactory : https://review.rdoproject.org
● The community platform for building RDO
● A Software Factory deployment branded for RDO
● Hosts all the distgit repositories + extra patches repositories
● All upstream changes acted upon with DLRN
● All distgit changes reviewed via Gerrit
● All distgit changes tested via Zuul + Jenkins + Nodepool + CBS
● New builds in CBS are automatically proposed when new releases are tagged upstream.

22. RDO workflow overview

23. Resulted in
● Master/ stable-branches yum repositories served https://trunk.rdoproject.org
● Consuming each bits by tripleo, kolla, packstack and puppet CI which run against different
project’s submitted patches to find issues early
● Making it easy for installers to test against latest upstream code so that they can get
released shortly after the services.
● Testing different stacks in different fashion through periodic and scenario jobs.
● Testing upstream gates jobs out of the gate using Weirdo.
● Leads to delivering more robust RDO cloud on time.

24. RDO stats for Ocata
● ~919 commits
● ~86 contributors
● RDO Ocata packages were available less than 12h after upstream release
● ~230 build failures caught by DLRN

25. RDO stats for Pike
● 1000+ commits
● 172 contributors
● Released in 2 days due to infra issues providing lessons to improve our infra

26. RDO Success Stories
● Cumulus Networks
● CERN
● Thoughtworks
● RDO Cloud on which develop OpenStack as well as running complete RDO infrastructure
● CentOS
● VEXXHOST

27. How to become a part of
RDO community?

28. Contributing to RDO also
leads to contributing to
OpenStack.

29. How to contribute:
● Checkout https://www.rdoproject.org/easyfix/
● Participate in #rdo meeting every wednesday at 08:00 P.M. (IST) on #rdo IRC channel
● Learn about new stuff during RDO office hour each Tuesday at 07:00 P.M. (IST)
● Improve RDO by participating in test days.
● Look forward to improve Tripleo, Kolla and OpenStack itself by taking a baby step
● Join #RDO mailing list

30. Thanks
Questions?
Tweet at @chkumar246

31. References:
● https://www.rdoproject.org/
● https://softwarefactory-project.io/sf/welcome.html
● http://ara.readthedocs.io/en/latest/
● https://review.rdoproject.org/sf/welcome.html
● http://cistatus.tripleo.org/
● https://docs.openstack.org/infra/
● https://www.openstack.org/videos/boston-2017/how-rdo-uses-software-factory-and-ansible
● https://www.rdoproject.org/user-stories/