Recommended
Recommended
More Related Content
Similar to Herding cats into boxes
Similar to Herding cats into boxes (15)
More from doughellmann
More from doughellmann (13)
Recently uploaded
Recently uploaded (20)
Herding cats into boxes
- 1. Herding Cats Into Boxes How OpenStack release management changes with Big Tent Doug Hellmann (@doughellmann) Thierry Carrez (@tcarrez)
- 3. Two approaches Pre-versioning 1.0alpha1, 1.0beta2, 1.0rc1… 1.0 Post-versioning 1.0, 1.0.1, 1.1.0… 2.0
- 4. At the beginning 6-month time-based releases YEAR.SEQUENCE (2014.1, 2014.2… 2015.1) Swift: feature-based, frequent releases X.Y.Z (1.1, 1.2, 1.2.1… 2.0) Post-versioning for stable release updates 2015.1.1 or 2.0.1
- 5. Client libraries Always backward-compatible Single release channel Semantic versioning
- 6. Oslo Incubator copy-and-paste Pre-versioning with alphas Post-versioning without alphas
- 9. Standardized library releases Release As Needed Semantic Versioning Release Day Guidelines Reviewable Release Requests
- 11. A push for intermediary releases Reduce tight coupling More flexibility
- 12. Death of common versioning Intermediary requires own versioning Less value for YEAR.SEQUENCE model Continued confusion if we keep both Switch once or switch later
- 14. What about stable branches ? Used to do synchronized point releases ...but what does that mean in the big tent ...but you can update just a piece
- 15. Stable branch point releases Tried getting rid of them completely ...but people still wanted reference points Tried tagging all commits ...but people feared the pollution of tag space
- 17. Version numbers nova 12.0.0 keystone 8.0.0 swift 2.5.0 neutron 7.0.0 heat 5.0.0 zaqar 1.0.0 ironic 4.2.0 ...
- 19. Release models release:cycle-with-milestones pre-versioned, one time-based release release:cycle-with-intermediary post-versioned, release as-needed
- 20. Other release models release:independent outside of release cycle and stable branches release:none no “release”
- 21. Liberty stable point releases Tag as-needed, or when OSSA Communicate through releases repo Encouraging regular releases Release notes
- 22. Reno in-tree release notes Compile notes from small files Scan branch history for inputs docs.openstack.org/developer/$PROJECT
- 23. Launchpad Good for planning Historical tracking features complex
- 24. Automation Run tagging script in CI More tools for liaisons
Editor's Notes
- Additional notes on https://etherpad.openstack.org/p/PMokx7f3dS Thank you for coming this morning. Our talk is titled “Herding Cats into Boxes” and we’re going to talk about changes to release management policies and processes as part of our community’s growth. My name is Doug Hellmann, and I’m the release management team project lead for Mitaka. I’ll talk a bit about some of our future plans, but first Thierry is going to introduce some of the history of the evolution of our release management. [TC] Rough outline: history, what changed in liberty, where we are today, where we’re going tomorrow
- [TTX] Release management is a bit like putting a cat into a box. All software needs to be released. All cats love boxes. But if you try to put a cat into a box yourself, that’s not easy. So try putting several cats into several boxes at the same time Releasing in open source is all about putting a stamp with a version number on the current state of the source code
- [TTX] There are generally two approaches when it comes to versioning In pre-versioning you decide in advance which version number you’re going to publish next. You go toward that target version and publish alphas, betas, and RCs along the way. This works well with time-based releases In post-versioning you decide which version number to use by the time you release, not before. That lets you pick meaningful version numbers, since at that point you know exactly how much changed since the last version. But that makes it a bit difficult to use alphas or betas along the way, so everything you tag is a full release. Or you have to use tricks to signal that some versions are actually not appropriate (like the odd/even scheme, or using 0.x versions until you’re mature) So what did we pick for OpenStack originally ?
- [TTX] For OpenStack very early on we settled on a time-based release model and a 6 month release schedule. Version numbers would be tied to dates, and we’d use pre-versioning to issue alphas and betas and RCs along the way. All projects would release a synchronized version at the end of the cycle. That worked very well for us, especially at the beginning where QA coverage wasn’t that great and we could really use external testing of beta artifacts Swift was always the cat that would not fit in that box. They wanted to release more often and follow a more feature-based release model. So they used a separate versioning system. We still wanted them to use pre-versioning with that scheme, but that created tension since they could not really predict the version number they would be releasing. Note that Swift would still issue a release every 6 months roughly at the time of the synchronized release, and maintain a stable branch for security and critical bugfix backports. So everyone would use one release channel per cycle, as we issue stable point versions to release those bugfixes. Those stable branch updates would all use post-versioning, like 2015.1.1 or 2.0.1 That was all relatively simple, as long as everything was bundled with the main service, including client libraries
- [TTX] But quickly we ended up wanting to release client libraries separately from the services. The idea was that they should not be tied to a specific version of the service: you could install the latest version and it would be always backwards-compatible with previous versions of the service So those could use a single release channel and release frequently outside of the constraints of the release cycle So they needed a different numbering system. For them we used semantic versioning (X.Y.Z) and a postversioning scheme. [DH]
- [DH] Our use of post-versioning grew out of our experience with the Oslo program. The Oslo common libraries started out as code extracted from the other projects into the incubator, where it was modified and then copied back into consuming applications. As that code matured we wanted to move to shared libraries. Early libraries like oslo.config and oslo.messaging used alpha version numbers in the middle of the release cycle, shifting to regular versions with stable branches at the end of the cycle. During the Icehouse to Kilo time frame Oslo added many more libraries. As part of that growth we realized that tagging and re-releasing that many libraries at the end of the cycle to drop the alpha versions would be very disruptive because the requirements in the server projects would have to be updated before their releases. We also had plenty of feedback from application developers that they did not want alpha version libraries. So we eventually stopped releasing alphas, and established a release model where we release full versions of the libraries and use the last release for each cycle to create a stable branch. And...
- [DH] ...that worked pretty well for Oslo. However, there were issues with releases of some of the libraries managed by other project teams, like clients and newer libraries from service teams.
- [DH] This was caused in part by further growth in the number of libraries, but also the number of teams managing those libraries. All of our service projects are expected to manage a client for communicating with the service, but since those teams were more focused on the services they produced, they were unfamiliar with the processes we established for managing releases in Oslo and sometimes that caused issues like poorly timed releases or poor version numbers. We decided to recentralize release management to address these issues. In order to do that, we needed to streamline the way we manage releases. We started by taking the release practices developed by the Oslo team...
- [DH] ...and applied them to the other libraries, including the clients. All libraries release at any point in the cycle (sometimes as often as weekly) and produce a single stable branch at the end of the cycle for use with that version of the services. All libraries now use Semantic Versioning numbering rules to help signal to consumers of the library the scope of changes in each release. We avoid releases late in the week, that might interfere with builds right before the weekend. And all of the standards are applied through a review process...
- [DH] ...because releases are requested with a patch to the openstack/releases git repository. Merging one of those patches triggers a release, at this point done by hand using some scripts managed by the release team. The library release process includes tagging the release, recording the release in launchpad with a milestone containing the resolved bugs and closed blueprints, and sending a release announcement. As application developers watched the the process improve...
- [DH] ...we started seeing more pressure for a similar model on the service side as well When we were a small community, with a small number of tightly coupled projects in the integrated release, it made sense to release them in lockstep. Now that we have so many more projects, and we have reduced that coupling, we want to let project teams work at their own pace, while still providing the stable branch mechanism for consumers who do not want to run from master. That means having our more mature projects move from the milestone system to a release model similar to what the Swift team has been using. That change requires some other changes, including to version numbers, right? [TTX]
- [TTX] Doing intermediary releases means releasing at your own rhythm, so projects using such a model can’t share versioning with others At that point we were left with a dilemma with versioning. We still had most projects under the common YEAR.SEQUENCE numbering, but expect a growing number of projects using specific numbers That created a lot of confusion, with users questioning if projects versioned independently were still part of “the release” Was the version defining the release ? We had the option of keeping the common number around until a project needed a specific versioning Or we could just switch everything and give every project its own version. During this cycle we were abandoning the concept of “the integrated release” anyway, making a common version number even less interesting
- [TTX] So we switched them all. We picked different numbers for the versions, based on the number of integrated releases the project had in the past. So Nova in Liberty would be 12.0.0, Heat would be 5.0.0 and Zaqar 1.0.0.
- [TTX] For stable branches, we had another kind of dilemma. When the “integrated release” was around we would regularly issue synchronized point release for all the projects in it. But that was a lot of effort and not a lot of people stepped up to help. With more projects getting stable branches under the big tent, we started questioning the value of synchronized point releases What would we end up synchronizing the release of, if we don’t have an integrated release anymore ? Each component has a different version now, so what does bumping them at the same time give us ? Doing that actually communicates the wrong message: that you need to update all the pieces together. But it should be possible to only upgrade one piece and have it still work just fine. So we considered getting rid of synchronized point releases completely
- [TTX] but when we proposed that, a number of users and packagers stepped up to save them. Their main argument was that there was a lot of different distributions of OpenStack and having common reference points to compare them was very useful. So we proposed another alternative, which was to release every single commit to a stable branch. After all, the branches are supposed to be always usable and the backports are supposed to be very safe. but then other people complained about the potential pollution to the tag space in git repositories, making that list unusable as we push hundreds of tags to stable branches So we could not come to an agreement on that, and settled for an intermediary solution, which Doug will expose in a few
- [TTX] OK, so what is the current situation today
- [TTX] I mentioned earlier that we denormalized the version numbers of the various components. That means the Liberty release looks like this It can be a bit confusing and difficult to keep track of what version maps to what release cycle. It gets even more difficult as we push stable point releases out. So I wanted us to create a tool to map versions to release series for every component. But by that time we already had the releases repository set up to review release requests, and that contained the version numbers for each release series all in one place So Doug applied a lot of sphinx magic over it and voilà!
- [TTX] This site shows a complete list of the version numbers for each deliverable in each series and a link to download the source for that version. We have imported the old version information as well, so anyone can look back at the history of the project. There is also a machine readable set of version information, since the requests are based on YAML files.
- [TTX] During the last 6 months we also worked on standardizing and documenting the various release models that are available for an OpenStack project cycle-with-milestones is the traditional one-release-per-cycle model. It’s good for young projects or complex projects that don’t have a good handle on QA, since you produce multiple development milestones and release candidates that encourage people to test your deliverables pre-release. It enforces a number of good practices around release management so that you don’t have to worry about it that much. cycle-with-intermediary is a generalization of the swift model: you release from time to time in the middle of the cycle, and you strive to do one release near the end of the cycle, to be included in the stable branches every 6 months. It’s good for more agile projects that want to get the features out to users as often as possible. The trade-off is that you directly produce releases: you need to be pretty confident about your automated test coverage and have internalized the QA process enough to be reasonably sure your release will be usable. It is therefore tailored to more mature (or very simple) projects. It’s important to note that projects following this model still follow the 6-month development cycle and still produce a release toward the end of the cycle. That is what we cut the end-of-cycle stable branch from, whatever the model followed.
- [TTX] There are other release models for specific deliverables the independent release model is for deliverables that won’t produce a stable branch at the end of the cycle and want to be completely free of the OpenStack development cycle. We as a community produce a lot of things that are not classic OpenStack services, and that model works well for that. Finally not all repositories need to be released, and we describe that using the “none” release model. Specs repositories for example are never released and therefore use this model. (We haven’t mentioned the last of the release tags, release:managed, which describes which projects are managed directly by the release management team) What about the stable liberty branch. I said earlier we would not do synchronized point releases anymore, what’s the plan there? [DH]
- [DH] The cost of coordinated stable releases has grown with the number of projects. We looked into dropping stable point releases entirely, but users still want to have official release so We compromised on continuing point releases but we will stop synchronizing the releases for Liberty, and plan to release each project as needed. For example, when a security fix is published, the project team would issue a point release to include that. The release will be triggered by project team members, through a proposed change to the releases repository. It’s still the release management team’s role to educate and encourage project teams to do regular releases We used to have a designated stable release manager, and one of the primary things they ended up doing was compiling release notes for the stable point release. Since we no longer want to have specific stable release managers, we also want to decentralize release notes production. To accomplish that...
- [DH] The release team has developed a new tool called “reno” for managing the notes as a collection of small files which are then compiled into a larger document. Using small files reduces merge conflicts when we backport changes from master to a stable branch, allowing us to cleanly copy the notes along with the code changes, which in turn means the notes can be written much earlier in the process, when the code is submitted. Reno integrates with Sphinx, the documentation build tool we use, so the release notes can be part of the project documentation on docs.openstack.org. Tools like reno and the release series version mapping site...
- [DH] ...will let us make some other process changes during the Mitaka cycle. The first of these is to scale back our use of Launchpad for change management. Launchpad has many good tracking features, but it can be complicated and using them correctly requires a deeper understanding of the tool than we want our average contributor to have to worry about. We frequently run into permission issues, unexpected name collisions, and other issues that require the release team to help sort out. This effort makes up a disproportionate amount of the work for the release management team. With the new tools, we can move our change management tracking to simpler processes where the changes can be reviewed before landing, instead of having to be cleaned up afterwards. We will also continue with the automation work started in liberty...
- [DH] ...especially by defining a job to run the tagging script when a new release request merges. We have a working session later today to identify the missing pieces of that work so we can finish them. When this is complete, we will be able to accept more projects under the release:managed tag, and include those projects in the release history site. In addition to the automation, we will also be spreading the load for release management by building tools release liaisons can use on their own Another way we will empower teams to handle their own releases is...
- [DH] ...by reducing our emphasis on strictly synchronizing milestones during development on the master branch. There are two reasons for this change. In the past we had a small number of tightly integrated projects, but now we have many more loosely coupled projects. Some of those projects, like Oslo, that work ahead of the schedule to enable other projects to use their work, and some projects like Horizon work a bit behind the schedule because they rely on other projects finishing work before they can consume it. Both of those examples are already on slightly different milestone schedules, and we’re generalizing this for our newer projects. We will still use a milestone-based schedule, but we will treat the milestones as reminder points rather than strict deadlines. We will rely on the project teams to manage their own releases, based on our guidance and support through tools. Those are our plans for Mitaka, ...
- thank you again for coming this morning. Before we take questions, we wanted to say that our talk today is dedicated to our cats, Malcolm, Duncan, and Patouille. And now we’d be happy to answer your questions.