Bring Your Project From a 10 Years to a 3 Months Release Cycle


Published on

Stefan Schmidt from the SRUK Open Source Group office talks about what it takes to bring a project like Enlightenment from zero release process to a clean 3 month release cycle.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Bring Your Project From a 10 Years to a 3 Months Release Cycle

  1. 1. Bring Your Project From a 10 Years to a 3 Months Release Cycle Stefan Schmidt
  2. 2. Agenda Introduction Release History Technological Side Social Influences Next Steps Summary
  3. 3. Introduction Enlightenment is not a web project Instead it is written in plain C Enlightenment is not developed by a StartUp Instead it is developed by the FOSS community and companies such as Samsung Enlightenment is not developed by a 5 person team in one office Instead it is developed by an international team distributed over many timezones All this might result in different understandings of agile development :-)
  4. 4. Release History
  5. 5. Release History The Enlightenment project started in 1997 by Carsten Haitzler Some early releases until E 0.16 A 12 years gap from first code to release E 0.17 Building a whole new set of libraries for it but no releases for them either for a long time More release driven now but still feature based and not time based schedule
  6. 6. Reasoning Most of the active developers at that time did not see a need for releases It was a project driven by volunteers which used the code straight out of CVS/SVN Some argued for it as the fastest agile development cycle ;-) Nobody stood up for taking care of the releases so it did not get done
  7. 7. What Changed? Different people joined the project which had more interested in releases Most Linux distributions wanted us to do releases before including our code Commercial parties also asked for releases With the pressure also the understanding grow that releases would help the project forward
  8. 8. How did we do it? That is what the rest of the talk is about From the technological side we improved our tools and infrastructure On the social side we steadily worked on giving releases a more prominent role
  9. 9. Technological Side
  10. 10. Tools and Automation One of the biggest roadblocks is to build trust in the code base you are going to release In the Enlightenment project with the two main libraries alone having around 850.000 lines of code You need a variation of tools that will help you to build up this trust SCM, automated builds and maybe continuous integration, static analyzers just to name a few
  11. 11. SCM and Understandable Commits As an very old project CVS was used in the beginning Switch to SVN for a newer tool but same old practices Large amounts of work committed as one change Many unrelated changes at once Switch to git After some struggling it helped to allow people working on different things without dumping everything into master Improved overall commits, code as well as commit messages People start to develop larger changes in branches
  12. 12. Automated Builds Setup automated builds with Jenkins Quick builds for every change pushed pushed to master Building for x86, x86_64, x32 with gcc as well as clang and also cross-compilation for windows with mingw Very useful as many developer only use one system for their work Many of the discovered problems are getting fixed instantly (see social pressure) In the dark times before we had this setup builds have been broken for some configurations for months During the nightly builds we run time intensive things like static analyzers and other testing
  13. 13. Static Analyzer Static analyzer can be very helpful for a plain C project Started with scan-build, part of clang, from the LLVM project Output quite noise and sadly a lot false positives Also experimented with klocwork but the proprietary nature made it hard to share results with the community Now also using Coverity which is also proprietary but offers a free service for FOSS projects Especially Coverity uncovered quite some issues which have been overlooked by code review You still have to deal with false positives but the signal to noise ratio is way better
  14. 14. Test Suite We have unit testing for a lot of components But with 358 tests we only cover 29.4% of our lines of code We are an UI toolkit so work is also ongoing on pixel perfect regression testing We are still weak on the automated testing side and need to improve One of the bigger worries before every release
  15. 15. Build Complexity By history the Enlightenment project was happy to add configuration options Over the last two years we tried to reduce these options to something maintainable Merged libs allow easier maintenance. 11 different libs have been merged into EFL All of these actions had direct influence on how much different configurations we can test The reduced set of libraries also eased the release process
  16. 16. Social Influences
  17. 17. General Social Changes Once we had IRC and email notification about broken builds social pressure grow to fix them Keep people in the loop and highlight positive changes (e.g. weekly QA newsletter) Most people have been ok with adapting to changes but you have to provide docs and guidance People follow the lead but would not have done it on their own. (e.g. backports of fixes to stable branches)
  18. 18. Towards Releases Easier once we came over a long time without a release Getting into the habit to do releases more often Core application now do releases a while after the libraries Happened naturally Switch to time based releases. Now fine tuning the process to identify problems Took over as release manager (Only because I build up trust to the code base due to all the automation) There need to be proof that it will really work Invest time to make a sane and realistic schedule
  19. 19. Next Steps
  20. 20. Changes to come ABI checker to avoid breakage surprises after a release Jenkins job that builds a magic developer branch Allowing developers to tests their changes for many configurations Thinking about adding arm builds to jenkins Thinking about running automated valgrind runs to detect memory issues (-fsanitize* could be an alternative) Fine tuning of the release process Think about regression test. Having them mandatory for fixes might blow our man power. Integrate the pixel perfect regression tester with jenkins for fast feedback
  21. 21. Changes that might never come Every commit runs through our jenkins testing before it goes into master (Qt project does this) More code review for core developers before changes hit the repo Hard to bring in as many developers want to avoid any delay before the code hits the repo Automate the release process (NEWS generation, announcement templates, upload tarballs, etc) A simple version bump and pushed git tag might be enough to get it all done :-)
  22. 22. Summary – What you should take home Automate as much as possible to keep the workload down and the project always in a good state Invest in all kind of tools to make your job easier Show the benefits of more regular releases and proof that it is possible Keep on fine tuning the process to avoid as much problems as possible Expect that people will only follow if you give them the lead and show how the process could work
  23. 23. Thank you.
  24. 24. References Imagery sources