Mikko Paukkila discusses optimizing software development processes to balance bureaucracy and flexibility. He advocates for continuous integration to find errors early and speed up feedback loops. Tools like Git, Jenkins, Gerrit enable CI by automating builds, testing and code reviews. Process optimizations include reducing time from change to product, automating more tests, and ensuring developers have easy environments and fast feedback. The goal is enabling smooth development flows from needs to requirements to changes to high quality products.

1. Making software
development
processes to
work for you
Optimizing investments to balance between bureaucracy and
total liberalism in software development
Mikko Paukkila – 9-Oct-2014

2. Background
• Mikko Paukkila is a Principal Systems Engineer in software build, release,
testing and integration area. He has experience of multiple software
development environments like Linux, Mac, Windows, Android, Qt, Symbian
etc. He has also wide knowledge of agile & lean processes and related tools
from small 10-20 people software&hardware development product projects
to very large programs of 1000+ developers. He has been driving deployment
of continuous integration processes in an international environment and has
been administrating related tools like Jenkins and Gerrit. Expert of SCM
tools like git. Mikko currently for Nokia Technologies for imaging technology
business.
• Disclaimer:
The presentation will contain a lot of opinions that are mine, not company’s.

3. Some words about software
development processes

4. From need to product
Purpose of software development processes&tools should be enabled smooth
flow between these phases:
1. From (customer) need to requirement
2. From requirement to a software change
3. From software change to high quality product
4. Product creation/finalization process
Requirement
management
Product
creation
Software
development
And not forgetting feedback loop.

5. Continuous Integration enables fast and
extensive feedback of software changes that
are in our release chain. This is done by
automated tools, but it requires that
developer commits software change(s) often
and react fast if there are problems in
builds.
CI is done to enable errors and integration
problems to be found earlier. This eventually
leads us to produce better quality software
already from the beginning and enables also
us to get our products and solutions faster to
the market.
Basic Continuous Integration system:
Continuous Integration
Developer practices:
1. Commit changes often
2. Follow the status
(build & test results)
3. React when needed
(debug, fix, merge etc.)
Automated
testingFast builds
Enablers of Continuous
Integration
Version control
system (git,…)
Build system
(e.g. Jenkins)
Test automation
system
Developers
+ other linked practices: Test Driven Development, …
Instant feedback and easy
access to the latest
environment and results
What? Why?
Error cost curve:
CI

6. Continuous Integration tools

7. Tool selections: SCM
• git is my option
• git is the most powerful SCM tool, but requires time to learn even most
important tricks of it
• Use that SCM tool that developers have mostly got used it or if you some
other specific requirements (like that your customers requires so)
• git submodules solution sucks – If you need hierarchy of repositories then
use for example Google’s repo tool
• What to consider?
 Distributed development – central SCM vs. distributed one?
 A lot of open source alternatives vs. commercial ones
commit e83c5163316f89bfbde7d9ab23ca2e25604af290
Author: Linus Torvalds <torvalds@ppc970.osdl.org>
Date: Thu Apr 7 15:13:13 2005 -0700
Initial revision of "git", the information manager from hell

8. Tool selections: repository hosting
• Host yourself or outside?
 Security
 Performance
 Price (also internal workforce costs)
• Gerrit
 Open source alternative with code review solution
 Qt and Google are using too, but have intenal forked versions
 Access management on quite detailed level and requires some learning
 git is the only supported version control tool
 https://android-review.googlesource.com/ , https://codereview.qt-project.org/
• Github
 SAAS or Github Enterprise
 Git only
• Gitorious
• Bitbucket by Atlassian
• Deveo
 Finnish version supporting git, Mercurial, Subversion

9. Continuous Integration system
• If your project is not having an automated build&test
system, then make sure you get it!
• Quite many open source alternatives:
 Buildbot
 Cruisecontrol
 Jenkins
 And commercial solutions too, like ElectricCommanders, Team
Foundation Server, Bamboo by Atlassian
• Unless you totally hate graphical interfaces or Java or
open source software, then Jenkins is your choice.

10. • Jenkins has probably the biggest Continuous Integration solution user space
and very wide support for multiple tools via plugins
• At least 61000 installations globally and 730+ plugins
(information from May 2013, Cloudbees)
• Java based, you can develop own plugins if needed
• Master – slave architecture
 You can have how many (slave) build machines that you like.
Operations are run normally via ssh.
 Commercial additions like Jenkins Enterprise by Cloudbees
http://pages.cloudbees.com/rs/cloudbees/images/Jenkins-Enterprise-by-
CloudBees.pdf

11. Software change flows

12. Code reviews! Code reviews! Code reviews!
• If you are running a project in which there is no code review practice, then think it again!
• Code review is one of the most valuable part of steps in release chain.
 Fast feedback
 One of the best ways to prevent quality to drop
• By code review (tool) you can:
 Get feedback from your colleagues
 Make better visible what changes are coming and prepare testing
 Run what ever prechecks that you need
 Visualize your changes
• What can go wrong?
 You code review tool is not part of chain – requires that developer need to do many additional steps to
get a change under review.
• Tool & practice recommendations
 http://en.wikipedia.org/wiki/Gerrit_%28software%29
 http://en.wikipedia.org/wiki/Crucible_%28software%29
 http://en.wikipedia.org/wiki/List_of_tools_for_code_review

13. Qt contribution model
Picture from: http://qt-project.org/wiki/Qt-Contribution-Guidelines
1)2)
3)
4) 5) 6)

14. Another release model
Reviewer
(approval,
optional verification)
Developer
New
commit Gerrit
Pre-build
&
test
Commit
Verification
Submit
CI build
& test
Environment
file for CI
built
environment
Ok?
Environment
file for
verified
environment
Ok?
Release verification
(manual testing)
Ok?
Developers
Commit gets already
available for other developers
CI version for
developers
Stable version for
developers
Longer build
& test
Testing
Development environment
levels
Bleeding edge Stable

15. Build phase
1. Fetch source code
2. Run checks that you can run already at this phase and which are not taking too
much time
 Like lint, sonar, valgring etc.
 Commercial options: Coverity etc.
3. Compile it
 Not only for your main target, but to all of your product’s target environments:
Linux? Windows? iOS? Mobile Platforms?
4. Provide feedback to the review system and directly to the developer if build fails
• Package your code & binaries if those are reusable in another continuation steps
• Run additional analyzer that are taking long time separately, like performance tools

16. Test phase
• Do you have already automated tests in place?
• Do you have already automated tests in place?
• Do you have already automated tests in place?
• There is a huge amount of test frameworks and test runners available for
various use cases. Choose what fits best for your technologies and processes.
• Normally test automation is not enough, but you need also some manual
testing

17. Places for optimization

18. CI
CI
Too complex and slow models
Applications
Middleware
OS
Adaptation
Chipset
CI for
platform
CI for
adaptation
(CI for Chipset SW)
CI
CI
CI
CI for
package /
package
wrapper
CI
Packag
e
CI for
package /
package
wrapper
CI
CI for
package /
package
wrapper
CI
CI for
package /
package
wrapper
CI
Deliveries via
packages
Package
• Gating is done for package or package wrapper depending on teams
• Multiple concurrent configurations shall be supported in all CI systems
• Asset Creation environment is set up as is reasonable for the program
Common CI
environments
CI Build & test setup
Package
CI
Package
CI
Package CI
Package
CI
Package
CI
Package
CI
Package
CI
Package CI
Package CI
Package
CI
Package
CI
Package
CI
Quality Gating level
(package)
CI
AssetCreationCIsetups
CI
CI for sub-release verification:
Package / Package wrapper
CI for pre-verification:
Asset Creation / Team CI
CI for release
verification
Developers
Software changes
Team
CommitGate
CI
Team
CI
Daily/weekly
releases
Daily/weekly
releases

19. Process optimizations
• How long it takes from software change to get the change into a product?
 Worse example: 1-2 weeks
 Best example: less than 15 minutes
 Optimizations:
 Build time
 Review time
 Too many steps in the process?
• Quality Gating
 Too heavy?
 Not at all?
• Test automation
 Do you really have it in place?

20. Requirements / Errors / User Stories /
Tasks / Defects / Epics / handling ’needs’
• We use Atlassian Jira
 Together with Agile plugin
• Various available solutions
 I have experince on Bugzilla, Rally, ScrumWorks, Lotus Notes, ...
• ”We have to use Scrum to be Agile?” – really?
 Are you solving your problems by forcing team(s) to use specific model?
 Agile/Scrum came with quite big hype
 There is no single right way to deploy agile processes
 You must be agile also by taking processes in use in an agile way.
Not only forcing the same way as those are defined in books.
 When Scrum is good? How about Kanban?
Scrum Continous
Integration
DevOps
Startup way of
working in big
companies
Waterfall
Lean
Agile
Kanban

21. Other communication solutions for
developers
• Messaging:
 Do you have a solution for developers to use for communication?
 Hipchat, IRC+Quassel (one to many and one to one)
 Lync, Skype (one to one + meetings)
 Important especially if working on multiple locations
• Information management
 Do developers have a solution to share instructions etc.
 TWiki, Concluence, Sharepoint, (Lotus Notes) etc.

22. Developer environment
• Can any developer repeat your product build?
• If not, then make it happen.
• It is better to package all tool that you need for compile
 Put those into git reposity or package those otherwise
 Support by installation scripts
• Do developers have a working IDE package and don’t need to spend hours in
installing & updating applications?

23. Technical Optimizations
• Speed?
 Put more money to build machines
 Start from changing from HDD to SDD -> normally huge improvement
 What steps can be run in parallel?
 Or reduce some step?
 Some technical optimizations: ccache, icecc, ElectricCloud, btrfs, network optimizations
• Money?
 Are you investment on okay level?
 Consider using more open source tools
• Maintenance
 Virtualization
 Automatic ’installation’: puppet, netboot
• Do you have all you tools integrated to your CI system?
 Also requirement and error management systems

24. Bureaucrazy or enhancement?
• Are your software development processes and tools more supporting
management?
• Or
• Are your software development processes and tools supporting software
development?
• Yes, visibility is important and tools for that, but not those tools that are
bringing still more steps for developer to get changes in.

25. Do you have the culture in place?
• Continuous Integration developer practices:
1. Commit changes often
2. Follow the status
(build & test results)
3. React when needed
(debug, fix, merge etc.)
• Do you have code review practice in use?
• Do you have practices for test automation?
• Do you allow others to touch ’any’ components?