presented at the Subversion Day Berlin 2010 agile development and the question "to branch or not to branch?"

1. Implementation of an Agile Process
for Multiple Teams Using SVN
Dr. Alexander Schwartz
June 11, 2010

2. About ...
Alexander Schwartz mobile.international
 mobile.international belongs to eBay  Other marketplaces
20+ years in IT industry  Main platform: mobile.de • RO: mobile.ro
• Biggest market place for • PL: pl.mobile.eu
automobiles in germany. • FR: ebay-automobile.fr
„used to be developer“ • IT: ebay-automobile.it
• One of the top-10 websites in
germany; 49 million visitor per moth

Software Development (IVW 02/2009)

Developer (C, C++, Java)

Tech Lead, Project Lead

Consultant, Trainer

Certified Scrum Master

Release Manager

Build Manager

Agile Configuration Manager

Agile Test Automation
Alex Schwartz 2

3. Outline
Our
experiences
Agile with SVN
Development
Scrum, Kanban,
Agile Dev w/ Lean
Branches?
NO ! YES! Our branching
pattern
Lean Release Release
Approaches Mgmt Dev
Process
Agile Configuration Mgmt
Alex Schwartz 3

4. Agile Development & Configuration Management
Dreilinden, April 30, 2010
Oliver Lorenz 4

5. Agile Process & “DONE”
potentially shippable
product increment
It’s “DONE”.
Alex Schwartz 5

6. Definition of Done (DoD)
Source: Gupta, Definition of Done: A Reference
Alex Schwartz 6

7. Agile & Branching?
DONE =
releasable
Agile Branching Approach #1
Avoid the use of branches.
(Branches are not necessary
since all features are done.)
Alex Schwartz 7

8. The Usual Quality Gap
Real World: Ideal World:
done means DONE = releasable
„still a lot to do“
?
►Process: Waterfall ►Process: Integrated Test Activities
Development Test Development T
Test
►Progress of Quality ►Progress of Quality
Releasable
Releasable
...
... Release
Mainline Codeline
Feature tested
Feature tested Release
... Codeline ...
Mainline Unit tests green
Unit tests green
Compiles
Compiles
Alex Schwartz 8

9. How to improve the common sence of DONE?
Many Aspects Strategy - Blue or the red pill?
 Process  Strategy A:
 Tools Brute force
 Infrastructure
 Skills  Strategy B:
 Mind set Step by step
• „QA as Quality Police“
• Developer: „I am not responsible
for quality.“ Action #1
Evangalize! (firm + carefully)
• Explain the need for an improved „done“ to all
As member of an agile team, I am stakeholders. Convince first developers.
responisble to deliver value to the
• Make tiny first steps for improvement
customer in time and quality.
+ get commitment
• Clarify requirements for Release Management, etc.
(Often, we overdo!)
Alex Schwartz 9

10. Our Code & Architecture
Architecture: • Java + Spring
Ebay like „pillar architecture“ • MySQL + replication
= separate major use cases • Build with Maven
• All code is built, packaged,
deployed + tested all
together
• In numbers:
– NLOC: ~ 1 million
– #modules: ~ 120
– #apps: ~ 50
– #servers: ~ 500
Alex Schwartz 10

11. Phase 1 of Our Scrum Transition
►CM: Mainline Situation Q2 2008
development • parallel development with multiple
distributed teams
• 50+ participants (25+ new/external)
Part-time • 10+ teams of different size
assignement • New marketplaces + apps
• One big codebase
Test system
►Process: SCRUM with waterfall QA Experiences
Development Test We delivered. (Almost in time.)
Reduced external and internal
►Progress of Quality (Done ...) quality + increased technical debt
Reduced test coverage.
Main impediment: low quality of
Release
Codeline
trunk blocks many teams.
Mainline
No one feels responsible for bugs.
Alex Schwartz 11

12. Use branches for agile development?
Yes,
it can make sense to use branches
in the context of agile development
with multiple teams.
Henrik Kniberg:
http://www.infoq.com/articles/agile-version-control
Alex Schwartz 12

13. How we do it..
Quality Gate:
Action #2 Quality Gate: releasable
ready for trunk
New Policies
(< releasable)
• Every project teams develops in a
project branch.
• No commits in the mainline.
• Every contribution for the mainline
(and the next release) has to satisfy
the conditions of a Quality Gate.
Dev Test
Test
Alex Schwartz 13

14. Phase 2 of Our Scrum Transition: Project Branches
►CM: Project
Experiences
branches
Increased quality of TRUNK.
Part-time
assignement Increased release flexibility
Increased release reliability
Increased release „throuput“
Increased responsibility
Increased understanding of DONE
►Process: SCRUM with small waterfall + big waterfall
Increased integration of testers (some
Dev Test
Test teams)
Small waterfall (some teams)
►Progress of Quality (Done ...) Merge overhead
Long R-test phase
Merge risks  re-test in R-test phase
Release Still a QA bottleneck
Mainline Codeline
Problem: completeness of test
environments  re-test in R-test phase
Alex Schwartz 14

15. Phase 3 of Our Scrum Transition:
►CM: Project Action #3
branches
• Reduced regression test phase lenght +
team size.
• Testers are assigned to their project team,
rather than to the R-test team.
• Be more strict: Verify the conditions of the
Quality Gate.
►Process: SCRUM with small waterfall + big waterfall Experiences
Dev Test
Test More conditions of quality gates
satisfied.
Decreased r-test phase length.
►Progress of Quality (Done ...)
Increased understanding of DONE
Increased integration of testers
Release
Codeline (most teams)
Mainline
Too big contributions
Too low external quality
Merge bugs
Alex Schwartz 15

16. What about Continuous Integration?
Continuous Integration = 2 concepts
(a) Continuous publish of small (b) Continuous build
changes for every active ... unit testing
codeline. ... packaging
... verification
... deployment in test system
... deployment in production
Rules: Rule:
• Integrate changes from Use Cont. Build+Test etc.
TRUNK often for every active codeline.
• Publish small chunks
Alex Schwartz 16

17. What about Continuous Integration and Testing?
Rule:
Use Cont. Build+Test etc.
for every active codeline. Challenges
- Create + maintain necessary infrastructure
- Who is maintaining it?
 Standard: Use... central team vs. project teams
• Metrics: test coverage + more
 Advanced: Try to automate ...
• Verification criteria for Quality Gates
• Merges
• CM tasks (creation of branches + CI build
configurations)
• Deployment
• DB Setup
 Monitoring
 Code Metrics (sonar.codehaus.org, ...)
 Branches + Merges
 Test system status
Alex Schwartz 17

18. Our Experiences with SVN
Overall: Possible? Yes, but we almost failed
Known tool 
Documentation 
Complexity of basic cmds 
Tool support around 
Browsing: FishEye 
Constant time branch creation 
Absence of bugs 
Merging 
• Speed 
• Reliability 
• Correctness 
• Easy of use 
Alex Schwartz 18

19. Our Problems (1)
• General • FishEye
– Broken working copies – Too slow or too less content
– Broken revisions on server – Major changes
– Server too slow – Not constant time bran
– Incomplete updates – Indexing takes veryching
– Hard to manipulate
• SVN replication
– Replication failures
– Dublicated revisions
Alex Schwartz 19

20. Our Problems (2) -- Merging
• Too many!
• SVN 1.5..... was a catastrophy
– Merge tracking was too buggy
– Q: Why merge tracking was
implemented for non-root
elements?
– No single merge was working out of
the box!
• Reintegrate merges (diff merge)
• Path replacements / evil twins – often not correct (not a copy!)
– Option „--reintegrate“ does not work
• Catch-up merges (change set based merge)
– Often not correct
– Convinience invocation does not wo
Alex Schwartz 20

21. Alternative Release Models – Our Release Cadence
Question:
Can we do better?
Alex Schwartz 21

22. The Lean Approach
• Focus on value stream
• Limit WIP (work in progress) ... since low WIP reduces cycle time
• Continous Deployment: Release often (several times a day !!!)
• Who is doing it?
• What about branching?
Alex Schwartz 22

23. The Lean Alternative to Branching: Feature Flags
if (feature_C_enabled) {
if (feature_B_enabled) ..
• No branches at all!
Alex Schwartz 23

24. References
Other Resources
Articles / Papers / Books
 CM Crossroads
 Henrik Kniberg:
Version Control for Multiple Agile Teams www.cmcrossroads.com
 Steve Berczuk. Robert Cowham, Brad Appleton: ...
An Agile Approach to Release Management
 Laura Wingerd, Christopher Seiwald:
High-level Best Practices in Software Configuration Management
 Mayank Gupta:
Definition of Done: A Reference, 2008
 Brad Appleton, Steve Berczuk, Ralph Cabrera, Robert Orenstein:
Steamed Lines: Branching Patterns for Parallel Software Development
 Henrik Kniberg:
Scrum And XP from the Trenches
 Lisa Crispin and Janet Gregory:
Agile Testing
 Laura Wingerd:
The Flow of Change
Alex Schwartz 24