Implementation of an agile process for multiple teams using SVN


Published on

presented at the Subversion Day Berlin 2010

agile development and the question "to branch or not to branch?"

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Implementation of an agile process for multiple teams using SVN

  1. 1. Implementation of an Agile Process for Multiple Teams Using SVN Dr. Alexander Schwartz June 11, 2010
  2. 2. About ... Alexander Schwartz  belongs to eBay  Other marketplaces 20+ years in IT industry  Main platform: • RO: • Biggest market place for • PL: automobiles in germany. • FR: „used to be developer“ • IT: • One of the top-10 websites in germany; 49 million visitor per moth  Software Development (IVW 02/2009)  Developer (C, C++, Java)  Tech Lead, Project Lead  Consultant, Trainer  Certified Scrum Master  Release Manager  Build Manager  Agile Configuration Manager  Agile Test Automation Alex Schwartz 2
  3. 3. Outline Our experiences Agile with SVN Development Scrum, Kanban, Agile Dev w/ Lean Branches? NO ! YES! Our branching pattern Lean Release Release Approaches Mgmt Dev Process Agile Configuration Mgmt Alex Schwartz 3
  4. 4. Agile Development & Configuration Management Dreilinden, April 30, 2010 Oliver Lorenz 4
  5. 5. Agile Process & “DONE” potentially shippable product increment It’s “DONE”. Alex Schwartz 5
  6. 6. Definition of Done (DoD) Source: Gupta, Definition of Done: A Reference Alex Schwartz 6
  7. 7. Agile & Branching? DONE = releasable Agile Branching Approach #1 Avoid the use of branches. (Branches are not necessary since all features are done.) Alex Schwartz 7
  8. 8. The Usual Quality Gap Real World: Ideal World: done means DONE = releasable „still a lot to do“ ? ►Process: Waterfall ►Process: Integrated Test Activities Development Test Development T Test ►Progress of Quality ►Progress of Quality Releasable Releasable ... ... Release Mainline Codeline Feature tested Feature tested Release ... Codeline ... Mainline Unit tests green Unit tests green Compiles Compiles Alex Schwartz 8
  9. 9. How to improve the common sence of DONE? Many Aspects Strategy - Blue or the red pill?  Process  Strategy A:  Tools Brute force  Infrastructure  Skills  Strategy B:  Mind set Step by step • „QA as Quality Police“ • Developer: „I am not responsible for quality.“ Action #1 Evangalize! (firm + carefully) • Explain the need for an improved „done“ to all As member of an agile team, I am stakeholders. Convince first developers. responisble to deliver value to the • Make tiny first steps for improvement customer in time and quality. + get commitment • Clarify requirements for Release Management, etc. (Often, we overdo!) Alex Schwartz 9
  10. 10. Our Code & Architecture Architecture: • Java + Spring Ebay like „pillar architecture“ • MySQL + replication = separate major use cases • Build with Maven • All code is built, packaged, deployed + tested all together • In numbers: – NLOC: ~ 1 million – #modules: ~ 120 – #apps: ~ 50 – #servers: ~ 500 Alex Schwartz 10
  11. 11. Phase 1 of Our Scrum Transition ►CM: Mainline Situation Q2 2008 development • parallel development with multiple distributed teams • 50+ participants (25+ new/external) Part-time • 10+ teams of different size assignement • New marketplaces + apps • One big codebase Test system ►Process: SCRUM with waterfall QA Experiences Development Test We delivered. (Almost in time.) Reduced external and internal ►Progress of Quality (Done ...) quality + increased technical debt Reduced test coverage. Main impediment: low quality of Release Codeline trunk blocks many teams. Mainline No one feels responsible for bugs. Alex Schwartz 11
  12. 12. Use branches for agile development? Yes, it can make sense to use branches in the context of agile development with multiple teams. Henrik Kniberg: Alex Schwartz 12
  13. 13. How we do it.. Quality Gate: Action #2 Quality Gate: releasable ready for trunk New Policies (< releasable) • Every project teams develops in a project branch. • No commits in the mainline. • Every contribution for the mainline (and the next release) has to satisfy the conditions of a Quality Gate. Dev Test Test Alex Schwartz 13
  14. 14. Phase 2 of Our Scrum Transition: Project Branches ►CM: Project Experiences branches Increased quality of TRUNK. Part-time assignement Increased release flexibility Increased release reliability Increased release „throuput“ Increased responsibility Increased understanding of DONE ►Process: SCRUM with small waterfall + big waterfall Increased integration of testers (some Dev Test Test teams) Small waterfall (some teams) ►Progress of Quality (Done ...) Merge overhead Long R-test phase Merge risks  re-test in R-test phase Release Still a QA bottleneck Mainline Codeline Problem: completeness of test environments  re-test in R-test phase Alex Schwartz 14
  15. 15. Phase 3 of Our Scrum Transition: ►CM: Project Action #3 branches • Reduced regression test phase lenght + team size. • Testers are assigned to their project team, rather than to the R-test team. • Be more strict: Verify the conditions of the Quality Gate. ►Process: SCRUM with small waterfall + big waterfall Experiences Dev Test Test More conditions of quality gates satisfied. Decreased r-test phase length. ►Progress of Quality (Done ...) Increased understanding of DONE Increased integration of testers Release Codeline (most teams) Mainline Too big contributions Too low external quality Merge bugs Alex Schwartz 15
  16. 16. What about Continuous Integration? Continuous Integration = 2 concepts (a) Continuous publish of small (b) Continuous build changes for every active ... unit testing codeline. ... packaging ... verification ... deployment in test system ... deployment in production Rules: Rule: • Integrate changes from Use Cont. Build+Test etc. TRUNK often for every active codeline. • Publish small chunks Alex Schwartz 16
  17. 17. What about Continuous Integration and Testing? Rule: Use Cont. Build+Test etc. for every active codeline. Challenges - Create + maintain necessary infrastructure - Who is maintaining it?  Standard: Use... central team vs. project teams • Metrics: test coverage + more  Advanced: Try to automate ... • Verification criteria for Quality Gates • Merges • CM tasks (creation of branches + CI build configurations) • Deployment • DB Setup  Monitoring  Code Metrics (, ...)  Branches + Merges  Test system status Alex Schwartz 17
  18. 18. Our Experiences with SVN Overall: Possible? Yes, but we almost failed Known tool  Documentation  Complexity of basic cmds  Tool support around  Browsing: FishEye  Constant time branch creation  Absence of bugs  Merging  • Speed  • Reliability  • Correctness  • Easy of use  Alex Schwartz 18
  19. 19. Our Problems (1) • General • FishEye – Broken working copies – Too slow or too less content – Broken revisions on server – Major changes – Server too slow – Not constant time bran – Incomplete updates – Indexing takes veryching – Hard to manipulate • SVN replication – Replication failures – Dublicated revisions Alex Schwartz 19
  20. 20. Our Problems (2) -- Merging • Too many! • SVN 1.5..... was a catastrophy – Merge tracking was too buggy – Q: Why merge tracking was implemented for non-root elements? – No single merge was working out of the box! • Reintegrate merges (diff merge) • Path replacements / evil twins – often not correct (not a copy!) – Option „--reintegrate“ does not work • Catch-up merges (change set based merge) – Often not correct – Convinience invocation does not wo Alex Schwartz 20
  21. 21. Alternative Release Models – Our Release Cadence Question: Can we do better? Alex Schwartz 21
  22. 22. The Lean Approach • Focus on value stream • Limit WIP (work in progress) ... since low WIP reduces cycle time • Continous Deployment: Release often (several times a day !!!) • Who is doing it? • What about branching? Alex Schwartz 22
  23. 23. The Lean Alternative to Branching: Feature Flags if (feature_C_enabled) { if (feature_B_enabled) .. • No branches at all! Alex Schwartz 23
  24. 24. References Other Resources Articles / Papers / Books  CM Crossroads  Henrik Kniberg: Version Control for Multiple Agile Teams  Steve Berczuk. Robert Cowham, Brad Appleton: ... An Agile Approach to Release Management  Laura Wingerd, Christopher Seiwald: High-level Best Practices in Software Configuration Management  Mayank Gupta: Definition of Done: A Reference, 2008  Brad Appleton, Steve Berczuk, Ralph Cabrera, Robert Orenstein: Steamed Lines: Branching Patterns for Parallel Software Development  Henrik Kniberg: Scrum And XP from the Trenches  Lisa Crispin and Janet Gregory: Agile Testing  Laura Wingerd: The Flow of Change Alex Schwartz 24