Here is a real story, not a fairy tale, to shine a hopeful light for those walking the Lean path in software organizations. A story of developers and testers growing together as problem-solvers, through the catalyzing application of some Lean principles, a few key ideas and shiny new tools from the Devops movement, to the wisdom of the gemba.

1. Devtest:
using Lean and Devops
practices to bring QA and
coders together
Laurent Bossavit
Twitter:@Morendil http://institut-agile.fr/

2. Devtest:
a love story
Laurent Bossavit
Twitter:@Morendil http://institut-agile.fr/

3. Once upon a time...

6. Who?
• ~ 2000 users, ~ 40000 transactions per day
• ~ 60 members on team, including
• ~ 30 developers
• ~ 10 testers (“qualifieurs”)

7. What?

8. Iterations,
task board...

9. Pair
programming

10. Continuous
integration

11. But not all was well...

12. Kaizen spirit ?

13. The actual process...

14. “Why is everything
taking so long?”

15. CORE profile
Development
USER
STORIES
REFACTORING, DATA UPDATES
REACT TO PRODUCTION ISSUES; ANALYSE DEFECTS; FIX DEFECTS
“Failure demand”: non-value adding,
perturbs other work
Based on the CORE profile tool, cf.
Value
adding
Predictable
http://www.slideshare.net/operaepartners/lean-it-strategy-lean-measurement-and-organizational-design

16. End user impact
• Constant background grumbling from end
users, eroding confidence in product
• Flaring up into occasional crises on “MEP”
days (deployments, roughly monthly)
• Substantial monetary impact: each failed
order is re-processed manually

17. Why are defects
slowing us so much?

18. “Staple yourself to a
bug”
(Vendeur via chaîne support)
Signalement via Océane
Qualif Soft
(remontée
de mocks)
Catalogue Dév
Defect #2875:
Migrations Optimale -> Open Pro KO
(aussi détecté par STELA début juin)
(repro)
(defect
fermé)
(analyse)
(defect
réouvert)
Dév
(correction)
(ano QC)
• Total lead time: 6 weeks

19. Best repeatable
performance?
90% > 1 day
75% > 3 days
65% > 5 days
50% > 8 days

20. Hidden costs of
production defects

21. “Passive cost windows”
• Bug introduced
• Bug shipped
• Bug detected
• Bug reported
• Bug fixed
• Fix shipped
lead time }Actual
} “Official” fix
impact

22. Red bin system

23. Existing, but under-used...

24. A delicate subject must
now come up...
(Persons of a delicate
disposition are advised
to cover their ears)

25. Bug reproduction!
"Calopteryx virgo paarung" by Holger Gröschl - http://www.naturspektrum.de/ns1.htm. Licensed under Creative Commons Attribution-Share Alike 2.0-de
via Wikimedia Commons - http://commons.wikimedia.org/wiki/File:Calopteryx_virgo_paarung.jpg#mediaviewer/File:Calopteryx_virgo_paarung.jpg

26. “Repro” the key to
improving the process?
• Minimum time for a repro: 1/2 day...
• ...often stretching to a few days...
• ...sometimes up to several weeks!

27. Setting goals for
“repro” lead time
Current:
1/2 day to
weeks
Target:
5 min

28. What’s so hard about
repro-ing defects?

29. Incomplete
logging...

31. What’s so hard about
repro? Take 2...

32. Standardized work:
software defects
• Repro needs inputs identical to production:
• same third-party inputs (web services)
• same path through application
• same configuration settings
• same database version
• same application version

33. Standardized work:
software defects
• After getting a good “repro” and only then:
• identify causes (debugger, static analysis)
• “necessary and sufficient conditions”
• capture as an automated test

34. What’s so hard about
repro? Take 3...

35. Bureaucracy...

36. Why are test environments
a scarce resource ?

37. Devops to the rescue!

38. Infrastructure as Code
http://bit.ly/jedi_iac

39. Infrastructure?
• Physical servers
• Virtual machines
• Network links
• Operating systems
• Application dependencies
• Application containers (e.g. JonAS)

40. “as code”?
• everything is a text file (no manual setup)
• under version control, with commit
messages
• ideally with automated tests

41. Servers on demand
http://martinfowler.com/bliki/SnowflakeServer.html
http://martinfowler.com/bliki/PhoenixServer.html

42. docker.io

43. New tooling

44. Action...

45. First results...
• Target of 5 minutes reached in
“pilot” phase
• Remaining challenges (technical)
• bring more of the system configuration
under version control
• closer integration of the tools involved

46. First results...
• Remaining challenges (non-technical)
• reach 5 minutes consistently
• align process with reality: count “red bin”
defects as official bugs, proactively
• apply “5 whys” to production bugs

47. Most importantly...
• Dev and QA really collaborating :)

48. How do we streamline defect
reproduction?

49. Sophisticated tooling
Visual Test

50. How do we get everyone to
care about production defects?

51. How the story ends...

52. Merci!