Stop the Line + Stop Feature DevelopmentLean practices for Software Product DevelopmentJuan Gutiérrez Plaza              |...
F-Secure – the company• Founded in 1988,  listed on NASDAQ OMX  Helsinki• Market cap ca 350 m€,  annual revenue ca 130 m€ ...
Products and Services                  Win, Mac, Linux, Android, iOS, RIM, Symbian, 20+ language versions3   21-Oct-2011  ...
Customers: 200+ operator partners globally    22    20    18    16    14    12    10     8     6     4     2     0    Oper...
About the AuthorsGabor Gunyho                          Régis Déau                              Juan Gutiérrez PlazaImprove...
What is this presentation all about?• No “recipe”• Just to share  how we did it        Image source: http://www.clker.com/...
The Project7   21-Oct-2011   © F-Secure Public
Project setup• About 12 teams (around 100 people)    •    Mostly in Helsinki, some in Kuala Lumpur, later also one in Pola...
Project timeline• Started: Dec 2009    • This presentation counts data from March 2010• Project Split and Spin-off: March ...
Stop the Line10   21-Oct-2011   © F-Secure Public
What is it?A practice coming from Lean that is originated from theToyota Production System (TPS) [6]                      ...
What is it? – The Line “Line” refers to production/assembly lines in automobile industry where       one station takes the...
What is it? - Stopping• If a problem is found, anyone can “pull the cord” that:     • Stops the line from moving ahead    ...
Why it happened? How to avoid it?• To get all the benefits of the Stop the Line practice, a root  cause analysis is done t...
Why to use it?• Focus on quality at all times• Identify recurrent (systemic) problems so they are solved  once and for all...
… and for us in SW. Development? (1/3)• Detection     • A Stop-the-Line is raised when         • A build is failing (e.g. ...
… and for us in SW. Development? (2/3)• Notification     • E-mail     • Stop-the-line radiator raises Stop-the-line flag f...
… and for us in SW. Development? (3/3)• Fixing     • A team or person claims the issue using the claim functionality in   ...
In short…19   21-Oct-2011   © F-Secure Public
In short… Problem is   Found                              Stop the                                Line                    ...
The First Implementation Fix• Rule #1 when the STL is on then do not commit new feature  development code to the module th...
Bug Handling &Stop Feature Development22   21-Oct-2011   © F-Secure Public
Old Model• High level concept:                    • Bug count usage:         • A bugs‟ warehouse                  • Measur...
QA (bug handling) process: Our Goal• Very fast track in closing new cases     • Get all new cases closed in less than 4 we...
New Model• Reversing the old decision-making order• Decision making order:          • Team members          • Team bug rev...
New Model• Using bug count limits – Stop Feature Development          • Work guidance:              • X bugs / team  STOP...
Stop Feature Development (SFD)• What is it, then?     • An enhancement for STL     • Line is stopped not only when tests a...
The QA process in a nutshell                                       Some valid bugs will                                   ...
The QA process in a nutshell• Bug is created: Choose the correct product area and prioritize the bug with  your best guess...
30   21-Oct-2011   © F-Secure Public
Statistics31   21-Oct-2011   © F-Secure Public
STL+SFD Events vs. Releases                                         STL + SFD            STL EnforcerMar‘10               ...
Sprint 29 – STL Root Cause Analysis Findings• 11 STL cases was tracked for Sprint 29• Frequent root cause categories:     ...
Total                                                    Ro                    Case 9                    Case 8           ...
Conclusions35   21-Oct-2011   © F-Secure Public
Conclusions• Overall quality of the product improved• Number of STL events decreased by time     • STL enforcer helped to ...
Questions?37   21-Oct-2011   © F-Secure Public
AcknowledgementsThe authors would like to thank the whole project team and the whole R&Dorganization of F-Secure and its m...
Working at F-Secure - Keeping it fun!• Coding Dojos• Team building events• Friday evening b**rs• ‟Hamburger‟ nights• Sport...
How does it look?40   21-Oct-2011   © F-Secure Public
Wanna work for F-Secure?     Would you like to work as…          • C++, JavaScript and Python developer?          • Qualit...
References[1] Schwaber, K., Beedle, M.: “Agile Software Development with Scrum”, Prentice Hall (2001)[2] Larman, C., Vodde...
Contact Informationjuan.gutierrez@f-secure.com                                                             http://creative...
Stop the line & Stop Feature Development practices
Upcoming SlideShare
Loading in …5
×

Stop the line & Stop Feature Development practices

2,245 views

Published on

F-Secure

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Stop the line & Stop Feature Development practices

  1. 1. Stop the Line + Stop Feature DevelopmentLean practices for Software Product DevelopmentJuan Gutiérrez Plaza | Gabor Gunyho | Régis DéauSenior Manager, Agile Practices Improvement Coach Manager, Testing Practices21-Oct-2011Protecting the irreplaceable | f-secure.com
  2. 2. F-Secure – the company• Founded in 1988, listed on NASDAQ OMX Helsinki• Market cap ca 350 m€, annual revenue ca 130 m€ (2010)• Headquartered in Helsinki, 18 country offices, presence in more than 100 countries• 812 people, 300+ in R&D, 5 R&D offices in 4 countries (2010)2 21-Oct-2011 © F-Secure Public
  3. 3. Products and Services Win, Mac, Linux, Android, iOS, RIM, Symbian, 20+ language versions3 21-Oct-2011 © F-Secure Public
  4. 4. Customers: 200+ operator partners globally 22 20 18 16 14 12 10 8 6 4 2 0 Operator revenue (mEur/quarter)4 21-Oct-2011 © F-Secure Public
  5. 5. About the AuthorsGabor Gunyho Régis Déau Juan Gutiérrez PlazaImprovement Coach with the “R&D Testing practices Manager at F-Secure Currently „Agile Practices Manager‟Global Methods” team at F-Secure, SDC unit, focusing on developing an at F-Secure‟s SDC unit, focusing onexperienced Agile and Lean product agile testing culture and improve the R&D transformation of the site.development expert, contributor and quality engineering practices for Experienced coach who has helpedreviewer of books on scaling Agile continuously improving the R&D different teams to improve in eng.and Lean SW development standards and process practices5 21-Oct-2011 © F-Secure Public
  6. 6. What is this presentation all about?• No “recipe”• Just to share how we did it Image source: http://www.clker.com/clipart-9889.html Text source: http://easteuropeanfood.about.com/od/hungariansoups/r/gulyasleves.htm6 21-Oct-2011 © F-Secure Public
  7. 7. The Project7 21-Oct-2011 © F-Secure Public
  8. 8. Project setup• About 12 teams (around 100 people) • Mostly in Helsinki, some in Kuala Lumpur, later also one in Poland • Mostly feature teams • Fairly mature in basic Scrum[1] and Agile engineering practices • Some experience in multi-team projects[2][3] but not on this scale• Major new product, significant changes in • Business model • Architecture • Longer-Term Planning[4][5], including new backlog tooling• Goal • Upgrading the demo platform every sprint8 21-Oct-2011 © F-Secure Public
  9. 9. Project timeline• Started: Dec 2009 • This presentation counts data from March 2010• Project Split and Spin-off: March 2010• Intermediate Public Release: Sept 2010 • Limited scope• Stop the Line practice: since Sept 2010 (1st draft in June) • Simplification of the practice: Oct 2010 • STL enforcer added on: March 2011• Stop Feature Development practice: since Sept 2010• Two-week sprints: 46 so far • Most resulted in a public Technology Preview release• Public Release Oct 20119 21-Oct-2011 © F-Secure Public
  10. 10. Stop the Line10 21-Oct-2011 © F-Secure Public
  11. 11. What is it?A practice coming from Lean that is originated from theToyota Production System (TPS) [6] Stop-the-Line Work is stopped if an abnormality is found. Work continues only when problem is fixed.11 21-Oct-2011 © F-Secure Public
  12. 12. What is it? – The Line “Line” refers to production/assembly lines in automobile industry where one station takes the output of the previous station as input Image source: http://www.fourwheeler.com/techarticles/body/129_0703_toyota_assembly_factory/photo_02.html12 21-Oct-2011 © F-Secure Public
  13. 13. What is it? - Stopping• If a problem is found, anyone can “pull the cord” that: • Stops the line from moving ahead • Signals the problem to everyone on the line pointing to the station in trouble Image source: http://www.resourcesystemsconsulting.com/blog/archives/78 Image source: http://www.flickr.com/photos/9516941@N08/3334795306/13 21-Oct-2011 © F-Secure Public
  14. 14. Why it happened? How to avoid it?• To get all the benefits of the Stop the Line practice, a root cause analysis is done to find what caused the problem• To avoid the same problem to happen again in the future, fix the root cause too14 21-Oct-2011 © F-Secure Public
  15. 15. Why to use it?• Focus on quality at all times• Identify recurrent (systemic) problems so they are solved once and for all• Avoid burying problems deep in the product where it‟s more difficult to fix it, potentially adding more problems on top of identified ones• Everybody is aware of the problem so anyone who can help, can contribute to fixing it15 21-Oct-2011 © F-Secure Public
  16. 16. … and for us in SW. Development? (1/3)• Detection • A Stop-the-Line is raised when • A build is failing (e.g. it doesnt compile or pass unit testing) • Automated smoke tests fail for 2 or more consecutive times • A problem prevents manual testing to be performed16 21-Oct-2011 © F-Secure Public
  17. 17. … and for us in SW. Development? (2/3)• Notification • E-mail • Stop-the-line radiator raises Stop-the-line flag for the “line” i.e., product area17 21-Oct-2011 © F-Secure Public
  18. 18. … and for us in SW. Development? (3/3)• Fixing • A team or person claims the issue using the claim functionality in radiator (since March „10) and then starts investigating it • Same or other team or person starts fixing the problem • Issues not claimed before next day are handled in the daily Scrum of Scrums and picked by teams • Team works on stop-the-line as high priority item until it is handled • When radiator no longer declares stop-the-line, team is freed from this responsibility• Prevention • Team worked on the stop-the-line case conducts a root cause analysis for selected cases and records findings then sends note to project mailing list18 21-Oct-2011 © F-Secure Public
  19. 19. In short…19 21-Oct-2011 © F-Secure Public
  20. 20. In short… Problem is Found Stop the Line Fix the problem immediately Root cause Analysis Fix the root cause20 21-Oct-2011 © F-Secure Public
  21. 21. The First Implementation Fix• Rule #1 when the STL is on then do not commit new feature development code to the module that has the STL, only commit bug fixes• Unfortunately not everyone was careful enough to follow this rule systematically so some non-related commits were done whilst STL events• To prevent the human mistakes an automated tool was introduced to enforce the rule #1, the “STL enforcer” • Hook was added in the repository that checks if the commit is done during a STL event, and if so, commits non-related with the fix, are rejected • Introduced in the middle of the project (March 2010)21 21-Oct-2011 © F-Secure Public
  22. 22. Bug Handling &Stop Feature Development22 21-Oct-2011 © F-Secure Public
  23. 23. Old Model• High level concept: • Bug count usage: • A bugs‟ warehouse • Measuring quality • Chief QE follows, reports and escalates (no real process to react)• Decision making order: • Bug life cycle: • Chief QE/Project bug review • Store all, prioritize continuously • Team bug review • Only high priority bugs get fixed • Team member • Rest in warehouse (>95% of all) • Maintenance gets all bugs that project did not have time to fix • Maintenance never fixes these23 21-Oct-2011 © F-Secure Public
  24. 24. QA (bug handling) process: Our Goal• Very fast track in closing new cases • Get all new cases closed in less than 4 weeks • Make decision quickly, closest the actual place of work• Avoid building a big batch (warehouse) of bugs by all means • To reduce recurring effort of prioritizing a long list24 21-Oct-2011 © F-Secure Public
  25. 25. New Model• Reversing the old decision-making order• Decision making order: • Team members • Team bug review • Team bug review with Product Owner (+other stakeholders if needed) • Project bug review25 21-Oct-2011 © F-Secure Public
  26. 26. New Model• Using bug count limits – Stop Feature Development • Work guidance: • X bugs / team  STOP new development in team • Y bugs / project  STOP the new development in whole project• Bug life cycle: • Extremely fast handling cycle: • Fix in this sprint • Fix in next sprint • Trash (w/ reason category) • For maintenance • Yes we fix • Trash (w/ reason category)26 21-Oct-2011 © F-Secure Public
  27. 27. Stop Feature Development (SFD)• What is it, then? • An enhancement for STL • Line is stopped not only when tests are not passing but when the number of non-critical bugs go over a threshold: • Per team • Per project• Why? • Care for quality even more • Prioritize bug fixing over new features• When? • >10 cases / team. Stop Feature Development for the team • >100 cases / project. Stop Feature Development for the whole project27 21-Oct-2011 © F-Secure Public
  28. 28. The QA process in a nutshell Some valid bugs will get trashed, but that is OK in this process!28 21-Oct-2011 © F-Secure Public
  29. 29. The QA process in a nutshell• Bug is created: Choose the correct product area and prioritize the bug with your best guess.• Decision: A team decides whether the bug is fixed in this sprint, next sprint or trashed.• Fix and test: A team fixes and tests their fix.• Closing: All new bugs are closed in less than 4 weeks.• Inside the teams: • Bugs not tracked if… • The bug is fixed and tested by the team within the sprint • The bug does not cross sprint or team boundaries29 21-Oct-2011 © F-Secure Public
  30. 30. 30 21-Oct-2011 © F-Secure Public
  31. 31. Statistics31 21-Oct-2011 © F-Secure Public
  32. 32. STL+SFD Events vs. Releases STL + SFD STL EnforcerMar‘10 Aug‘11 Jan‘11 Jun‘11 Now 32 21-Oct-2011 © F-Secure Public
  33. 33. Sprint 29 – STL Root Cause Analysis Findings• 11 STL cases was tracked for Sprint 29• Frequent root cause categories: 1. Blind commits (or insufficiently tested commits) 2. Code/environment changes broke Test Automations 3. Large commits• Actions that can prevent similar case in future: 1. Ensure sufficient testing before commit, commit to branch if needed 2. Monitor the radiator for smoke test results after commit (delay the commit to next day if you plan to leave office soon) 3. Developers should test final builds manually more often 4. Make smaller and incremental commits33 21-Oct-2011 © F-Secure Public
  34. 34. Total Ro Case 9 Case 8 Case 7 Case 6 Case 5 Case 4 Case 3 Case 2 Case 1 ot Case 1034 C au s eC at eg or Co y de 3 1 1 121-Oct-2011 br / e n ok v i r e o La TA nm rg en e tc 2 1 1 co ha m m ng es Ha its lf im 1 1 pl em Bl en in te d d fe© F-Secure Public 4 1 1 1 1 co m at m ur es its Te st E 1 1 (IT nv ) ir o nm Ho en w tC fu ca ha tu th n ng re is es ? be Ha pr rd ev en 3 1 1 1 pr / N te e v ot d Detailed case breakdown en w o in M tion rth on t 3 ito effo he 1 1 1 rR rt ad iat Sm or al 2 1 1 co ler & m m inc i re Fa ts m st en er ta l 1 1 sh TA or Ha te n rd Co T w m A cy are p 1 1 im let cle to pl e f em ea De en ture ve tat i 3 1 1 1 bu lope on ild rs s m sh En o ou su re o ld t r fte es 5 1 1 1 1 1 be e s n t Re fo uff re ici d Ed com ent uc m tes a it tin to te 1 1 m de g on ve ito lop r er on w ha t
  35. 35. Conclusions35 21-Oct-2011 © F-Secure Public
  36. 36. Conclusions• Overall quality of the product improved• Number of STL events decreased by time • STL enforcer helped to avoid some mistakes• Not releasing every two weeks BECAME AN EXCEPTION and not a rule• New bug handling process helped on focusing on important bugs• SFD keeps the level of open bugs in a manageable number• After a settle down period, these practices changed the mindset of the people to be more quality focus• Next step: • STL and SFD are “brakes” to avoid accidents, now we are learning how to drive at high speed safely (avoid making so many bugs)36 21-Oct-2011 © F-Secure Public
  37. 37. Questions?37 21-Oct-2011 © F-Secure Public
  38. 38. AcknowledgementsThe authors would like to thank the whole project team and the whole R&Dorganization of F-Secure and its management for making this presentationpossible and support the data collection and publishingWe‟d like to thank especially • Petri Kuikka • Risto Kumpulainen • Pekka Kiviniemi • Ferrix Hovifor their contribution in the bug handling process, Continuous Integration andTest Automation system and radiator design and implementation and datavisualization38 21-Oct-2011 © F-Secure Public
  39. 39. Working at F-Secure - Keeping it fun!• Coding Dojos• Team building events• Friday evening b**rs• ‟Hamburger‟ nights• Sports club: Football, squash, badminton…• Trainings and book club• Silent and… Musical hours ;-)• Low hierarchy, small teams• Innovation day and demo once per month• Large and small company benefits39 21-Oct-2011 © F-Secure Public
  40. 40. How does it look?40 21-Oct-2011 © F-Secure Public
  41. 41. Wanna work for F-Secure? Would you like to work as… • C++, JavaScript and Python developer? • Quality Engineer? • Scrum Master? • Product Owner? We are looking for GOOD PEOPLE! If you want to move to Bordeaux, LET ME KNOW! If you are more interested in Helsinki, Oulu, San José or Kuala Lumpur, check the position opens at: http://www.f-secure.com/en_EMEA-Corp/careers/open-positions/41 21-Oct-2011 © F-Secure Public
  42. 42. References[1] Schwaber, K., Beedle, M.: “Agile Software Development with Scrum”, Prentice Hall (2001)[2] Larman, C., Vodde, B.: “Scaling Lean & Agile Development: Thinking and Organizational Tools for Large-Scale Scrum”, Addison-Wesley Professional (2008)[3] Larman, C., Vodde, B.: “Practices for Scaling Lean & Agile Development: Large, Multisite, and Offshore Product Development with Large-Scale Scrum”, Addison-Wesley Professional (2010)[4] Leffingwell, D.: “Scaling Software Agility: Best Practices for Large Enterprises”, Addison-Wesley Professional (2007)[5] Leffingwell, D.: “Agile Software Requirements: Lean Requirements Practices for Teams, Programs, and the Enterprise”, Addison-Wesley Professional (2011)[6] Womack, J.P., Jones, D.T., Roos, D.: The machine that changed the world (1990, 2007)[7] Poppendieck, M., Poppendieck, T.: “Implementing Lean Software Development: from Concept to Cash”, Addison-Wesley (2007)42 21-Oct-2011 © F-Secure Public
  43. 43. Contact Informationjuan.gutierrez@f-secure.com http://creativecommons.org/licenses/by-nd/3.0/gabor.gunyho@f-secure.comregis.deau@f-secure.comThe authors did their best to attribute the authors of texts and images, and to recognize any copyrights, see moredetails of copyrights, license terms and conditions for each source under the reference link provided. If you thinkthat anything in this material should be changed, added or removed, please contact the authors at the addressesabove43 21-Oct-2011 © F-Secure Public

×