Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Similar to Weaning Legagy Platform From Offshore Qa
Similar to Weaning Legagy Platform From Offshore Qa (20)
Weaning Legagy Platform From Offshore Qa
- 1. Weaning a Legacy Platform from Traditional Offshore QA January 31, 2012
- 2. The Agile Quality Ideal •Unit Tests •Integration Tests •Story/Acceptance Tests •The development team “owns” quality
- 3. The Agile Quality Ideal – Where is QA? Traditional QA Responsibilities: • Verifying functionality • Verifying bug fixes • Regression Testing Nearly obsolete?
- 4. The Agile Quality Ideal – A possible role for QA
- 5. Why was offshore QA a good idea in the past? Little or no test coverage Fear of refactoring Code rot Changes introduce bugs Copious regression testing needed
- 6. Why was offshore QA a good idea in the past? Manual regression testing Offshore QA •3X cheaper •Overnight feedback
- 7. Why is offshore QA an impediment now? • Development team produces functionality • QA team assures quality
- 8. Why is it so hard to abandon the offshore QA model? • Low test coverage and brittle code are realities – They don’t change overnight • All-hands-on-deck cleanup effort is impractical – Business must go on
- 9. Practical steps to wean platform from offshore QA Three-step plan: 1. Establish development team as a Quality organization 2. Establish development team as a “Quality Assurance” organization 3. Find a new value-adding role for offshore QA staff
- 10. Constant improvement of code cleanliness / test coverage a. Opportunistic legacy rescue b. Scheduled/budgeted cleanup c. Culture of bug prevention
- 11. Opportunistic Legacy Rescue: The Hippocratic Oath First, do no harm
- 12. Opportunistic Legacy Rescue: The Boy Scout Principle Always leave the campsite cleaner than how you found it.
- 13. The legacy rescue catch-22
- 14. The legacy code change algorithm From Michael Feathers, Working Effectively with Legacy Code 1. Identify change points 2. Find test points 3. Break dependencies (safe refactoring) 4. Write tests 5. Make changes and refactor
- 15. Opportunistic Legacy Rescue: Not enough • Business expects a certain velocity • Large-scale refactoring is expensive • Can’t absorb within context of story development
- 16. Budget for cleanup • One pair dedicated to large-scale refactoring and code cleanup • Team prioritizes cleanup items to tackle • Pair addresses issues one-by-one
- 17. Creating a culture of bug prevention • Scrum Kanban • Newly-introduced bug goes to top of priority queue • Bug backlog stays in control • Bugs become painful
- 18. Feeling the pain • Developers: Increase efforts toward bug prevention • Business owners: Support bug prevention efforts (e.g. refactoring)
- 19. Practical steps to wean platform from offshore QA Three-step plan: 1. Establish development team as a Quality organization 2. Establish development team as a “Quality Assurance” organization 3. Find a new value-adding role for offshore QA staff
- 20. Adoption of “QA-type” responsibilities by the development team a. QA lead in team room b. Co-owned Selenium test suite c. “QA hats” for developers
- 21. QA lead in team room • Owns QA process • Provides direction to QA team • Provides instant, face-to-face feedback to developers • Diffuses us vs. them conflicts • Collaborates with developers in writing Selenium tests • Coordinates exploratory testing
- 22. Selenium test suite • Co-owned by Development and QA • Developers responsible for adding tests as per their judgment • Selenium tests run on CI server • Failures require immediate attention
- 23. Developers put on “QA hats” 1. Peer verification 2. Exploratory testing
- 24. Peer verification • In workflow, between DEV Completion and Business Verification
- 25. Exploratory testing by team • ½ hour per week
- 26. Benefits of developers wearing QA hats • Developers assume ownership of quality • Quality gaps become more visible to developers • Team develops mindset toward quality • Detect bugs that would otherwise have gone offshore • Offshore team’s load is lightened
- 27. Practical steps to wean platform from offshore QA Three-step plan: 1. Establish development team as a Quality organization 2. Establish development team as a “Quality Assurance” organization 3. Find a new value-adding role for offshore QA staff
- 28. But we like our offshore QA team • Long-standing relationship • Domain knowledge • Inexpensive • We want to make their current role irrelevant, not make them irrelevant • How can we leverage them to provide value in our new Agile world?
- 29. Re-training offshore QA team to take on value-adding roles a. Second line of defense b. Performance testing c. “User-centric” regression tests
- 30. Second line of defense
- 31. Writing, maintaining and executing performance tests
- 32. Writing and maintaining automated “user-centric” regression tests
- 33. The offshore QA challenge Update your skills or become obsolete
- 34. So how are we doing? Size of offshore QA team Before November 2010: 16 November 2010 – June 2011: 8 Since June 2011: 4 Unit test coverage June 2011: 18% January 2012: 36%
- 35. Production high/critical bugs created vs. resolved
- 36. All high/critical bugs created vs. resolved
- 37. Q&A 37 Making Web Applications Accessible
Editor's Notes
- 1 or 2 QA guys in the team room for the “hard stuff”Load testingNon-standard environmentsWhatever cannot be run on CI server
- Feathers definition of legacy code: Code without test coverage
- NOTE: Offshore QA not nearly as bad as offshore developmentOvernight feedback beats 24-hour cycle, but Agile strives for instant feedbackOffshore testers are used to getting their context from copious functional specifications. Agile replaces these specifications with an ongoing dialog between product owners and developers. Testers are not privy to this dialog and are therefore missing the context surrounding the stories.Us vs. Them: Developers produce functionality, Testers assure quality -- Contention built in (contention is anti-agile)Developers abdicate ownership of quality = Less incentive to prevent bugs and produce clean code
- Three processes that gradually reduce, and eventually eliminate, the need for third-party regression testingCreate the quality – Establish development team as a Quality organizationBridge the Developer/QA gap -- Development team is a Quality Assurance organizationFind a new value-added role for QA (the hard stuff)
- Creating quality
- Class has cyclomatic complexity of 110. What’s another IF-condition? NO!!! No new code without a test.CI job can go red when complexity goes up or coverage goes down.
- Now that we are not making it worse, take one small step and make it a little better.The campsite, not the state park.If you trip on something, you are responsible for cleaning it up.
- Code should not be refactored without test coverageBUT we cannot write tests without first refactoring
- Legacy rescue takes a long time / boy-scouting must be limited
- Getting business buy-in: Make results visible-- Cost of bugs-- Benefits of refactoring
- SCRUM – Pressure to commit to stories for iteration. Bugs get pushed to end.Kanban – No iteration commitments
- Pain = Awareness / Warning Imagine if a heart attack was not painful
- Firststep – blurs on-shore/off-shore distinction Advocate for quality in team room Collects metrics on bug patterns
- Painful but worthwhile – 1 legitimate failure justifies 10 false negatives Duplicate offshore QA’s regression testing – make it less relevant
- Focus on Border conditionsMultiple browsersPrecise fulfillment of requirements as statedAmbiguities in requirements
- QA staffer or developer coordinates Each developer assigned a functional area and environment (operating system/browser) Developers report any problems to QA lead
- These are the “hard stuff” discussed earlier
- Try to break itUtilize domain knowledge – awareness of soft spots
- Independent of Team Room context (just needs projected/actual usage data) Technological domain unto itself – can keep abreast emerging technologies and tools, own process (may be too much for development team to absorb) Best run at night – leverage time zone difference CONTROVERSIAL: Should development team abdicate ownership?
- Automate what they used to do manually Developmentteam’s Selenium tests: Response to individual bugs and “stories” We are not really story-driven QA team’s Selenium tests End-to-end user scenarios Like performance testing, offshore team can keep abreast of technologies and tools, and take ownership of process (share technological findings with DEV team) Based on observed, not required behavior, so can be built and maintained outside context of team room
- Does the story have a happy ending? In truth, the story has no ending at all, for when it comes to process improvement, there is no Definition of Done. But there are a lot of happy people in our team room, in our executive offices and in libraries and universities world-wide.
- Test coverage is a means. The end goal is quality. We began instituting most of the practices discussed in September. The bug rate began to flatten in November.
- Here the flattening of the bug rate is even more dramatic. Again, we saw the change in November.