Your SlideShare is downloading. ×
Establishing Release Quality Levels and Release Acceptance Tests
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Establishing Release Quality Levels and Release Acceptance Tests

2,155
views

Published on

This presentation introduces the critically important concept of release quality levels: predefined measures of quality that an Agile team can hit. It draws from Jim Highsmith's excellent work on the …

This presentation introduces the critically important concept of release quality levels: predefined measures of quality that an Agile team can hit. It draws from Jim Highsmith's excellent work on the danger of low intrinsic quality. It includes examples of release quality levels established used by VeriSign in their Agile processes.

Published in: Technology, Business

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,155
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
25
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • www.enthiosys.com Copyright (c) 2001-2008 by Luke Hohmann
  • www.enthiosys.com Copyright (c) 2003-2005 by Luke Hohmann
  • The table below provides detailed descriptions of the Release Levels from a product quality perspective. Further details on the Release Levels can be found in the SDLC documentation.
  • <- 20% -> <------ 10% ------> Research shows that there is a chance of 4 additional errors being introduced with each error that gets corrected! Error Feedback Ratio is a critical parameter measuring the quality in any software organization. A large ratio indicates the process is becoming unstable.
  • what are covered by ATs?
  • Simple, procedural steps
  • www.enthiosys.com Copyright (c) 2003-2006 by Luke Hohmann
  • Transcript

    • 1. Establishing Release Quality Levelsand Release Acceptance Tests Insight Forum April 21, 2009© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 1
    • 2. About Luke Hohmann• Founder/CEO of Enthiosys – Agile Product Management consulting – Customer needs, roadmaps, business model – Product management mentoring and training• Agile product guy – VP Bus Dev (Aladdin), VP Eng &Product Dev’t (Aurigin), VP Systems Eng (EDS Fleet Services) – Board of Agile Alliance• Author, speaker, blogger – “Innovation Games” – “Beyond Software Architecture” – “Journey of the Software Professional” – agile PM blog at www.Enthiosys.com © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 2
    • 3. Goals1. Provide a framework for thinking about quality2. Create better results by building the right quality3. Practical advice to help you succeed What did I forget? © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 3 3
    • 4. Agenda• Some discussion – what’s a release? quality?• The iron triangle or the quality box? NOT• Release quality levels A• Release acceptance tests TOOLS TALK © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 4
    • 5. Discussions (5 min)• What’s a release?• What’s quality? © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 5
    • 6. What do we offer? And why?• We offer things to our market ecosystem (customers, analysts, etc.) to achieve goals – A release is offered to generate revenue – Lo-fi prototypes are offered to improve designs – Betas are offered to gain pre-release feedback (and maybe generate revenue…)• To be offered a thing has to have a “quality” that is suitable to helping us realize the goal © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 6
    • 7. What is Quality?• “Quality is conformance to user requirements.” —Phillip Crosby, Quality is Free (1980s)• “Quality is the absence of defects that would make software stop completely or produce unacceptable results.” —Capers Jones, Applied Software Measurement (1991)• “Quality is achieving excellent levels of fitness for use, conformance to requirements, reliability, and maintainability.” —Watts Humphrey, Managing the Software Process (1980s)• “Quality is value to some person.” —Jerry Weinberg, Quality Software Management (1992) © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 7
    • 8. Is that enough?• What about the manner in which we create “the thing”? What about the code?• What if the code is just plain ugly?• And if you’re using an Agile method, what if your code doesn’t pass the green bar? © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 8
    • 9. Quality is a Relationship • High Performance is High Quality – to users who notice low performance. • Elegant Coding is High Quality – to developers who place high value on the opinions of their peers. • Zero Defects is High Quality – to users who would be disturbed by those defects. • Lots of Features is High Quality – to marketers who believe that features sell products. © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 9©2008 Information Architects, Inc.
    • 10. The Dichotomy of Quality • Intrinsic or Extrinsic? – Does quality exist in the things we observe or is it subjective, existing only in the eye of the observer? “You take your analytic knife, put the point directly on the term Quality and just tap, not hard, gently, and the whole world splits, cleaves, right in two—hip and square, classic and romantic, technological and humanistic—and the split is clean.” —Robert Pirsig Zen and the Art of Motorcycle Maintenance (1974) © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 10©2008 Information Architects, Inc.
    • 11. Total Quality is therefore… Extrinsic Quality = Value Total Quality Intrinsic Quality = Craftsmanship © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 11©2008 Information Architects, Inc., Adapted and Extended
    • 12. Agenda• Some discussion – what’s a release? quality?• The iron triangle or the quality box?• Release quality levels• Release acceptance test © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 12
    • 13. Should the triangle be a box? The Traditional Iron Triangle The Quality Box Value Quality Scope (Intrinsic quality) (Extrinsic quality) Cost Schedule Cost Schedule © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 13
    • 14. It is a systemThe Traditional Iron Triangle The Quality Box Value Quality Scope (Intrinsic quality) (Extrinsic quality) Cost Schedule Cost Schedule © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 14
    • 15. What can affect the system?• Flip-chart exercise © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 15
    • 16. Choosing to affect the system The Quality Box Extrinsic Quality Value Quality • Which features present?(Extrinsic quality) (Intrinsic quality) • How well do the work? • How well have they been tested? • Do you have enough? • Is this release ready when I need it? Intrinsic Quality • Craftsmanship of the code • Maintainability Cost Schedule • Level of defects (works as specified) Let’s hold these as fixed… © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 16
    • 17. Agenda• Some discussion – what’s a release? quality?• The iron triangle or the quality box?• Release quality levels• Release acceptance tests © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 17
    • 18. Release quality level• A predetermined quality level chosen by product management that sets expectations regarding the intrinsic and extrinsic quality of a release.• Enables organizations to make confident decisions about their context• Negotiated with development © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 18
    • 19. Example from VeriSign Managed Security Services Release Level Description Release Level 5 • All functionality fully certified. • Near-zero probability of high severity errors and a low probability that medium severity errors. Release Level 4 • All new and modified functionality fully certified, except where risk is deemed minimal (e.g. internal facing reports, low probability use cases). • Features not fully certified documented in the test plan and/or project charter. Strategic regression testing of existing functionality performed. • Low probability that high severity errors will be identified in production. Project carries a moderate probability that medium severity errors. Release Level 3 • All new and modified functionality at least partially certified. • Features not fully certified documented in the test plan and/or project charter. • Partial regression testing performed. • Moderate probability for high severity errors. Higher probability for medium severity errors. Release Level 2 • Most new and/or modified functionality partially certified. • Features not fully certified communicated to stakeholders. • Partial regression testing performed. • Higher probability that high and medium severity errors will be identified. Release Level 1 • New and/or modified features not certified by QA. • Regression testing may or may not occur. • Level 1 releases may be available only for demo or controlled access purposes. • High risk for high severity errors. © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 19©2007 VeriSign, Inc. Presented at Agile 2007 conference with Enthiosys 19
    • 20. Set each scale separately Intrinsic Quality Extrinsic Quality5 • Everything certified! 5 • Customers will rave about this! • Lots of automation!4 • All new stuff tested by QA 4 • Consistency very high • Full regression testing • Simple and sophisticated tasks3 • All new stuff tested by QA 3 • Documented • Almost full regression testing • Complete for basic tasks2 • Most new stuff tested by QA 2 • Not documented • Partial regression tested • Complete for basic tasks1 • Not tested by QA 1 • Not documented • Not regression tested • Possibly incomplete © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 20
    • 21. Lower bound for intrinsic quality is very high • Leading edge enterprises employ technologies that can approach 99% cumulative defect removal rates. • The norm for US firms is a cumulative defect removal rate of 75%. • A cumulative defect removal rate of 95% on a project appears to be a nodal point where several other benefits accrue. For projects of similar size and type, these projects: – have the shortest schedules. – have the lowest quantity of effort in terms of person-months – have the highest levels of user satisfaction after release —Capers Jones, Applied Software Measurement (1991) © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 21©2008 Information Architects, Inc.
    • 22. Why is Intrinsic Quality so Important? • The Impact of code quality on testing • Error Location Dynamics • Error Feedback Ratio • Technical Debt © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 22©2008 Information Architects, Inc.
    • 23. Impact of Code Quality on Testing Development: 10 days, Test time= 4 people, 4 KLOC, 2 days 1 d/KLOC Development: 10 days, How long to test? Assume Test time= ½ day to find & fix per defect. 30 days 4 people, 4 KLOC, 15 d/KLOC Outcome: no time to finish testing, technical debt increases! © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 23©2008 Information Architects, Inc.
    • 24. Error Location Dynamics 64 56 Difficult errors 48 take longer to find: 40 1 hr/d to 50 hr/d Errors Located 32 24 16 8 0 0 2 4 6 8 10 12 Time © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 24©2008 Information Architects, Inc.
    • 25. Error Feedback Ratio The time to finish removing errors The time to finish removing errors is critically dependent on the error is critically dependent on the error feedback ratio. The three simulations feedback ratio. The three simulations differ only in their feedback ratios. differ only in their feedback ratios. A 20% difference in feedback ratio A 20% difference in feedback ratio leads to an 88% difference in leads to an 88% difference in completion time, but the next 10% completion time, but the next 10% increase leads to aa112% increase. increase leads to 112% increase. ERROR FEEDBACK: Errors put into a system when attempting to correct other faults. ERROR FEEDBACK RATIO: The number of problems created per fix. EFR = ERRORS CREATED / ERRORS RESOLVED © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 25©2008 Information Architects, Inc.
    • 26. Technical Debt Customer • Once on far right of curve, Responsiveness all choices are hard • If nothing is done, it justCost of Change (CoC) Actual gets worse CoC • In applications with high technical debt, estimating is nearly impossible Product Technical Debt • Only 3 strategies Release – Do nothing, it gets worse Optimal CoC – Replace, high cost/risk – Incremental refactoring, 1 2 3 4 5 6 7 8 commitment to invest Years © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 26
    • 27. Agenda• Some discussion – what’s a release? quality?• The iron triangle or the quality box?• Release quality levels• Release acceptance tests © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 27
    • 28. Lots of Testing Options! Customer Usability Tests Testing Business Intent Exploratory (Fit, Fitnesse) Run each time a story Testing scenario is completed Assess each iteration Programmer Property Tests Testing Design Intent Response, Security (xUnit) Scaling,… Run at least once each development episode Assess each iteration © From Enthiosys Inc. http://www.testing.com/cgi-bin/blog/2003/08/21#agile-testing-project-1 2009, Brian Marick All rights reserved. www.enthiosys.com or 650.528.4000 28
    • 29. Release Acceptance Tests• Tests owned and written by the product manager / product owner / customer to verify that a story is complete and correct.• Should be (mostly) automated (more later)• Popularized by Agile methods, useful everywhere! © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 29
    • 30. Benefits of Acceptance Tests• Increase team confidence that the system is correct• Help PMs think through requirements• Informs developer estimates• Concrete definition of completion• Early involvement of QA• Automated regression testing for free © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 30
    • 31. ATs test the users’ experience• UTs should test application logic in detail generally in isolation• ATs should test: but not... – Interaction & Flow Usability – Performance Look & Feel – Error Handling – Security © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 31
    • 32. ATDD Cycle © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 32
    • 33. Acceptance Tests are “Quirky”• Written by PMs/Customers... But executed in code? – ATs require a customer meta-language generic enough to capture requirements, abstract enough to be maintainable – Simple enough to be easily understood – They are interpreted by various frameworks – Requires external systems and “spike” development• Automation tools dictate form/format of tests – http://www.fitnesse.org/ – http://seleniumhq.org/ – http://robotframework.org © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 33
    • 34. Fitnesse Structure © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 34
    • 35. Fitnesse test example Each row is one testFitnesse supports manydifferent testing tablestyles that can beorganized into test suites. © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 35
    • 36. Writing good acceptance tests• Tell a story of accomplishing a goal – Goal  test title – Activities  rows in your test suite• Expect them to be stable, even if the underlying system is going through a lot of change• Don’t reference domain objects © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 36
    • 37. Consider “Given-When-Then”• Given some context When something happens Then some behavior• Given an innovation games planner with a scheduled party When they cancel the party Then send a cancellation email to the players and the facilitator, a cancellation confirmation to the planner, and ensure that the game play URL points to the “cancelled party” page © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 37
    • 38. Overcoming common challenges• You don’t have to automate everything• Get started, but don’t expect quick turns• Help your product managers by writing a few• Make AT writing a natural part of the conversation © 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 38
    • 39. Thank You !!! Luke Hohmann Founder & CEO Enthiosys, Inc. 615 National Ave., Ste 220 Mountain View, CA 94043 cell: (408) 529-0319 www.enthiosys.com lhohmann@enthiosys.com Innovation Through Understanding®© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 39