More Related Content Similar to Establishing Release Quality Levels and Release Acceptance Tests (20) More from Luke Hohmann (20) Establishing Release Quality Levels and Release Acceptance Tests1. Establishing Release Quality Levels
and Release Acceptance Tests
Insight Forum
April 21, 2009
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
1
2. About Luke Hohmann
• Founder/CEO of Enthiosys
– Agile Product Management consulting
– Customer needs, roadmaps, business model
– Product management mentoring and training
• Agile product guy
– VP Bus Dev (Aladdin), VP Eng &Product Dev’t
(Aurigin), VP Systems Eng (EDS Fleet Services)
– Board of Agile Alliance
• Author, speaker, blogger
– “Innovation Games”
– “Beyond Software Architecture”
– “Journey of the Software Professional”
– agile PM blog at www.Enthiosys.com
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
2
3. Goals
1. Provide a framework for thinking about quality
2. Create better results by building the right quality
3. Practical advice to
help you succeed
What
did I
forget?
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
3 3
4. Agenda
• Some discussion – what’s a release? quality?
• The iron triangle or the quality box?
NOT
• Release quality levels
A
• Release acceptance tests TOOLS
TALK
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
4
5. Discussions (5 min)
• What’s a release?
• What’s quality?
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
5
6. What do we offer? And why?
• We offer things to our market ecosystem
(customers, analysts, etc.) to achieve goals
– A release is offered to generate revenue
– Lo-fi prototypes are offered to improve designs
– Betas are offered to gain pre-release feedback
(and maybe generate revenue…)
• To be offered a thing has to have a “quality” that is
suitable to helping us realize the goal
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
6
7. What is Quality?
• “Quality is conformance to user requirements.”
—Phillip Crosby, Quality is Free (1980s)
• “Quality is the absence of defects that would make software
stop completely or produce unacceptable results.”
—Capers Jones, Applied Software Measurement (1991)
• “Quality is achieving excellent levels of fitness for use,
conformance to requirements, reliability, and maintainability.”
—Watts Humphrey, Managing the Software Process
(1980s)
• “Quality is value to some person.”
—Jerry Weinberg, Quality Software Management (1992)
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
7
8. Is that enough?
• What about the manner in which we create “the
thing”? What about the code?
• What if the code is just plain ugly?
• And if you’re using an Agile method, what if your
code doesn’t pass the green bar?
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
8
9. Quality is a Relationship
• High Performance is High Quality
– to users who notice low performance.
• Elegant Coding is High Quality
– to developers who place high value on the opinions of
their peers.
• Zero Defects is High Quality
– to users who would be disturbed by those defects.
• Lots of Features is High Quality
– to marketers who believe that features sell products.
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
9
©2008 Information Architects, Inc.
10. The Dichotomy of Quality
• Intrinsic or Extrinsic?
– Does quality exist in the things we
observe or is it subjective, existing
only in the eye of the observer?
“You take your analytic knife, put the point directly
on the term Quality and just tap, not hard, gently,
and the whole world splits, cleaves, right in two—hip
and square, classic and romantic, technological and
humanistic—and the split is clean.”
—Robert Pirsig
Zen and the Art of Motorcycle Maintenance (1974)
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
10
©2008 Information Architects, Inc.
11. Total Quality is therefore…
Extrinsic Quality = Value
Total Quality
Intrinsic Quality = Craftsmanship
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
11
©2008 Information Architects, Inc., Adapted and Extended
12. Agenda
• Some discussion – what’s a release? quality?
• The iron triangle or the quality box?
• Release quality levels
• Release acceptance test
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
12
13. Should the triangle be a box?
The Traditional Iron Triangle The Quality Box
Value Quality
Scope (Intrinsic quality)
(Extrinsic quality)
Cost Schedule Cost Schedule
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
13
14. It is a system
The Traditional Iron Triangle The Quality Box
Value Quality
Scope (Intrinsic quality)
(Extrinsic quality)
Cost Schedule Cost Schedule
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
14
15. What can affect the system?
• Flip-chart exercise
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
15
16. Choosing to affect the system
The Quality Box Extrinsic Quality
Value Quality • Which features present?
(Extrinsic quality) (Intrinsic quality) • How well do the work?
• How well have they been tested?
• Do you have enough?
• Is this release ready when I need it?
Intrinsic Quality
• Craftsmanship of the code
• Maintainability
Cost Schedule • Level of defects (works as specified)
Let’s hold these as fixed…
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
16
17. Agenda
• Some discussion – what’s a release? quality?
• The iron triangle or the quality box?
• Release quality levels
• Release acceptance tests
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
17
18. Release quality level
• A predetermined quality level chosen by
product management that sets
expectations regarding the intrinsic and
extrinsic quality of a release.
• Enables organizations to make confident
decisions about their context
• Negotiated with development
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
18
19. Example from VeriSign Managed Security Services
Release Level Description
Release Level 5 • All functionality fully certified.
• Near-zero probability of high severity errors and a low probability that medium severity errors.
Release Level 4 • All new and modified functionality fully certified, except where risk is deemed minimal (e.g.
internal facing reports, low probability use cases).
• Features not fully certified documented in the test plan and/or project charter. Strategic
regression testing of existing functionality performed.
• Low probability that high severity errors will be identified in production. Project carries a
moderate probability that medium severity errors.
Release Level 3 • All new and modified functionality at least partially certified.
• Features not fully certified documented in the test plan and/or project charter.
• Partial regression testing performed.
• Moderate probability for high severity errors. Higher probability for medium severity errors.
Release Level 2 • Most new and/or modified functionality partially certified.
• Features not fully certified communicated to stakeholders.
• Partial regression testing performed.
• Higher probability that high and medium severity errors will be identified.
Release Level 1 • New and/or modified features not certified by QA.
• Regression testing may or may not occur.
• Level 1 releases may be available only for demo or controlled access purposes.
• High risk for high severity errors.
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
19
©2007 VeriSign, Inc. Presented at Agile 2007 conference with Enthiosys
19
20. Set each scale separately
Intrinsic Quality Extrinsic Quality
5 • Everything certified! 5 • Customers will rave about this!
• Lots of automation!
4 • All new stuff tested by QA 4 • Consistency very high
• Full regression testing • Simple and sophisticated tasks
3 • All new stuff tested by QA 3 • Documented
• Almost full regression testing • Complete for basic tasks
2 • Most new stuff tested by QA 2 • Not documented
• Partial regression tested • Complete for basic tasks
1 • Not tested by QA 1 • Not documented
• Not regression tested • Possibly incomplete
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
20
21. Lower bound for
intrinsic quality is very high
• Leading edge enterprises employ technologies that can
approach 99% cumulative defect removal rates.
• The norm for US firms is a cumulative defect removal
rate of 75%.
• A cumulative defect removal rate of 95% on a project
appears to be a nodal point where several other benefits
accrue. For projects of similar size and type, these
projects:
– have the shortest schedules.
– have the lowest quantity of effort in terms of person-months
– have the highest levels of user satisfaction after release
—Capers Jones, Applied Software Measurement (1991)
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
21
©2008 Information Architects, Inc.
22. Why is
Intrinsic Quality so Important?
• The Impact of code quality on testing
• Error Location Dynamics
• Error Feedback Ratio
• Technical Debt
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
22
©2008 Information Architects, Inc.
23. Impact of Code Quality on Testing
Development: 10 days,
Test time=
4 people, 4 KLOC, 2 days
1 d/KLOC
Development: 10 days, How long to test? Assume Test time=
½ day to find & fix per defect. 30 days
4 people, 4 KLOC,
15 d/KLOC
Outcome: no time to finish testing,
technical debt increases!
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
23
©2008 Information Architects, Inc.
24. Error Location Dynamics
64
56
Difficult errors
48 take longer to find:
40 1 hr/d to 50 hr/d
Errors Located
32
24
16
8
0
0 2 4 6 8 10 12
Time
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
24
©2008 Information Architects, Inc.
25. Error Feedback Ratio
The time to finish removing errors
The time to finish removing errors
is critically dependent on the error
is critically dependent on the error
feedback ratio. The three simulations
feedback ratio. The three simulations
differ only in their feedback ratios.
differ only in their feedback ratios.
A 20% difference in feedback ratio
A 20% difference in feedback ratio
leads to an 88% difference in
leads to an 88% difference in
completion time, but the next 10%
completion time, but the next 10%
increase leads to aa112% increase.
increase leads to 112% increase.
ERROR FEEDBACK: Errors put into a system when attempting to correct
other faults.
ERROR FEEDBACK RATIO: The number of problems created per fix.
EFR = ERRORS CREATED / ERRORS RESOLVED
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
25
©2008 Information Architects, Inc.
26. Technical Debt
Customer • Once on far right of curve,
Responsiveness
all choices are hard
• If nothing is done, it just
Cost of Change (CoC)
Actual gets worse
CoC
• In applications with high
technical debt, estimating
is nearly impossible
Product Technical Debt • Only 3 strategies
Release – Do nothing, it gets worse
Optimal CoC – Replace, high cost/risk
– Incremental refactoring,
1 2 3 4 5 6 7 8 commitment to invest
Years
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
26
27. Agenda
• Some discussion – what’s a release? quality?
• The iron triangle or the quality box?
• Release quality levels
• Release acceptance tests
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
27
28. Lots of Testing Options!
Customer Usability
Tests Testing
Business Intent
Exploratory
(Fit, Fitnesse)
Run each time a story Testing
scenario is completed Assess each iteration
Programmer Property
Tests Testing
Design Intent Response,
Security
(xUnit) Scaling,…
Run at least once each
development episode Assess each iteration
© From Enthiosys Inc. http://www.testing.com/cgi-bin/blog/2003/08/21#agile-testing-project-1
2009, Brian Marick All rights reserved. www.enthiosys.com or 650.528.4000
28
29. Release Acceptance Tests
• Tests owned and written by the product
manager / product owner / customer to verify that
a story is complete and correct.
• Should be (mostly) automated (more later)
• Popularized by Agile methods, useful everywhere!
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
29
30. Benefits of Acceptance Tests
• Increase team confidence that the system is correct
• Help PMs think through requirements
• Informs developer estimates
• Concrete definition of completion
• Early involvement of QA
• Automated regression testing for free
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
30
31. ATs test the users’ experience
• UTs should test application logic in detail
generally in isolation
• ATs should test: but not...
– Interaction & Flow Usability
– Performance Look & Feel
– Error Handling
– Security
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
31
32. ATDD Cycle
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
32
33. Acceptance Tests are “Quirky”
• Written by PMs/Customers... But executed in code?
– ATs require a customer meta-language generic enough to
capture requirements, abstract enough to be maintainable
– Simple enough to be easily understood
– They are interpreted by various frameworks
– Requires external systems and “spike” development
• Automation tools dictate form/format of tests
– http://www.fitnesse.org/
– http://seleniumhq.org/
– http://robotframework.org
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
33
34. Fitnesse Structure
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
34
35. Fitnesse test example
Each row is one test
Fitnesse supports many
different testing table
styles that can be
organized into test suites.
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
35
36. Writing good acceptance tests
• Tell a story of accomplishing a goal
– Goal test title
– Activities rows in your test suite
• Expect them to be stable, even if the underlying
system is going through a lot of change
• Don’t reference domain objects
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
36
37. Consider “Given-When-Then”
• Given some context
When something happens
Then some behavior
• Given an innovation games planner with a
scheduled party
When they cancel the party
Then send a cancellation email to the players
and the facilitator, a cancellation confirmation
to the planner, and ensure that the game play
URL points to the “cancelled party” page
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
37
38. Overcoming common challenges
• You don’t have to automate everything
• Get started, but don’t expect quick turns
• Help your product managers by writing a few
• Make AT writing a natural part of the conversation
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
38
39. Thank You !!!
Luke Hohmann
Founder & CEO
Enthiosys, Inc.
615 National Ave., Ste 220
Mountain View, CA 94043
cell: (408) 529-0319
www.enthiosys.com
lhohmann@enthiosys.com
Innovation Through Understanding®
© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
39
Editor's Notes www.enthiosys.com Copyright (c) 2001-2008 by Luke Hohmann www.enthiosys.com Copyright (c) 2003-2005 by Luke Hohmann The table below provides detailed descriptions of the Release Levels from a product quality perspective. Further details on the Release Levels can be found in the SDLC documentation. <- 20% -> <------ 10% ------> Research shows that there is a chance of 4 additional errors being introduced with each error that gets corrected! Error Feedback Ratio is a critical parameter measuring the quality in any software organization. A large ratio indicates the process is becoming unstable. what are covered by ATs? Simple, procedural steps www.enthiosys.com Copyright (c) 2003-2006 by Luke Hohmann