To requirements and beyond...

2
Keynote Outline
Let’s break the ice!
Why do you need good requirements?
How can you better structure and what for?
Some tips and tricks
A TRH! Oh great!...erh remind me…what for exactly?
Pilot, prioritize, communicate, follow-up
Yes, but I’m an Analyst, not a TM or a TA…
That’s all folks!

3
WHO’S WHO?
Mais que diable allait-il faire dans cette galère? - Molière

4
A Schizophrenic View on Me, Myself and I, …
I’m 20 !
Olivier Denoo
I’m Belgian

5
Keynote Outline
That’s all folks!

7
Functional Defect
Project : Louvois
Late payments, wrong salary calculation…
The French Ministry was unable to fix the software issue…
Families are appalled
346 Mio € Loss – no usage
French Servicemen hit by a payroll software issue

8
Security Defect
Orange TelCo blamed – Privacy rules over hacking

9
User Satisfaction issue
Windows 8 – Even Vista was better!

10
User / Customer
Issue
Technical Issue
Functional Issue
Ow! Let’s go fishing!
Production
Maintenance
Poor Non-
Functional Testing
Poor Code
quality
Poor Technical
architecture
Bug - Defect
Failure - Issue
Insufficient
involvement
Poor business
processes
Communication
Needs
Market
Poor Requirements
Poor Functional
Architecture
Poor functional
Testing
Insufficient
involvement
Poor Business
Processes
Communication
Needs
Market Poor Non-
Functional Testing
Poor Code
quality
Poor Functional
Testing
Production
Maintenance
Poor Technical
Architecture
Poor Requirements
Poor Functional
Architecture
CMMI® for DEV

11
Keynote Outline
That’s all folks!

12
LANGUAGE & PATTERNS
A matter of

14
How Do We (testers) speak?
• Constraints:
– Time
– Costs
– Resources
• Requirements:
– Coverage
– Test depth
Coverage
Depth
Costs, Resources
Time

16
Understand them
Speak their language
Fulfil their needs
Put yourself into their shoes

17
Keynote Outline
That’s all folks!

18
Use Patterns…
• …and make TESTABLE requirements:
– FUNCTIONAL
• As ROLE I need ACTION in order to OBJECTIVE +
context qualifyers
• As user I need to access my personal file in order to complete it
with the necessary info
– SYSTEM
• When TRIGGER the system NAME must
ACTION/PERMISSION to WHOM/WHAT in order
to OBJECTIVE + qualifyers
• When book is available, BorrowBook must allow subscribers to
borrow if subscription paid and limit not reached

20
Often (very often)…
• Absence of references
• Few measures and metrics (or none)
– Unknown coverage
– Unclear efficiency
– No capitalization
• Decisions = estimations
• Management cannot evaluate
properly

21
What Would we Want to Know?
• What’s left to do?
• Am I on track?
• What can I improve?
• What is MY quality?
• Do defects get solved?
• What did I learn?
Not measuring
= Not knowing

22
• No clear specification
• What is really needed?
– Basic or advanced functionality?
– Usability?
– Performance?
– Security?
• When do we have enough tests?
– How critical is the functionality?
– Deep or shallow test?
• How long to design, execute the tests?
• What should we test first?
• How can we draw a conclusion from the tests?
Difficulties Encountered

23
Keynote Outline
That’s all folks!

24
A TRH???
« Logic : a good tool that is sold to us with no user manual »
- Pierre Véron

25
Hierarchical tree view of application
requirements
Business process thinking
How to reach business goals
(notion of benefit)
Process-oriented
Tracing down from process to
functions
Quality reference
Structuring with a TRH
test requirement 1
test requirement 1.1
test requirement 1.2.1

26
The Fundamental Test Process
Test planning and control
Test analysis and design
Test implementation
and execution
Evaluating exit criteria
and reporting
Test closure activities
TRH used throughout
these stages

27
Unit
testing
Integration
testing
System
testing
Acceptance
testing
Requirements
Specification
Functional
Specification
Global
design
Detailed
design
Code
Business
Objectives
The TRH and the V-model
Test requirement
Business requirement
System requirement

28
The Problem with Requirements
• Requirements form the basis for testing
• Requirements are not always testable
– Often imprecise, ambiguous
– Often incomplete
– Can be contradicting or totally wrong
– Sometimes, requirements simply don’t exist
• Challenges
– make imprecise, incomplete requirements testable
– Identify wrong, missing requirements

29
How would you test a train ticket application?
A Small Example …

30
Constructing the Hierarchy
HOW?
Let’s bring
the Money In
Sell tickets
Exchange
tickets
Refund
tickets

31
Constructing the Hierarchy
HOW?
Let’s bring
the Money In
Sell tickets
Exchange
tickets
Refund
tickets
International
National Full price
Off peak
Reduced - Young
Reduced - Sr
Reduced Loyalty
HOW?

32
Asking How, Creatively
• Use alternative formulations:
– What needs to happen?
– Are there any related problems?
– What are you likely to have done previously/next?
– What needs to happen when this fails?
– How does the user know this?
– How could things be screwed up?
• Use basic test specification techniques to
stimulate responses.
– “Manage things” → CRUD
(create/read/update/delete)
– Equivalence partitioning
– Boundaries
– Negative cases
• Make sure sub-requirements cover parent 100%

33
Establish Business Context
Let’s bring
the Money In
Sell tickets
Exchange
tickets
Refund
tickets
International
National Full price
Off peak
Reduced - Young
Reduced - Sr
Reduced Loyalty
WHY?
WHY?
WHY?

34
Asking Why, Unaggressively
• The question “why?” by itself can be quite
intimidating (or even look foolish)
• Use alternative formulations:
– What is the purpose of that?
– Can you explain why you need it to do that?
– What’s the relation to your business?
– What business problem is this solving?
– What was the thinking behind that?
– What is the underlying reason for that?
– What would happen if that feature was not
available?
– …

35
How to Get There?
• Previous Knowledge & Experience
• Reading and retro-analysis
• Face-to-face meetings
• Workshops
NEVER ASSUME !!!
- Loyalty is not about calculating
& redeeming points
It is about COMMUNICATION

36
Keynote Outline
A TRH! Oh great!...erh remind me what for exactly?
That’s all folks!

37
A TRH What for?
• Prioritise
• Communicate
• Pilot
• Estimate
• Test strategy
backbone
test requirement 1
1
1
1
2
2
3
3

38
Test Implementation and Execution
TRH
Test
design
spec
Test
design
spec
Test
procedure
spec
Test
case
spec
Test
procedure
spec
Test
Plan
bugs
Baseline
planning
Progress
report

39
TRH and Test Design
TEST SPECIFICATION ID: XYZ
PURPOSE: Verify that search function
examines articles.
APPROACH REFINEMENTS: …

40
TRH and Process
Test procedure
Scenario
Data
Validation
points
Physical Design
WHAT ?
Test procedure
Scenario
Data
Validation
points
Test case1
Test case2
Test case3
Test case5
Test case4
Test Requirements Hierarchy
WHY / WHAT?
test requirement 1
Logical Design
WHAT?

41
Who’s the Pilot?…Damn’ it’s the TRH!
• Nb testable requirements (leaves)
– Identified / Developed / written
– % Coverage(s)
– % Executed
Progress (t vs. t-1 vs. planned)
– Risk / Impact (before and after execution)
– Planning / Effort

42
In Practice
• Planning - Execution
– 100 requirements to test
– 100 persons*days budget
– Writing / Execution = 60/40
• Quality
– 50% requirements coverage
– 50% passed product
– Grey zones (50 shades)?

43
Software Tools…Who Cares?
• There are (too) many of them
– From Excel to Doors via TestLink & others
– Integrated in suites (HP, IBM) or not
– OpenSource or Commercial
• Most (at least the known ones) are good
• Tools are NOT the issue

44
…Provided they Allow…
• Real-time share and follow
• Information and reporting
(collect and process)
• Decisions
• Anticipation
• Adapt the tool to YOUR organisation

45
Word Processor in Outline Mode

48
…but How to Use them Matters
• They must be robust…
– The « in-house » requirements tracker
• …fit to your needs and organisation
• …and should allow to share and pilot
– Customer / supplier « encode twice »

49
Requirements
• Total Nb & Testable >> coverage
• Repartition / severity
• Repartition / phase / version
• Repartition / priority
• Nb incorrect or maintained
requirements
• Nb scenarios / Requirement
• Nb unused requirements
• Nb added requirements
• Nb changed requirements

50
Test Scenarii – Test Designs
• Total & Testable (Nb)
• Répartition / severity
• Repartition / phase or version
• Nb scenarii reviewed and/or incorrect
• Nb scenarii / Requirement
• Nb scenarii unused
• Nb scenarii added
Measurements details
Fine but complex vs.coarse but blurry estimations
Differs from phase to phase depending on goals

51
Defects
• Total Nb
• Repartition / severity
• Repartition / status
• Repartition / responsible
• Nb found / unit time
• Nb false positive
• Nb unsolved
• Nb found next phase

52
Adjust your Strategy
You are HERE
Time
Scope
EXPECTED Deadline
EXTENDED Deadline
100%
0%
50%
1
3
2
66%
Productivity

53
Different Mitigation Options
• Option 2
– Limit Scope (shrink till you drop)
• Quality drop
• Increased risk
– Let it slip (the whoosh sound of dealines)
• Late market presence
• Extra costs
• Option 3
– Increase resources (Stakhanov – Taylor)
• Extra costs
• 2 women + 4,5 mths = NO baby

54
One Example
• Mitigating options 2 (slippage) and 3 (increase ressources)
– Use what you already know to predict and plan
• Risk oriented strategy
– Sanity check
– Flexible resources
– Planning by roles and capability
• Use existing resources
• Identify and manage activities
– Test new functionality
– Re-test (bug fix)
– Regression test (re-validation)
– …/…
• Multiply PC (automation)
– Execute based on risk / priority
• Most important first
• Most important = deeper and longer
BA

55
Estimation Model
Estimated
(found)
Fixed
Planning
Exported
(next phase)
Inherited
(previous phase)
TEST Phase n

56
Test Campaigns
1 2 3 4 5 6 7
Go / No Go
SP 2.1 SP 2.2 SP 2.3 ?
Production?
Production?
h1 h2 h3
Inherited from
SP1.x
h1 h2
Σhi
Σhi
d2 d3
Manual Tests (n) Manual tests (n+1) + automated (n)
• ! Performance
• ! Availability
• ! Stability
27/03 02/05 21/05
Inherited
from SP2.0
d1 0 ?
?
?
Correction > c1 Correction > c2
SC
21/03 18/04
8 ?
2/04
di = Nb estimated defcts / phase
ci = Nb fixed defects in the same phase
hi = Nb inherited defects
Σ’hi
Σ’hi

57
And if you Have…TIME
• Today (status time)
– Consumed vs, attended resources
– Forecast based upon experience
• Mean time (write / fix / execute)
• Mean time (code / analyse / fix)
• Application up time
• Application performance
• MTBE, MTBF
Good stop criterion…
if enough maturity

58
Ressources metrics
• Time span / person (per profile)
– # Hrs / test activity
• Preparation
• Planning
• Design
• Environment
• Execution
• Maintenance
• Fixing tests
• Reporting
• Overhead
expenditure

59
Work Product Trend
0
50
100
150
200
250
300
350
400
450
JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
planned designed executed passed

60
Do NOT Overperform

Date 27/08
Sprint 2
Release 3
# days 15/23
15
# Testers Effective Cum. % planned resources for SP2.3
Execution test designs 5,5
Check printed documents 3
Ad hoc tests 5 63%
Quotation (Product Management) 0 100%
Re-test SP1 + SP2 ad hoc defects 0 100%
Traduction 0 100%
Total resources coverage Release 3 13,5 62%
Availability - Lost time
Availability of the application that day 100,0% week 1/5 week 2/5 week 3/5 week 4/5 week 5/5
Average availability for SP2.3 85,7% 80,1% 78,6% 95,6% 85,7%
Test Weather
Report
58%

Test execution (2849 data sets)
# data sets foreseen that date 170 Found defects
# data sets tested that date 90
% data set coverage SP2.3 - against planned ds 53% OK NOK Not testable Intermediate
# data sets executed SP2.3 2148 1850 233 0 65
86,1% 10,8% 0,0% 3,0%
% data set coverage SP2.3 - against total ds 75,4% Docs waiting for check
# SP2 defects foreseen to be re-tested during SP2.3 445
# SP2 defects re-tested that date 22 OK NOK Not testable
# SP2 defects re-tested while executing ds SP2.3 211 184 26 1
87,2% 12,3% 0,5%
% re-test defects coverage 47%
Average datasets/p/d foreseen for SP2.3 19,9 *Average test design (15 ds/p/d) and check document (29 ds/p/d)
Average datasets/p/d effective that day 16,4
Average datasets/p/d effective in SP2.3 17,3
Re-test SP2 AD HOC defects
# SP2 ad hoc defects foreseen to be re-tested during SP2.3 106
# SP2 ad hoc defects foreseen that date 0 NOK + OK +
# SP2 ad hoc defects re-tested that date 0 OK NOK New defect New defect
# SP2 ad hoc defects re-tested in SP2.3 99 81 16 0 0
81,8% 16,2% 0,0% 0,0%
% SP2 ad hoc defects re-test coverage 93%
Average defects re-tested/p/d foreseen for SP2.3 10,0
Average defects re-tested/p/d effective that day -
Average datasets/p/d effective in SP2.3 -
% data sets executed SP2.3
Cumul # SP2 ad hoc defects re-tested in SP2.3
Cumul # SP2 defects re-tested while executing ds SP2.3
# Defects
# encoded defects that day 18 Urgent High Medium Low
# active defects in Defect Tracker for SP2.3 102 20 51 24 7 102
# active defects in Defect Tracker 411 29 257 110 15 411
Open IT exam Pend. TEST Sub-total Active Whole
IT Factory 65 0 21 86
Ready TLF Pend. TLF User Exam
Test Factory 2 297 26 325
Closed Deferred Archived
Not active defects 454 47 1507 2008
Test exam
0 0
Remarks:
Today :
Finalised retest of fiches linked to "marques-modèles" has been done
Application has been better regarding response time but a lot of instabilities (T000): see daily urgent defects sheet for details.
Program for tomorrow :
Same as today
2419
411
 Fitness
Program
Data set foreseen / executed
0
500
1000
1500
2000
2500
3000
5/aug
7/aug
9/aug
13/aug
19/aug
21/aug
23/aug
27/aug
29/aug
2/sep
4/sep
6/sep
Executed Cumulated DS
DS foreseen this day (intial
planning)
120
140
0
500
1000
1500
2000
2500
3000
Availability of the application
91%92%
24%
100%
93%
64%
89%
82%
86%
100%100%100%
92%
71%
100%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
5/aug
7/aug
9/aug
11/aug
13/aug
15/aug
17/aug
19/aug
21/aug
23/aug
25/aug
27/aug
29/aug
31/aug
2/sep
4/sep
6/sep
Availability
Retest defects SP1 foreseen / executed
0
20
40
60
80
100
120
140
5/aug
7/aug
9/aug
13/aug
19/aug
21/aug
23/aug
27/aug
29/aug
2/sep
4/sep
6/sep
Re-tested this daySP1 + SP2 ad
hoc
Foreseen to be retested this day
(intial planning)

61
Keynote Outline
That’s all folks!

62
Yes, Sure…However
• You are also accountable for Quality
• You can set-up and follow what everyone
understands
– Added value
– Business means
• You bring the missing link to life

63
I Can Do Everything – I’m An Expert

64
Keynote Outline
That’s all folks!

To requirements and beyond...

Recommended

Recommended

More Related Content

Similar to To requirements and beyond...

Similar to To requirements and beyond... (20)

More from SQALab

More from SQALab (20)

Recently uploaded

Recently uploaded (20)

To requirements and beyond...