The document summarizes Colin Hammond's experience with automating requirements sizing using ScopeMaster over the past few months. Some key points: 1. ScopeMaster can find 50% or more of requirements defects automatically by analyzing text for completeness, clarity, consistency and other quality attributes. This allows defects to be found and fixed early before coding begins. 2. Early defect removal can significantly improve project quality and outcomes, with estimates of avoiding thousands of dollars in rework costs per defect found. It also enables faster development by removing defects upfront. 3. Reaction to ScopeMaster's automated sizing and quality checks has been mixed depending on role, with agilists and some developers initially skeptical but QA managers and

1. First Few Months of
Automated Sizing
C o s m i c S p e c i a l I n t e r e s t G r o u p . 2 0 1 8
C o l i n H a m m o n d

2. •A problem to solve
•The birth of ScopeMaster
•Reactions
•Just the beginning
Agenda

3. Journey with automating size
measurement
IT Career
Colin Hammond
colin@albiontech.com
20+ companies
80+ projects

4. $400Bn (USA) is spent on software dev every year (Standish)
Size is the single most significant Metric
FP really do work! SP don’t
Failure remains little changed, esp on large projects (Standish)
Bug fixing is the biggest activity
16-20% defects are in the requirements (Capers Jones)
Removing bugs Early is very cost effective.
Estimates Vary wildly Agile and vendors maintain the mystique
COSMIC SIG attendees know this….but not everyone does.
Background

5. 20101990 - 2010
PM with
No metrics
Quality sells sizing, not so
2016
“Need to automate”
2016-17
LOTS of trial and
error
The “fishing”
conversation
December 2017 February 2018
“Daughter” conversation.
CFPS
PageAnalyzer
20162012-2015
ProjectSizer

6. Free form
requirements
or user stories
(import CSV)
Measure CFP estimateAnalyses the text
“Within and across the
requirements”
Quality Potential defects
Estimates Project level
Reword and refine
SAAS analyser of Software Requirements or User Stories
Version 1
What is it?
Very fast NLP
+ rules
+ search
techniques

7. Subject
Focus
Verb Object
“As a user I want to display orders”

8. Writes and runs ± 400 checks/minute
Indicative numbers.
Performance varies
depending on many factors
Clear(unambiguous)
Complete
Concise
Consistent
50% + of all
requirements defects
Automated requirements checks

9. Demo
Complete
Concise
Consistent
Clear
Correct
Current

10. 90 Stories
1 Person
16 hours
150 defects found & fixed!
Case Study

11. Size CFP
ScopeMaster Potential
requirements defects
ScopeMaster Potential
requirements Defects
per CFP
Ingredients * 300 138 0.46
Transport * 1800 1080 0.6
Oracle CR * 56 33.6 0.6
Auto generated Test
300 60 0.2
QA reviewed system * 270 218.7 0.81
Public Domain
Contracts System
872 837.12 0.96
Requirement Sets
I guarantee to find some using ScopeMaster

12. Experience to Date
Seeing Poor Quality Inputs:
Very Brief
2 word FURs
Too Verbose
100 words = 3 CFP
30 CFP / per requirement.
Inconsistent
Same OOI given three different names
within one FUR
Volume
100 sets of requirements
Around 50,000 CFP
Exposure
Mostly business applications
Very few embedded systems
Very few enhancements

13. Manager would not have
wanted to know the size,
else his project might have
been cancelled
QA Managers who have/
take no responsibility for
requirements quality
Some Negativity / Opinions
Waterfall associated

14. Reactions to Sizing & Tool
Agilists (anti-old)
QA Managers
Resistent Indifferent or Sceptical Open
Dev houses (“enjoying” ambiguity)
Mistrust FP/CFP
CFPS strongly aligned to IFPUG Open minded metrics enthusiasts
Commissioning Business
Developers
BA’s
Cosmic Enthusiasts
Estimators
PM’s*
Disbelief
Anti
Too busy
Happy
ignorant

15. Prevention
Pre-test removal
Test
95%+
Test
80% 1,000 left 250 left
Testing Quality
1000 FP Solution
Shift the conversation

16. This is a big deal!
Bad fixes
14.9%
Documents
10.9%
Security
6.0%
Architecture
2.0% Design
22.5%
Code
26.8%
Requirements
16.9%
1,000 FP Application
Source Capers Jones: 2012

17. Find
Value
Fix Rework avoided
$100 $150 $1000’s+ +

18. Requirements Inspections
Grooming find 2 defects per hour (4 people)
Fix half a day per defect (1 person)
5-6 hours effort
to find and fix a problem
5 hours
1 person can find and fix 4 defects per hour with ScopeMaster
15 minutes
to find and fix a problem
15 minutes

19. Functional Sizing - Speed
125 - 500 FP / Day 500 -2500 FP/ day
4x faster

20. The opportunity
2x Productivity
Given a day rate of $500
Productivity Benefit = circa
$50,000 p.a per BA
Global 20m Developers
10m Developers (English)
1m Business Analysts
Global benefit is $50bn p.a.
(looking only at BA
productivity)

21. Challenges
Short Term
Teaching the value of Subject Verb Object
Separating Subject Verb Object from
acceptance criteria
Explaining Cosmic E,X,R,W
Medium Term
Analysing acceptance criteria

22. $
Time
Develop
$
Profit from
being first to
market
Breakeven point
Early to Market with Strategic IT innovation

23. $
Time
Impact of delay is compound
Develop
Market
opportunity
missed by being
late
Breakeven point
Delayed

24. $
Time
Impact of delay is compound
Develop
Market
opportunity
missed by being
late
Breakeven point
Delayed
$

25. 1.Quality
1.Finds ±50% of requirements defects, before coding
2.Performs testing that might not get done
3.Quality early = less cost and shorter schedules
2.Fewer failed projects
3.Vendor Management
1.Stronger negotiation position
2.Requirements arbitration
3.Improve outsourcing contract management
4.Strategic
1.Gets your software to market faster
2.First mover advantage in any given market place
5.Measurement
1.Promotes FSM adoption, which is good practice
2.Measure faster than manually
3.Measure properly without learning the boring bit
6.Learning / Improvement
1.Encourages/teaches people to write good user stories
Benefits

26. Doing what
to what
get this right first
“As a user I need to update personnel details so that …bla, bla”
Story Quality, what is good?
“Update requirements”
“Maintain calendar entries”
“As a user I want toDisplay orders”
“Delete invoice”
“Search for companies”
“Book a room”

27. •Requirements Quality
•Size (COSMIC by stealth)
Now Future
• UX
• Requirements suggestions engine (re-use, security)
• Test suggestions
• Custom ontology
• NFRs
• More ML
• Multilingual
• IFPUG
• Tighter JIRA etc integration
• Machine Learning
• Local Benchmarking
• Consultancy toolkit
• Requirements Management
• Industry Benchmarking
• Levelling the playing field
• BDD
• Requirements -> Code
• Speech -> Code

28. Thanks!
Colin Hammond. colin@albiontech.com

29. B E H AV I O U R D R I V E N D E V E L O P M E N T - B D D
1: Feature: Some terse yet descriptive text of what is desired
2: Textual description of the business value of this feature
3: Business rules that govern the scope of the feature
4: Any additional information that will make the feature easier to
understand
5:
6: Scenario: Some determinable business situation
7: Given some precondition
8: And some other precondition
9: When some action by the actor
10: And some other action
11: And yet another action
12: Then some testable outcome is achieved
13: And something else we can check happens too
14:
15: Scenario: A different situation
Gherkin Syntax

30. Find
defects
early
Does QA work that often
doesn’t happen

31. Clear(unambiguous)
Complete
Concise
Consistent
Correct
Current
Requirements Quality