Measurement Process: Improving the ISO 15939 Standard

24° International Workshop on Software Measurement
(IWSM) and 9th International Conference on Software
Process and Product Measurement (MENSURA)
Rotterdam (Netherlands) - October 6-8, 2014
Measurement Process: Improving the ISO 15939 Standard
www.eng.it
Luigi BBuugglliioonnee
Alain AAbbrraann

Measurement Process Goals of the presentation
1. Discuss the value arising from a Measurement Plan
2. Introduce suggestions for a refined Measurement
Information Model (MIM)
3. Propose a recommended content for a Measurement
Plan
2 IWSM-MENSURA 2014 – October 6-8, 2014 www.eng.it
© 2014 Luigi Buglione Alain Abran

ETS - GELOG At a glance
ggeelologg.e.ettssmmttl.lc.caa

Engineering At a glance
wwwwww.e.enngg.i.tit

Agenda
Measurement Process
• Introduction
– Three steps back…
– Some basic questions…
• ISO 15939 – Measurement Process
– Context
– What is included in the current version (2007)
– What could be included in a next version
• Improving the MIM
– Refining and Improving the MIM
• Measurement Plan
– The ‘5Ws + 2H’ proposal
– The BMP technique
• Summary Next Steps
• QA

Introduction Three Steps back...
YYoouu ccaannnnoott ccoonnttrrooll wwhhaatt yyoouu
ccaannnnoott mmeeaassuurree bbuutt......
......YYoouu ccaannnnoott mmeeaassuurree wwhhaatt
yyoouu ccaannnnoott ddeeffiinnee bbuutt......
...YYoouu ccaannnnoott ddeeffiinnee wwhhaatt yyoouu
ddoonn’’tt kknnooww...

Introduction Some basic questions....
When measuring in your organization, do you have a
‘measurement plan’ or not?
Do you properly define your measures with needed details for a precise
measurement or are they simply listed? E.g. What is a LOC?
When establishing your set of measures, do you start from your own
goal or from a predefined list (e.g. PSM) or are you goal-oriented?
Do you balance your measures by multiple criteria or are you mostly
focusing on time and costs? If yes, why not also other perspectives
(e.g. quality, risk, etc.)?

Agenda
Measurement Process
• Introduction
– Context
– Refining and Improving the MIM
• QA

ISO/IEC 15939 Context
• Specific ISO standard on Measurement
1° Ed: 1999, 2° Ed: 2007
• ‘Explodes’ the ones in ISO 12207, 15288 or 15504 process models
• Adopt the VIM (Int. Vocabulary of Metrologogy) terms and
glossary

ISO/IEC 15939 What is included in the current version
SSoommee ssttrreennggtthhss......
• Information Needs Products
Introduced a MIM (Measurement
Information Model)
MIM structures a series of measurement-related
concepts
Several levels: from the ‘information
needs’ till measurable attributes
• Metrology Alignment
Partial alignment with language/concepts
proposed in Metrology
Breakout in the Software Engineering
domain
E.g. VIM (International Vocabulary of
Metrology)
Measurement Experience Base
(MEB)
A good estimation could be done using
your own historical data
MEB is the organizational database data
from activities
Present from the first 1999 edition of the
standard
Fundamental for re-establishing and
tuning threshold values from data

ISO/IEC 15939 What could be included in a next version
SSoommee iimmpprroovveemmeenntt ppooiinnttss......
• MIM structure
Now: The measurement concepts included in the ISO 15939 standard are not
fully included in the current MIM graphical representation
Tomorrow: MIM picture could be split into ‘phases’ and specify better roles
and moments in time during the life cycle for detecting issues for
improvements
• Clause 5.2 - ‘Plan the measurement process’
Now: single key measurement tasks may be collected into a measurement
plan
Tomorrow: could refer to such coordination but using proper criteria for
obtaining a valuable measurement plan as the combination of multiple
viewpoints/perspectives (stakeholders) introduce formally the ‘measurement
plan’ definition and concept in the normative part (currently ‘weak’ §5.2.6.1
+ Annex F)
Clause 4.2.3 – ‘Select measures’
Now: asks for criteria to select measures, but with no suggestions about ‘how’
Tomorrow: insert explicitly possible criteria (e.g. balancing criteria/number of
measures by perspectives, overall cost, information value, etc.)

Agenda
Measurement Process
• Introduction
– Context
– Refining the MIM
– Improving the MIM
• QA

Improving the MIM Refining the MIM – Elements Structure
• Phases
Three main phases: Data Analysis, Data
Preparation, Data Collection
Each phase will include some of the
activities of the current MIM
• Use of measurement results
Analysis of relationships across different
measurement results
Assessment against reference contexts
for evaluation and/or decision-making
‘Hidden’ details to be elicited and
made visible
Decision criteria
Assumptions
Expected relationships
Estimates or evaluation
Numerical thresholds or targets
Statistical confidence limits
...

Improving the MIM Improving the MIM – Additional concepts
• Standard Reference Model
Generic/Accepted model of Info Needs
It’d include:
Set of individual relationships, jointly with
measures to be considered (e.g.
evaluation and/or decision criteria)
Algorithm combining measures in implied
criteria
• Organizational Reference Context
Aligned with the Standard Reference
Model
It’d include:
Decision criteria (w/target values,
evaluation scales)
Reference values according to contexts
(benchmarking domains, application
types, etc. – www.isbsg.org)
Data Analysis
As currently defined but...
It’d be better feeded by data/information
coming from the Standard Model and the
Organizational Reference Context

Agenda
Measurement Process
• Introduction
– Context
• QA

Measurement Plan The ‘5Ws+2H’ proposal
• The original ‘5W+H’ journalist rule
Typically a good paper should contain:
Who, What, Why, When, Where, How
Who: not well expressed now RACI matrix for
describing who should collect, prepare, and analyze data
• The 2° H – a possible addition
‘How Much’ threshold(s) value(s)
As in Control Charts (TQM), possibly two (2) thresholds:
UCL (Upper Control Level)
LCL (Lower Control Level)
Threshold(s) should be periodically (re)evaluated from
MEB data
• Utility and applicability
Same suggestion for a good (whatever) plan
It could be a possible refinement for the current Appendix
F (‘Example elements of measurement planning’)
In such way it could be possible to check easily with a
structured manner what has/hasn’t included in the
measurement plan
Not necessarily the ‘measurement plan’ should be part of a
project/quality plan, it could/should be a separate plan for
including a proper level of detail

Measurement Plan Balancing Multiple Perspectives (BMP)
• Not too few, not too many...
A common-sense rule, always valid in the ‘real life’, less respected
in the business world ‘sustainability’
Not forgetting that ‘you cannot control what you cannot measure’
... what is the right number of measures?
...and what is the right number of viewpoints to consider?
Goal: optimize the TCO, based on a positive ROI/VOI
• How much does it cost to measure?
Measurement IS NOT part of Project Management
E.g. CMMI MEA, ISO/IEC 15504-2 MAN.5, etc...
Did you forecast the proper budget on Measurement?
• How many viewpoints to consider at the same time?
Stakeholders’ analysis and involvement is fundamental to success
Are we looking to the right stakeholders?
Are we achieving ‘value’ from our product/services?
...we need to properly mix all needed elements in our
‘Measurement Plan’ in order to be successful but ‘balanced’
Don’t forget to analyze ‘counter-effects’ (pros cons)

BMP: Balancing Multiple Perspectives
• Balancing Multiple Perspectives (BMP)
Multi-perspective approach proposed in 2005
www.semq.eu/leng/modtechbmp.htm
Measurement Plan
• Measurement procedure:
Determine the dimensions of interest in the
project.
Determine the list of the most
representative measures associated with
each dimension.
For each of the measures selected, identify
which other control variables might be
impacted negatively.
Figure out the best combination of
indicators and the causal relations between
them in order to build a measurement plan
for the project.

BMP: Causal Relationships
• Not sufficient to perform steps#1 and #2 (design a measurement plan)
• Next (required) step: establish coherent and proper relationships among
goals through measures (i.e. the BSC strategic map) for achieving both
single perspective goals, as well as overall organizational ones
NNBB: also explicitly
required by ISO
9001:2008
(§4.1 lett.c)
Measurement Plan

Agenda
Measurement Process
• Introduction
– Context
• QA

Summary Next Steps
Measur. Framework
• Measurement in the practice
Measurement is one of the most needed and relevant processes, often under-evaluated
It helps to manage a project but is separated from the Project Management process
Measuring helps in determining the ‘how much’ we need to produce for better estimates
time (effort/duration) and costs/prices, reducing more and more subjective views and errors
• ISO/IEC 15939 – up today
ISO standard on the Measurement Process
Aligned with Metrology rules and standards (e.g. VIM, GUM)
First edition (1999), second (current) edition: 2007
A third edition is on the way...some possible improvements suggested
• ISO/IEC 15939 – suggestions for tomorrow
MIM – refining the structure (phases), introducing roles (RACI) and thresholds (from MEB)
Measurement plan – introduce a template, applying the 5W’s+2H rules and BMP technique
All models are wrong. Some models
are useful.
(GGGGeeeeoooorrrrggggeeee BBBBooooxxxx,,,, MMMMaaaatttthhhheeeemmmmaaaattttiiiicccciiiiaaaannnn,,,, 1111999911119999-2222000011113333)
Next Steps
Refine this proposal into more operative terms for possible introduction into next ISO 15939
revision process
...let’s start measure introducing these suggestions!
All models are wrong. Some models
are useful.
(GGGGeeeeoooorrrrggggeeee BBBBooooxxxx,,,, MMMMaaaatttthhhheeeemmmmaaaattttiiiicccciiiiaaaannnn,,,, 1111999911119999-2222000011113333)

Measurement Process Lessons Learned...

Q A
Measurement Process
BBeeddaannkktt vvoooorr uuww aaaannddaacchhtt!!
TThhaannkkss ffoorr yyoouurr aatttteennttiioonn!!

Our Contact Data
Luigi
Buglione
Engineering Ingegneria Informatica/ETS
luigi.buglione@eng.it
Alain
Abran
ETS Montréal
alain.abran@etsmtl.ca
Measurement Process

Measurement Process: Improving the ISO 15939 Standard

More Related Content

What's hot

Viewers also liked

Similar to Measurement Process: Improving the ISO 15939 Standard

More from Luigi Buglione

Recently uploaded

Measurement Process: Improving the ISO 15939 Standard