Iwsm2014 measurement process - improving the iso 15939 standard (luigi buglione & alain abran)

24° International Workshop on Software
Measurement (IWSM) and 9th International
Conference on Software Process and Product
Measurement Process: Improving the ISO 15939 Standard
www.eng.it
Measurement (MENSURA)
Rotterdam (Netherlands) - October 6-8, 2014
Luigi BBuugglliioonnee
Alain AAbbrraann

Measurement Process Goals of the presentation
1. Discuss the value arising from a Measurement Plan
2. Introduce suggestions for a refined Measurement
Information Model (MIM)
3. Propose a recommended content for a Measurement
Plan
2 IWSM-MENSURA 2014 – October 6-8, 2014 www.eng.it
© 2014 Luigi Buglione & Alain Abran

ETS - GELOG At a glance
ggeelologg..eettssmmttl.l.ccaa

Engineering At a glance
wwwwww..eenngg..itit

Agenda
Measurement Process
• Introduction
– Three steps back…
– Some basic questions…
• ISO 15939 – Measurement Process
– Context
– What is included in the current version (2007)
– What could be included in a next version
• Improving the MIM
– Refining and Improving the MIM
• Measurement Plan
– The ‘5Ws + 2H’ proposal
– The BMP technique
• Summary & Next Steps
• Q&&A

Introduction Three Steps back...
YYoouu ccaannnnoott ccoonnttrrooll wwhhaatt yyoouu
ccaannnnoott mmeeaassuurree bbuutt......
......YYoouu ccaannnnoott mmeeaassuurree wwhhaatt
yyoouu ccaannnnoott ddeeffiinnee bbuutt......
......YYoouu ccaannnnoott ddeeffiinnee wwhhaatt yyoouu
ddoonn’’tt kknnooww......

Introduction Some basic questions....
When measuring in your organization, do you have a ‘measurement
plan’ or not?
Do you properly define your measures with needed details for a precise
measurement or are they simply listed? E.g. What is a LOC?
When establishing your set of measures, do you start from your own
goal or from a predefined list (e.g. PSM) or are you goal-oriented?
Do you balance your measures by multiple criteria or are you mostly
focusing on time and costs? If yes, why not also other perspectives
(e.g. quality, risk, etc.)?

Agenda
Measurement Process
• Introduction
– Context
– Refining and Improving the MIM
• Q&&A

ISO/IEC 15939 Context
• Specific ISO standard on Measurement
 1° Ed: 1999, 2° Ed: 2007
• ‘Explodes’ the ones in ISO 12207, 15288 or 15504 process models
• Adopt the VIM (Int. Vocabulary of Metrologogy) terms and glossary

ISO/IEC 15939 What is included in the current version
SSoommee ssttrreennggtthhss......
• Information Needs & Products
 Introduced a MIM (Measurement Information
Model)
 MIM structures a series of measurement-related
concepts
 Several levels: from the ‘information needs’
till measurable attributes
• Metrology Alignment
 Partial alignment with language/concepts
proposed in Metrology
 Breakout in the Software Engineering domain
 E.g. VIM (International Vocabulary of
Metrology)
 Measurement Experience Base
(MEB)
 A good estimation could be done using your
own historical data
 MEB is the organizational database data
from activities
 Present from the first 1999 edition of the
standard
 Fundamental for re-establishing and tuning
threshold values from data

ISO/IEC 15939 What could be included in a next version
SSoommee iimmpprroovveemmeenntt ppooiinnttss......
• MIM structure
 Now: The measurement concepts included in the ISO 15939 standard are not
fully included in the current MIM graphical representation
 Tomorrow: MIM picture could be split into ‘phases’ and specify better roles
and moments in time during the life cycle for detecting issues for
improvements
• Clause 5.2 - ‘Plan the measurement process’
 Now: single key measurement tasks may be collected into a measurement
plan
 Tomorrow: could refer to such coordination but using proper criteria for
obtaining a valuable measurement plan as the combination of multiple
viewpoints/perspectives (stakeholders)  introduce formally the ‘measurement
plan’ definition and concept in the normative part (currently ‘weak’  §5.2.6.1
+ Annex F)
 Clause 4.2.3 – ‘Select measures’
 Now: asks for criteria to select measures, but with no suggestions about ‘how’
 Tomorrow: insert explicitly possible criteria (e.g. balancing criteria/number of
measures by perspectives, overall cost, information value, etc.)

Agenda
Measurement Process
• Introduction
– Context
– Refining the MIM
– Improving the MIM
• Q&&A

Improving the MIM Refining the MIM – Elements & Structure
• Phases
 Three main phases: Data Analysis, Data
Preparation, Data Collection
 Each phase will include some of the activities
of the current MIM
• Use of measurement results
 Analysis of relationships across different
measurement results
 Assessment against reference contexts for
evaluation and/or decision-making
 ‘Hidden’ details to be elicited and
made visible
 Decision criteria
 Assumptions
 Expected relationships
 Estimates or evaluation
 Numerical thresholds or targets
 Statistical confidence limits
 ...

Improving the MIM Improving the MIM – Additional concepts
• Standard Reference Model
 Generic/Accepted model of Info Needs
 It’d include:
 Set of individual relationships, jointly with measures to
be considered (e.g. evaluation and/or decision criteria)
 Algorithm combining measures in implied criteria
• Organizational Reference Context
 Aligned with the Standard Reference Model
 It’d include:
 Decision criteria (w/target values, evaluation scales)
 Reference values according to contexts
(benchmarking domains, application types, etc. –
www.isbsg.org)
 Data Analysis
 As currently defined but...
 It’d be better feeded by data/information coming from
the Standard Model and the Organizational Reference
Context

Agenda
Measurement Process
• Introduction
– Context
• Q&&A

Measurement Plan The ‘5Ws+2H’ proposal
• The original ‘5W+H’ journalist rule
 Typically a good paper should contain:
 Who, What, Why, When, Where, How
 Who: not well expressed now  RACI matrix for describing who should
collect, prepare, and analyze data
• The 2° H – a possible addition
 ‘How Much’  threshold(s) value(s)
 As in Control Charts (TQM), possibly two (2) thresholds:
 UCL (Upper Control Level)
 LCL (Lower Control Level)
 Threshold(s) should be periodically (re)evaluated from MEB data
• Utility and applicability
 Same suggestion for a good (whatever) plan
 It could be a possible refinement for the current Appendix F
(‘Example elements of measurement planning’)
 In such way it could be possible to check easily with a structured
manner what has/hasn’t included in the measurement plan
 Not necessarily the ‘measurement plan’ should be part of a
project/quality plan, it could/should be a separate plan for
including a proper level of detail

Measurement Plan Balancing Multiple Perspectives (BMP)
• Not too few, not too many...
 A common-sense rule, always valid in the ‘real life’, less respected in the
business world  ‘sustainability’
 Not forgetting that ‘you cannot control what you cannot measure’
 ... what is the right number of measures?
 ...and what is the right number of viewpoints to consider?
 Goal: optimize the TCO, based on a positive ROI/VOI
• How much does it cost to measure?
 Measurement IS NOT part of Project Management
 E.g. CMMI MEA, ISO/IEC 15504-2 MAN.5, etc...
 Did you forecast the proper budget on Measurement?
• How many viewpoints to consider at the same time?
 Stakeholders’ analysis and involvement is fundamental to success
 Are we looking to the right stakeholders?
 Are we achieving ‘value’ from our product/services?
 ...we need to properly mix all needed elements in our ‘Measurement Plan’ in order to be successful but
‘balanced’
 Don’t forget to analyze ‘counter-effects’ (pros & cons)

BMP: Balancing Multiple Perspectives
• Balancing Multiple Perspectives (BMP)
 Multi-perspective approach proposed in 2005
 www.semq.eu/leng/modtechbmp.htm
Measurement Plan
• Measurement procedure:
 Determine the dimensions of interest in the
project.
 Determine the list of the most
representative measures associated with
each dimension.
 For each of the measures selected, identify
which other control variables might be
impacted negatively.
 Figure out the best combination of
indicators and the causal relations between
them in order to build a measurement plan
for the project.

BMP: Causal Relationships
• Not sufficient to perform steps#1 and #2 (design a measurement plan)
• Next (required) step: establish coherent and proper relationships
among goals through measures (i.e. the BSC strategic map) for
achieving both single perspective goals, as well as overall organizational
ones
NNBB: also explicitly
required by ISO
9001:2008 (§4.1
lett.c)
Measurement Plan

Agenda
Measurement Process
• Introduction
– Context
• Q&&A

Summary && Next Steps
Measur. Framework
• Measurement in the practice
 Measurement is one of the most needed and relevant processes, often under-evaluated
 It helps to manage a project but is separated from the Project Management process
 Measuring helps in determining the ‘how much’ we need to produce for better estimates
time (effort/duration) and costs/prices, reducing more and more subjective views and errors
• ISO/IEC 15939 – up today
 ISO standard on the Measurement Process
 Aligned with Metrology rules and standards (e.g. VIM, GUM)
 First edition (1999), second (current) edition: 2007
 A third edition is on the way...some possible improvements suggested
• ISO/IEC 15939 – suggestions for tomorrow
 MIM – refining the structure (phases), introducing roles (RACI) and thresholds (from MEB)
 Measurement plan – introduce a template, applying the 5W’s+2H rules and BMP technique
All models are wrong. Some models are useful.
(George Box, Mathematician , 1919-2013)
 Next Steps
 Refine this proposal into more operative terms for possible introduction into next ISO 15939
revision process
 ...let’s start measure introducing these suggestions!
All models are wrong. Some models are useful.
(George Box, Mathematician , 1919-2013)

Measurement Process Lessons Learned...

Q && A
Measurement Process
BBeeddaannkktt vvoooorr uuww aaaannddaacchhtt!!
TThhaannkkss ffoorr yyoouurr aatttteennttiioonn!!

Our Contact Data
Luigi
Buglione
Engineering Ingegneria
Informatica/ETS
luigi.buglione@eng.it
Alain
Abran
ETS Montréal
alain.abran@etsmtl.ca
Measurement Process

Iwsm2014 measurement process - improving the iso 15939 standard (luigi buglione & alain abran)

More Related Content

Viewers also liked

Similar to Iwsm2014 measurement process - improving the iso 15939 standard (luigi buglione & alain abran)

More from Nesma

Recently uploaded

Iwsm2014 measurement process - improving the iso 15939 standard (luigi buglione & alain abran)