Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Measurement Process: Improving the ISO 15939 Standard

912 views

Published on

Over the past few years ISO has published a number of specific standards detailing processes included in a generic form in software development life cycle models. ISO 15939 on the Measurement process itself is an example of such specific ISO standard. This paper presents some suggestions for improvements to its Measurement Information Model and to the measurement plan within the planning process of ISO 15939.

Published in: Software
  • Be the first to comment

Measurement Process: Improving the ISO 15939 Standard

  1. 1. 24° International Workshop on Software Measurement (IWSM) and 9th International Conference on Software Process and Product Measurement (MENSURA) Rotterdam (Netherlands) - October 6-8, 2014 Measurement Process: Improving the ISO 15939 Standard www.eng.it Luigi BBuugglliioonnee Alain AAbbrraann
  2. 2. Measurement Process Goals of the presentation 1. Discuss the value arising from a Measurement Plan 2. Introduce suggestions for a refined Measurement Information Model (MIM) 3. Propose a recommended content for a Measurement Plan 2 IWSM-MENSURA 2014 – October 6-8, 2014 www.eng.it © 2014 Luigi Buglione Alain Abran
  3. 3. ETS - GELOG At a glance 3 IWSM-MENSURA 2014 – October 6-8, 2014 www.eng.it © 2014 Luigi Buglione Alain Abran ggeelologg.e.ettssmmttl.lc.caa
  4. 4. Engineering At a glance 4 IWSM-MENSURA 2014 – October 6-8, 2014 www.eng.it © 2014 Luigi Buglione Alain Abran wwwwww.e.enngg.i.tit
  5. 5. Agenda Measurement Process 5 IWSM-MENSURA 2014 – October 6-8, 2014 www.eng.it © 2014 Luigi Buglione Alain Abran • Introduction – Three steps back… – Some basic questions… • ISO 15939 – Measurement Process – Context – What is included in the current version (2007) – What could be included in a next version • Improving the MIM – Refining and Improving the MIM • Measurement Plan – The ‘5Ws + 2H’ proposal – The BMP technique • Summary Next Steps • QA
  6. 6. Introduction Three Steps back... YYoouu ccaannnnoott ccoonnttrrooll wwhhaatt yyoouu ccaannnnoott mmeeaassuurree bbuutt...... ......YYoouu ccaannnnoott mmeeaassuurree wwhhaatt yyoouu ccaannnnoott ddeeffiinnee bbuutt...... ...YYoouu ccaannnnoott ddeeffiinnee wwhhaatt yyoouu ddoonn’’tt kknnooww... 6 IWSM-MENSURA 2014 – October 6-8, 2014 www.eng.it © 2014 Luigi Buglione Alain Abran
  7. 7. Introduction Some basic questions.... When measuring in your organization, do you have a ‘measurement plan’ or not? Do you properly define your measures with needed details for a precise measurement or are they simply listed? E.g. What is a LOC? When establishing your set of measures, do you start from your own goal or from a predefined list (e.g. PSM) or are you goal-oriented? Do you balance your measures by multiple criteria or are you mostly focusing on time and costs? If yes, why not also other perspectives (e.g. quality, risk, etc.)? 7 IWSM-MENSURA 2014 – October 6-8, 2014 www.eng.it © 2014 Luigi Buglione Alain Abran
  8. 8. Agenda Measurement Process 8 IWSM-MENSURA 2014 – October 6-8, 2014 www.eng.it © 2014 Luigi Buglione Alain Abran • Introduction – Three steps back… – Some basic questions… • ISO 15939 – Measurement Process – Context – What is included in the current version (2007) – What could be included in a next version • Improving the MIM – Refining and Improving the MIM • Measurement Plan – The ‘5Ws + 2H’ proposal – The BMP technique • Summary Next Steps • QA
  9. 9. ISO/IEC 15939 Context • Specific ISO standard on Measurement 9 IWSM-MENSURA 2014 – October 6-8, 2014 www.eng.it © 2014 Luigi Buglione Alain Abran 1° Ed: 1999, 2° Ed: 2007 • ‘Explodes’ the ones in ISO 12207, 15288 or 15504 process models • Adopt the VIM (Int. Vocabulary of Metrologogy) terms and glossary
  10. 10. ISO/IEC 15939 What is included in the current version SSoommee ssttrreennggtthhss...... • Information Needs Products Introduced a MIM (Measurement 10 IWSM-MENSURA 2014 – October 6-8, 2014 www.eng.it © 2014 Luigi Buglione Alain Abran Information Model) MIM structures a series of measurement-related concepts Several levels: from the ‘information needs’ till measurable attributes • Metrology Alignment Partial alignment with language/concepts proposed in Metrology Breakout in the Software Engineering domain E.g. VIM (International Vocabulary of Metrology) Measurement Experience Base (MEB) A good estimation could be done using your own historical data MEB is the organizational database data from activities Present from the first 1999 edition of the standard Fundamental for re-establishing and tuning threshold values from data
  11. 11. ISO/IEC 15939 What could be included in a next version SSoommee iimmpprroovveemmeenntt ppooiinnttss...... 11 IWSM-MENSURA 2014 – October 6-8, 2014 www.eng.it © 2014 Luigi Buglione Alain Abran • MIM structure Now: The measurement concepts included in the ISO 15939 standard are not fully included in the current MIM graphical representation Tomorrow: MIM picture could be split into ‘phases’ and specify better roles and moments in time during the life cycle for detecting issues for improvements • Clause 5.2 - ‘Plan the measurement process’ Now: single key measurement tasks may be collected into a measurement plan Tomorrow: could refer to such coordination but using proper criteria for obtaining a valuable measurement plan as the combination of multiple viewpoints/perspectives (stakeholders) introduce formally the ‘measurement plan’ definition and concept in the normative part (currently ‘weak’ §5.2.6.1 + Annex F) Clause 4.2.3 – ‘Select measures’ Now: asks for criteria to select measures, but with no suggestions about ‘how’ Tomorrow: insert explicitly possible criteria (e.g. balancing criteria/number of measures by perspectives, overall cost, information value, etc.)
  12. 12. Agenda Measurement Process 12 IWSM-MENSURA 2014 – October 6-8, 2014 www.eng.it © 2014 Luigi Buglione Alain Abran • Introduction – Three steps back… – Some basic questions… • ISO 15939 – Measurement Process – Context – What is included in the current version (2007) – What could be included in a next version • Improving the MIM – Refining the MIM – Improving the MIM • Measurement Plan – The ‘5Ws + 2H’ proposal – The BMP technique • Summary Next Steps • QA
  13. 13. Improving the MIM Refining the MIM – Elements Structure • Phases Three main phases: Data Analysis, Data Preparation, Data Collection Each phase will include some of the activities of the current MIM • Use of measurement results Analysis of relationships across different measurement results Assessment against reference contexts for evaluation and/or decision-making ‘Hidden’ details to be elicited and made visible Decision criteria Assumptions Expected relationships Estimates or evaluation Numerical thresholds or targets Statistical confidence limits ... 13 IWSM-MENSURA 2014 – October 6-8, 2014 www.eng.it © 2014 Luigi Buglione Alain Abran
  14. 14. Improving the MIM Improving the MIM – Additional concepts • Standard Reference Model Generic/Accepted model of Info Needs It’d include: 14 IWSM-MENSURA 2014 – October 6-8, 2014 www.eng.it © 2014 Luigi Buglione Alain Abran Set of individual relationships, jointly with measures to be considered (e.g. evaluation and/or decision criteria) Algorithm combining measures in implied criteria • Organizational Reference Context Aligned with the Standard Reference Model It’d include: Decision criteria (w/target values, evaluation scales) Reference values according to contexts (benchmarking domains, application types, etc. – www.isbsg.org) Data Analysis As currently defined but... It’d be better feeded by data/information coming from the Standard Model and the Organizational Reference Context
  15. 15. Agenda Measurement Process 15 IWSM-MENSURA 2014 – October 6-8, 2014 www.eng.it © 2014 Luigi Buglione Alain Abran • Introduction – Three steps back… – Some basic questions… • ISO 15939 – Measurement Process – Context – What is included in the current version (2007) – What could be included in a next version • Improving the MIM – Refining the MIM – Improving the MIM • Measurement Plan – The ‘5Ws + 2H’ proposal – The BMP technique • Summary Next Steps • QA
  16. 16. Measurement Plan The ‘5Ws+2H’ proposal • The original ‘5W+H’ journalist rule Typically a good paper should contain: Who, What, Why, When, Where, How Who: not well expressed now RACI matrix for describing who should collect, prepare, and analyze data • The 2° H – a possible addition ‘How Much’ threshold(s) value(s) As in Control Charts (TQM), possibly two (2) thresholds: UCL (Upper Control Level) LCL (Lower Control Level) Threshold(s) should be periodically (re)evaluated from 16 IWSM-MENSURA 2014 – October 6-8, 2014 www.eng.it © 2014 Luigi Buglione Alain Abran MEB data • Utility and applicability Same suggestion for a good (whatever) plan It could be a possible refinement for the current Appendix F (‘Example elements of measurement planning’) In such way it could be possible to check easily with a structured manner what has/hasn’t included in the measurement plan Not necessarily the ‘measurement plan’ should be part of a project/quality plan, it could/should be a separate plan for including a proper level of detail
  17. 17. Measurement Plan Balancing Multiple Perspectives (BMP) • Not too few, not too many... A common-sense rule, always valid in the ‘real life’, less respected in the business world ‘sustainability’ Not forgetting that ‘you cannot control what you cannot measure’ ... what is the right number of measures? ...and what is the right number of viewpoints to consider? Goal: optimize the TCO, based on a positive ROI/VOI • How much does it cost to measure? Measurement IS NOT part of Project Management E.g. CMMI MEA, ISO/IEC 15504-2 MAN.5, etc... Did you forecast the proper budget on Measurement? • How many viewpoints to consider at the same time? Stakeholders’ analysis and involvement is fundamental to success Are we looking to the right stakeholders? Are we achieving ‘value’ from our product/services? ...we need to properly mix all needed elements in our ‘Measurement Plan’ in order to be successful but ‘balanced’ Don’t forget to analyze ‘counter-effects’ (pros cons) 17 IWSM-MENSURA 2014 – October 6-8, 2014 www.eng.it © 2014 Luigi Buglione Alain Abran
  18. 18. BMP: Balancing Multiple Perspectives • Balancing Multiple Perspectives (BMP) Multi-perspective approach proposed in 2005 www.semq.eu/leng/modtechbmp.htm 18 IWSM-MENSURA 2014 – October 6-8, 2014 www.eng.it © 2014 Luigi Buglione Alain Abran Measurement Plan • Measurement procedure: Determine the dimensions of interest in the project. Determine the list of the most representative measures associated with each dimension. For each of the measures selected, identify which other control variables might be impacted negatively. Figure out the best combination of indicators and the causal relations between them in order to build a measurement plan for the project.
  19. 19. BMP: Causal Relationships • Not sufficient to perform steps#1 and #2 (design a measurement plan) • Next (required) step: establish coherent and proper relationships among goals through measures (i.e. the BSC strategic map) for achieving both single perspective goals, as well as overall organizational ones 19 IWSM-MENSURA 2014 – October 6-8, 2014 www.eng.it © 2014 Luigi Buglione Alain Abran NNBB: also explicitly required by ISO 9001:2008 (§4.1 lett.c) Measurement Plan
  20. 20. Agenda Measurement Process 20 IWSM-MENSURA 2014 – October 6-8, 2014 www.eng.it © 2014 Luigi Buglione Alain Abran • Introduction – Three steps back… – Some basic questions… • ISO 15939 – Measurement Process – Context – What is included in the current version (2007) – What could be included in a next version • Improving the MIM – Refining the MIM – Improving the MIM • Measurement Plan – The ‘5Ws + 2H’ proposal – The BMP technique • Summary Next Steps • QA
  21. 21. Summary Next Steps Measur. Framework • Measurement in the practice Measurement is one of the most needed and relevant processes, often under-evaluated It helps to manage a project but is separated from the Project Management process Measuring helps in determining the ‘how much’ we need to produce for better estimates time (effort/duration) and costs/prices, reducing more and more subjective views and errors • ISO/IEC 15939 – up today ISO standard on the Measurement Process Aligned with Metrology rules and standards (e.g. VIM, GUM) First edition (1999), second (current) edition: 2007 A third edition is on the way...some possible improvements suggested • ISO/IEC 15939 – suggestions for tomorrow MIM – refining the structure (phases), introducing roles (RACI) and thresholds (from MEB) Measurement plan – introduce a template, applying the 5W’s+2H rules and BMP technique All models are wrong. Some models are useful. (GGGGeeeeoooorrrrggggeeee BBBBooooxxxx,,,, MMMMaaaatttthhhheeeemmmmaaaattttiiiicccciiiiaaaannnn,,,, 1111999911119999-2222000011113333) 21 IWSM-MENSURA 2014 – October 6-8, 2014 www.eng.it © 2014 Luigi Buglione Alain Abran Next Steps Refine this proposal into more operative terms for possible introduction into next ISO 15939 revision process ...let’s start measure introducing these suggestions! All models are wrong. Some models are useful. (GGGGeeeeoooorrrrggggeeee BBBBooooxxxx,,,, MMMMaaaatttthhhheeeemmmmaaaattttiiiicccciiiiaaaannnn,,,, 1111999911119999-2222000011113333)
  22. 22. Measurement Process Lessons Learned... 22 IWSM-MENSURA 2014 – October 6-8, 2014 www.eng.it © 2014 Luigi Buglione Alain Abran
  23. 23. Q A Measurement Process BBeeddaannkktt vvoooorr uuww aaaannddaacchhtt!! TThhaannkkss ffoorr yyoouurr aatttteennttiioonn!! 23 IWSM-MENSURA 2014 – October 6-8, 2014 www.eng.it © 2014 Luigi Buglione Alain Abran
  24. 24. Our Contact Data Luigi Buglione Engineering Ingegneria Informatica/ETS luigi.buglione@eng.it 24 IWSM-MENSURA 2014 – October 6-8, 2014 www.eng.it © 2014 Luigi Buglione Alain Abran Alain Abran ETS Montréal alain.abran@etsmtl.ca Measurement Process

×