Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Pitfalls and Countermeasures in Software Quality Measurements and Evaluations

206 views

Published on

Hironori Washizaki, "Pitfalls and Countermeasures in Software Quality Measurements and Evaluations," 5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ), Keynote, Nanjing, Dec 4, 2017

Published in: Software
  • Be the first to comment

Pitfalls and Countermeasures in Software Quality Measurements and Evaluations

  1. 1. Pitfalls and Countermeasures in Software Quality Measurements and Evaluations Hironori Washizaki washizaki@waseda.jp Prof., Global Software Engineering Laboratory, Waseda University Visiting Prof., National Institute of Informatics Director, SYSTEM INFORMATION CO., LTD. Vice-Chair, IEEE CS Japan Chapter Chair, SEMAT Japan Chapter Convenor, ISO/IEC/JTC1/SC7/WG20 PC Co-Chair, APSEC 2018 Nagoya! 5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ) Nanjing, Dec 4, 2017 H. Washizaki, Pitfalls and Countermeasures in Software Quality Measurements and Evaluations, Advances in Computers, 106, 2017
  2. 2. Metric and Measurement • Mapping attributes to values/names on scales – Quality control by single metric – Estimating metric values by other metrics • You cannot control what you cannot measure! Req. Def. Design Impl. Testing Func. Spec. Function point Module design Coupling Source code Complexity LOC Defect report Defect density Test case N. tests passed Effort, time Cyclomatic complexity 0 0.2 0.4 0.6 0.8 1 0 0.05 0.1 0.15 0.2 0.25 0.3 Reuserate #Header files included 2
  3. 3. Pitfalls and Countermeasures 3 Pitfall Countermeasure Negative Hawthorne effects Goal-orientation Multidimensional measurements Organization misalignment Visualization of relationships among organizational goals, strategies, and measurements Exhaustive identification of rationales Uncertain future Prediction incorporating uncertainty Measurement program improvement by machine learning Self-certified quality Standard-based evaluation Pattern-based evaluation H. Washizaki, Pitfalls and Countermeasures in Software Quality Measurements and Evaluations, Advances in Computers, 106, 2017
  4. 4. Pitfalls and Countermeasures 4 Pitfall Countermeasure Negative Hawthorne effects Goal-orientation Multidimensional measurements Organization misalignment Visualization of relationships among organizational goals, strategies, and measurements Exhaustive identification of rationales Uncertain future Prediction incorporating uncertainty Measurement program improvement by machine learning Self-certified quality Standard-based evaluation Pattern-based evaluation H. Washizaki, Pitfalls and Countermeasures in Software Quality Measurements and Evaluations, Advances in Computers, 106, 2017
  5. 5. Hawthorne Effect 5nicolasdsampson.com, Observe And Learn: The Magic Of Paying Attention http://nicolasdsampson.com/wp-content/uploads/2012/10/2010_12_06_observe-learn-magic-paying-attention.jpg
  6. 6. Goal-Question-Metric (GQM) Paradigm • Goal-oriented framework for identifying goals and necessary corresponding metrics • Goal: measurement goals • Question: questions for evaluating goal achievement • Metric: objective or subjective metrics for obtaining necessary quantitative data to answer questions Identifying metrics Interpreting measurement results Metric Goal Question Collected data Goal achievement evaluation Answer Measured value R. van Solingen, E. Berghout, “The Goal/Question/Metric Method” McGraw-Hill Education, 1999
  7. 7. 7 GQM-based Multidimensional Measurements H. Washizaki, R. Namiki, T. Fukuoka, Y. Harada, H. Watanabe, "A Framework for Measuring and Evaluating Program Source Code Quality", 8th Int’l Conference on Product Focused Software Development and Process Improvement (Profes2007) Reliability Maintain -ability Portability Efficiency Reusability Analysability Changeability Testability Goal Metric Depth of call graph T Cyclomatic Number min Estimated static path length min Measured by some tools Measured by some tools Are functions not complicated? Question Sub- Question Is call- nesting not too deep? Is logic not too complex? IndependentIndependent SpecificSpecific Are entities named properly? (Scale type: T threshold, min smaller is better, max larger is better)
  8. 8. SEMAT-based Multidimensional measurements 8 Software System Opportunity Stakeholders Requirements Work Team Way of Working Customer Solution Endeavour Ivar Jacobson, Pan-Wei Ng, Paul E. McMahon, Ian Spence, Svante Lidman, “The Essence of Software Engineering: The SEMAT Kernel,” Communications of the ACM, vol.55, no.12, pp.42-49, December 2012.
  9. 9. alpha as Project Measurement 9  State 1  XXXXXXXXXXXXXXXXXX  XXXXXXXXXXX  XXXXXXXXXXXX  State 2  XXXXXXXXXXXXXXXXXX  XXXXXXXXXXX  XXXXXXXXXXXX  State 3  XXXXXXXXXXXXXXXXXX  XXXXXXXXXXX  XXXXXXXXXXXX  ……..  ……. Checklist Adopted/modified from I. Jacobson, et al.: Tutorial: Essence - Kernel and Language for Software Engineering Practices, ICSE'13 alpha Indicator 1 2 3 4 5
  10. 10. Example from ITA WG on project failuers 10 • An employee in charge of Bank office inquires "registered customer information cannot be browsed from the terminal“. • It was because a batch processing for the previous day has ended abnormally due to wrong data input. • It took about two hours to recover the data, and employees at each office had to handle customer inquiry manually. • For that reason, we had received plenty of complaints from customers who had been waiting for a long time! H. Washizaki, “Analyzing and refining project failure cases from wider viewpoints by using SEMAT Essence,” Essence Conference in Seoul, 2017.
  11. 11. 11 Delay in service deployment Delay in service deployment Huge data inputs manually Huge data inputs manually Wrong data inputWrong data input Defects in operation manual Defects in operation manual Misunderstanding of abnormal state as normal state Misunderstanding of abnormal state as normal state Problem Cause Root cause Requirement s: NG “Coherent” Software System: NG “Demonstrable” Automate data input Automate data input Countermeasure Frequent rule- violations Frequent rule- violationsWork: NG “Under Control” Way of Working: NG “In Use” Review and modify rules Review and modify rules Monitor and enforce compliance with rules Monitor and enforce compliance with rules Opportunity: NG “Benefit Accrued” Team: NG “Collaborating” Stakeholders: NG “Satisfied in Use”
  12. 12. Pitfalls and Countermeasures 12 Pitfall Countermeasure Negative Hawthorne effects Goal-orientation Multidimensional measurements Organization misalignment Visualization of relationships among organizational goals, strategies, and measurements Exhaustive identification of rationales Uncertain future Prediction incorporating uncertainty Measurement program improvement by machine learning Self-certified quality Standard-based evaluation Pattern-based evaluation H. Washizaki, Pitfalls and Countermeasures in Software Quality Measurements and Evaluations, Advances in Computers, 106, 2017
  13. 13. Organization misalignment 13 We increase customer satisfaction by quality and usability improvement! Top managementWe reduce customer- reported defects by improving testing process and product maintainability. Development team We track defect data and code quality metrics. Quality assurance team
  14. 14. GQM+Strategies M1: Customer satisfaction survey M1: Customer satisfaction survey Measurement DataGoals and Strategies BusinessLevelSoftwareLevel Goal: Increase customer satisfaction by 10% Goal: Increase customer satisfaction by 10% Strategy: Improve product quality Strategy: Improve product quality Strategy: Improve usability of product Strategy: Improve usability of product Goal: Reduce customer- reported defects by 20% Goal: Reduce customer- reported defects by 20% No sub-level goal defined Strategy: Improve efficiency of system testing Strategy: Improve efficiency of system testing Strategy: Improve maintainability of software Strategy: Improve maintainability of software Isolated strategy M2: Field defect data M2: Field defect data M3: Code quality metrics (McCabe, coupling, cohesion) M3: Code quality metrics (McCabe, coupling, cohesion) Isolated data 14 • Alignment and tracing among goal, strategy and data Context & Assumption Context & Assumption
  15. 15. Context-Assumption-Matrix [IEICE’16] 15Takanobu Kobori, Hironori Washizaki, et al., “Exhaustive and efficient identification of rationales using GQM+Strategies with stakeholder relationship analysis,” IEICE Transactions on Information and Systems, Vol.E99-D, No.9, pp.2219-2228, 2016. Target View Hospital Insurance company Health sensor maker ・・・ Hospital Insurance company Health sensor maker ・・・ Insurance company can offer personalized products by having medical info… Context / Assumption Insurance company Medical info provider Personal medical info. Customized product Resource Goal depends
  16. 16. 16 Insurance company G S A S Health sensor maker Medical info. provider Expectation from the view of insurance company Expectation from the view of health sensor maker Receive medical info. Personalized products developed Strategy Insurance product and fee can be personalized based on personal medical info. Goal Context / Assumption Sales of new products Personal fitness Confirm personal info. Management system of receiver of info. Secure management of medical info. Strategy Must ensure personal info. guideline Goal Context / Assumption Insurance company Medical info. provider Confirming management systems Pati ent Getting consent Conflict, redundancy, complement Conflict, redundancy, complement
  17. 17. 17 Towards clear dependency and alignment G S A S Insurance company Medical info. provider Personal medical info. Realizing personalized insurance Objective of medical info. use Managing personal info. Clarify objective of use of medical info. Strategy Establish personal info. management system Strategy Medical info. provider Secure management of medical info. Strategy Must ensure personal info. guideline Goal Context / Assumption Health sensor maker Insurance company Receive medical info. Personalized products developed Strategy Goal Context / Assumption Sales of new products Personal fitness Insurance product and fee can be personalized based on personal medical info.
  18. 18. S1 S2 S3 S5 S6 S4 S7 S1 S2 S3 S4 S6 S5 S7 Interpretive Structural Modeling (ISM) 18 Impact S1 S2 S3 S4 S5 S6 S7 S1 1 0 0 0 0 0 0 S2 0 1 0 0 0 0 0 S3 1 1* 1 0 1 0 0 S4 1 1* 0 1 0 0 1 S5 1* 1 1 0 1 0 0 S6 1* 1 0 1 0 1 1* S7 1* 1 0 1 0 0 1 Power. Hierarchical structuring 可到達行列 S1 S2 S3 S4 S5 S6 S7 S1 1 0 0 0 0 0 0 S2 1 1 0 0 0 0 0 S3 1 0 1 0 1 0 0 S4 1 0 0 1 0 0 1 S5 0 1 1 0 1 0 0 S6 0 1 0 1 0 1 0 S7 0 1 0 1 0 0 1 Relation matrix Reachability matrix
  19. 19. ISM-based Alignment [HICSS’16] S1 S2 S3 S4 ・ ・ S1 S2 S3 S4 ・・ 1 1 (1) Decompose into elements (2)Restru cturing (3)Analysis& alignment S5 S1 S2 S3 S4 S6 S7 Hierarchical structure S5 S1 S2 S3 S4 S6 S7 ISM 1 Elements (especially strategies) 1 1 1 1 S5 S1 S2 S3 S4 S6 S7 GQM+Strategies G GG G Reachability matrix Conflict specified • Alignment for single GQM+Strategies model • Future: alignment over areas and stakeholders Yohei Aoki, Takanobu Kobori, Hironori Washizaki, et al., “Identifying Misalignment of Goals and Strategies across Organizational Units by Interpretive Structural Modeling,” 49th Hawaii International Conference on System Sciences (HICSS), 2016
  20. 20. Pitfalls and Countermeasures 20 Pitfall Countermeasure Negative Hawthorne effects Goal-orientation Multidimensional measurements Organization misalignment Visualization of relationships among organizational goals, strategies, and measurements Exhaustive identification of rationales Uncertain future Prediction incorporating uncertainty Measurement program improvement by machine learning Self-certified quality Standard-based evaluation Pattern-based evaluation H. Washizaki, Pitfalls and Countermeasures in Software Quality Measurements and Evaluations, Advances in Computers, 106, 2017
  21. 21. Uncertain Future 21
  22. 22. Identifying parts that are hard to maintain… How large? ELOC N. functions Measurement System Improvement by ML N. Tsuda, et al. Iterative Process to Improve GQM Models with Metrics Thresholds to Detect High-risk Files, SANER 2015 Review Quality measurement Machine learning Goal Question Measurement 22 10 25 300 150 N. functions ELOC OK NG 71 N. functions ELOC OK NG Improvement X
  23. 23. ]] #Issues Actual Predicted DayKiyoshi Honda, Hironori Washizaki and Yoshiaki Fukazawa, “Generalized Software Reliability Model Considering Uncertainty and Dynamics: Model and Applications”, International Journal of Software Engineering and Knowledge Engineering (IJSEKE), 2016. Logistic Gompertz Non-homogeneous Poisson process(NHPP) Software reliability model (SRM) as actionable metric 23
  24. 24. Prediction with uncertainty 0 10 20 30 40 50 60 70 80 90 0 2 4 6 8 10 12 14 #Issues Time ActualData Our model 24 Kiyoshi Honda, Hironori Washizaki and Yoshiaki Fukazawa, “Generalized Software Reliability Model Considering Uncertainty and Dynamics: Model and Applications”, International Journal of Software Engineering and Knowledge Engineering (IJSEKE), 2016.
  25. 25. Prediction with uncertainty 0 10 20 30 40 50 60 70 80 90 0 2 4 6 8 10 12 14 #Issues Time ActualData Our model - + 25 Lower (worst case) Upper (best case) T1 T2 Kiyoshi Honda, Hironori Washizaki and Yoshiaki Fukazawa, “Generalized Software Reliability Model Considering Uncertainty and Dynamics: Model and Applications”, International Journal of Software Engineering and Knowledge Engineering (IJSEKE), 2016.
  26. 26. Uncertainty patterns and prediction 26 0 0.5 1 ConstantIncrease Decrease Kiyoshi Honda, Hironori Washizaki and Yoshiaki Fukazawa, “Generalized Software Reliability Model Considering Uncertainty and Dynamics: Model and Applications”, International Journal of Software Engineering and Knowledge Engineering (IJSEKE), 2016.
  27. 27. Pitfalls and Countermeasures 27 Pitfall Countermeasure Negative Hawthorne effects Goal-orientation Multidimensional measurements Organization misalignment Visualization of relationships among organizational goals, strategies, and measurements Exhaustive identification of rationales Uncertain future Prediction incorporating uncertainty Measurement program improvement by machine learning Self-certified quality Standard-based evaluation Pattern-based evaluation H. Washizaki, Pitfalls and Countermeasures in Software Quality Measurements and Evaluations, Advances in Computers, 106, 2017
  28. 28. Self-Certified Quality 28 Image: How to Spot a Fake: Avoiding Degree Mill Scams https://wenr.wes.org/2015/06/spot-fake-avoiding-degree-mill-scams
  29. 29. ISO/IEC 25000 SQuaRE-based Quality Measurement and Benchmark 29 Q1. Any path going through internal servers only? Q2. Any path going through outside servers? Q3. Any P2P communications? … G. The events or actions cannot be repudiated later through communication channels (paths). M. Signed communication path ratio = #Signed_paths / #Total_paths E.g. Non-repudiation Waseda U. Team Vendor 1 Concretize SQuaRE measurements by GQM 2 Prepare measurement methods: data forms, static analysis, questionnaire, user- testing 3 Conduct code static analysis, user-testing Fill data forms, questionnaire 4 Measure and evaluate quality Scores by using percentile E.g., Top 30% = 0.7 HighLow #Products Measured value H. Nakai, N. Tsuda, K. Honda, H. Washizaki and Y. Fukazawa, Initial Framework for a Software Quality Evaluation based on ISO/IEC 25022 and ISO/IEC 25023, IEEE International Conference on Software Quality, Reliability & Security (QRS 2016)
  30. 30. 21 Japanese Products Measurement 30 Functional suitability Performance efficiency Compatibility Usability Reliability Security Maintainability Portability Effectiveness Efficiency Satisfaction Freedom from risk Context coverage • Low security and compatibility in some products • Necessary to address these in IoT era H. Nakai, N. Tsuda, K. Honda, H. Washizaki and Y. Fukazawa, Initial Framework for a Software Quality Evaluation based on ISO/IEC 25022 and ISO/IEC 25023, IEEE International Conference on Software Quality, Reliability & Security (QRS 2016)
  31. 31. Relationships among characteristics 31 Internal/External Quality Quality in Use p-value < 0.1p-value < 0.1 性能効率性 互換性 使用性 信頼性 セキュリティ 保守性 移植性 有効性 効率性 満足性 リスク回避性 利用状況網羅性 機能適合性 0. 31 0. 19 -0. 72 0. 37 -0. 05 0. 50 0. 31 -0. 14 0. 52 1. 00 1. 00 1. 00 性能効率性 0. 44 0. 24 0. 36 -0. 17 0. 37 0. 32 0. 32 -0. 10 -0. 50 -0. 50 -0. 50 互換性 0. 04 0. 17 -0. 06 0. 36 -0. 04 -0. 14 0. 05 -0. 50 -0. 50 -0. 50 使用性 0. 17 -0. 21 0. 11 0. 44 -0. 09 -0. 20 -1. 00 -1. 00 -1. 00 信頼性 0. 30 0. 41 0. 45 -0. 08 0. 11 1. 00 1. 00 1. 00 セキュリティ -0. 06 0. 19 0. 64 -0. 34 0. 50 0. 50 0. 50 保守性 0. 26 -0. 29 0. 01 1. 00 1. 00 1. 00 移植性 -0. 21 0. 67 0. 50 0. 50 0. 50 有効性 0. 03 -1. 00 -1. 00 -1. 00 効率性 1. 00 1. 00 1. 00 満足性 1. 00 1. 00 リスク回避性 1. 00 Func. Perf. Comp. Usa. Relia. Sec. Main. Port. Effe. Effic. Sati. Free. Perf. Comp. Usa. Relia. Sec. Main. Port. Effe. Effic. Sati. Free. Cont. • Negative correlation between usability and functionality. • Need to adopt user-centered development H. Nakai, N. Tsuda, K. Honda, H. Washizaki and Y. Fukazawa, Initial Framework for a Software Quality Evaluation based on ISO/IEC 25022 and ISO/IEC 25023, IEEE International Conference on Software Quality, Reliability & Security (QRS 2016)
  32. 32. Appropriate design Inappropriate design Security Patterns and Testing 32 Role-based access control (RBAC) pattern
  33. 33. TESEM: Test Driven Secure Modeling Tool [ARES’13][ARES’13][IJSSE’14][ICST’15][Information’16] 33 Security Design Pattern Problem Solution Context Test design as requirement ! create Actor ! create UI : ! create Subject.. Test Script Test case testing [ARES’13] Validating Security Design Pattern Applications Using Model Testing, Int’l Conf. Availability, Reliability and Security [ARES’14] Verification of Implementing Security Design Patterns Using a Test Template, Conf. Availability, Reliability and Security [IJSSE’14] Validating Security Design Pattern Applications by Testing Design Models, Int’l J. Secure Software Engineering 5(4) [ICST’15] TESEM: A Tool for Verifying Security Design Pattern Applications by Model Testing, IEEE ICST’15 Tools Track [Information’16] Implementation Support of Security Design Patterns Using Test Templates, Information 7(2) testing
  34. 34. 34 window.onload = setEventHandler; function setEventHandler() { $(“reg_type”).onchange = calcPrice; ・・・ $(“reg_addcart”).onclick = addCart; }; function calcPrice() { ・・・ }; function addCart() { if(isValidInput()) { reqRunTrans(); } else { alert(“Invalid user inputs”); } }; function reqRunTrans() { new Ajax.Request(“runTrans.php”, { method: “GET”, parameters: getParams(), onSuccess: succeeded }); }; function succeeded() { disableAll(); jumpToConfirm(); }; window.onload = setEventHandler; function setEventHandler() { $(“reg_type”).onchange = calcPrice; ・・・ $(“reg_addcart”).onclick = addCart; }; function calcPrice() { ・・・ }; function addCart() { if(isValidInput()) { reqRunTrans(); } else { alert(“Invalid user inputs”); } }; function reqRunTrans() { new Ajax.Request(“runTrans.php”, { method: “GET”, parameters: getParams(), onSuccess: succeeded }); }; function succeeded() { disableAll(); jumpToConfirm(); };
  35. 35. 35 Finite State Machine Extraction …………………………………………… …………………………………………… function setEventHandler() { $(“addcart”).onclick=addCart; …………………………………………… …………………………………………… …………………………………………… …………………………………………… function setEventHandler() { $(“addcart”).onclick=addCart; …………………………………………… …………………………………………… onclick setEventHandler addCart E.g. User Action [Mahamoff’06] “Prevent multiple calls of specific user event handler” Y. Maezawa, K. Nishiura, H. Washizaki, S. Honiden, Validating Ajax Applications Using a Delay-Based Mutation Technique”, 29th IEEE/ACM International Conference on Automated Software Engineering (ASE 2014) Y. Maezawa, H. Washizaki, Y. Tanabe and S. Honiden , “Automated Verification of Pattern-based Interaction Invariants in Ajax Applications, 28th IEEE/ACM International Conference on Automated Software Engineering (ASE2013) Duplicate order Duplicate order [Mahamoff’06] M. Mahamoff, “Ajax Design Patterns”, O’Reilly Media Inc., 2006.
  36. 36. 36 window.onload = setEventHandler; function setEventHandler() { $(“reg_type”).onchange = calcPrice; ・・・ $(“reg_addcart”).onclick = addCart; }; function calcPrice() { ・・・ }; function addCart() { if(isValidInput()) { $(“addCart”).disabled = true; reqRunTrans(); } else { alert(“Invalid user inputs”); $(“addCart”).disabled = false; } }; function reqRunTrans() { new Ajax.Request(“runTrans.php”, { method: “GET”, parameters: getParams(), onSuccess: succeeded }); }; function succeeded() { disableAll(); jumpToConfirm(); }; window.onload = setEventHandler; function setEventHandler() { $(“reg_type”).onchange = calcPrice; ・・・ $(“reg_addcart”).onclick = addCart; }; function calcPrice() { ・・・ }; function addCart() { if(isValidInput()) { $(“addCart”).disabled = true; reqRunTrans(); } else { alert(“Invalid user inputs”); $(“addCart”).disabled = false; } }; function reqRunTrans() { new Ajax.Request(“runTrans.php”, { method: “GET”, parameters: getParams(), onSuccess: succeeded }); }; function succeeded() { disableAll(); jumpToConfirm(); };
  37. 37. Pitfalls and Countermeasures 37 Pitfall Countermeasure Negative Hawthorne effects Goal-orientation Multidimensional measurements Organization misalignment Visualization of relationships among organizational goals, strategies, and measurements Exhaustive identification of rationales Uncertain future Prediction incorporating uncertainty Measurement program improvement by machine learning Self-certified quality Standard-based evaluation Pattern-based evaluation
  38. 38. SamurAI Coding IPSJ 6th International AI Programing Contest World Final March 14 2018 Tokyo http://samuraicoding.info APSEC 2018 25th Asia-Pacific Software Engineering Conference Nara Dec 4-7 (due: June) PC Chair: H. Washizaki Int. Journal of Agile and Extreme Software Development (IJAESD) Editor-in-Chief: H. Washizaki COMPSAC 2018 42nd IEEE Computer Society Int’l Conf. Computers, Software & Applications Tokyo July 23-27 (due: Jan 15)

×