Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

IPA RISE 2015-16: Waseda Software Quality Benchmark

1,785 views

Published on

Introduced Waseda Software Quality Benchmark (WSQB) based on SQuaRE as a result of IPA/RISE Research at ISO SC7/WG6 Plenary Meeting Kuantan

Published in: Software
  • Be the first to comment

  • Be the first to like this

IPA RISE 2015-16: Waseda Software Quality Benchmark

  1. 1. IPA RISE 2015-16 Waseda Software Quality Benchmark Professor, Waseda University Visiting Professor, National Institute of Informatics Outside Director, SYSTEM INFORMATION ISO/IEC/JTC1 SC7/WG20 Convenor washizaki@waseda.jp Hironori Washizaki In corporation with Prof. Azuma and Mr. Komiyama from ISO/IEC/JTC1/SC7/WG6, Mr. Tsuda and Dr. Honda from Waseda U., and Computer Software Association Japan (CSAJ)
  2. 2. Executive Summary 2 • 21 package software and SaaS • Waseda Software Quality Benchmark (WSQB2017) http://www.washi.cs.waseda.ac.jp/?page_id=3479 • ISO/IEC 25000 SQuaRE- based concrete measurement • Relationships among different quality characteristics 0 20 40 60 80 100 機能適合性 性能効率性 互換性 使用性 信頼性 セキュリティ 保守性 移植性 0 5 10 1 2 3 4 5 6 12 vendors 21 products IV&V org. Delegate Result ISO/IEC JTC1/SC7/WG6 SQuaRE editorsCooperation Eval. result
  3. 3. Quality Evaluation Framework 3 Q1. Any path going through internal servers only? Q2. Any path going through outside servers? Q3. Any P2P communications? … G. The events or actions cannot be repudiated later through communication channels (paths). M. Signed communication path ratio = #Signed_paths / #Total_paths E.g. Non-repudiation Waseda U. Team Vendor 1 Concretize SQuaRE measurements by GQM 2 Prepare measurement methods: data forms, static analysis, questionnaire, user- testing 3 Conduct code static analysis, user-testing Fill data forms, questionnaire 4 Measure and evaluate quality Scores by using percentile E.g., Top 30% = 0.7 HighLow Histogram Measured value
  4. 4. Internal/External Quality • 66 measurements (i.e., metrics) • Measurement coverage: 34% of 66×21 products 4 #Measurements defined 4 10 2 9 13 6 8 5 0 1 0 0 3 0 4 1 機能適合性 性能効率性 互換性 使用性 信頼性 セキュリティ 保守性 移植性 測定値あり 測定値なしMeasured NOT measured Functional suitability Performance efficiency Compatibility Usability Reliability Security Maintainability Portability
  5. 5. Quality in Use User questionnaire • Standard questionnaire – E.g., Are you satisfied … ? • 3 products User testing • 10 products 5 Waseda U. Team Vendor 1 Declare functions 2 Extract core functions 3 Define normal test cases 4 Define exceptional test cases 5 Conduct test 6 Measure quality Coverage 24% Measured NOT measured
  6. 6. Result: Quality Characteristics 6 Functional suitability Performance efficiency Compatibility Usability Reliability Security Maintainability Portability Effectiveness Efficiency Satisfaction Freedom from risk Context coverage
  7. 7. Relationships among characteristics 7 Internal/External Quality Quality in Use p-value < 0.1 性能効率性 互換性 使用性 信頼性 セキュリティ 保守性 移植性 有効性 効率性 満足性 リスク回避性 利用状況網羅性 機能適合性 0. 31 0. 19 -0. 72 0. 37 -0. 05 0. 50 0. 31 -0. 14 0. 52 1. 00 1. 00 1. 00 性能効率性 0. 44 0. 24 0. 36 -0. 17 0. 37 0. 32 0. 32 -0. 10 -0. 50 -0. 50 -0. 50 互換性 0. 04 0. 17 -0. 06 0. 36 -0. 04 -0. 14 0. 05 -0. 50 -0. 50 -0. 50 使用性 0. 17 -0. 21 0. 11 0. 44 -0. 09 -0. 20 -1. 00 -1. 00 -1. 00 信頼性 0. 30 0. 41 0. 45 -0. 08 0. 11 1. 00 1. 00 1. 00 セキュリティ -0. 06 0. 19 0. 64 -0. 34 0. 50 0. 50 0. 50 保守性 0. 26 -0. 29 0. 01 1. 00 1. 00 1. 00 移植性 -0. 21 0. 67 0. 50 0. 50 0. 50 有効性 0. 03 -1. 00 -1. 00 -1. 00 効率性 1. 00 1. 00 1. 00 満足性 1. 00 1. 00 リスク回避性 1. 00 Func. Perf. Comp. Usa. Relia. Sec. Main. Port. Effe. Effic. Sati. Free. Perf. Comp. Usa. Relia. Sec. Main. Port. Effe. Effic. Sati. Free. Cont.
  8. 8. Analysis by Contexts 8 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 機能 適… 性能 効… 互換 性 使用 性 信頼 性 セキュ リティ 保守 性 移植 性 有効 性 効率 性 満足 性 リスク 回… 利用 状… パッケージ (n=17) クラウド(n=4) 区別なし (n=21) Package Cloud ALL Functional suitability Performance efficiency Compatibility Usability Reliability Security MaintainabilityPortability Effectiveness Efficiency Satisfaction Freedom from risk Context coverage
  9. 9. Recommendations 9 Waseda Software Quality Benchmark http://www.washi.cs.waseda.ac.jp/?page_id=3479 Industry ISO/IEC JTC1/SC7/WG6 Continuous measurement and PSQ-Certification Improve SQuaRE 4. Incorporate concrete measurements into SQuaRE 5. Address properties specific to Agile and Cloud 1. Low security and compatibility in some products. Necessary to address these in IoT era. 2. Negative correlation between usability and functionality. Need to adopt user-centered development. 3. Limited data and goals. Necessary to measure/benchmark Improve quality management
  10. 10. Publications  K. Honda, H. Washizaki and Y. Fukazawa, “Generalized Software Reliability Model Considering Uncertainty and Dynamics: Model and Applications,” International Journal of Software Engineering and Knowledge Engineering (IJSEKE), pp.1-29, 2017.  K. Honda, H. Washizaki, Y. Fukazawa, et al., “Detection of Unexpected Situations by Applying Software Reliability Growth Models to Test Phases,” 26th IEEE International Symposium on Software Reliability Engineering (ISSRE 2015), Industry Track  H. Nakai, N. Tsuda, K. Honda, H. Washizaki, and Y. Fukazawa, “Initial Framework for a Software Quality Evaluation based on ISO/IEC 25022 and ISO/IEC 25023,” Poster, The 2016 IEEE International Conference on Software Quality, Reliability & Security (QRS 2016)  K. Honda, N. Nakamura, H. Washizaki and Y. Fukazawa, “An Industrial Case Study of Project Management Using Cross Project Software Reliability Growth Model,” Poster, The 2016 IEEE International Conference on Software Quality, Reliability & Security (QRS 2016)  K. Honda, N. Nakamura, H. Washizaki and Y. Fukazawa, “Case Study: Project Management Using Cross Project Software Reliability Growth Model,” IEEE International Workshop on Trustworthy Computing, 2016  K. Honda, N. Nakamura, H. Washizaki and Y. Fukazawa, “Case Study: Project Management Using Cross Project Software Reliability Growth Model Considering System Scale,” 27th IEEE International Symposium on Software Reliability Engineering (ISSRE 2016), Industry Track,  H. Nakai, N. Tsuda, K. Honda, H. Washizaki, Y. Fukazawa, “Evaluating Software Product Quality based on SQuaRE Series,” IEEE TENCON 2016 10

×