Quality assurance of non-functional requirements  Mika Mäntylä TKK/SoberIT/SPRG T-76.5650 Software engineering seminar
Outline <ul><li>Introduction </li></ul><ul><li>QA of performance and usability (short) </li></ul><ul><li>QA of evolvabilit...
QA of … Reliability Interoperability Performance Usability Scalability Security Availability Maintainability Modifiability...
…  with TDD Module Testing Load testing Exploratory testing Bounce checker Test case based testing Code review Load tester...
TDD Module Testing Load testing Exploratory testing Test case based testing Code review Load tester Pair-programming Unit ...
TDD Module Testing Load testing Exploratory testing Test case based testing Code review Load tester Pair-programming Unit ...
TDD Module Testing Load testing Exploratory testing Test case based testing Code review Load tester Pair-programming Unit ...
TDD Module Testing Load testing Exploratory testing Test case based testing Code review Load tester Pair-programming Unit ...
TDD Module Testing Load testing Exploratory testing Test case based testing Code review Load tester Pair-programming Unit ...
TDD Module Testing Load testing Exploratory testing Test case based testing Code review Load tester Pair-programming Unit ...
TDD Module Testing Load testing Exploratory testing Test case based testing Code review Load tester Pair-programming Unit ...
Summary <ul><li>Qualities and methods have many to many relationship </li></ul><ul><li>Often relationships between qualiti...
Outline <ul><li>Introduction </li></ul><ul><li>QA of performance and usability (short) </li></ul><ul><li>QA of evolvabilit...
QA of performance <ul><li>Requirements analysis </li></ul><ul><ul><li>Estimate data volumes and user volume & behavior </l...
Nixon B. A., 2000
QA of Usability <ul><li>Usability – Quality in Use </li></ul><ul><ul><li>The capability of the software to be  understood ...
Outline <ul><li>Introduction </li></ul><ul><li>QA of performance and usability (short) </li></ul><ul><li>QA of evolvabilit...
Quality assurance of evolvability <ul><li>Design </li></ul><ul><li>Company Standards / Coding conventions </li></ul><ul><l...
Method 1 for evolvability: Design <ul><li>Design for maintainable software </li></ul><ul><ul><li>Principles </li></ul></ul...
Method 2 for evolvability: Coding conventions <ul><li>Program layout has great impact to program comprehension  </li></ul>...
Method 3 for evolvability: Code review <ul><li>Spreads knowledge of the system </li></ul><ul><ul><li>Increases evolvabilit...
Method 4 for evolvability: Tools <ul><li>Code metrics </li></ul><ul><ul><li>E.g. Lines of code, number of coupling, execut...
Method 5 for evolvability: Improve <ul><li>Terms </li></ul><ul><ul><li>Refactoring - Controlled way to improve the softwar...
Method 6 for evolvability: Knowledge distribution <ul><li>Further development is harder, if expertise is lost </li></ul><u...
Method 7 for evolvability: Environment support <ul><li>Good unit test harness increases evolvability </li></ul><ul><ul><li...
Quality assurance of evolvability <ul><li>Design </li></ul><ul><li>Company Standards / Coding conventions </li></ul><ul><l...
Outline <ul><li>Introduction </li></ul><ul><li>QA of performance and usability (short) </li></ul><ul><li>QA of evolvabilit...
Code review’s effect to qualities <ul><li>Spreads knowledge of the system </li></ul><ul><ul><li>Increases evolvability ind...
Pair programming’s effect to qualities <ul><li>Increases understandability </li></ul><ul><li>Makes design better (evolvabi...
Exploratory testing’s effect to qualaties <ul><li>Single study: Exploratory testing (ET) vs Test Case Testing (TCT) </li><...
Outline <ul><li>Introduction </li></ul><ul><li>QA of performance and usability (short) </li></ul><ul><li>QA of evolvabilit...
Teams with different goals 1 1 5 5 2-3 Team5 (output) 2 2 3 3 4 Team4 (clarity) 4 4 1 2 5 Team3 (memory) 5 3 2 1 2-3 Team2...
Practical approach 1 2 3 3 2 3 Multi-platform build 3 3 1 3 1 2 Load Tests 3 3 2 1 2 1 Test sessions 1 1 2 3 2 1 Code Revi...
Summary <ul><li>Many quality attributes and many quality assurance methods exist </li></ul><ul><ul><li>Lecture series inst...
Topics <ul><li>Consulting plan for a company </li></ul><ul><li>How to test Unit tests effect to quality X </li></ul>
Upcoming SlideShare
Loading in …5
×

Quality assurance and testing of quality attributes

2,178 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,178
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
32
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Quality assurance and testing of quality attributes

  1. 1. Quality assurance of non-functional requirements Mika Mäntylä TKK/SoberIT/SPRG T-76.5650 Software engineering seminar
  2. 2. Outline <ul><li>Introduction </li></ul><ul><li>QA of performance and usability (short) </li></ul><ul><li>QA of evolvability (long) </li></ul><ul><li>How are qualities effected by </li></ul><ul><ul><li>code review, </li></ul></ul><ul><ul><li>pair-programming </li></ul></ul><ul><ul><li>exploratory testing </li></ul></ul><ul><li>Practical approach </li></ul>
  3. 3. QA of … Reliability Interoperability Performance Usability Scalability Security Availability Maintainability Modifiability Testability Safety Survivability Confidentiality Portability Dependability Timeliness Integrity Fault-tolerance
  4. 4. … with TDD Module Testing Load testing Exploratory testing Bounce checker Test case based testing Code review Load tester Pair-programming GUI robot Unit testing Functional testing Acceptance testing External beta-testers Usability testing Static analysis Iteration Demo Structural testing
  5. 5. TDD Module Testing Load testing Exploratory testing Test case based testing Code review Load tester Pair-programming Unit testing Functional testing Acceptance testing External beta-testers Usability testing Static analysis Iteration Demo Structural testing Reliability Interoperability Performance Usability Scalability Security Availability Maintainability Modifiability Testability Safety Survivability Confidentiality Portability Dependability Timeliness Integrity Fault-tolerance Qualities Methods
  6. 6. TDD Module Testing Load testing Exploratory testing Test case based testing Code review Load tester Pair-programming Unit testing Functional testing Acceptance testing External beta-testers Usability testing Static analysis Iteration Demo Structural testing Reliability Interoperability Performance Usability Scalability Security Availability Maintainability Modifiability Testability Safety Survivability Confidentiality Portability Dependability Timeliness Integrity Fault-tolerance
  7. 7. TDD Module Testing Load testing Exploratory testing Test case based testing Code review Load tester Pair-programming Unit testing Functional testing Acceptance testing External beta-testers Usability testing Static analysis Iteration Demo Structural testing Reliability Interoperability Performance Usability Scalability Security Availability Maintainability Modifiability Testability Safety Survivability Confidentiality Portability Dependability Timeliness Integrity Fault-tolerance
  8. 8. TDD Module Testing Load testing Exploratory testing Test case based testing Code review Load tester Pair-programming Unit testing Functional testing Acceptance testing External beta-testers Usability testing Static analysis Iteration Demo Structural testing Reliability Interoperability Performance Usability Scalability Security Availability Maintainability Modifiability Testability Safety Survivability Confidentiality Portability Dependability Timeliness Integrity Fault-tolerance Qualities and methods have many to many relationship
  9. 9. TDD Module Testing Load testing Exploratory testing Test case based testing Code review Load tester Pair-programming Unit testing Functional testing Acceptance testing External beta-testers Usability testing Static analysis Iteration Demo Structural testing Reliability Interoperability Performance Usability Scalability Security Availability Maintainability Modifiability Testability Safety Survivability Confidentiality Portability Dependability Timeliness Integrity Fault-tolerance Relationships between qualities and methods are poorly understood
  10. 10. TDD Module Testing Load testing Exploratory testing Test case based testing Code review Load tester Pair-programming Unit testing Functional testing Acceptance testing External beta-testers Usability testing Static analysis Iteration Demo Structural testing Reliability Interoperability Performance Usability Scalability Security Availability Maintainability Modifiability Testability Safety Survivability Confidentiality Portability Dependability Timeliness Integrity Fault-tolerance Method may not be the most important factor affecting a quality
  11. 11. TDD Module Testing Load testing Exploratory testing Test case based testing Code review Load tester Pair-programming Unit testing Functional testing Acceptance testing External beta-testers Usability testing Static analysis Iteration Demo Structural testing Reliability Interoperability Performance Usability Scalability Security Availability Maintainability Modifiability Testability Safety Survivability Confidentiality Portability Dependability Timeliness Integrity Fault-tolerance Specific methods for specific qualities
  12. 12. Summary <ul><li>Qualities and methods have many to many relationship </li></ul><ul><li>Often relationships between qualities and methods are poorly understood </li></ul><ul><li>Method may not be the key factor when assuring particular quality </li></ul><ul><ul><li>Code review’s effect on security </li></ul></ul><ul><ul><ul><li>Largely dependent on the reviewer’s instructions </li></ul></ul></ul><ul><li>Specific methods exist for some qualities </li></ul><ul><ul><li>Usability testing </li></ul></ul><ul><ul><li>Load testing </li></ul></ul>
  13. 13. Outline <ul><li>Introduction </li></ul><ul><li>QA of performance and usability (short) </li></ul><ul><li>QA of evolvability (long) </li></ul><ul><li>How are qualities effected by </li></ul><ul><ul><li>code review, </li></ul></ul><ul><ul><li>pair-programming </li></ul></ul><ul><ul><li>exploratory testing </li></ul></ul><ul><li>Practical approach </li></ul>
  14. 14. QA of performance <ul><li>Requirements analysis </li></ul><ul><ul><li>Estimate data volumes and user volume & behavior </li></ul></ul><ul><ul><ul><li>Peak usage vs. Regular usage </li></ul></ul></ul><ul><ul><ul><li>Realistic data volumes </li></ul></ul></ul><ul><ul><li>Acceptable response time </li></ul></ul><ul><ul><li>Hardware restrictions </li></ul></ul><ul><ul><ul><li>With clustered server you can always buy more servers with mobile phone you cannot </li></ul></ul></ul><ul><li>Design & Code </li></ul><ul><ul><li>Testability of the application with load testing tools </li></ul></ul><ul><ul><li>Design solutions </li></ul></ul><ul><ul><ul><li>Bottlenecks, key algorithms, clustering </li></ul></ul></ul><ul><li>Testing </li></ul><ul><ul><li>Data gathered in RA is crucial </li></ul></ul><ul><ul><li>Automate and parameterizes scripts </li></ul></ul><ul><ul><li>Measure </li></ul></ul><ul><ul><li>Use profiler to identify bottlenecks </li></ul></ul>
  15. 15. Nixon B. A., 2000
  16. 16. QA of Usability <ul><li>Usability – Quality in Use </li></ul><ul><ul><li>The capability of the software to be understood learned used and liked by the user under specified conditions (ISO 9126-1) </li></ul></ul><ul><li>Understand context and users </li></ul><ul><li>Design for Usability </li></ul><ul><ul><li>Field observation, interviews, prototypes </li></ul></ul><ul><li>Usability reviews, e.g. </li></ul><ul><ul><li>Heuristic evaluation </li></ul></ul><ul><ul><li>Cognitive walkthrough with task scenarios </li></ul></ul><ul><li>Usability testing, e.g. </li></ul><ul><ul><li>Thinking aloud protocol in usability lab </li></ul></ul><ul><ul><li>Co-discovery (Pair usability testing) </li></ul></ul><ul><ul><li>Surveys and questionnaires </li></ul></ul>
  17. 17. Outline <ul><li>Introduction </li></ul><ul><li>QA of performance and usability (short) </li></ul><ul><li>QA of evolvability (long) </li></ul><ul><li>How are qualities effected by </li></ul><ul><ul><li>code review, </li></ul></ul><ul><ul><li>pair-programming </li></ul></ul><ul><ul><li>exploratory testing </li></ul></ul><ul><li>Practical approach </li></ul>
  18. 18. Quality assurance of evolvability <ul><li>Design </li></ul><ul><li>Company Standards / Coding conventions </li></ul><ul><li>Code inspections </li></ul><ul><li>Assessing software evolvability with tools </li></ul><ul><li>Improving software evolvability </li></ul><ul><li>Knowledge of the system </li></ul><ul><li>Environment support </li></ul>
  19. 19. Method 1 for evolvability: Design <ul><li>Design for maintainable software </li></ul><ul><ul><li>Principles </li></ul></ul><ul><ul><ul><li>Low and loose coupling </li></ul></ul></ul><ul><ul><ul><ul><li>System level, E.g. </li></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>No database specific functionality </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><li>Subsystem level, E.g. </li></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Clear separation of responsibilities </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>Interaction through Interfaces </li></ul></ul></ul></ul></ul><ul><ul><ul><li>High cohesion </li></ul></ul></ul><ul><ul><ul><li>Information hiding </li></ul></ul></ul><ul><ul><li>Most probable change scenarios should be evaluated </li></ul></ul><ul><ul><li>Different software component have, e.g. GUI vs. Encryption </li></ul></ul><ul><ul><ul><li>different lifetimes </li></ul></ul></ul><ul><ul><ul><li>different modification rates </li></ul></ul></ul><ul><li>Collaborative design </li></ul><ul><li>Design review </li></ul>
  20. 20. Method 2 for evolvability: Coding conventions <ul><li>Program layout has great impact to program comprehension </li></ul><ul><li>Documentation </li></ul><ul><ul><li>Naming of code elements </li></ul></ul><ul><ul><ul><li>Should reflect their intent </li></ul></ul></ul><ul><ul><ul><li>Naming standard </li></ul></ul></ul><ul><ul><li>Commenting </li></ul></ul><ul><ul><ul><li>Studies have shown that too much comments are also harmful ! </li></ul></ul></ul><ul><ul><ul><li>Use when naming does not clearly state the intent </li></ul></ul></ul><ul><ul><ul><ul><li>e.g. “Check needed because of bug in Java virtual machine” </li></ul></ul></ul></ul><ul><li>Structure – Rules for acceptable structure </li></ul><ul><ul><li>Explicit metrics </li></ul></ul><ul><ul><ul><li>“ Do not write methods over 100 lines” </li></ul></ul></ul><ul><ul><li>Or heuristics </li></ul></ul><ul><ul><ul><li>“ Make sure all methods perform single explicit task” </li></ul></ul></ul>
  21. 21. Method 3 for evolvability: Code review <ul><li>Spreads knowledge of the system </li></ul><ul><ul><li>Increases evolvability inderectly </li></ul></ul><ul><li>Distribution of CR issue (performance under functionality) </li></ul>
  22. 22. Method 4 for evolvability: Tools <ul><li>Code metrics </li></ul><ul><ul><li>E.g. Lines of code, number of coupling, execution paths </li></ul></ul><ul><ul><li>Quantify the vague concept of evolvability </li></ul></ul><ul><ul><li>Predict future development effort </li></ul></ul><ul><ul><li>Also industrial adoption </li></ul></ul><ul><li>Reverse-engineering </li></ul><ul><ul><li>i.e. identify the system's components and their interrelationships, e.g. code -> UML </li></ul></ul><ul><ul><li>Analyze whether the design matches the actual source </li></ul></ul><ul><li>Metrics </li></ul><ul><ul><li>Pro’s </li></ul></ul><ul><ul><ul><li>Provides objective numbers </li></ul></ul></ul><ul><ul><ul><li>Does not require in depth understanding of the system </li></ul></ul></ul><ul><ul><li>Con’s </li></ul></ul><ul><ul><ul><li>Does not say anything about the correctness (quality) </li></ul></ul></ul><ul><ul><ul><ul><li>e.g. Does the class have the correct associations </li></ul></ul></ul></ul><ul><li>Reverse-engineering </li></ul><ul><ul><li>Pro’s </li></ul></ul><ul><ul><ul><li>Provides higher level view of the source code </li></ul></ul></ul><ul><ul><ul><li>Can be compared with the intended design </li></ul></ul></ul><ul><ul><li>Con’s </li></ul></ul><ul><ul><ul><li>Requires in depth understanding of the system </li></ul></ul></ul>
  23. 23. Method 5 for evolvability: Improve <ul><li>Terms </li></ul><ul><ul><li>Refactoring - Controlled way to improve the software’s structure without changing its observable behavior </li></ul></ul><ul><ul><li>Re-engineering - Examination and alteration of a subject system to reconstitute it in a new form and the subsequent implementation of the new form </li></ul></ul><ul><ul><li>Re-writing – Recreate the product starting with clean slate </li></ul></ul><ul><li>All of the above can come handy during software evolution </li></ul><ul><ul><li>Refactoring - continuous activity performed by all software developers </li></ul></ul><ul><ul><li>Re-engineering - when current system structure is no longer usable </li></ul></ul><ul><ul><li>Rewriting </li></ul></ul><ul><ul><ul><li>When re-engineering is not enough </li></ul></ul></ul><ul><ul><ul><li>When moving to different new technology/language </li></ul></ul></ul>
  24. 24. Method 6 for evolvability: Knowledge distribution <ul><li>Further development is harder, if expertise is lost </li></ul><ul><ul><li>Expertise = solutions and locations to domain problems, domain knowledge, software architecture, design decisions </li></ul></ul><ul><li>Good HRM policy can help maintain knowledge </li></ul><ul><li>Regardless there is risk of losing the people with knowledge </li></ul><ul><ul><li>Retirement, moving to area, dying, changing career etc. </li></ul></ul><ul><li>Strategies to keep vital knowledge of the system </li></ul><ul><ul><li>Documenting </li></ul></ul><ul><ul><li>Spreading knowledge to several people </li></ul></ul>
  25. 25. Method 7 for evolvability: Environment support <ul><li>Good unit test harness increases evolvability </li></ul><ul><ul><li>Increases courage to </li></ul></ul><ul><ul><ul><li>improve software structure </li></ul></ul></ul><ul><ul><ul><li>add new features </li></ul></ul></ul><ul><ul><li>Reduces developers test efforts </li></ul></ul><ul><li>Other environment support can have similar effect </li></ul><ul><ul><li>“ We don't need unit test as we have excessive number of testers” </li></ul></ul>
  26. 26. Quality assurance of evolvability <ul><li>Design </li></ul><ul><li>Company Standards / Coding conventions </li></ul><ul><li>Code inspections </li></ul><ul><li>Assessing software evolvability with tools </li></ul><ul><li>Improving software evolvability </li></ul><ul><li>Knowledge of the system </li></ul><ul><li>Environment support </li></ul>
  27. 27. Outline <ul><li>Introduction </li></ul><ul><li>QA of performance and usability (short) </li></ul><ul><li>QA of evolvability (long) </li></ul><ul><li>How are qualities effected by </li></ul><ul><ul><li>code review, </li></ul></ul><ul><ul><li>pair-programming </li></ul></ul><ul><ul><li>exploratory testing </li></ul></ul><ul><li>Practical approach </li></ul>
  28. 28. Code review’s effect to qualities <ul><li>Spreads knowledge of the system </li></ul><ul><ul><li>Increases evolvability inderectly </li></ul></ul><ul><li>Distribution of CR issue (performance under functionality) </li></ul>
  29. 29. Pair programming’s effect to qualities <ul><li>Increases understandability </li></ul><ul><li>Makes design better (evolvability) </li></ul>
  30. 30. Exploratory testing’s effect to qualaties <ul><li>Single study: Exploratory testing (ET) vs Test Case Testing (TCT) </li></ul><ul><li>ET found more usability and GUI issues </li></ul><ul><li>TCT found more performance issues </li></ul>1059 111 % 503 556 Total 502 110 % 239 263 Wrong function 24 380 % 5 19 Usability 120 82 % 66 54 Technical defect 80 95 % 41 39 Performance 194 102 % 96 98 Missing function 8 167 % 3 5 Inconsistency 119 143 % 49 70 GUI 12 200 % 4 8 Documentation Total ET/TCT (%) TCT ET Type
  31. 31. Outline <ul><li>Introduction </li></ul><ul><li>QA of performance and usability (short) </li></ul><ul><li>QA of evolvability (long) </li></ul><ul><li>How are qualities effected by </li></ul><ul><ul><li>code review, </li></ul></ul><ul><ul><li>pair-programming </li></ul></ul><ul><ul><li>exploratory testing </li></ul></ul><ul><li>Practical approach </li></ul>
  32. 32. Teams with different goals 1 1 5 5 2-3 Team5 (output) 2 2 3 3 4 Team4 (clarity) 4 4 1 2 5 Team3 (memory) 5 3 2 1 2-3 Team2 (statem.) 3 5 4 4 1 Team1 (effort) Rank output Rank clarity Rank memory Rank statem. Rank effort
  33. 33. Practical approach 1 2 3 3 2 3 Multi-platform build 3 3 1 3 1 2 Load Tests 3 3 2 1 2 1 Test sessions 1 1 2 3 2 1 Code Review Portability Maintainability Efficiency Usability Reliability Functionality   Company specific list of practices <ul><li>Company specific list of quality attributes, e.g.: </li></ul><ul><li>Only developing Windows application get rid of Portability </li></ul><ul><li>Developing crypto library remove Usability take in Security </li></ul>
  34. 34. Summary <ul><li>Many quality attributes and many quality assurance methods exist </li></ul><ul><ul><li>Lecture series instead of single lecture? </li></ul></ul><ul><li>Plenty of research on assuring certain qualities </li></ul><ul><ul><li>Usability, Security, Evolvability (=Maintainability) </li></ul></ul><ul><li>Lack of research on qa methods effects on qualities </li></ul><ul><ul><li>Effects of pair-programming, exploratory testing to qualities </li></ul></ul><ul><li>Exists specific methods for specific qualities </li></ul><ul><ul><li>Usability testing </li></ul></ul>
  35. 35. Topics <ul><li>Consulting plan for a company </li></ul><ul><li>How to test Unit tests effect to quality X </li></ul>

×