• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Quality assurance and testing of quality attributes
 

Quality assurance and testing of quality attributes

on

  • 1,666 views

 

Statistics

Views

Total Views
1,666
Views on SlideShare
1,662
Embed Views
4

Actions

Likes
0
Downloads
14
Comments
0

1 Embed 4

http://www.slideshare.net 4

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Quality assurance and testing of quality attributes Quality assurance and testing of quality attributes Presentation Transcript

    • Quality assurance of non-functional requirements Mika Mäntylä TKK/SoberIT/SPRG T-76.5650 Software engineering seminar
    • Outline
      • Introduction
      • QA of performance and usability (short)
      • QA of evolvability (long)
      • How are qualities effected by
        • code review,
        • pair-programming
        • exploratory testing
      • Practical approach
    • QA of … Reliability Interoperability Performance Usability Scalability Security Availability Maintainability Modifiability Testability Safety Survivability Confidentiality Portability Dependability Timeliness Integrity Fault-tolerance
    • … with TDD Module Testing Load testing Exploratory testing Bounce checker Test case based testing Code review Load tester Pair-programming GUI robot Unit testing Functional testing Acceptance testing External beta-testers Usability testing Static analysis Iteration Demo Structural testing
    • TDD Module Testing Load testing Exploratory testing Test case based testing Code review Load tester Pair-programming Unit testing Functional testing Acceptance testing External beta-testers Usability testing Static analysis Iteration Demo Structural testing Reliability Interoperability Performance Usability Scalability Security Availability Maintainability Modifiability Testability Safety Survivability Confidentiality Portability Dependability Timeliness Integrity Fault-tolerance Qualities Methods
    • TDD Module Testing Load testing Exploratory testing Test case based testing Code review Load tester Pair-programming Unit testing Functional testing Acceptance testing External beta-testers Usability testing Static analysis Iteration Demo Structural testing Reliability Interoperability Performance Usability Scalability Security Availability Maintainability Modifiability Testability Safety Survivability Confidentiality Portability Dependability Timeliness Integrity Fault-tolerance
    • TDD Module Testing Load testing Exploratory testing Test case based testing Code review Load tester Pair-programming Unit testing Functional testing Acceptance testing External beta-testers Usability testing Static analysis Iteration Demo Structural testing Reliability Interoperability Performance Usability Scalability Security Availability Maintainability Modifiability Testability Safety Survivability Confidentiality Portability Dependability Timeliness Integrity Fault-tolerance
    • TDD Module Testing Load testing Exploratory testing Test case based testing Code review Load tester Pair-programming Unit testing Functional testing Acceptance testing External beta-testers Usability testing Static analysis Iteration Demo Structural testing Reliability Interoperability Performance Usability Scalability Security Availability Maintainability Modifiability Testability Safety Survivability Confidentiality Portability Dependability Timeliness Integrity Fault-tolerance Qualities and methods have many to many relationship
    • TDD Module Testing Load testing Exploratory testing Test case based testing Code review Load tester Pair-programming Unit testing Functional testing Acceptance testing External beta-testers Usability testing Static analysis Iteration Demo Structural testing Reliability Interoperability Performance Usability Scalability Security Availability Maintainability Modifiability Testability Safety Survivability Confidentiality Portability Dependability Timeliness Integrity Fault-tolerance Relationships between qualities and methods are poorly understood
    • TDD Module Testing Load testing Exploratory testing Test case based testing Code review Load tester Pair-programming Unit testing Functional testing Acceptance testing External beta-testers Usability testing Static analysis Iteration Demo Structural testing Reliability Interoperability Performance Usability Scalability Security Availability Maintainability Modifiability Testability Safety Survivability Confidentiality Portability Dependability Timeliness Integrity Fault-tolerance Method may not be the most important factor affecting a quality
    • TDD Module Testing Load testing Exploratory testing Test case based testing Code review Load tester Pair-programming Unit testing Functional testing Acceptance testing External beta-testers Usability testing Static analysis Iteration Demo Structural testing Reliability Interoperability Performance Usability Scalability Security Availability Maintainability Modifiability Testability Safety Survivability Confidentiality Portability Dependability Timeliness Integrity Fault-tolerance Specific methods for specific qualities
    • Summary
      • Qualities and methods have many to many relationship
      • Often relationships between qualities and methods are poorly understood
      • Method may not be the key factor when assuring particular quality
        • Code review’s effect on security
          • Largely dependent on the reviewer’s instructions
      • Specific methods exist for some qualities
        • Usability testing
        • Load testing
    • Outline
      • Introduction
      • QA of performance and usability (short)
      • QA of evolvability (long)
      • How are qualities effected by
        • code review,
        • pair-programming
        • exploratory testing
      • Practical approach
    • QA of performance
      • Requirements analysis
        • Estimate data volumes and user volume & behavior
          • Peak usage vs. Regular usage
          • Realistic data volumes
        • Acceptable response time
        • Hardware restrictions
          • With clustered server you can always buy more servers with mobile phone you cannot
      • Design & Code
        • Testability of the application with load testing tools
        • Design solutions
          • Bottlenecks, key algorithms, clustering
      • Testing
        • Data gathered in RA is crucial
        • Automate and parameterizes scripts
        • Measure
        • Use profiler to identify bottlenecks
    • Nixon B. A., 2000
    • QA of Usability
      • Usability – Quality in Use
        • The capability of the software to be understood learned used and liked by the user under specified conditions (ISO 9126-1)
      • Understand context and users
      • Design for Usability
        • Field observation, interviews, prototypes
      • Usability reviews, e.g.
        • Heuristic evaluation
        • Cognitive walkthrough with task scenarios
      • Usability testing, e.g.
        • Thinking aloud protocol in usability lab
        • Co-discovery (Pair usability testing)
        • Surveys and questionnaires
    • Outline
      • Introduction
      • QA of performance and usability (short)
      • QA of evolvability (long)
      • How are qualities effected by
        • code review,
        • pair-programming
        • exploratory testing
      • Practical approach
    • Quality assurance of evolvability
      • Design
      • Company Standards / Coding conventions
      • Code inspections
      • Assessing software evolvability with tools
      • Improving software evolvability
      • Knowledge of the system
      • Environment support
    • Method 1 for evolvability: Design
      • Design for maintainable software
        • Principles
          • Low and loose coupling
            • System level, E.g.
              • No database specific functionality
            • Subsystem level, E.g.
              • Clear separation of responsibilities
              • Interaction through Interfaces
          • High cohesion
          • Information hiding
        • Most probable change scenarios should be evaluated
        • Different software component have, e.g. GUI vs. Encryption
          • different lifetimes
          • different modification rates
      • Collaborative design
      • Design review
    • Method 2 for evolvability: Coding conventions
      • Program layout has great impact to program comprehension
      • Documentation
        • Naming of code elements
          • Should reflect their intent
          • Naming standard
        • Commenting
          • Studies have shown that too much comments are also harmful !
          • Use when naming does not clearly state the intent
            • e.g. “Check needed because of bug in Java virtual machine”
      • Structure – Rules for acceptable structure
        • Explicit metrics
          • “ Do not write methods over 100 lines”
        • Or heuristics
          • “ Make sure all methods perform single explicit task”
    • Method 3 for evolvability: Code review
      • Spreads knowledge of the system
        • Increases evolvability inderectly
      • Distribution of CR issue (performance under functionality)
    • Method 4 for evolvability: Tools
      • Code metrics
        • E.g. Lines of code, number of coupling, execution paths
        • Quantify the vague concept of evolvability
        • Predict future development effort
        • Also industrial adoption
      • Reverse-engineering
        • i.e. identify the system's components and their interrelationships, e.g. code -> UML
        • Analyze whether the design matches the actual source
      • Metrics
        • Pro’s
          • Provides objective numbers
          • Does not require in depth understanding of the system
        • Con’s
          • Does not say anything about the correctness (quality)
            • e.g. Does the class have the correct associations
      • Reverse-engineering
        • Pro’s
          • Provides higher level view of the source code
          • Can be compared with the intended design
        • Con’s
          • Requires in depth understanding of the system
    • Method 5 for evolvability: Improve
      • Terms
        • Refactoring - Controlled way to improve the software’s structure without changing its observable behavior
        • Re-engineering - Examination and alteration of a subject system to reconstitute it in a new form and the subsequent implementation of the new form
        • Re-writing – Recreate the product starting with clean slate
      • All of the above can come handy during software evolution
        • Refactoring - continuous activity performed by all software developers
        • Re-engineering - when current system structure is no longer usable
        • Rewriting
          • When re-engineering is not enough
          • When moving to different new technology/language
    • Method 6 for evolvability: Knowledge distribution
      • Further development is harder, if expertise is lost
        • Expertise = solutions and locations to domain problems, domain knowledge, software architecture, design decisions
      • Good HRM policy can help maintain knowledge
      • Regardless there is risk of losing the people with knowledge
        • Retirement, moving to area, dying, changing career etc.
      • Strategies to keep vital knowledge of the system
        • Documenting
        • Spreading knowledge to several people
    • Method 7 for evolvability: Environment support
      • Good unit test harness increases evolvability
        • Increases courage to
          • improve software structure
          • add new features
        • Reduces developers test efforts
      • Other environment support can have similar effect
        • “ We don't need unit test as we have excessive number of testers”
    • Quality assurance of evolvability
      • Design
      • Company Standards / Coding conventions
      • Code inspections
      • Assessing software evolvability with tools
      • Improving software evolvability
      • Knowledge of the system
      • Environment support
    • Outline
      • Introduction
      • QA of performance and usability (short)
      • QA of evolvability (long)
      • How are qualities effected by
        • code review,
        • pair-programming
        • exploratory testing
      • Practical approach
    • Code review’s effect to qualities
      • Spreads knowledge of the system
        • Increases evolvability inderectly
      • Distribution of CR issue (performance under functionality)
    • Pair programming’s effect to qualities
      • Increases understandability
      • Makes design better (evolvability)
    • Exploratory testing’s effect to qualaties
      • Single study: Exploratory testing (ET) vs Test Case Testing (TCT)
      • ET found more usability and GUI issues
      • TCT found more performance issues
      1059 111 % 503 556 Total 502 110 % 239 263 Wrong function 24 380 % 5 19 Usability 120 82 % 66 54 Technical defect 80 95 % 41 39 Performance 194 102 % 96 98 Missing function 8 167 % 3 5 Inconsistency 119 143 % 49 70 GUI 12 200 % 4 8 Documentation Total ET/TCT (%) TCT ET Type
    • Outline
      • Introduction
      • QA of performance and usability (short)
      • QA of evolvability (long)
      • How are qualities effected by
        • code review,
        • pair-programming
        • exploratory testing
      • Practical approach
    • Teams with different goals 1 1 5 5 2-3 Team5 (output) 2 2 3 3 4 Team4 (clarity) 4 4 1 2 5 Team3 (memory) 5 3 2 1 2-3 Team2 (statem.) 3 5 4 4 1 Team1 (effort) Rank output Rank clarity Rank memory Rank statem. Rank effort
    • Practical approach 1 2 3 3 2 3 Multi-platform build 3 3 1 3 1 2 Load Tests 3 3 2 1 2 1 Test sessions 1 1 2 3 2 1 Code Review Portability Maintainability Efficiency Usability Reliability Functionality   Company specific list of practices
      • Company specific list of quality attributes, e.g.:
      • Only developing Windows application get rid of Portability
      • Developing crypto library remove Usability take in Security
    • Summary
      • Many quality attributes and many quality assurance methods exist
        • Lecture series instead of single lecture?
      • Plenty of research on assuring certain qualities
        • Usability, Security, Evolvability (=Maintainability)
      • Lack of research on qa methods effects on qualities
        • Effects of pair-programming, exploratory testing to qualities
      • Exists specific methods for specific qualities
        • Usability testing
    • Topics
      • Consulting plan for a company
      • How to test Unit tests effect to quality X