Outsourced Product Development: Product QA Services
Upcoming SlideShare
Loading in...5

Outsourced Product Development: Product QA Services






Total Views
Views on SlideShare
Embed Views



6 Embeds 138

http://www.pramatiservices.com 120
http://www.slideshare.net 12
http://pramatiservices.com 3
http://blog.pramatiservices.com 1 1
http://www.coolqatester.blogspot.com 1



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

Outsourced Product Development: Product QA Services Outsourced Product Development: Product QA Services Presentation Transcript

  • Overview of Product QA
  • Product QA: Why it is tough
    • Product is more than software
      • Software application
      • Platform and variations
      • Documentation: online help, manuals, tech briefs, examples, training
      • Add-ins and third party product integrations
      • Installation experience
      • Handling upgrades and updates
  • Product QA: Why it is tough
    • Quality is more than lack of bugs
      • Does it solve the problem when it works?
      • Is it practical to use? Is it designed for efficient use?
      • Does it keep working or works only under ideal conditions?
      • Is it economical to build, maintain and respond rapidly to customer/user needs?
    View slide
    • QA is everything you do to minimize risk of failure and promote excellence
    • Help producing products that satisfy target customer base
    Role of QA in Product Company View slide
  • Role of QA in Product Company
    • Broad functions of QA
      • Risk management: release planning, release approval
      • Bring customer involvement: early release programs
      • Act as skillful developers: mimic tough customer!
      • Process definition and improvement
      • Inspection and testing
      • Experience-based improvement: contribute to product definition, many finer points
    • Should play an important role throughout The Life of a Product
  • Structure and Function of QA 5 4 3 2 1 6 Planning Design Coding Testing Release Post-Release Process Definition Process Improvement QA Planning Risk Assessment QA Certification Release Inspection Test Development Test Suite Design Product Definition Inputs Early Release Program Test Execution Code Review Design Review
  • Non-testing Functions
    • What QA is required to do
    • QA procedures, expectations from other teams
    • Effort estimation and team planning
    • Resource planning, tools and software procurement
    QA Planning
    • Assessment of the state of the processes
    • Setting process goals and reviewing
    • Recommendations for improvement with justification
    • Action plan for implementing improvements
    Process Definition & Improvement
    • Design the tests in parallel to software design
    • Tests need organization into independent test suites
    • Build the tests before you build the software
    • If you find a bug, write a test that proves the bug exists, fix the bug and then rerun the test to prove it is fixed
    Test Suite Design
    • Goal: Clarification and accept/reject design decisions
    • Starts with a presentation of the design for the component/module to a selected panel of developers
    • Peer review is commonly used, with most relevant engineers from other parts of the product included
    • Discussion and coordination of comments incorporation
    Test Suite Design Review
    • Check-in review done with peers of the dev team
      • Including new features and bug fixes
    • Code walkthrough by dev across streams, including QA
      • review meeting with prepared participants
      • Sample source files are selected for review
    • Focus is on finding defects, before they slip through to testing phases
      • Lower costs by catching them early
    Check-in Review, Walkthrough
    • Need to ensure smooth porting of current customers
    • Test ISV applications available internally
    • Public program to increase the test base
      • Alpha and Beta testing
    Early Release Programs
    • QA evaluates quality, independent of developers
      • Developers should not determine how much quality is sufficient
    • QA acts as internal customers and verifies release readiness
    Release Readiness
  • Product Testing
  • Our Test Classification Verify performance and scalability of the product; also produce data on installation requirements Load Negative testing to ensure that the product functions gracefully under unexpected conditions Reliability Testing the product for practicality, efficiency of use and ease of learning of the product interface Usability Test suite for compliance to industry standards Compliance Verify that functional requirements of the product are as per specifications Functional What purpose the test is serving Final set of testing done by QA to approve GA release Final testing Testing done with the help of external partners, early release customers, etc. Beta testing Testing done on code release to QA Alpha testing Basic set of tests run after every nightly build, to ensure that nothing is broken Sanity testing When a test is being run Test uses product internals, even calls unexposed functions White box Test does not use knowledge of product code and design Black box How a test is written Description Class Based on
    • Key aspects to cover (they are endless!)
      • Correctness
      • Interoperability
      • Compatibility
      • Compliance
      • Security
    • Think of permutations
      • Functions vs. input data vs. states
      • Product vs. platforms
      • System vs. external factors
      • Testing vs. versions of product
    Functional Testing
    • Emphasis on when product is expected to be declared “compliant” with Industry Standard (such as J2EE™)
    • Compliance test suite (TCK for J2EE) might be made available
    • Compliance to standards does not mean the product is functionally tested for use
    Compliance Testing
    • Performance testing
      • Performance that can be achieved on a configuration setup
      • Deterioration with load
      • Variations across platforms
      • Writing a sizing guide with suggested deployment configurations
      • Generating a near real-world load
    • Scalability testing
      • How does the system scale with increase in platform capacity
      • Fail over and load balancing under clustered configurations
    • Benchmarking
      • Comparative performance testing and analysis with competitor products
      • Public benchmarking, such as SPECjAppserver™ or ECperf™
    Load Testing
  • Performance Reports
    • Maturity: how the product expects erroneous user handling
    • Fault-tolerance: Ability to perform under faulty hardware/platform conditions with degraded performance but near full functionality
    • Integrity: How the system cannot be broken by malicious users
    Reliability Testing Aspects
  • Reliability Testing Aspects...
    • Recoverability: ability of the product to recover to a consistent state when power outage, network fault or other disaster happens
    • Safety: protection of user data, backup, logging, etc.
    • Understandability: clarity of interface, intuitiveness
    • Learnability: ease of learning, matches user’s current understanding of other products
    • Consistency: across versions, all across the product
    Usability Testing Aspects
  • Usability Testing Aspects...
    • Its not only about GUI, its also about file formats, XML, alternatives such as command line tools
    • We normally work with external consultants who are specialized in usability engineering
    • Efficiency
      • Storage
      • Processing
    • Maintainability
      • Code organization, branching
      • Modularity for ease of changes, maintenance, re-factoring
    • Portability
      • Platform support
      • Reusability of code
    Other Aspects of Testing Some of these aspects may not be important to end-customers, but very important for engineering
    • Document the design
    • Use internal error checking
    • Test units before integrating them
    • Tell the tester what’s new or flaky
    • Test the build yourself, first
    • Evolve product in functional layers
    Making Testing Easier
  • Tools and Infrastructure
    • Test automation helps repeatability of testing
    • Useful test automation is a major software project
    • Engineers are able to detect more problems, more accurately
    Testing Automation
  • Test Case Management System
  • J2EE™ Compliance Testing
    • J2EE CTS Kit
      • From Sun
    • Standards Verification
    • Automated Harness
    • Augment CTS
      • For Complex scenarios
    • Built on Studio Framework
      • Homegrown
    • Ease of writing new tests
    • Automated Discovery of new tests
    • Extensive tests metadata
      • Compose new plans/suites
    • Integrated into the Build System
      • For automated post-build testing
    Test Management Framework
  • Test Management Framework
  • Reliability Testing
  • Distributed Stress Testing System
  • Thank you.