Product QA – A Test engineering perspective Ensuring products are tested the way they will be used

Product QA: Why it is Tough Product is more than software Software application Platform and variations Documentation: online help, manuals, tech briefs, examples, training Add-ins and third party product integrations Installation experience Handling upgrades and updates Quality is more than lack of bugs Does it solve the problem when it works? Is it practical to use? Is it designed for efficient use? Does it keep working or works only under ideal conditions? Is it economical to build, maintain and respond rapidly to customer/user needs?

Product is more than software

Software application

Platform and variations

Documentation: online help, manuals, tech briefs, examples, training

Add-ins and third party product integrations

Installation experience

Handling upgrades and updates

Quality is more than lack of bugs

Does it solve the problem when it works?

Is it practical to use? Is it designed for efficient use?

Does it keep working or works only under ideal conditions?

Is it economical to build, maintain and respond rapidly to customer/user needs?

Role of QA in Product Company QA is everything you do to minimize risk of failure and promote excellence Help producing products that satisfy target customer base Broad functions of QA Risk management: release planning, release approval Bring customer involvement: early release programs Act as skillful developers: mimic tough customer! Process definition and improvement Inspection and testing Experience-based improvement: contribute to product definition, many finer points Should play an important role throughout The Life of a Product

QA is everything you do to minimize risk of failure and promote excellence

Help producing products that satisfy target customer base

Broad functions of QA

Risk management: release planning, release approval

Bring customer involvement: early release programs

Act as skillful developers: mimic tough customer!

Process definition and improvement

Inspection and testing

Experience-based improvement: contribute to product definition, many finer points

Should play an important role throughout The Life of a Product

Hawk-Eye Methodology-Precision Delivered 5 4 3 2 1 6 Planning Design Coding Testing Release Post-Release Process Definition Process Improvement QA Planning Risk Assessment QA Certification Release Inspection Test Development Test Suite Design Product Definition Inputs Early Release Program Test Execution Code Review Design Review

Our offerings

Services

Business landscape

Non-testing Functions

QA Planning What QA is required to do QA procedures, expectations from other teams Effort estimation and team planning Resource planning, tools and software procurement

What QA is required to do

QA procedures, expectations from other teams

Effort estimation and team planning

Resource planning, tools and software procurement

Process Definition & Improvement Assessment of the state of the processes Setting process goals and reviewing Recommendations for improvement with justification Action plan for implementing improvements

Assessment of the state of the processes

Setting process goals and reviewing

Recommendations for improvement with justification

Action plan for implementing improvements

Test Suite Design Design the tests in parallel to software design Tests need organization into independent test suites Build the tests before you build the software If you find a bug, write a test that proves the bug exists, fix the bug and then rerun the test to prove it is fixed

Design the tests in parallel to software design

Tests need organization into independent test suites

Build the tests before you build the software

If you find a bug, write a test that proves the bug exists, fix the bug and then rerun the test to prove it is fixed

Test Suite Design Review Goal: Clarification and accept/reject design decisions Starts with a presentation of the design for the component/module to a selected panel of developers Peer review is commonly used, with most relevant engineers from other parts of the product included Discussion and coordination of comments incorporation

Goal: Clarification and accept/reject design decisions

Starts with a presentation of the design for the component/module to a selected panel of developers

Peer review is commonly used, with most relevant engineers from other parts of the product included

Discussion and coordination of comments incorporation

Check-in Review & Code Walkthrough Check-in review done with peers of the dev team Including new features and bug fixes Code walkthrough by dev across streams, including QA review meeting with prepared participants Sample source files are selected for review Focus is on finding defects, before they slip through to testing phases Lower costs by catching them early

Check-in review done with peers of the dev team

Including new features and bug fixes

Code walkthrough by dev across streams, including QA

review meeting with prepared participants

Sample source files are selected for review

Focus is on finding defects, before they slip through to testing phases

Lower costs by catching them early

Early Release Programs Need to ensure smooth porting of current customers Test ISV applications available internally Public program to increase the test base Alpha and Beta testing

Need to ensure smooth porting of current customers

Test ISV applications available internally

Public program to increase the test base

Alpha and Beta testing

Release Readiness QA evaluates quality, independent of developers Developers should not determine how much quality is sufficient QA acts as internal customers and verifies release readiness

QA evaluates quality, independent of developers

Developers should not determine how much quality is sufficient

QA acts as internal customers and verifies release readiness

Our Test Classification Based on Class Description How a test is written Black box Test does not use knowledge of product code and design White box Test uses product internals, even calls unexposed functions When a test is being run Sanity testing Basic set of tests run after every nightly build, to ensure that nothing is broken Alpha testing Testing done on code release to QA Beta testing Testing done with the help of external partners, early release customers, etc. Final testing Final set of testing done by QA to approve GA release What purpose the test is serving Functional Verify that functional requirements of the product are as per specifications Compliance Test suite for compliance to industry standards Load Verify performance and scalability of the product; also produce data on installation requirements Reliability Negative testing to ensure that the product functions gracefully under unexpected conditions Usability Testing the product for practicality, efficiency of use and ease of learning of the product interface

Functional Testing Key aspects to cover (they are endless!) Correctness Interoperability Compatibility Compliance Security … Think of permutations Functions vs. input data vs. states Product vs. platforms System vs. external factors Testing vs. versions of product

Key aspects to cover (they are endless!)

Correctness

Interoperability

Compatibility

Compliance

Security

…

Think of permutations

Functions vs. input data vs. states

Product vs. platforms

System vs. external factors

Testing vs. versions of product

Compliance Testing Emphasis on when product is expected to be declared “compliant” with Industry Standard (such as J2EE™) Compliance test suite (TCK for J2EE) might be made available Compliance to standards does not mean the product is functionally tested for use

Emphasis on when product is expected to be declared “compliant” with Industry Standard (such as J2EE™)

Compliance test suite (TCK for J2EE) might be made available

Compliance to standards does not mean the product is functionally tested for use

Load Testing Performance testing Performance that can be achieved on a configuration setup Deterioration with load Variations across platforms Writing a sizing guide with suggested deployment configurations Generating a near real-world load Scalability testing How does the system scale with increase in platform capacity Fail over and load balancing under clustered configurations Benchmarking Comparative performance testing and analysis with competitor products Public benchmarking, such as SPECjAppserver™ or ECperf™

Performance testing

Performance that can be achieved on a configuration setup

Deterioration with load

Variations across platforms

Writing a sizing guide with suggested deployment configurations

Generating a near real-world load

Scalability testing

How does the system scale with increase in platform capacity

Fail over and load balancing under clustered configurations

Benchmarking

Comparative performance testing and analysis with competitor products

Public benchmarking, such as SPECjAppserver™ or ECperf™

Performance Reports

Reliability Testing Aspects to test Maturity: how the product expects erroneous user handling Fault-tolerance: Ability to perform under faulty hardware/platform conditions with degraded performance but near full functionality Integrity: How the system cannot be broken by malicious users Recoverability: ability of the product to recover to a consistent state when power outage, network fault or other disaster happens Safety: protection of user data, backup, logging, etc.

Aspects to test

Maturity: how the product expects erroneous user handling

Fault-tolerance: Ability to perform under faulty hardware/platform conditions with degraded performance but near full functionality

Integrity: How the system cannot be broken by malicious users

Recoverability: ability of the product to recover to a consistent state when power outage, network fault or other disaster happens

Safety: protection of user data, backup, logging, etc.

Usability Testing Aspects to cover Understandability: clarity of interface, intuitiveness Learnability: ease of learning, matches user’s current understanding of other products Consistency: across versions, all across the product Its not only about GUI, its also about file formats, XML, alternatives such as command line tools We normally work with external consultants who are specialized in usability engineering

Aspects to cover

Understandability: clarity of interface, intuitiveness

Learnability: ease of learning, matches user’s current understanding of other products

Consistency: across versions, all across the product

Its not only about GUI, its also about file formats, XML, alternatives such as command line tools

We normally work with external consultants who are specialized in usability engineering

Other Aspects for Testing Some of these aspects may not be important to end-customers, but very important for engineering Key aspects include Efficiency Storage Processing Maintainability Code organization, branching Modularity for ease of changes, maintenance, refactoring Portability Platform support Reusability of code

Some of these aspects may not be important to end-customers, but very important for engineering

Key aspects include

Efficiency

Storage

Processing

Maintainability

Code organization, branching

Modularity for ease of changes, maintenance, refactoring

Portability

Platform support

Reusability of code

Making Testing Easier Document the design Use internal error checking Test units before integrating them Tell the tester what’s new or flaky Test the build yourself, first Evolve product in functional layers

Document the design

Use internal error checking

Test units before integrating them

Tell the tester what’s new or flaky

Test the build yourself, first

Evolve product in functional layers

Tools and Infrastructure

Test Automation Test automation helps repeatability of testing Useful test automation is a major software project Engineers are able to detect more problems, more accurately

Test automation helps repeatability of testing

Useful test automation is a major software project

Engineers are able to detect more problems, more accurately

Industry standard skills

In-House Tools

BrighTest Automation tool developed in-house Leverages Selenium and Java API based approach Easily customizable to suite specific project requirements Ability to store reports in our own format for future use. Easy to install. Easily redefined parameters Customized Alerts and Reports. Weekly and Nightly build integrations Applicable to testing of web based applications and products as well as Social Platforms Pilot usage on for a couple of happy customer engagements. Extensively used to test Pramati’s Enterprise 2.0 product Qontext .

Automation tool developed in-house

Leverages Selenium and Java

API based approach

Easily customizable to suite specific project requirements

Ability to store reports in our own format for future use.

Easy to install.

Easily redefined parameters

Customized Alerts and Reports.

Weekly and Nightly build integrations

Applicable to testing of web based applications and products as well as Social Platforms

Pilot usage on for a couple of happy customer engagements.

Extensively used to test Pramati’s Enterprise 2.0 product Qontext .

Test Case Management System

J2EE™ Compliance Testing J2EE CTS Kit From Sun Standards Verification Automated Harness

J2EE CTS Kit

From Sun

Standards Verification

Automated Harness

Test Management Framework Augment CTS For Complex scenarios Built on Studio Framework Homegrown Ease of writing new tests Automated Discovery of new tests Extensive tests metadata Compose new plans/suites Integrated into the Build System For automated post-build testing

Augment CTS

For Complex scenarios

Built on Studio Framework

Homegrown

Ease of writing new tests

Automated Discovery of new tests

Extensive tests metadata

Compose new plans/suites

Integrated into the Build System

For automated post-build testing

Reliability Testing

Blade- Distributed Stress Testing System

Thank you. [email_address]