Basics of Structured Software Testing


Published on

1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • The CMMI has had an impact on global software development far greater than that of any other standard. However, even with the breadth of CMMI’s impact, the Standish Group still reports disturbing results for software development. Build 1 – 30% of projects are cancelled prior to completion Build 2 – 54% are over budget Build 3 – 66% are not considered successful Build 4 – and 90% are delivered late Build 5 – Is it any wonder that the U.S. National Institute of Standards and Technology reports that the United States alone wastes almost $60 billion per year on software problems?
  • Ask the question first, and let trainees come up with answer. Show panel afterwards. The panel shows the “negative” approach first (finding defects). This is an important part of testing but it’s just a part. There’s more to testing. The examples slowly evolve to a more positive and comprehensive description of the meaning of testing. A possible definition: Testing is a process of planning, preparing and measuring aiming at establishing characteristics of an information system. Testing identifies the gap between actual and desired status of a system.
  • Ten behoeve van een juiste afbakening wordt door middel van deze slide ingegaan op de vraag "Wat is testen niet?". Testen en implementeren zijn tegengestelde activiteiten. Testresultaten torpederen de implementatieplannen nogal eens. Testen is niet vrijgeven of accepteren. Het testen levert advies over de kwaliteit. De vrijgave beslissing is aan anderen, veelal de opdrachtgever van de test. Testen is ook geen foutherstel, slechts bij de programmatest is testen en herstellen in een hand te leggen. Bij alle andere testen moet het principe gehanteerd worden, dat er niet getest wordt door de persoon die heeft gebouwd en niet hersteld wordt door de persoon die heeft getest. Testen is niet goedkoop, de testkosten schommelen, afhankelijk van het systeemtype, tussen de 20 en 40 procent van de ontwikkelingskosten. Aan de andere kant is testen misschien toch wel goedkoop. Een goed uitgevoerde test kan een kwalitatief beter systeem opleveren waardoor tijdens productie minder fouten/storingen zullen optreden. Testen is niet een fase na ontwikkeling, het behelst een serie activiteiten die moeten worden uitgevoerd vanaf een pril stadium van ontwikkeling. Testen is in eerste instantie niet bedoeld om het systeem te toetsen op volledigheid en juistheid van de functionaliteit. Dit wil niet zeggen dat er tijdens het testen niet ingegaan mag worden op onjuistheden in de functionele specificaties. Het mag niet het hoofddoel worden en er moet procedureel anders mee worden omgegaan. Implementation is a phase after testing Acceptance is done by business people, never by the test team. The test manager advices. The test team finds defects, developers do defect repair Testing is expensive, but in the long run it will save you money, if done in a structured manner Test planning, specification and preparation is done prior to and during development. Only the execution (and completion) phases take place after development Business analysts and users define the desired functionality. The test team checks if the system meets these requirements. Testers do not doubt the desired functionality
  • The industry has not yet reached a level of maturity where testing is obsolete (will probably never happen).
  • An information system (test object) is more than software alone. It also includes hardware, documentation etc. Quality characteristics are used to determine what needs to be tested.
  • Gold left-most triangle - TMap is based on a business-driven test management (BDTM) approach. Right side parallelogram - TMap describes a structured test process. Gray bottom triangle - TMap contains a complete tool box. Arrow all around - TMap is an adaptive test method. < Details on next slides >
  • Gold left-most triangle - TMap is based on a business-driven test management (BDTM) approach. Right side parallelogram - TMap describes a structured test process. Gray bottom triangle - TMap contains a complete tool box. Arrow all around - TMap is an adaptive test method. < Details on next slides >
  • The risks of the system = product risks, the risk of defect occurring in production. The short of BDTM is that there is a business case for what gets tested. Costs of defects vs. the costs of testing. Steps: Defining test of scope and gathering test goals. Determining risk class for each combination of characteristic and object part. Determining whether a combination of characteristic and object part must be tested thoroughly or lightly. An overall estimate is then made for the test and a planning set up. Allocating test techniques to the combinations of characteristic and object part. Throughout the test process, the test manager reviews with various stakeholders
  • Master Test Plan is developed. Manage the entire test process (planning & control of testing). Acceptance & System Tests –Lifecycle model for these business-focused tests. Development tests – testing against design Supporting processes – make sure that test processes comply with overall quality initiatives
  • In the Planning phase, the test manager formulates a coherent approach that is supported by the client to adequately execute the test assignment. This is laid down in the test plan. Control phase - the activities in the test plan are executed, monitored, and adjusted if necessary. The Setting up and maintaining infrastructure phase aims to provide the required test infrastructure that is used in the various TMap phases and activities. The Preparation phase aims to have access to a test basis, agreed with the client of the test, of adequate quality to design the test cases. The tests are specified in the Specification phase and executed in the Execution phase. This provides insight into the quality of the test object. The test assignment is concluded in the Completion phase. This phase offers the opportunity to learn lessons from experiences gained in the project. Furthermore activities are executed to guarantee reuse of products.
  • Coverage – requirements that are agreed must be tested HAVE been tested
  • Basics of Structured Software Testing

    1. 1. Test Management Approach (TMap) Chris Hampton, PMP Practice Manager, Software Control & Testing Sogeti USA September 2007
    2. 3. Agenda: <ul><li>The State of, and Business Case for, Software Quality </li></ul><ul><li>What is testing? </li></ul><ul><li>Sogeti’s Test Management Approach ® (Tmap ® ) </li></ul><ul><li>Sogeti’s Test Process Improvement ® (TPI ® ) Methodology </li></ul>
    3. 4. Poor State of Software Delivery “ Software failure costs the American economy $59.5 billion annually.” Source: NIST (2002) 30% of projects cancelled early 54% of projects over budget 66% of projects rated unsuccessful 90% of projects are delivered late Source: Standish 2003
    4. 5. The Business Case For QA <ul><li>Compliance </li></ul><ul><ul><li>Basel II, Sarbanes Oxley, HIPAA, etc… </li></ul></ul><ul><ul><li>Reduced liability from defective software and defective process </li></ul></ul><ul><li>Cost Savings </li></ul><ul><ul><li>Quality processes now reduce support costs later </li></ul></ul><ul><ul><li>Lost revenue </li></ul></ul><ul><li>Satisfaction </li></ul><ul><ul><li>Higher User satisfaction </li></ul></ul><ul><ul><li>Improved business alignment </li></ul></ul><ul><li>Growth </li></ul><ul><ul><li>Better quality and measurements allows development optimization </li></ul></ul><ul><ul><li>Increase ability to deliver </li></ul></ul>
    5. 6. Where Quality Problems Originate Source: James Martin 56% 7% 10% 27%
    6. 7. How Quality Provides Tremendous Savings April 16, 2010 Unit Test Production Code COST TO REPAIR DEFECT Planning Analysis System Design $140 Unit Test $1000 Code $14000 Time cost and resource effort increase exponentially later in the lifecycle Source : Cutter Consortium/Forrester/Sogeti Integration Test $2500 System Test $4500 Certification Test $7000
    7. 8. What is testing? <ul><li>Testing is a process aimed at: </li></ul><ul><li>Finding defects in a controlled manner </li></ul><ul><li>Detecting the level of quality of the test object </li></ul><ul><li>Demonstrating the gap between specifications and the actual product </li></ul><ul><li>Demonstrating that the end product functions as called for in the requirements </li></ul>
    8. 9. Testing is not . . . . <ul><li>Testing is not: </li></ul><ul><li>Implementation </li></ul><ul><li>Acceptance </li></ul><ul><li>Defect repair </li></ul><ul><li>A phase after development – although testing includes a phase after development </li></ul><ul><li>Training on a new system </li></ul>
    9. 10. Why do we test ? <ul><li>To mitigate business risks that have been identified: </li></ul><ul><ul><li>Validate software quality </li></ul></ul><ul><ul><li>Verify integration of business processes </li></ul></ul><ul><ul><li>Reduce time-to-market </li></ul></ul><ul><ul><li>Competitive purposes </li></ul></ul><ul><li>To ensure business usability of the software: </li></ul><ul><ul><li>Release low-error/known-error software </li></ul></ul><ul><ul><li>Move high quality software (meets or exceeds quality expectations) into production </li></ul></ul><ul><ul><li>Demonstrate the usefulness of new technology </li></ul></ul>
    10. 11. Testing of information s ystems <ul><li>Application Software </li></ul><ul><li>Hardware </li></ul><ul><li>System Software </li></ul><ul><li>Procedures </li></ul><ul><li>Documentation </li></ul><ul><li>Functionality </li></ul><ul><li>Continuity </li></ul><ul><li>Performance </li></ul><ul><li>Usability </li></ul><ul><li>Interoperability (between different applications and systems) </li></ul>What do we test?
    11. 12. Structured Software Testing <ul><li>Testing everything is impractical </li></ul><ul><ul><li>Funding limitations </li></ul></ul><ul><ul><li>Limited time and resources </li></ul></ul><ul><ul><li>Diminishing value after a certain point </li></ul></ul><ul><li>Isn’t there a more effective way? </li></ul><ul><li>Yes! Structured software testing: a risk-based, quality-centric approach to testing </li></ul>
    12. 13. Test Management Approach (TMap ® ) <ul><li>Sogeti’s methodology for structured testing of software products. </li></ul><ul><li>TMap® has evolved to be the standard for software testing in Europe and quickly gaining traction in the US. </li></ul><ul><ul><li>It is being used by more than three hundred Dutch, Belgium and German organizations. </li></ul></ul><ul><ul><li>Industry adoption includes : </li></ul></ul><ul><ul><ul><li>Financial Services </li></ul></ul></ul><ul><ul><ul><li>Insurance </li></ul></ul></ul><ul><ul><ul><li>Government </li></ul></ul></ul><ul><ul><ul><li>Consumer Electronics </li></ul></ul></ul><ul><ul><ul><li>Telecommunications </li></ul></ul></ul><ul><ul><ul><li>Medical systems </li></ul></ul></ul>
    13. 14. 4 Essentials of TMap ® <ul><li>Business-driven test management (BDTM) approach </li></ul><ul><li>Structured test process </li></ul><ul><li>Complete tool box </li></ul><ul><li>Adaptive test method </li></ul>BDTM puts focus on risk-based testing—only testing what has impacts to the organization.
    14. 15. Business Driven Test Management <ul><li>Choices must be made in what is tested and how thoroughly. </li></ul><ul><li>This is an economic decision based on: </li></ul><ul><ul><li>The risks that an organization thinks it will incur </li></ul></ul><ul><ul><li>The availability of time and resources </li></ul></ul><ul><ul><li>The results the customer wishes to achieve. </li></ul></ul><ul><li>The choices based on risks, result, time and cost constitutes the basis for the BDTM approach. </li></ul>
    15. 16. The BDTM Process <ul><li>Formulate the test goals. </li></ul><ul><li>Determine the risk class for each combination of characteristic and object part. </li></ul><ul><li>Determining the depth of testing required. </li></ul><ul><ul><li>Based on risk/resource trade off </li></ul></ul><ul><li>Develop the test estimate and test plan. </li></ul><ul><li>Allocate the appropriate test techniques. </li></ul><ul><li>Throughout the test process, the test manager provides the stakeholders with insight into and control options over test process and test object. </li></ul>
    16. 17. Risk Definition <ul><li>Testing should be based on the mitigation of risk and validation of expected quality defined by the business requirements </li></ul><ul><li>Definition of risk </li></ul><ul><ul><li>A “risk” is a chance of a failure occurring, related to expected damage to the project (should the failure occur) </li></ul></ul><ul><li>Risk Categories </li></ul><ul><ul><li>Risks identified in different categories such as business risks, project risks, product risks, process risks </li></ul></ul><ul><li>Risk Ranking </li></ul><ul><ul><li>Risk are ranked in criticality relative to one another, by instituting a method of risk ranking (or “risk rating”) </li></ul></ul>
    17. 18. How Risk Ranking Works <ul><li>Assemble a team of people representing various roles from the project </li></ul><ul><ul><li>PMs, Business Leads/SMEs, Test Leads, Tech Leads </li></ul></ul><ul><li>Create an initial list of risks </li></ul><ul><li>Team assigns a numeric value to each risk = the probability of occurrence of each risk </li></ul><ul><li>Team assigns a second value to each risk, representing the impact on the project/organization </li></ul><ul><li>Multiply the two values together (the probability of occurrence X the impact) </li></ul><ul><li>The result is a relative value for each risk </li></ul><ul><li>Order the risks by their relative risk values </li></ul><ul><ul><li>“ risk rating” or “risk ranking” </li></ul></ul><ul><ul><li>Ranking helps manage the most critical risks, especially those falling in the middle tier of the ranking </li></ul></ul><ul><li>Periodically update the list of risks </li></ul>
    18. 19. Risks – quality characteristics matrix High Priority Low Priority Medium Priority Low High Risks Medium High Priority/ Low Risk Low Priority/ High Risk Low Priority/ Low Risk High Priority/ High Risk Medium Priority/ Low Risk Medium Priority/ Medium Risk Medium Priority/ High Risk High Priority/ Medium Risk Low Priority/ Medium Risk Multi-Level Matrix
    19. 20. Ten steps for creating a test case Step 1 Step 2 Step 3 Step 4 Step 5 Step 10 Step 9 Step 8 Step 7 Step 6 Identify sub-systems / sub-processes Identify risks Rank risks Agree on relevant quality characteristics Rank relevant quality characteristics Create subsystem / quality characteristic pairings Create test scenarios for each subsystem Associate test scenarios with risks Select suitable test techniques CREATE TEST CASES Facilitated Sessions FUNCTIONAL AREA
    20. 21. Structured Test Process <ul><li>Master Test Plan, managing the total test process </li></ul><ul><ul><li>Acceptance and System Tests </li></ul></ul><ul><ul><li>Developmental Test </li></ul></ul><ul><ul><li>Supporting Processes </li></ul></ul><ul><li>Lifecycle Model </li></ul>
    21. 22. Master Test Plan <ul><li>Developed by the test manager in partnership with the appropriate stakeholders </li></ul><ul><li>Jointly define: </li></ul><ul><ul><li>What will be tested for each test level, </li></ul></ul><ul><ul><li>When it will be tested </li></ul></ul><ul><ul><li>To what degree of thoroughness. </li></ul></ul><ul><li>This plan constitutes the basis for the test plans for the separate test levels. </li></ul><ul><ul><li>Unit Testing </li></ul></ul><ul><ul><li>Systems Integration Testing </li></ul></ul><ul><ul><li>Acceptance Testing </li></ul></ul><ul><ul><li>Developmental Testing </li></ul></ul>
    22. 23. Life Cycle Model <ul><li>The life cycle model is a generic model. </li></ul><ul><ul><li>Can be applied to all test levels and test types </li></ul></ul><ul><ul><li>Used in parallel with the life cycle models for system development. </li></ul></ul><ul><li>Test activities are divided across seven phases: </li></ul><ul><ul><li>Planning </li></ul></ul><ul><ul><li>Control </li></ul></ul><ul><ul><li>Setting up and maintaining infrastructure </li></ul></ul><ul><ul><li>Preparation, </li></ul></ul><ul><ul><li>Specification </li></ul></ul><ul><ul><li>Execution </li></ul></ul><ul><ul><li>Completion </li></ul></ul>
    23. 24. TMap  - S tructured testing lifecycle P&C P S E C <ul><ul><li>PREPARATION </li></ul></ul><ul><ul><li>Testability review </li></ul></ul><ul><li>Allocate techniques </li></ul><ul><ul><li>Requirements review </li></ul></ul><ul><ul><li>Data cycle test </li></ul></ul><ul><ul><li>Process cycle test </li></ul></ul><ul><ul><li>Other tests </li></ul></ul><ul><ul><li>Specify test cases </li></ul></ul><ul><ul><li>Create/assess infrastructure </li></ul></ul><ul><ul><li>Test specification </li></ul></ul><ul><ul><li>Data flow test </li></ul></ul><ul><ul><li>EXECUTION </li></ul></ul><ul><ul><li>Pretest </li></ul></ul><ul><li>(Re)test </li></ul><ul><li>Checking </li></ul><ul><li>Assessing </li></ul><ul><li>Debugger </li></ul><ul><li>Record and playback </li></ul><ul><li>Monitoring </li></ul><ul><li>Preserve testware </li></ul><ul><li>Evaluate process </li></ul><ul><ul><li>PLANNING & CONTROL </li></ul></ul><ul><ul><li>Inspection and study -Test strategy </li></ul></ul><ul><li>Develop test strategy -Test estimation </li></ul><ul><ul><ul><li>Risk analysis -Reports </li></ul></ul></ul><ul><li>Test Estimation (TPA) -Management tools </li></ul><ul><ul><li>Setup organization -Budgeting tools </li></ul></ul><ul><li>Prepare test plan -Defect management </li></ul><ul><li>Management and control tool </li></ul><ul><ul><li>SPECIFICATION </li></ul></ul><ul><ul><li>Create test scripts </li></ul></ul><ul><li>Create infrastructure </li></ul><ul><ul><li>COMPLETION </li></ul></ul><ul><ul><li>Preserve testware for future use </li></ul></ul><ul><li>Evaluate process </li></ul>
    24. 25. Complete Tool Kit <ul><li>Techniques </li></ul><ul><ul><li>Many techniques can be used in the test process. A test technique is a combination of actions to produce a test product in a universal manner. </li></ul></ul><ul><li>TMap provides techniques for the following: </li></ul><ul><ul><li>Test estimation </li></ul></ul><ul><ul><li>Defect management </li></ul></ul><ul><ul><li>Creating metrics </li></ul></ul><ul><ul><li>Product risk analysis </li></ul></ul><ul><ul><li>Test design </li></ul></ul><ul><ul><li>Product evaluation. </li></ul></ul>
    25. 26. Adaptive Test Method <ul><li>Complete And Flexible </li></ul><ul><li>Works with any SDLC </li></ul><ul><ul><li>Iterative </li></ul></ul><ul><ul><li>RAD </li></ul></ul><ul><ul><li>Agile </li></ul></ul><ul><ul><li>Waterfall </li></ul></ul><ul><ul><li>RUP </li></ul></ul><ul><li>Works with any technology </li></ul><ul><ul><li>Mainframe </li></ul></ul><ul><ul><li>Web </li></ul></ul><ul><ul><li>ERP </li></ul></ul><ul><ul><li>SOA </li></ul></ul>
    26. 28. TMap ® – Differentiators <ul><li>Business-Driven Test Management </li></ul><ul><li>Risk-based test strategy </li></ul><ul><li>Efficient testing—coverage without overlap </li></ul><ul><li>Testing off of critical path as much as possible </li></ul><ul><li>Criteria and metrics about production readiness </li></ul><ul><li>Management of testing to project timelines </li></ul><ul><li>Compliments industry tools </li></ul>Proven methodology applied in hundreds of companies internationally over the past 14 years!
    27. 29. Benefits of Structure Testing <ul><li>Defects are found early , costing less time/money to reach production  delivering a higher quality product </li></ul><ul><li>Required quality of the various test objects is tested for and validated , by focusing on testing for quality as a risk mitigation strategy </li></ul><ul><li>By keeping a larger percentage of the testing process/effort off the critical path , faster time-to-market results </li></ul><ul><li>Structured software testing is more cost-effective and efficient than non-structured testing approaches </li></ul><ul><li>Sound test coverage is provided , without the need to overlap phases </li></ul><ul><li>Establishes a test organization that is prepared and efficient </li></ul><ul><li>Delivers a repeatable process </li></ul>
    28. 30. Test Process Improvement (TPI  ) <ul><li>The Sogeti TPI® Model was developed to facilitate a stepwise improvement of the testing process. </li></ul><ul><ul><li>Developed from practical experience from 30+ years of Sogeti work in testing. </li></ul></ul><ul><ul><li>Offers a frame of reference to determine the strengths and weaknesses of your current testing process. </li></ul></ul><ul><ul><li>Covers 20 key areas that need specific improvement to achieve a well define testing process. </li></ul></ul><ul><li>Key TPI Areas: </li></ul>Test Strategy Life-Cycle Model Moment of Involvement Estimating and Planning Test Specification Techniques Static Test Techniques Metrics Test Automation Test Environment Office Environment Commitment and Motivation Test Functions and Training Scope of Methodology Communication Reporting Defect Management Testware Management Test Process Management Evaluation and Low-Level Testing
    29. 31. Typical TPI  Process Steps <ul><li>Interview key people </li></ul><ul><li>Describe the ‘As-Is’ process including Strengths & Weaknesses </li></ul><ul><li>Describe the ‘To-Be’ process </li></ul><ul><li>Document process improvement actions </li></ul><ul><li>Define expected benefits in terms of Objectives </li></ul><ul><li>Plan for Implementation of the Improvement Actions (short-term and long-term) </li></ul>
    30. 32. TPI  - Test Maturity Matrix Increasing Maturity
    31. 33. Resources <ul><li>Web Sites </li></ul><ul><ul><li>Sogeti USA Web Site: </li></ul></ul><ul><ul><li>TMap Web Site: </li></ul></ul><ul><li>TMap ® certification (Netherlands) </li></ul><ul><ul><li>Foundation </li></ul></ul><ul><ul><li>TMap ® professional Advanced/Expert </li></ul></ul><ul><ul><li>TMap ® management Advanced/Expert </li></ul></ul><ul><li>Books </li></ul>
    32. 34. Sogeti SCT Services <ul><li>Managed Testing Services (MTS) </li></ul><ul><li>Quality Management Services </li></ul><ul><li>Test Automation Services </li></ul><ul><li>Testing Professional Services </li></ul><ul><li>SAP Testing Services </li></ul><ul><li>3000+ people worldwide in SCT space </li></ul><ul><li>30+ yrs experience </li></ul>
    33. 35. Sogeti’s Geographical Presence <ul><ul><ul><li>A workforce of over 18,000 staff spread over 200 locations </li></ul></ul></ul>France Spain Sweden Belux Germany Switzerland NL Ireland India UK Denmark USA
    34. 36. Thank you! Chris Hampton, PMP Practice Manager, Software Control & Testing Sogeti USA [email_address] Cell: (214) 549-7613 Sogeti Office: (972) 892-3400