SWE 637, Tools


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

SWE 637, Tools

  1. 1. Introduction to Software Testing Chapter 9.3 Challenges in Testing Software Test Criteria and the Future of Testing Paul Ammann & Jeff Offutt http://www.cs.gmu.edu/~offutt/softwaretest/
  2. 2. Chapter 9 Outline <ul><li>Testing for Emergent Properties: Safety and Security </li></ul><ul><li>Software Testability </li></ul><ul><li>Test Criteria and the Future of Software Testing </li></ul>Test Criteria and the Future of Software Testing Introduction to Software Testing (Ch 9.2) © Ammann & Offutt
  3. 3. Testing in the 1980s and 1990s <ul><li>Software engineering was very different </li></ul><ul><ul><li>Low quality software was normal and expected </li></ul></ul><ul><ul><li>The cost of building better software outweighed the benefits </li></ul></ul><ul><li>Software was smaller </li></ul><ul><ul><li>Most software was bundled , shrink-wrapped , contracted </li></ul></ul><ul><ul><li>Industry became dominated by one monopolistic vendor </li></ul></ul><ul><li>About a dozen books </li></ul><ul><ul><li>A few broad survey books, one or two on testing , design , requirements , etc </li></ul></ul><ul><li>In the early 1980s, one semester course was enough to learn most of the knowledge in software engineering </li></ul><ul><li>Safety critical software was a tiny part of the industry </li></ul>Testing, especially high-end testing, simply was not very important Introduction to Software Testing (Ch 9.2) © Ammann & Offutt
  4. 4. Cost of Late Testing Requirements Test Planning Design Review Development Unit Testing Functional Testing System Testing Production Cost per Fault 1X 1X 1X 5X 10X 50X $6K $13K $20K $101K $363K $252K Current State Fault Discovery 20% 13% 6% 20% 5% 36% 10% Fault Origination 40% 50% Planning & Requirements Design Development Testing UAT Deploy to Production Source – Software Engineering Institute; Carnegie Mellon University; Handbook CMU/SEI-96-HB-002; page 56-58 Introduction to Software Testing (Ch 9.2) © Ammann & Offutt
  5. 5. Testing in the 2000s <ul><li>The field has dramatically changed </li></ul><ul><li>Today’s software market : </li></ul><ul><ul><li>is much bigger </li></ul></ul><ul><ul><li>is more competitive </li></ul></ul><ul><ul><li>has more users </li></ul></ul><ul><ul><li>used in more places </li></ul></ul><ul><li>The web offers a new deployment platform </li></ul><ul><ul><li>Very competitive and very available to more users </li></ul></ul><ul><li>Enterprise applications means bigger programs and more users </li></ul><ul><li>Embedded software is now ubiquitous … check your pockets ! </li></ul><ul><li>Paradoxically, free software increases our expectations ! </li></ul><ul><li>Security has now become essential … </li></ul>Introduction to Software Testing (Ch 9.2) © Ammann & Offutt
  6. 6. Problems Leading to Security Vulnerabilities <ul><li>1980 : Security was mostly about math </li></ul><ul><ul><li>Cryptography </li></ul></ul><ul><li>1990 : Security was mostly about protecting the database </li></ul><ul><li>2000 : Security was mostly about bullet-proofing the network </li></ul><ul><li>Today : Most security vulnerabilities are a result of faults in the software </li></ul>Software testing is fast becoming essential to an essential problem in the software industry Software Security Introduction to Software Testing (Ch 9.2) © Ammann & Offutt
  7. 7. Testing Research <ul><li>1980s : Criteria and algorithms for unit testing </li></ul><ul><li>1990s : Test criteria for other technologies and “ levels ” </li></ul><ul><li>2000s : More automation for test criteria and more technologies </li></ul>A major point of this book is that we have enough criteria <ul><li>Test criteria are uniform across different software artifacts </li></ul><ul><ul><li>A graph is a graph is a graph … </li></ul></ul><ul><ul><li>36 criteria on 4 structures </li></ul></ul><ul><li>So are we done with testing research ? </li></ul><ul><ul><li>Can I retire now … </li></ul></ul>Introduction to Software Testing (Ch 9.2) © Ammann & Offutt
  8. 8. Testing Research – Are We Done ? <ul><li>We need more work on how to test modern technologies </li></ul><ul><ul><li>The ideas in chapter 7 are still developing </li></ul></ul><ul><ul><li>OO, web, GUIs, real-time, embedded, … </li></ul></ul><ul><ul><li>How to create the structures and how to adapt the criteria ? </li></ul></ul><ul><li>Which criteria to use when ? </li></ul><ul><ul><li>Cost / benefit tradeoffs among criteria are not known </li></ul></ul><ul><li>Which structure should be used when ? </li></ul><ul><li>Technology transition to industry </li></ul><ul><ul><li>How best to automate the testing research ideas ? </li></ul></ul><ul><ul><li>How to insert new testing techniques into the development process ? </li></ul></ul><ul><ul><li>How to effectively and efficiently balance research theory with practical needs ? </li></ul></ul>No ! Introduction to Software Testing (Ch 9.2) © Ammann & Offutt
  9. 9. Testing Research – Going Forward <ul><li>Automatic test data generation is one of the hardest problems </li></ul><ul><ul><li>But also the core of testing – getting values to satisfy criteria </li></ul></ul><ul><ul><li>Research started in the 1970s – major advances in the 1980s, 1990s and 2000s </li></ul></ul><ul><ul><li>Still very little help from industry-quality tools </li></ul></ul><ul><li>Testability , reliability , and other related areas are wide open </li></ul><ul><li>We know a lot about testing new software – much less about regression testing </li></ul><ul><ul><li>Which happens to account for most of the effort in industry … </li></ul></ul><ul><li>Research in software testing is increasing </li></ul><ul><ul><li>Currently lots of funding in Europe and Asia – from both government and industry sources </li></ul></ul><ul><ul><li>New, well attended venues (eg ICST) </li></ul></ul>Introduction to Software Testing (Ch 9.2) © Ammann & Offutt
  10. 10. Future of Software Testing <ul><li>Increased specialization in testing teams will lead to more efficient and effective testing </li></ul><ul><li>Testing and QA teams will have more technical expertise </li></ul><ul><li>Developers will have more knowledge about testing and motivation to test better </li></ul><ul><li>Agile processes puts testing first—putting pressure on both testers and developers to test better </li></ul><ul><li>Testing and security are starting to merge </li></ul>Introduction to Software Testing (Ch 9.2) © Ammann & Offutt
  11. 11. Summary <ul><li>With more attention from </li></ul><ul><ul><li>Industry </li></ul></ul><ul><ul><li>Research </li></ul></ul><ul><ul><li>Education … </li></ul></ul>We are clearly entering a golden age for software testing Finally we have the knowledge, resources, and motivation to make testing a technical discipline Introduction to Software Testing (Ch 9.2) © Ammann & Offutt