Your SlideShare is downloading. ×
0
Testing: A Roadmap Mary Jean Harrold Sunita Bodigutla CS 589 Fall 2003
Definition <ul><li>A verification method that applies a controlled set of conditions and stimuli for the purpose of findin...
Quality Assurance <ul><li>ISO 9000:2000  </li></ul><ul><ul><li>&quot;Part of quality management focused on providing confi...
Testing <ul><li>Designing test cases </li></ul><ul><li>executing the software with those test cases </li></ul><ul><li>exam...
Expensive <ul><li>Testing accounts for  50%  of the cost of software development </li></ul><ul><li>Higher for critical sys...
What this paper is about <ul><li>A report written in 2000 </li></ul><ul><li>assesses the state of the art in software test...
Testing Component based Systems Testing based on  Precode artifacts Demonstrating effectiveness of testing techniques Test...
Testing Component based Systems Testing based on  Precode artifacts Demonstrating effectiveness of testing techniques Test...
Component Based Systems <ul><li>Increased size and complexity of software systems has led to component based systems </li>...
Component User Component Reusable Service Semantics Repository System System Under Assemble Select Composible Executable U...
Component Based Systems (contd.) <ul><li>COTS  : Commercial off the shelf software </li></ul><ul><ul><li>e.g. Microsoft Ac...
Problems with Testing COTS <ul><li>Unavailability of source code </li></ul><ul><li>Technical Heterogeneity </li></ul><ul><...
Component Based Software <ul><li>Techniques for testing by Component provider- a number of them </li></ul><ul><li>Techniqu...
Test Adequacy <ul><li>A  test adequacy criterion  is a systematic criterion that is used to determine whether a test suite...
Future Directions <ul><li>Efficient techniques and tools for </li></ul><ul><ul><li>testing aspects like security, dependab...
Testing Component based Systems Testing based on  Precode artifacts Demonstrating effectiveness of testing techniques Test...
Integration test strategies and architectural styles fig 6
Fault type and Integration Test Strategy fig 7
Testing based on Architecture specifications <ul><li>Formal notation for architecture specification ‘ CHAM ’ : Chemical Ab...
Testing based on Architecture specifications (contd.) <ul><li>[4] Using the formal description of SA, a transition graph d...
Future Directions <ul><li>Techniques for  </li></ul><ul><ul><li>automatic generation of test cases from formal specificati...
Testing Component based Systems Testing based on  Precode artifacts Demonstrating effectiveness of testing techniques Test...
Testing Evolving Software <ul><li>Regression Testing:  ensuring that no new errors are introduced into previously tested c...
Future Directions <ul><li>Techniques to </li></ul><ul><ul><li>assist in managing the test suites </li></ul></ul><ul><ul><l...
Testing Component based Systems Testing based on  Precode artifacts Demonstrating effectiveness of testing techniques Test...
Test Criteria <ul><li>Effective for certain classes of faults </li></ul><ul><ul><li>data flow </li></ul></ul><ul><ul><li>c...
Modified Condition/Decision Coverage <ul><li>Definition: </li></ul><ul><ul><li>Every point of entry and exit in the progra...
Modified Condition/Decision Coverage (contd.) <ul><li>Simple example </li></ul><ul><ul><li>A  and  B </li></ul></ul><ul><u...
Future directions <ul><li>Find ways  </li></ul><ul><ul><li>to determine interaction between the various test- selection cr...
Testing Component based Systems Testing based on  Precode artifacts Demonstrating effectiveness of testing techniques Test...
Effective Processes for Testing <ul><li>Process used for planning and implementing it </li></ul><ul><li>Test plan  </li></...
Using Testing Artifacts <ul><li>Execution Traces </li></ul><ul><ul><li>Information about which </li></ul></ul><ul><ul><ul>...
Other topics <ul><li>Methods and Tools </li></ul><ul><ul><li>Scalability </li></ul></ul><ul><ul><li>Computational tradeoff...
Conclusion <ul><li>Overview </li></ul><ul><li>List of techniques </li></ul><ul><li>No comparison  </li></ul><ul><li>Future...
<ul><li>[1] Antonia Bertolino, Andrea Polini,  “A Framework for Component Deployment Testing” </li></ul><ul><li>[2] A.Orso...
fig 12. Sequential Multiphase Compiler Architecture
fig 13.Transition Graph for the Sequential Compile
fig 14. Concurrent Reduced Graph
Thank You
Upcoming SlideShare
Loading in...5
×

Sunita Bodigutla

658

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
658
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
18
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Sunita Bodigutla"

  1. 1. Testing: A Roadmap Mary Jean Harrold Sunita Bodigutla CS 589 Fall 2003
  2. 2. Definition <ul><li>A verification method that applies a controlled set of conditions and stimuli for the purpose of finding errors. </li></ul><ul><li>The paper says: </li></ul><ul><ul><li>It’s a process that is performed to support quality assurance </li></ul></ul>
  3. 3. Quality Assurance <ul><li>ISO 9000:2000 </li></ul><ul><ul><li>&quot;Part of quality management focused on providing confidence that quality requirements will be fulfilled“ </li></ul></ul><ul><ul><li>It is a planned program consisting of: all actions necessary to provide adequate confidence that an item or product conforms to established technical requirements; and, activities designed to evaluate the process by which products are developed or manufactured. </li></ul></ul><ul><ul><li>A product does what it is expected to. </li></ul></ul>
  4. 4. Testing <ul><li>Designing test cases </li></ul><ul><li>executing the software with those test cases </li></ul><ul><li>examining the results produced by those executions </li></ul>
  5. 5. Expensive <ul><li>Testing accounts for 50% of the cost of software development </li></ul><ul><li>Higher for critical systems </li></ul><ul><li>Efficient ways to perform testing to control the cost </li></ul>
  6. 6. What this paper is about <ul><li>A report written in 2000 </li></ul><ul><li>assesses the state of the art in software testing </li></ul><ul><li>outlines future directions in software testing </li></ul><ul><li>gives pointers to software testing resources </li></ul>
  7. 7. Testing Component based Systems Testing based on Precode artifacts Demonstrating effectiveness of testing techniques Testing Evolving Software creating effective testing processes Other testing approaches Using testing artifacts Methods and Tools Empirical Studies Practical Testing Methods, Tools, and Processes for Development of High-Quality Software Fundamental Research Status of Testing Methods, Tools, and Processes fig 1. Software Testing Roadmap
  8. 8. Testing Component based Systems Testing based on Precode artifacts Demonstrating effectiveness of testing techniques Testing Evolving Software creating effective testing processes Other testing approaches Using testing artifacts Methods and Tools Empirical Studies Practical Testing Methods, Tools, and Processes for Development of High-Quality Software Fundamental Research Status of Testing Methods, Tools, and Processes fig 2
  9. 9. Component Based Systems <ul><li>Increased size and complexity of software systems has led to component based systems </li></ul><ul><li>Component Developer vs. Component user </li></ul>
  10. 10. Component User Component Reusable Service Semantics Repository System System Under Assemble Select Composible Executable Understand Assembly Compile fig 3
  11. 11. Component Based Systems (contd.) <ul><li>COTS : Commercial off the shelf software </li></ul><ul><ul><li>e.g. Microsoft Active X, MFC, Java Applets etc </li></ul></ul><ul><li>Why COTS- </li></ul><ul><ul><li>saves money </li></ul></ul><ul><ul><li>saves time </li></ul></ul><ul><ul><li>component reuse </li></ul></ul><ul><ul><li>standardization </li></ul></ul><ul><ul><li>simplified design </li></ul></ul>
  12. 12. Problems with Testing COTS <ul><li>Unavailability of source code </li></ul><ul><li>Technical Heterogeneity </li></ul><ul><li>Enterprise Heterogeneity </li></ul><ul><li>Dynamic evolution </li></ul><ul><li>Component Trust Problem </li></ul>
  13. 13. Component Based Software <ul><li>Techniques for testing by Component provider- a number of them </li></ul><ul><li>Techniques available to component user-very few </li></ul><ul><li>Theory of test adequacy by Rosenblum </li></ul>
  14. 14. Test Adequacy <ul><li>A test adequacy criterion is a systematic criterion that is used to determine whether a test suite provides an adequate amount of testing for a component under test </li></ul><ul><li>C-adequate-for-P for adequate unit testing of a component </li></ul><ul><li>C-adequate-on-M for adequate integration testing of a component-based system </li></ul>fig 4
  15. 15. Future Directions <ul><li>Efficient techniques and tools for </li></ul><ul><ul><li>testing aspects like security, dependability and safety </li></ul></ul><ul><ul><li>representing and computing the types of testing information (meta-data) that the component-user needs </li></ul></ul><ul><ul><ul><li>Java Beans: BeanInfo </li></ul></ul></ul><ul><ul><li>storing the information </li></ul></ul><ul><ul><li>using the information </li></ul></ul><ul><ul><li>A Framework for component deployment [1] called ‘component deployment testing (CDT) framework’ </li></ul></ul><ul><ul><ul><li>component runtime introspection </li></ul></ul></ul><ul><ul><ul><li>reflection API of Java </li></ul></ul></ul>
  16. 16. Testing Component based Systems Testing based on Precode artifacts Demonstrating effectiveness of testing techniques Testing Evolving Software creating effective testing processes Other testing approaches Using testing artifacts Methods and Tools Empirical Studies Practical Testing Methods, Tools, and Processes for Development of High-Quality Software Fundamental Research Status of Testing Methods, Tools, and Processes fig 5
  17. 17. Integration test strategies and architectural styles fig 6
  18. 18. Fault type and Integration Test Strategy fig 7
  19. 19. Testing based on Architecture specifications <ul><li>Formal notation for architecture specification ‘ CHAM ’ : Chemical Abstract Machine. </li></ul><ul><ul><li>Architecture: a set of components (molecules) </li></ul></ul><ul><ul><li>states and interactions governed by rules (reactions) </li></ul></ul><ul><ul><li>CHAM is specified by defining molecules m1,m2,…, solutions s1, s2,….. and transformation rules T, T’,….. </li></ul></ul><ul><ul><li>Solutions are multisets of molecules interpreted as defining the states of CHAM </li></ul></ul><ul><ul><li>Transformation rules describe how the system dynamically evolves through reaction steps </li></ul></ul>
  20. 20. Testing based on Architecture specifications (contd.) <ul><li>[4] Using the formal description of SA, a transition graph depicting all possible behaviors of the system is arrived at. </li></ul><ul><li>A suitable set of reduced graphs are identified, each subgraph specific architectural properties of the system. </li></ul><ul><li>For each subgraph, test cases are derived to cover all the transitions in it. </li></ul>
  21. 21. Future Directions <ul><li>Techniques for </li></ul><ul><ul><li>automatic generation of test cases from formal specification of software architecture </li></ul></ul><ul><ul><li>evaluating software architectures for testability </li></ul></ul>
  22. 22. Testing Component based Systems Testing based on Precode artifacts Demonstrating effectiveness of testing techniques Testing Evolving Software creating effective testing processes Other testing approaches Using testing artifacts Methods and Tools Empirical Studies Practical Testing Methods, Tools, and Processes for Development of High-Quality Software Fundamental Research Status of Testing Methods, Tools, and Processes fig 8
  23. 23. Testing Evolving Software <ul><li>Regression Testing: ensuring that no new errors are introduced into previously tested code </li></ul><ul><li>Expensive </li></ul><ul><li>Selective Regression Testing: which subset of a test suite must be rerun to test a new version of the system </li></ul>
  24. 24. Future Directions <ul><li>Techniques to </li></ul><ul><ul><li>assist in managing the test suites </li></ul></ul><ul><ul><li>reduce the size of test suites </li></ul></ul><ul><ul><li>prioritize test cases </li></ul></ul>
  25. 25. Testing Component based Systems Testing based on Precode artifacts Demonstrating effectiveness of testing techniques Testing Evolving Software creating effective testing processes Other testing approaches Using testing artifacts Methods and Tools Empirical Studies Practical Testing Methods, Tools, and Processes for Development of High-Quality Software Fundamental Research Status of Testing Methods, Tools, and Processes fig 9
  26. 26. Test Criteria <ul><li>Effective for certain classes of faults </li></ul><ul><ul><li>data flow </li></ul></ul><ul><ul><li>complex boolean expressions: Modified Condition/decision coverage </li></ul></ul>
  27. 27. Modified Condition/Decision Coverage <ul><li>Definition: </li></ul><ul><ul><li>Every point of entry and exit in the program has been invoked atleast once </li></ul></ul><ul><ul><li>every condition in a decision in the program has taken on all possible outcomes atleast once </li></ul></ul><ul><ul><li>each condition has been shown to independently affect the decisions outcome </li></ul></ul>
  28. 28. Modified Condition/Decision Coverage (contd.) <ul><li>Simple example </li></ul><ul><ul><li>A and B </li></ul></ul><ul><ul><li>Extension of truth table to pairs table </li></ul></ul><ul><ul><li>The test set { (T T), </li></ul></ul><ul><ul><li>(T F), (F T)} is adequate for MC/DC coverage </li></ul></ul>fig 10 F F F 4 1 F F T 3 1 F T F 2 2 3 T T T 1 B A Result AB Number
  29. 29. Future directions <ul><li>Find ways </li></ul><ul><ul><li>to determine interaction between the various test- selection criteria and find ways to combine them to perform more effective testing </li></ul></ul>
  30. 30. Testing Component based Systems Testing based on Precode artifacts Demonstrating effectiveness of testing techniques Testing Evolving Software creating effective testing processes Other testing approaches Using testing artifacts Methods and Tools Empirical Studies Practical Testing Methods, Tools, and Processes for Development of High-Quality Software Fundamental Research Status of Testing Methods, Tools, and Processes fig 11
  31. 31. Effective Processes for Testing <ul><li>Process used for planning and implementing it </li></ul><ul><li>Test plan </li></ul><ul><li>Perpetual testing </li></ul><ul><li>Microsoft uses a different model </li></ul><ul><ul><li>synchronize and stabilize </li></ul></ul><ul><li>Future directions </li></ul><ul><ul><li>validate the existing models </li></ul></ul><ul><ul><li>integration of various quality techniques and tools </li></ul></ul>
  32. 32. Using Testing Artifacts <ul><li>Execution Traces </li></ul><ul><ul><li>Information about which </li></ul></ul><ul><ul><ul><li>statements were executed </li></ul></ul></ul><ul><ul><ul><li>paths were executed </li></ul></ul></ul><ul><ul><ul><li>values particular variables acquired </li></ul></ul></ul><ul><li>Artifacts can be stored for use in retesting the software after it is modified </li></ul><ul><ul><li>dynamic program slices </li></ul></ul><ul><ul><li>heuristic approaches </li></ul></ul><ul><ul><li>List of invariants </li></ul></ul><ul><ul><li>Regression Testing </li></ul></ul><ul><ul><li>Visualization technique </li></ul></ul>
  33. 33. Other topics <ul><li>Methods and Tools </li></ul><ul><ul><li>Scalability </li></ul></ul><ul><ul><li>Computational tradeoffs </li></ul></ul><ul><ul><li>Automatic generation </li></ul></ul><ul><li>Empirical Studies </li></ul><ul><li>Testing Resources </li></ul>
  34. 34. Conclusion <ul><li>Overview </li></ul><ul><li>List of techniques </li></ul><ul><li>No comparison </li></ul><ul><li>Future directions </li></ul><ul><li>Testing is a wide area </li></ul><ul><li>Dynamic field </li></ul>
  35. 35. <ul><li>[1] Antonia Bertolino, Andrea Polini, “A Framework for Component Deployment Testing” </li></ul><ul><li>[2] A.Orso, M.J.Harrold and D.Rosenblum, “Component Metadata for Software Engineering Tasks”, in W.Emmerich and S.Tai(Eds) EDO 2000, LNCS 1999, pp. 129-144 </li></ul><ul><li>[3] J.A.Stafford and A.L.Wolf, “Annotating components to support component based static analysis of software systems “, Proc. the Grace Hopper Celeb. of women in computing 2001 </li></ul><ul><li>[4] Antonia Bertolino, Paola Inverardi, Henry Muccini, Andrea Rosetti, “An Approach to Integration Testing based on Architectural Descriptions” </li></ul>References
  36. 36. fig 12. Sequential Multiphase Compiler Architecture
  37. 37. fig 13.Transition Graph for the Sequential Compile
  38. 38. fig 14. Concurrent Reduced Graph
  39. 39. Thank You
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×