Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Ordenação Topológica

497 views

Published on

  • Be the first to comment

  • Be the first to like this

Ordenação Topológica

  1. 1. Evolving an Elective Software Testing Course: Lessons Learned Edward L. Jones Florida A&M University Tallahassee, FL USA 3rd Workshop on Teaching Software Testing
  2. 2. Agenda <ul><li>Course Overview </li></ul><ul><li>Student Background </li></ul><ul><li>Driving Principles </li></ul><ul><li>Overview of Assignments </li></ul><ul><li>Course Reflection </li></ul><ul><li>Improvements </li></ul><ul><li>Assignment Walkthroughs </li></ul>
  3. 3. Course Overview DESCRIPTION: The purpose of this course is to build skills necessary to perform software testing at the function, class and application level. Students will be taught concepts of black-box (functional and boundary) and white-box (coverage-based) testing, and will apply these concepts to small programs and components (functions and classes). Students will also be taught evaluative techniques such as coverage and mutation testing (error seeding). This course introduces the software engineering discipline of software quality engineering and the legal and societal issues of software quality.
  4. 4. Programming Focus AUDIENCE: Not software testers but software developers. What distinguishes the course approach is that is stresses the programming aspect of software testing. A goal is to enhance and expand students’ programming skills to support activities across the testing lifecycle: C++ programming and Unix shell script programming to automate aspects of software testing. Students just needed a course to take ...
  5. 5. Conceptual Objectives <ul><li>The student shall understand </li></ul><ul><ul><li>The software testing lifecycle </li></ul></ul><ul><ul><li>The relationship between testing, V&V, SQA </li></ul></ul><ul><ul><li>Theoretical/practical limits of software testing </li></ul></ul><ul><ul><li>The SPRAE testing framework </li></ul></ul><ul><ul><li>Concepts and techniques for black-/white-box testing </li></ul></ul><ul><ul><li>Test case design from behavioral model </li></ul></ul><ul><ul><li>Design patterns for test automation </li></ul></ul><ul><ul><li>Test coverage criteria </li></ul></ul><ul><ul><li>Issues of software testing management </li></ul></ul>
  6. 6. Performance Objectives <ul><li>The student shall be able to: </li></ul><ul><ul><li>Use the Unix development environment </li></ul></ul><ul><ul><li>Write simple Unix shell scripts </li></ul></ul><ul><ul><li>Design functional and boundary test cases </li></ul></ul><ul><ul><li>Develop manual test scripts </li></ul></ul><ul><ul><li>Conduct tests and document results </li></ul></ul><ul><ul><li>Write test drivers to automate function, object and application testing </li></ul></ul><ul><ul><li>Evaluate test session results; write problem reports </li></ul></ul>
  7. 7. Learning/Evaluation Activities <ul><li>80% practice / 20% concepts </li></ul><ul><li>Lectures (no text) </li></ul><ul><li>Laboratory assignments </li></ul><ul><ul><li>Unix commands and tools </li></ul></ul><ul><ul><li>Testing tasks </li></ul></ul><ul><li>Examinations </li></ul><ul><ul><li>2 Online tests </li></ul></ul><ul><ul><li>Final (online) </li></ul></ul><ul><li>Amnesty period (1 test / 2 labs) </li></ul>
  8. 8. Student Background <ul><li>Reality: </li></ul><ul><ul><li>20 students </li></ul></ul><ul><ul><li>Not particularly interested in testing </li></ul></ul><ul><ul><li>Low programming skill/experience </li></ul></ul><ul><li>Ideal: </li></ul><ul><ul><li>An interest in software testing </li></ul></ul><ul><ul><li>Strong programming skills </li></ul></ul><ul><ul><li>Scientific method (observation, hypothesis forming) </li></ul></ul><ul><ul><li>Sophomore or junior standing </li></ul></ul><ul><ul><li>Desire for internship in software testing </li></ul></ul>
  9. 9. My Perspective on Teaching Testing <ul><li>Testing is not just for testers! </li></ul><ul><li>In ideal world, fewer testers required </li></ul><ul><ul><li>Developers have tester’s skills/mentality </li></ul></ul><ul><ul><li>Testing overlays development process </li></ul></ul><ul><li>No silver bullet … just bricks </li></ul><ul><ul><li>Simple things provide leverage </li></ul></ul><ul><ul><li>No one-size-fits-all </li></ul></ul><ul><li>Be driven by a few sound principles </li></ul>
  10. 10. Driving Principles <ul><li>Testing for Software Developers </li></ul><ul><ul><li>Duality of developer and tester </li></ul></ul><ul><li>Few Basic Concepts </li></ul><ul><ul><li>Testing lifecycle </li></ul></ul><ul><ul><li>Philosophy / Attitudes (SPRAE) </li></ul></ul><ul><li>Learn By Doing </li></ul><ul><ul><li>Different jobs across the lifecycle </li></ul></ul>
  11. 11. A Testing Lifecycle Analysis Design Implementation Execution Evaluation Specification Test Strategy/Plan Test Script, Data, Driver Defect Data Problem Reports Test Results Test Cases
  12. 12. Experience Objectives <ul><li>Student gains experience at each lifecycle stage </li></ul><ul><li>Student uses/enhances existing skills </li></ul><ul><li>Student applies different testing competencies </li></ul><ul><li>Competencies distinguish novices from the experienced </li></ul>
  13. 13. A Framework for Practicing Software Testing <ul><ul><li>S pecification the basis for testing </li></ul></ul><ul><ul><li>P remeditation (forethought, techniques) </li></ul></ul><ul><ul><li>R epeatability of test design, execution, and evaluation (equivalence v. replication) </li></ul></ul><ul><ul><li>A ccountability via testing artifacts </li></ul></ul><ul><ul><li>E conomy (efficacy) of human, time and computing resources </li></ul></ul>
  14. 14. Key Test Practices <ul><li>Practitioner -- performs defined test </li></ul><ul><li>Builder -- constructs test “machinery” </li></ul><ul><li>Designer -- designs test cases </li></ul><ul><li>Analyst -- sets test goals, strategy </li></ul><ul><li>Inspector -- verifies process/results </li></ul><ul><li>Environmentalist -- maintains test tools & environment </li></ul><ul><li>Specialist -- performs test life cycle. </li></ul>
  15. 15. Test Products <ul><li>Test Report (informal) of manual testing </li></ul><ul><li>Test Scripts for manual testing </li></ul><ul><li>Test Log (semi-formal) </li></ul><ul><li>Application Test Driver (Unix shell script) </li></ul><ul><li>Unit/Class Test Driver (C++ program) </li></ul><ul><li>Test Data Files </li></ul><ul><li>Test Results (automated) </li></ul><ul><li>Bug Fix Log (informal) </li></ul>
  16. 16. Specification Products <ul><li>Narrative specification </li></ul><ul><li>Specification Diagrams </li></ul><ul><li>Specification Worksheet (pre/post conditions) </li></ul><ul><li>Decision Tables </li></ul><ul><li>Control Flow Graphs </li></ul>
  17. 17. Assignments Target Skills <ul><li>Observation Skills </li></ul><ul><ul><li>Systematic exploration of software behavior </li></ul></ul><ul><li>Specification Skills </li></ul><ul><ul><li>Describe expected or actual behavior </li></ul></ul><ul><li>Programming Skills </li></ul><ul><ul><li>Coding for development of test machinery </li></ul></ul><ul><li>Test Design Skills </li></ul><ul><ul><li>Derive test cases from specification using technique </li></ul></ul><ul><li>Team Skills </li></ul><ul><ul><li>Work with other testers </li></ul></ul>
  18. 18. Course Reflection <ul><li>Testing is programming intensive </li></ul><ul><li>Testing requires analytical skills and facility with mathematical tools </li></ul><ul><li>Testing generates data management problem that is amenable to automation </li></ul><ul><li>Testing gives students advantage in entry-level positions </li></ul><ul><li>Students take this course too late </li></ul>
  19. 19. Failed Course Expectations <ul><li>Students test at all levels </li></ul><ul><ul><li>No “in-the-large” application (e.g., web-based) </li></ul></ul><ul><li>Students develop intuitive testing skills </li></ul><ul><ul><li>On largest project, concepts did not transfer </li></ul></ul><ul><ul><li>1 in 3 students show “knack” for testing </li></ul></ul><ul><li>Impact of balance of concept and experience </li></ul><ul><ul><li>Poor performance on exams with problems like those in labs </li></ul></ul><ul><li>Test case design skills low </li></ul><ul><ul><li>Homework needed v. labs (programming) </li></ul></ul><ul><li>Mentoring (timely feedback) did not occur </li></ul><ul><ul><li>Students left to own devices too much </li></ul></ul>
  20. 20. Why These Outcomes? <ul><li>Formalisms important, but difficult </li></ul><ul><ul><li>Provide the behavior model (e.g., decision table) </li></ul></ul><ul><ul><li>Basis for systematic test case design, automation </li></ul></ul><ul><li>Lack of textbook </li></ul><ul><ul><li>Students need concepts + lots of examples </li></ul></ul><ul><li>Poor availability when students were working </li></ul><ul><ul><li>Students worked at last minute </li></ul></ul><ul><ul><li>Not always around </li></ul></ul><ul><ul><li>Automated grading lacked 1-1 feedback </li></ul></ul><ul><li>Standards-rich/tool-poor environment a distraction </li></ul><ul><li>Assigned work too simple?? </li></ul>
  21. 21. Proposed Changes <ul><li>Improve lecture notes and example bank </li></ul><ul><ul><li>Find and refine </li></ul></ul><ul><ul><li>Resources and workbook </li></ul></ul><ul><li>Outside-in: testing in-the-large before in-the-small </li></ul><ul><li>Recitation/laboratory for discussion and feedback </li></ul><ul><li>Increase use of testing tools (no-cost) </li></ul><ul><li>Increase use of collection of code/applications </li></ul><ul><li>Examination testbank for practice, learning </li></ul>
  22. 22. Assignment Walkthroughs <ul><li>(see paper) </li></ul>
  23. 23. Assignment Walkthroughs <ul><li>Blind Testing </li></ul><ul><li>Test Documentation </li></ul><ul><li>Specification </li></ul><ul><li>Test Automation via Shell Scripts </li></ul><ul><li>Unit Test Automation (Driver) </li></ul><ul><li>White-Box Unit Testing </li></ul><ul><li>Class Testing </li></ul>
  24. 24. Blind Testing I <ul><li>Objective: Explore behavior of software without the benefit of a specification </li></ul><ul><li>Given: Executables + general description </li></ul><ul><li>Results: Students not systematic in exploration or in generalizing observed behavior </li></ul><ul><ul><li>Hello  output based on length of input </li></ul></ul><ul><ul><li>Add  1-digit modulus 10 adder, input exception </li></ul></ul><ul><ul><li>Pay  pay calculation with upper bound pay amount </li></ul></ul>
  25. 25. Blind Testing II <ul><li>Programming Objective: Student writes program that matches the observed behavior of Blind Testing I </li></ul><ul><li>Test Objective: Observations on Blind Testing I used as “test cases” for reverse-engineered program. </li></ul><ul><li>Results: Students did not see the connection; </li></ul><ul><ul><li>Did not replicate the recorded behavior </li></ul></ul><ul><ul><li>Did not recognize (via testing) failure to replicate </li></ul></ul>
  26. 26. SUPPLEMENTAL SLIDES <ul><li>Student work </li></ul>
  27. 27. SCALING UP The heart of the approach is to use a decision table as a thinking tool . The most critical task in this process is to identify all the stimuli and responses. When there are many logical combinations of stimuli, the decision table can become large, indicating that the unit is complex and hard to test .
  28. 28. IDENTIFYING BEHAVIOR Approaches <ul><ul><li>Work backwards </li></ul></ul><ul><ul><ul><li>Identify each response </li></ul></ul></ul><ul><ul><ul><li>Identify conditions that provoke response </li></ul></ul></ul><ul><ul><ul><li>Identify separate stimuli </li></ul></ul></ul><ul><ul><li>Work forward </li></ul></ul><ul><ul><ul><li>Identify stimuli </li></ul></ul></ul><ul><ul><ul><li>Identify how each stimulus influences what unit does </li></ul></ul></ul><ul><ul><ul><li>Specify the response </li></ul></ul></ul>
  29. 29. IDENTIFYING STIMULI <ul><ul><li>Arguments passed upon invocation </li></ul></ul><ul><ul><li>Interactive user inputs </li></ul></ul><ul><ul><li>Internal, secondary data </li></ul></ul><ul><ul><ul><li>global or class variables </li></ul></ul></ul><ul><ul><li>External data (sources) </li></ul></ul><ul><ul><ul><li>file or database status variables </li></ul></ul></ul><ul><ul><ul><li>file or database data </li></ul></ul></ul><ul><ul><li>Exceptions </li></ul></ul>
  30. 30. IT PAYS TO BE A GOOD STIMULUS DETECTIVE <ul><ul><li>Failure to identify stimuli results in an incomplete, possibly misleading test case </li></ul></ul><ul><ul><li>The search for stimuli exposes </li></ul></ul><ul><ul><ul><li>interface assumptions -- a major source of integration problems </li></ul></ul></ul><ul><ul><ul><li>incomplete design of unit </li></ul></ul></ul><ul><ul><ul><li>inadequate provision for exception handling </li></ul></ul></ul>
  31. 31. IDENTIFYING RESPONSES <ul><ul><li>Arguments/Results passed back on exit </li></ul></ul><ul><ul><li>Interactive user outputs </li></ul></ul><ul><ul><li>Internal, secondary data </li></ul></ul><ul><ul><ul><li>updated global or class variables </li></ul></ul></ul><ul><ul><li>External data (sinks) </li></ul></ul><ul><ul><ul><li>output file or database status variables </li></ul></ul></ul><ul><ul><ul><li>output file or database data </li></ul></ul></ul><ul><ul><li>Exceptions </li></ul></ul>
  32. 32. IT PAYS TO BE A GOOD RESPONSE DETECTIVE <ul><ul><li>Failure to identify responses results in </li></ul></ul><ul><ul><ul><li>incomplete understanding of the software under test </li></ul></ul></ul><ul><ul><ul><li>shallow test cases </li></ul></ul></ul><ul><ul><ul><li>incomplete expected results </li></ul></ul></ul><ul><ul><ul><li>incomplete test &quot;success&quot; verification -- certain effects not checked </li></ul></ul></ul><ul><ul><li>To test, one must know all the effects </li></ul></ul>
  33. 33. A SKETCHING TOOL Black-Box Schematic Stimulus Type Response Type Software under Test Argument Inputs Globals Database Exception Argument Outputs Globals Database Exception
  34. 34. BEFORE CONTINUTING Much of the discussion so far involves how to identify what software does . We have introduced thinking tools for systematically capturing our findings. These thought processes and tools can be used anywhere in the lifecycle, e.g., in software design ! One Stone for Two Birds!!
  35. 35. Specialist I - Competencies Practitioner Test Designer Test Analyst Test Inspector Test Environmentalist Test SPECIALIST 1 Test Builder 1 1 1 1 1 1 2 2 2 2 2 2 2 3 3 3 3 3 3 3 4 4 4 4 4 4 4 5 5 5 5 5 5 5 Test Practitioner ... ... ... ... ... ... ...
  36. 36. BOUNDARY TESTING DESIGN METHODOLOGY <ul><ul><li>Specification </li></ul></ul><ul><ul><li>Identify elementary boundary conditions </li></ul></ul><ul><ul><li>Identify boundary points </li></ul></ul><ul><ul><li>Generate boundary test cases </li></ul></ul><ul><ul><li>Update test script (add boundary cases). </li></ul></ul>
  37. 37. EXAMPLE: Pay Calculation (1) Specification <ul><ul><li>Compute pay for employee, given the number of hours worked and the hourly pay rate. For hourly employees (rate < 30), compute overtime at 1.5 times hourly rate for hours in excess of 40. Salaried employees (rate >= 30) are paid for exactly 40 hours. </li></ul></ul>
  38. 38. EXAMPLE B (2) Identify Behaviors <ul><ul><li>Case 1: Hourly AND No overtime </li></ul></ul><ul><ul><ul><li>(Rate < 30) & (Hours <= 40) </li></ul></ul></ul><ul><ul><ul><li>Expect Pay = Hours * Rate </li></ul></ul></ul><ul><ul><li>Case 2: Hourly AND Overtime </li></ul></ul><ul><ul><ul><li>(Rate < 30) & (Hours > 40) </li></ul></ul></ul><ul><ul><ul><li>Expect Pay = 40*Rate+1.5*Rate*(Hours - 40) </li></ul></ul></ul><ul><ul><li>Case 3: Salaried (Rate >= 30) </li></ul></ul><ul><ul><ul><li>Expect Pay = 40 * Rate </li></ul></ul></ul>
  39. 39. DECISION TABLE Columns define Behaviors Condition c1: Rate < 30 | Y Y N N c2: Hours <= 40 | Y N Y N Action a1: Pay = Straight time | X a2: Pay = Overtime | X a3: Pay = Professional | X X
  40. 40. EXAMPLE B (3) Create Test Cases <ul><ul><li>One test case per column of decision table </li></ul></ul><ul><ul><ul><li>Case 1: Hourly, No Overtime </li></ul></ul></ul><ul><ul><ul><li>Case 2: Hourly, Overtime </li></ul></ul></ul><ul><ul><ul><li>Case 3: Salaried, No Extra Hours </li></ul></ul></ul><ul><ul><ul><li>Case 4: Salaried, Extra Hours </li></ul></ul></ul><ul><ul><li>Order the test cases by column </li></ul></ul>
  41. 41. EXAMPLE B (4) Write Test Script Step Stimuli Expected Response Hours Rate Pay = 1 2 30 50 10 10 300 550 3 30 40 1600 4 50 40 1600
  42. 42. Testing Modules -- Drivers A test driver executes a unit with test case data and captures the results. Driver Test set Data External Effects Unit Arguments Results Test Set Results
  43. 43. Implementing Test Drivers <ul><li>Complexity </li></ul><ul><ul><li>Arguments/Results only </li></ul></ul><ul><ul><li>Special set-up required to execute unit </li></ul></ul><ul><ul><li>External effects capture/inquiry </li></ul></ul><ul><ul><li>Oracle announcing &quot;PASS&quot;/&quot;FAIL&quot; </li></ul></ul><ul><li>Major Benefits </li></ul><ul><ul><li>Automated, repeatable test script </li></ul></ul><ul><ul><li>Documented evidence of testing </li></ul></ul><ul><ul><li>Universal design pattern </li></ul></ul>
  44. 44. Test Driver for Unit Pay Driver D_pay uses unit_environment E; { declare Hrs, Rate, expected; testcase_no = 0; open tdi_file(&quot;tdi-pay.txt&quot;); open trs_file(&quot;trs-pay.txt&quot;); while (more data in tdi_file) { read(tdi_file, Hrs, Rate); read(tdi_file, expected); testresult = pay(Hrs, Rate); write (trs_file, testcase_no++, Hrs, Rate, expected, testresult); } //while close tdi_file, trs_file; } //driver
  45. 45. Test Driver Files (Pay) Test Data File File name: tdi-pay.txt Format: (test cases only) rate hours expected-pay File content: 10 40 400 10 50 550 10 0 0 ------ Note: No environment setup. Test Results File File name: trs-pay.txt Format: case# rate hours exp-pay act-pay File content: 1 10 40 400 400 2 10 50 550 500 3 10 0 0 0 ------ Note: Results file must be inspected for failures . Pass Fail Pass Test Script!!
  46. 46. Testing Classes -- Drivers (Black-Box) Class Test Driver Method Args /Results Test set Data Test Set Results Class Class-state Method(s)
  47. 47. Example -- Stack Class class Stack { public: Stack(); void push(int n); int pop(); int top(); bool empty(); private: int Size; int Top; int Values[100]; }; Notes: (1) Class state -- variables Size, Top and the first 'Size' values in array Values. (2) Methods push and pop modify class state; top and empty inquire about the state. (3) Stack does not require any test environment of its own. (4) Class state HIDDEN from test, i.e., black box.
  48. 48. Test Driver Files (Stack class) Test Data File (tdi-stack.txt) File content: ----- 1 8 1 7 3 7 2 7 2 8 4 true ------ Note: No test environment setup. Methods: 1-push, 2-pop, 3-top, 4-empty Test Results File (trs-stack.txt) File content: ----- 1 1 8 2 1 7 3 3 7 8 4 2 7 8 5 2 8 7 6 4 1 ------ Note: Results file must be inspected for pass/fails . --- Top should be 7 . --- Push . --- Pop, should be 8 . --- Stack should be empty. Fail Fail Fail Pass

×