Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1. CS 160: Software Engineering April 8 Class Meeting Department of Computer Science San Jose State University Spring 2008 Instructor: Prof. Ron Mak http://
  2. 2. Unofficial Field Trip? <ul><li>Computer History Museum in Mt. View. </li></ul><ul><ul><li>See </li></ul></ul><ul><ul><li>You provide your own transportation to and from the museum. </li></ul></ul><ul><ul><li>Completely voluntary, no credit. </li></ul></ul><ul><li>Approximately 2 hours during a Saturday afternoon later this month or early next month. </li></ul><ul><ul><li>Docent-led tour of various computer systems and artifacts from the 1940s through the 1990s. </li></ul></ul><ul><ul><ul><li>Eniac, Enigma, SAGE, IBM 360, Cray supercomputers, etc. </li></ul></ul></ul><ul><ul><li>See, hear, and touch an actual IBM 1401 computer system (c. 1959-1965) that has been restored and is operational: </li></ul></ul><ul><ul><ul><li>800 cpm card reader </li></ul></ul></ul><ul><ul><ul><li>flashing console lights </li></ul></ul></ul><ul><ul><ul><li>spinning tape drives with vacuum columns </li></ul></ul></ul><ul><ul><ul><li>600 lpm line printer </li></ul></ul></ul>
  3. 3. Software Reliability <ul><li>Reliable software has a low probability of failure while operating for a specified period of time under specified operating conditions. </li></ul><ul><ul><li>The specified time period and the specified operating conditions are part of the nonfunctional requirements. </li></ul></ul><ul><ul><li>For some applications, reliability may be more important than the other functional and nonfunctional requirements. </li></ul></ul><ul><ul><ul><li>mission-critical applications </li></ul></ul></ul><ul><ul><ul><li>medical applications </li></ul></ul></ul>
  4. 4. Software Reliability (cont’d) <ul><li>Reliable software is the result of good software quality assurance (SQA) throughout an application’s life cycle. </li></ul><ul><ul><li>Design </li></ul></ul><ul><ul><ul><li>good requirements elicitation </li></ul></ul></ul><ul><ul><ul><li>good object-oriented design and analysis </li></ul></ul></ul><ul><ul><ul><li>good architecture </li></ul></ul></ul><ul><ul><li>Development </li></ul></ul><ul><ul><ul><li>good management (e.g., source control, reasonable schedules) </li></ul></ul></ul><ul><ul><ul><li>good coding practices (e.g., design patterns) </li></ul></ul></ul><ul><ul><ul><li>good testing practices </li></ul></ul></ul><ul><ul><li>Deployment </li></ul></ul><ul><ul><ul><li>good preventive maintenance (e.g., training) </li></ul></ul></ul><ul><ul><ul><li>good monitoring </li></ul></ul></ul><ul><ul><ul><li>good failure analysis (when failures do occur) </li></ul></ul></ul>
  5. 5. Software Testing <ul><li>Some key questions: </li></ul><ul><ul><li>What is testing? </li></ul></ul><ul><ul><li>What is a successful test? </li></ul></ul><ul><ul><li>Who does testing? </li></ul></ul><ul><ul><li>When does testing occur? </li></ul></ul><ul><ul><li>What are the different types of testing? </li></ul></ul><ul><ul><li>What testing tools are available? </li></ul></ul><ul><ul><li>How do you know your tests covered everything? </li></ul></ul><ul><ul><li>When can you stop testing? </li></ul></ul>
  6. 6. What is testing? <ul><li>Testing is a systematic procedure to discover faults in software in order to prevent failure . </li></ul><ul><ul><li>Failure: A deviation of the software’s behavior from its specified behavior (as per its requirements) </li></ul></ul><ul><ul><ul><li>Can be minor to major (such as a crash) </li></ul></ul></ul><ul><ul><li>Erroneous state: A state that the operating software is in that will lead to a failure. </li></ul></ul><ul><ul><ul><li>Example: low on memory </li></ul></ul></ul><ul><ul><li>Fault: What caused the software to enter an erroneous state. </li></ul></ul><ul><ul><ul><li>Also known as: defect, bug </li></ul></ul></ul><ul><ul><ul><li>Example: a memory leak </li></ul></ul></ul>
  7. 7. What is a successful test? <ul><li>Testing is the opposite of coding. </li></ul><ul><ul><li>Coding: Create software and try to get it to work. </li></ul></ul><ul><ul><li>Testing: Break the software and demonstrate that it doesn’t work. </li></ul></ul><ul><li>Testing and coding require different mind sets. </li></ul><ul><ul><li>It can be very difficult for developers to test their own code. </li></ul></ul><ul><ul><li>If you wrote the code, you psychologically want it to work and not see it break. </li></ul></ul><ul><ul><li>Since you know how the code should be used, you may not think to try using it in ways other than as you intended. </li></ul></ul><ul><li>A successful test is one that finds bugs. </li></ul>
  8. 8. Who does testing? <ul><li>Developers </li></ul><ul><ul><li>As difficult as it may be, you must test your own code . </li></ul></ul><ul><ul><li>Test each other’s code (peer testing) </li></ul></ul><ul><ul><ul><li>test interfaces </li></ul></ul></ul><ul><li>Users </li></ul><ul><li>Testers </li></ul><ul><ul><li>Members of the Testing or Quality Assurance (QA) department. </li></ul></ul><ul><ul><li>Software engineers who did not write the code. </li></ul></ul><ul><ul><li>Manual writers and trainers who create examples and demos. </li></ul></ul>
  9. 9. When does testing occur? <ul><li>Recall the Old Waterfall Model : </li></ul><ul><li>In the new Agile Methodology, testing is part of each and every iteration . </li></ul><ul><ul><li>Therefore, testing occurs throughout development, not just at the end. </li></ul></ul>XXX Requirements Design Implementation Testing
  10. 10. What are the different types of testing? <ul><li>Usability testing </li></ul><ul><ul><li>Developers and users test the user interface. </li></ul></ul><ul><li>Unit testing </li></ul><ul><ul><li>Developers test an individual “unit”. </li></ul></ul><ul><ul><li>Unit: A small set of related components, such as to implement a use case. </li></ul></ul><ul><li>Integration testing </li></ul><ul><ul><li>Developers test how their units work together with other units. </li></ul></ul><ul><li>System testing </li></ul><ul><ul><li>Test how an entire system works. </li></ul></ul><ul><ul><li>Includes performance testing and stress testing. </li></ul></ul>
  11. 11. Alpha Testing vs. Beta Testing <ul><li>Alpha testing </li></ul><ul><ul><li>Usability and system testing of a nearly complete application in the development environment . </li></ul></ul><ul><li>Beta testing </li></ul><ul><ul><li>Usability and system testing of a complete or nearly complete application in the user’s environment . </li></ul></ul><ul><li>Today, it is not uncommon to release an application to the public for beta testing. </li></ul><ul><ul><li>New releases of web-based applications are put “into beta” by software companies. </li></ul></ul>
  12. 12. Usability Testing: The German Newton <ul><li>The Apple Newton was an early PDA developed in the early 1990s. </li></ul><ul><ul><li>Besides the English version, there was a German and a Japanese version. </li></ul></ul><ul><ul><li>It was too far ahead of its time and was killed by Steve Jobs after he returned to Apple. </li></ul></ul><ul><li>A key feature of the Newton was handwriting recognition : ‹‹ Handschrifterkennung ›› </li></ul><ul><ul><li>The Newton recognized successive words in a sentence using an algorithm that tracked the movement of the stylus. </li></ul></ul><ul><li>Cultural differences between Americans and Germans </li></ul><ul><ul><li>The way the Germans write their letters (e.g., the letter h). </li></ul></ul><ul><ul><li>The way the Germans write words (e.g., ‹‹ Gesch ä ftsreise ›› “business trip”). </li></ul></ul><ul><ul><li>Philosophy about personal handwriting. </li></ul></ul>
  13. 13. Usability Testing: NASA Mars Rover Mission <ul><li>Usability testing for the Collaborative Information Portal (CIP) software system. </li></ul><ul><li>NASA’s Jet Propulsion Laboratory (JPL) conducted several Operational Readiness Tests (ORTs) before the actual rovers landed. </li></ul><ul><ul><li>A working rover was inside of a large “sandbox” in a separate warehouse building at JPL. </li></ul></ul><ul><ul><li>Mission control personnel communicated with and operated the sandbox rover as if it were on Mars. </li></ul></ul><ul><ul><ul><li>A built-in delay simulated the signal travel time between Earth and Mars. </li></ul></ul></ul><ul><ul><li>Mission scientists accessed and analyzed the data and images downloaded by the sandbox rover. </li></ul></ul><ul><li>All mission software, including CIP, was intensely tested in this simulated environment. </li></ul>
  14. 14. Unit Testing <ul><li>Each unit test focuses on components created by a developer. </li></ul><ul><ul><li>Should be done by the developer before checking in the code. </li></ul></ul><ul><ul><li>Easier to find and fix bugs when there are fewer components. </li></ul></ul><ul><ul><li>Bottom-up testing. </li></ul></ul><ul><li>Developers create test cases to do unit tests. </li></ul>
  15. 15. Unit Testing: Test Cases <ul><li>Test case: A set of input values for the unit and a corresponding set of expected output values. </li></ul><ul><ul><li>Use case -> test case </li></ul></ul><ul><ul><li>Do unit testing on the participating objects of the use case. </li></ul></ul><ul><li>A test case can be run within a testing framework (AKA test bed, test harness) consisting of a test driver and one or more test stubs. </li></ul><ul><ul><li>Test driver: Simulates the part of the system that calls the unit. Calls the unit and passes in the input values. </li></ul></ul><ul><ul><li>Test stub: Simulates the components that the unit depends on. </li></ul></ul><ul><ul><ul><li>When called by the unit, a test stub responds in a “reasonable” manner. </li></ul></ul></ul>
  16. 16. Unit Testing: Testing Framework UNIT TEST DRIVER TEST STUB TEST STUB TEST STUB
  17. 17. Black Box Testing vs. White Box Testing <ul><li>Black box testing </li></ul><ul><ul><li>Deals only with the input/output behavior of the unit. </li></ul></ul><ul><ul><li>The internals of the unit are not considered. </li></ul></ul><ul><li>White box testing </li></ul><ul><ul><li>Tests the internal behavior of the unit. </li></ul></ul><ul><ul><ul><li>execution paths </li></ul></ul></ul><ul><ul><ul><li>state transitions </li></ul></ul></ul>
  18. 18. Unit Testing: Equivalence Testing <ul><li>Minimize the number of test cases by partitioning the possible inputs into equivalence classes . </li></ul><ul><ul><li>Assumption: For each equivalence class, the unit behaves the same way for any input from the class. </li></ul></ul><ul><ul><li>Example: A calendar unit that takes a year and a month as input. </li></ul></ul><ul><ul><ul><li>Year equivalence classes: leap years and non-leap years </li></ul></ul></ul><ul><ul><ul><li>Month equivalence classes: months with 30 days, months with 31 days, and February (28 or 29 days) </li></ul></ul></ul><ul><li>Black box test. </li></ul>
  19. 19. Unit Testing: Boundary Testing <ul><li>Test the boundaries (the extremes, or “edges”) of the equivalence classes. </li></ul><ul><li>Calendar component examples: </li></ul><ul><ul><li>Month 0 and month 13. </li></ul></ul><ul><ul><li>Months with the incorrect number of days. </li></ul></ul><ul><ul><li>Invalid leap years (e.g., not a multiple of 4) </li></ul></ul><ul><li>Black box test. </li></ul>
  20. 20. Unit Testing: Monte Carlo Testing <ul><li>Monte Carlo (a famous casino in Monaco) refers to the random generation of input values . </li></ul><ul><ul><li>Used when the input values for a unit are too numerous and cannot be partitioned into equivalence classes. </li></ul></ul><ul><ul><li>The test driver generates random values from the input domain and passes them to the unit. </li></ul></ul><ul><ul><li>Any erroneous output narrows your search for the bug. </li></ul></ul><ul><li>Black box test. </li></ul>
  21. 21. Unit Testing: Path Testing <ul><li>Analyze the execution paths within the unit. </li></ul><ul><ul><li>Draw a flow graph. </li></ul></ul><ul><li>Generate input values to force the unit to follow each and every path at least once . </li></ul><ul><li>White box test. </li></ul>
  22. 22. Unit Testing: State Transition Testing <ul><li>Similar to path testing. </li></ul><ul><li>Analyze the states that the unit can be in. </li></ul><ul><ul><li>Draw a UML statechart diagram. </li></ul></ul><ul><li>Generate input data that forces the unit to transition into each and every state at least once . </li></ul><ul><li>White box test. </li></ul>
  23. 23. Unit Testing Tool: JUnit <ul><li>JUnit: A unit testing framework for Java components. </li></ul><ul><ul><li>A member of the xUnit family of testing frameworks for various programming languages. </li></ul></ul><ul><li>Open source: Download from </li></ul><ul><li>JUnit demo. </li></ul>
  24. 24. JUnit Demo <ul><li>Class to test: </li></ul>package cs160.util; public class Calculator { public double add (double first, double second) { return first + second; } public double subtract (double first, double second) { return second - first; // oops! } public double multiply (double first, double second) { return first * second; } }
  25. 25. JUnit Demo (cont’d) <ul><li>Unit test cases: </li></ul>package cs160.test; import cs160.util.Calculator; import junit.framework.TestCase ; public class CalculatorTester extends TestCase { public void testAdd () { Calculator calculator = new Calculator(); double result = calculator.add(40, 30); assertEquals (70.0, result, 0.0); } public void testSubtract () { Calculator calculator = new Calculator(); double result = calculator.subtract(40, 30); assertEquals (10.0, result, 0.0); } }
  26. 26. Unit Testing Tool: Clover <ul><li>Clover: A unit testing tool that tells you what parts of your code you have actually executed (“covered”). </li></ul><ul><ul><li>Unfortunately, not open source. </li></ul></ul><ul><ul><li>Download a 30-day trial from </li></ul></ul><ul><li>Clover demo. </li></ul>
  27. 27. Integration Testing <ul><li>After your code has passed all the unit tests, you must do integration testing to see how your unit works with other units </li></ul><ul><ul><li>Other units either written by you or by other developers. </li></ul></ul><ul><li>Create an Ant script that builds the part of the application that contains: </li></ul><ul><ul><li>Your unit </li></ul></ul><ul><ul><li>The units that call your unit </li></ul></ul><ul><ul><li>The units that your unit calls </li></ul></ul>
  28. 28. Regression Tests <ul><li>Sad fact of life: When you fix one bug, you might very well introduce one or more new bugs. </li></ul><ul><ul><li>Tests that have passed previously may now show failures – the system has regressed . </li></ul></ul><ul><li>Regression tests: Re-run earlier integration tests to make sure they still pass. </li></ul><ul><ul><li>Do this every time you make a change. </li></ul></ul><ul><ul><li>Create Ant scripts to run the tests. </li></ul></ul><ul><ul><li>Schedule regression tests to run periodically automatically, e.g., overnight every night. </li></ul></ul>
  29. 29. System Testing <ul><li>Test the entire application within its operating environment. </li></ul><ul><ul><li>Installation testing </li></ul></ul><ul><ul><li>Functional testing </li></ul></ul><ul><ul><li>Performance testing </li></ul></ul><ul><li>Part of beta testing. </li></ul><ul><li>Acceptance testing . </li></ul><ul><ul><li>Signoff by stakeholders, clients, and customers. </li></ul></ul>
  30. 30. When can you stop testing? <ul><li>“ Testing can prove the presence of bugs, but never their absence .” </li></ul><ul><li>Stop testing when: </li></ul><ul><ul><li>All the regression tests pass. </li></ul></ul><ul><ul><li>Testing finds only “acceptable” bugs. </li></ul></ul><ul><ul><ul><li>put on the Known Bugs list </li></ul></ul></ul><ul><ul><ul><li>have workarounds </li></ul></ul></ul><ul><ul><li>When you’ve run out of time. </li></ul></ul>
  31. 31. Next class meeting… <ul><li>Logging and monitoring </li></ul><ul><li>Stress testing </li></ul><ul><li>Test-Driven Development (TDD) </li></ul><ul><li>Failure analysis </li></ul><ul><li>Creating test plans </li></ul>