Seminar on Software Testing

Software Testing As software quality control Beat Fluri Zurich SoftSummer 2009 Summer School on Complex Software Systems University of Zurich Department of Informatics software evolution & architecture lab

Me, myself, and I Beat Fluri Senior research associate (UZH) ETH Zurich UZH Studies of Computerscience Doctorate in area of Software Evolution Analysis Lecturer @ UZH Software Testing ﬂuri.computerscience.ch Material of this seminar Dr. Beat Fluri © 2009 2

Agenda Introduction into Software Testing Continuous Integration Unit Testing Integration Testing System and GUI Testing Regression Testing Tools you may like in you CI environment Dr. Beat Fluri © 2009 3

Introduction

What is software testing? Design and implementation of a special kind of software system; one that exercises another software system with the intent to ﬁnding bugs. Rober V. Binder, TOOSMPT Dr. Beat Fluri © 2009 6

Test design Identify, model, and analyze the responsibilities of the system under test On different level of abstraction: methods, classes, packages, subsystems, system Design test cases based on this external perspective Functional testing Add test cases based on code analysis, suspicions, and heuristics Structural testing Develop expected results for each test case or choose an approach to evaluate the pass or no pass status of each test case Dr. Beat Fluri © 2009 7

Test execution Implementation under test is minimally operational Bottom-up to reduce test driver and test double overhead Test suite execution: result is evaluated as pass or no pass Coverage tool to instrument the implementation under test Deﬁne goal: 95% statements, 80% branches, 70% conditions, etc. If necessary, develop additional tests to exercise uncovered code Stop: coverage goal is met and all tests pass Dr. Beat Fluri © 2009 8

Why is software testing so hard? Input/state space public Line(int x0, int y0, int x1, int y1) Input per int: 232 different values: 232 x 232 x 232 x 232 = 2128 1000 lines per second: 1028 years Execution sequences for (int i = 0; i < n; i++) { if (a.get(i) == b.get(i)) { x[i] += 100; 2n + 1 paths with n iterations } else { x[i] /= 2; } } Dr. Beat Fluri © 2009 9

Why is software testing so hard? Coincidental correctness public class Account { protected long lastTxDate; public long daysSinceLastTx() { long today = Calendar.getInstance().getTimeInMillis(); return 1 + (today - lastTxDate) / (24 * 3600 * 1000); } } public class DepositAccount extends Account { public long daysSinceLastTx() { long today = Calendar.getInstance().getTimeInMillis(); return (today - lastTxDate) / (24 * 3600 * 1000); } } Dr. Beat Fluri © 2009 10

Why is software testing so hard? Undecidability problem (halting problem) There isn’t any program that can decide whether the execution of another program will halt or not For simple program it is decidable but not in general Dr. Beat Fluri © 2009 11

Why software testing? Find bugs early Airbag does not work: ﬁnd out before or after releasing the car? Part of quality assurance Would you trust the airbag software if it wasn’t tested? Supports changing a software system Verify component integrity after a change Dr. Beat Fluri © 2009 12

Why is testing neglected? Expensive (software testing is hard) Test case design and selection Building test environment Integration environment Few methodologies for test automation Not part of development process Hard to include for speciﬁc projects (customer won’t pay extra effort) Not part of the software engineering culture Missing testing environment Dr. Beat Fluri © 2009 13

Short glossary Test case Set of inputs, execution conditions, and expected result (pass and no pass criterion) Test suite Set of test cases, usually related to a testing goal Test driver Class or utility program that applies test cases Double (stub, mock) Partial, temporary implementation of a component Test harness Substitution for parts of the deployment environment, i.e., a system of test drivers and other tools to support test execution Dr. Beat Fluri © 2009 14

Short glossary Oracle A mechanism to evaluate the actual result of a test case as pass or no pass Equivalence class Set of input values: if one value is processed correctly/incorrectly, all other values will also be processed correctly/incorrectly Coverage Percentage of elements required by a test strategy that have been exercised by a given test suite Dr. Beat Fluri © 2009 15

Short glossary Fault model Identifies relationships and components that are most likely to have faults Test model Testable representation of the relationships among elements of an implementation based on a fault model Test strategy Algorithm or heuristic to create test cases from an implementation or test model Test design Produces a test suite using a test strategy: identification of interesting test inputs, placing these test inputs into a sequence, and defining the expected results Dr. Beat Fluri © 2009 16

Short glossary Fault-directed testing Testing that seeks to reveal implementation faults Conformance-directed testing Testing that seeks to establish conformance to requirements or speciﬁcations Dr. Beat Fluri © 2009 17

V-Model Speciﬁcation Implementation Acceptance Test User Delivery needs System Test System Spec System Integration Test Subsystem Subsystem Design/Spec Unit Test Unit Unit Design/Spec Dr. Beat Fluri © 2009 18

Continuous Integration

Continuous Integration - What ... is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly. Martin Fowler, Article Source: http://martinfowler.com/articles/continuousIntegration.html Dr. Beat Fluri © 2009 21

Continuous Integration - How Version control repository: CVS, Subversion, Git, Mercurial, etc. Build tools: Ant, Maven, Make, Gant, Grails, Rake, etc. Continuous integration environment (server): CruiseControl, Continuum, Hudson Our experience Subversion (http://subversion.tigris.org/) Apache Maven (http://maven.apache.org/) Hudson (https://hudson.dev.java.net/) Dr. Beat Fluri © 2009 23

Continuous Integration - Why Reduce risks Defects are detected when they are introduced Measure the health of a software Environment always the same and build starts clean => no assumptions Reduce repetitive manual processes (safe time and costs) Process runs the same every time Ordered process: compile, unit tests, integration tests, qa, etc. Generate deployable software at any time and at any place Going back in the build history to deploy older, maybe stable, builds Source: Paul M. Duval. Continuous Integration. Pearson Education, Inc., 2007 Dr. Beat Fluri © 2009 24

Continuous Integration - Why Enable better project visibility Decisions on quality improvements and tests are shown immediately Ability to notice trends in various quality metrics (# bugs; # checkstyle, findbugs, pmd violations; code coverage of test cases) Establish greater confidence in the software product from the development team Making progress is visible and encourages developers Confidence increases if the increase of overall product quality is visible Same code basis for every developer: reduces communication overhead Go to build #x, do you see... Source: Paul M. Duval. Continuous Integration. Pearson Education, Inc., 2007 Dr. Beat Fluri © 2009 25

uDoo - Model <<interface>> <<interface>> <<interface>> IEventProvider ITodoListModel IObservable Todo * <<interface>> TodoList Dispatcher * IObserver 1..* Project EventID * Event Dr. Beat Fluri © 2009 27

uDoo - Controller <<interface>> ITodoListController <<interface>> ITodoCommand TodoListController * execute() undo() AddProject DueDateChange PriorityChange RemoveProject Tod doToProject DescriptionChange MoveTodo ProjectTitleChange RemoveTodo * Todo 2 Project <<interface>> ITodoListModel Dr. Beat Fluri © 2009 28

Demo uDoo with Maven and Hudson

Unit Testing

Unit Testing Speciﬁcation Implementation User Test units in isolation Delivery needs Concentrate on possible bugs within smallest possible units System Spec System Subsystem Subsystem Design/Spec Unit Test Unit Unit Design/Spec Dr. Beat Fluri © 2009 31

Unit Testing public void doSomething(int number) 1 Method scope Constraints on input 3 Method & class scope Inheritance private List<Todo> fTodos; Polymorphism 2 Class scope Encapsulation State dependant behavior Dr. Beat Fluri © 2009 32

1 Method Scope Test Constraints on Input

Method Scope - General Approach Decompose the speciﬁcation into independently testable features Identify representative values (domain analysis) Identify constraints for all input variables (boundaries) Select test values for each variable in each boundary Select test values for variables not given in the boundary Generate test case speciﬁcations Determine expected results for these inputs Generate test cases and instantiate tests Dr. Beat Fluri © 2009 34

Domain Analysis - On, Off, In Points On point A value that lies on a boundary Off point A value that lies not on the boundary In point A value that lies not on the boundary and satisﬁes all boundary conditions Dr. Beat Fluri © 2009 35

Domain Analysis - On, Off, In Points Open boundary: x > 10 On point: 10 Off point: 11 On point makes the boundary condition false Off point makes the boundary condition true Closed boundary: x <= 100 On point: 100 Off point: 101 On point makes the boundary condition true Off point makes the boundary condition false Dr. Beat Fluri © 2009 36

Domain Analysis - On, Off, In Points List empty: List.size() == 0 loaded: List.size() > 0 && List.size() < List.MAX_CAPACITY full: List.size() == List.MAX_CAPACITY Abstract state on point Smallest possible change in any state variable would produce a state change empty: empty list loaded: list with one element list with (List.MAX_CAPACITY - 1) elements full: full list Dr. Beat Fluri © 2009 37

Domain Analysis - On, Off, In Points Abstract state off point empty: list with one element loaded: empty list list with List.MAX_CAPACITY elements full: list with (List.MAX_CAPACITY - 1) elements Abstract state in point empty: empty list loaded: list with (0 < x < List.MAX_CAPACITY) elements list with (0 < x < List.MAX_CAPACITY) elements full: full list Dr. Beat Fluri © 2009 38

One-by-One (1x1) Selection Criteria One on point and one off point for each relational condition x > 10: on point = 10; off point = 11 One on point and two off points for each strict equality condition x == 10: on point = 10; off point1 = 9; off point2 = 11 One on point and one off point for each non-scalar type x == “Testing”: on point = “Testing”; off point = “Testing!” Dr. Beat Fluri © 2009 39

One-by-One (1x1) Selection Criteria One on point and at least one off point for each abstract state invariant !list.isEmpty() on point: list with one element off point: empty list Don’t repeat identical tests for adjacent subdomains Adjacent subdomains may share boundaries State 1: x >= 10, on point = 10, off point = 9 State 2: x < 10, on point = 10, off point = 11 Test points: 9, 10, 11 Dr. Beat Fluri © 2009 40

Domain Test Matrix Boundary Test Cases Variable Condition Type 1 2 3 4 5 6 On 10 > 10 Off 11 x On 100 <= 100 Off 101 Typical In 15 90 On false !isEmpty() aList Off true Typical In true true true true Expected result Reject Accept Accept Reject Reject Accept Dr. Beat Fluri © 2009 41

Category-Partition Test Pattern Design method scope test suites based on input/output analysis Method to test: TodoList.addTodo(Todo todo, Project project) Procedure Decompose the speciﬁcation into independently testable features 1. Identify the functions of the method under test 2. Identify the input and output parameters of each function Identify representative values 3. Identify categories for each input parameter 4. Partition each category into choices 5. Identify constraints on choices Generate test case speciﬁcations 6. Generate test cases by enumerating all choice combinations 7. Develop expected results for each test case Dr. Beat Fluri © 2009 42

Category-Partition Test Pattern 1. Identify the functions of the method under test Adds the given Todo to the given Project and returns true Does nothing if the given Project does not exist and returns false 2. Identify the input/output parameters of each testable function Input: Given Project Input: Given Todo Input: Content of the todo-list Output: true or false Output: Given Todo is in given Project or not Dr. Beat Fluri © 2009 43

Category-Partition Test Pattern 3. Identify categories for each input parameter Parameter Category Todo value Project value Todo-list special case 4. Partition each category into choices Parameter Category Choices Todo value null not null Project value null not null Todo-list special case contains Project does not contain Project Dr. Beat Fluri © 2009 44

Category-Partition Test Pattern 5. Identify constraint on choices If todo-list throws an exception for (todo != null) and (project == null) it will also throw one for (todo == null) and (project == null) If either (todo == null) or (project == null) the state of the todo-list does not care (DC) 6. Generate test cases by enumerating all choice combinations 2 x 2 x 2 choices: 8 test cases Remove cases identiﬁed by Step 5: 4 test cases Dr. Beat Fluri © 2009 45

Category-Partition Test Pattern Parameter/Choices Expected Result TC Todo Project Todo-list Returned Exception Todo-list 1 DC null DC NullPointerException !todo 2 DC not null DC NullPointerException !todo 3 not null not null project true todo 4 not null not null !project false !todo Legend project todo-list contains project !project todo-list does not contain project todo project contains todo !todo project does not contain todo Dr. Beat Fluri © 2009 46

2 Class Scope Test Encapsulation

2 Class Scope Test State dependent behavior

State Dependent Behavior Non-modal classes Does not impose any constraints on the sequence of message accepted, i.e., does not have a state Quasi-modal classes Imposes sequential constraints on message acceptance that change with the content of the object (Collections) Modal classes Places both message and domain constraints on the acceptable sequence of a message (Account) Dr. Beat Fluri © 2009 50

Invariant Boundaries Test Pattern Select test-efficient test value combinations for classes, interfaces, and components composed of complex and primitive data types Procedure Decompose the specification into independently testable features 1. Define the class invariants Identify representative values 2. Develop on and off points for each condition using 1x1 selection criteria Generate test case specifications 3. Complete test suite by developing in points for variables not referenced in a condition 4. Represent all test cases in a domain test matrix Dr. Beat Fluri © 2009 51

Invariant Boundaries Test Pattern Elevator Example Elevator can only run if the following invariants are met 0 < weight <= MAX_WEIGHT LOWEST_FLOOR <= selectedFloor <= HIGHEST_FLOOR selectedFloor != currentFloor door.isClosed !inEmergency Dr. Beat Fluri © 2009 52

Elevator - Domain Test Matrix Boundary Test Cases Variable Condition Type 1 2 3 4 5 6 7 On 0 > 0 Off 1 weight On max <= MAX_WEIGHT Off max+1 Typical In 23 50 66 On low >= LOWEST_FLOOR Off low-1 On high <= HEIGHEST_FLOOR Off selectedFloor On != currentFloor Off In 1 2 0 -1 2 6 1 Typical In 0 3 5 6 On isClosed() door Off Typical In true true true true true true true On false inEmergency Off Typical In false false false false false false false Expected result Reject Accept Accept Reject Accept Reject Accept Dr. Beat Fluri © 2009 53

Elevator - Domain Test Matrix Boundary Test Cases Variable Condition Type 8 9 10 11 12 13 14 15 On > 0 Off weight On <= MAX_WEIGHT Off Typical In 23 55 56 54 10 11 18 14 On >= LOWEST_FLOOR Off On <= HEIGHEST_FLOOR Off high+1 selectedFloor On 0 -1 != currentFloor Off 0 In 3 -1 4 0 3 Typical In -1 0 0 1 5 4 2 On true isClosed() door Off false Typical In true true true true true true On false false inEmergency Off true Typical In false false false false false false Expected result Reject Accept Accept Reject Accept Reject Accept Reject Dr. Beat Fluri © 2009 54

Modal Class Test Pattern public class Account { private int balance; private Date lastActivity; public void open() {} public int balance() {} public void credit(int amount) {} public void debit(int amount) {} public void freeze() {} public void unfreeze() {} public void close() {} } Dr. Beat Fluri © 2009 55

Modal Class Test Pattern Constraints on message sequences Account: debit is not accepted if the balance is <= 0 Account: freeze is not accepted if account is closed or already frozen Modal class test 1. Method scope test 2. Alpha-omega cycle 3. Generate state model and transition tree 4. Elaborate transition tree with a full expansion of conditional transition variants 5. Tabulate events and actions along each path to form test cases 6. Develop test data for each path by using Invariant Boundaries 7. Develop sneak path test cases Dr. Beat Fluri © 2009 56

Modal Class Test Pattern State model of Account balance credit [balance >= 0] debit [balance < 0] Overdrawn ! credit balance balance Open freeze Frozen unfreeze debit [today - lastActivity > 5y] balance credit Inactive [balance == 0] close [balance == 0] close Closed " Dr. Beat Fluri © 2009 57

Modal Class Test Pattern Transition tree of Account balance Open debit Open credit Open balance Overdrawn debit [balance < 0] credit Overdrawn Overdrawn credit [balance >= 0] Open Open balance freeze Frozen Frozen unfreeze Open balance [today - lastActivity > 5y] Inactive Inactive close Closed [balance == 0] close Closed Dr. Beat Fluri © 2009 58

Modal Class Test Pattern balance Conditional Transitions Open credit Open debit [balance > 0] Open balance debit [balance == 0] Overdrawn Open credit [balance < 0] Overdrawn debit [balance < 0] Overdrawn credit [balance > 0] Open credit [balance == 0] Open Open balance freeze Frozen Frozen unfreeze Open balance [today - lastActivity > 5y] Inactive Inactive [balance == 0] close Closed [balance != 0] close/error Open [balance != 0] close/error Inactive [balance == 0] close Closed Dr. Beat Fluri © 2009 59

Modal Class Test Pattern Sneak Path Bug that allows an illegal message to be accepted, resulting in an illegal transition Test sneak path Send illegal messages Example: Account in state overdrawn Send message close Expected responses to sneak paths Message should be rejected State of the object should be unchanged Dr. Beat Fluri © 2009 60

Modal Class Test Pattern Test case specification For each path in the final transition tree specify one test case Test Case/Event Path Expected TC Level 1 Level 2 Level 3 Terminal State 1 New Open 2 New balance Open 3 New debit [balance < 0] balance Overdrawn Test case specification For each sneak path specify one test case Test Case/Event Path Expected Result TC Test State Test Event State Exception 1 Inactive freeze Inactive IllegalEventException 2 Overdrawn close Overdrawn IllegalEventException 3 Frozen debit Frozen IllegalEventException Dr. Beat Fluri © 2009 61

3 Method & Class Scope Test Inheritance and polymorphism

Inheritance-Related Bugs Incorrect Initialization Superclass initialization is omitted or incorrect Inadvertent Bindings Privates are invisible Non-private are invisible if subclass has member with the same name Missing Override A subclass-speciﬁc implementation of a superclass method is omitted Java: use abstract classes Dr. Beat Fluri © 2009 63

Inheritance-Related Bugs Naked Access A superclass instance variable is visible in a subclass and subclass methods update variables directly Square Peg in a Round Hole A subclass is incorrectly located in a hierarchy For example: Square subclass of Rectangular Naughty Children A subclass either does not accept all messages that the superclass accepts or leaves the object in a state that is illegal in the superclass Dr. Beat Fluri © 2009 64

Inheritance-Related Bugs Worm Holes A subclass computes values that are not consistent with the superclass invariant or superclass state invariants Gnarly Hierarchies A subclass creates new holes, boundaries, or intersections that are inconsistent with the superclass domain For example, strengthen precondition (in Eiffel, e.g., not possible) Weird Hierarchies Inheritance is abused as a kind of code-sharing macro to support hacks without regard to the resulting semantics Fat Interface A subclass inherits methods that are inappropriate or irrelevant Dr. Beat Fluri © 2009 65

Polymorphic Message Test Pattern Develop tests for a client of a polymorphic server that exercise all client bindings to the server Class hierarchy that overrides methods of the superclass Client may fail to meet all preconditions for all possible bindings The implementation of a server class is changed or extended Approach (different than in Binder) Determine all possible dynamic types Call method with each possible dynamic type Dr. Beat Fluri © 2009 66

Polymorphic Message Test Pattern Movie 1 Price title: String getCharge(days: int) getCharge(days: int) getFrequentRenterPoints(days: int) getFrequentRenterPoints(days: int) ChildrensPrice RegularPrice getCharge(days: int) getCharge(days: int) @Test getFrequentRenterPoints(days: int) public void getChargeTest() { NewReleasePrice Movie m = new Movie(new ChildrensPrice()); Assert.assertThat(m.getCharge(1), is(2)); getCharge(days: int) getFrequentRenterPoints(days: int) m = new Movie(new NewReleasePrice()); Assert.assertThat(m.getCharge(1), is(4)); m = new Movie(new RegularPrice()); Assert.assertThat(m.getCharge(1), is(3)); } @Test public void getFrequentRenterPointsTest() { // similar to getChargeTest() for ChildrensPrice and NewReleasePrice } Adapted from: Martin Fowler. Refactoring. Addison-Wesley Professional, 1999. Dr. Beat Fluri © 2009 67

Polymorphic Server Test Pattern Design a test suite at class scope that veriﬁes LSP compliance for polymorphic server hierarchy LSP: Liskov substitution principle If for each object o1 of type S there is an object o2 of type T such that for all programs P deﬁned in terms of T, the behavior of P is unchanged when o1 is substituted for o2, then S is a subtype of T. Barbara Liskov, Data Abstraction and Hierarchy, Keynote OOPSLA 1987 Hierarchy under test is nonmodal No constraints on the sequence of messages accepted Goal: exercise overridden methods in a polymorphic hierarchy Dr. Beat Fluri © 2009 68

Polymorphic Server Test Pattern Test each method in its deﬁning class and all subclasses that override it Reuse of the test suite for an overridden superclass method It could, but check whether additional test cases are necessary to test functionality and to satisfy coverage criteria Overriding a superclass method used by a superclass method Additionally test subclass inherited methods within the context of the subclass Check for conformance with LSP Make a client stub (simulate a client in test suite) Clients of a polymorphic server can substitute any subtype without causing a failure Dr. Beat Fluri © 2009 69

Polymorphic Server Test class ASTNode { class ASTNodeTest { final void accept(ASTVisitor v) { @Test void accept0Test() {/*..*/} v.preVisit(this); @Test void acceptTest() {/*..*/} accept0(v); } v.postVisit(this); class IfStatementTest { } @Test void accept0Test() {/*..*/} void accept0(ASTVisitor v) {} @Test } void acceptTest() { class IfStatement extends ASTNode { ASTNode a = new IfStatement(); void accept0(ASTVisitor v) { a.accept(new VisitorStub()); boolean children = v.visit(this); // ... if (children) { acceptChild(v, getExpression()); } acceptChild(v, getThenStatement()); } acceptChild(v, getElseStatement()); class WhileStatementTest { } @Test void accept0Test() {/*..*/} v.endVisit(this); @Test } void acceptTest() { } ASTNode a = new WhileStatement(); class WhileStatement extends ASTNode { a.accept(new VisitorStub()); void accept0(ASTVisitor v) { // ... // ... } } } } Dr. Beat Fluri © 2009 70

Structural Testing Find structural problems in source code

Structural Testing Test cases that satisfy coverage criteria Simple, but effective, coverage criteria Statement coverage Branch coverage A Condition coverage B C D F E X G I J H K Dr. Beat Fluri © 2009 72

Statement coverage All statements in a method have been executed at least once Why is statement coverage is not enough Statement coverage can be achieved without exercising all branches Single loop iteration can cover all statements but may not reveal loop control bugs Statement coverage can be achieved without exercising all true-false combinations of conditions Dr. Beat Fluri © 2009 73

Branch Coverage Every path from a node is executed at least once Why is branch coverage not enough Branch coverage can be achieved without exercising all true-false combinations of conditions Branch coverage does not ensure that all entry- exit paths are executed int foo(int x) { if (a == b) ++x; if (x == y) --x; return x; } Dr. Beat Fluri © 2009 74

Condition Coverage Condition coverage Each basic condition to be evaluated as true and false at lease once It is not required to test all branches if (x > 0 || x <= 10) {} (true || false) and (false || true) Not possible: Short-circuit evaluation Branch/condition coverage Condition coverage + each branch taken at least once Still not all true-false combinations of conditions exercised (true || false) and (false || true) and (false || false) Dr. Beat Fluri © 2009 75

Test in Isolation Mock’em

What about Testing in Isolation? Always possible; but dependencies must be simulated Stubs or mocks Stubs Provide answers to calls made during tests. Re-implemented interface of the original class Mocks Stubs + expectations which form a speciﬁcation of the calls they are expected to receive Dr. Beat Fluri © 2009 78

When to Use Mocks Classical way: Use real objects if possible Mocks if using real objects is awkward or if real objects cannot be used Focus on result of behavior Cost/beneﬁt balance Mockist way: Always use a mock for any object Problem: strong coupling to the implementation Mocking model objects is overkill Focus on how a unit/component is implemented to write expectations Source: http://martinfowler.com/articles/mocksArentStubs.html Dr. Beat Fluri © 2009 79

Tools for Mocking EasyMock http://www.easymock.org based on reflection mocking of static, final, or private methods; constructors not possible JMock http://www.jmock.org based on reflection mocking of static, final, or private methods; constructors not possible Dr. Beat Fluri © 2009 80

Tools for Mocking Amock http://amock.blogspot.com/ based on aspect oriented programming mocking of static, ﬁnal, or private methods; constructors possible AspectJ must be used JMockit (0.95) https://jmockit.dev.java.net/ based on instrumentation in Java 1.5 (VM arg: -javaagent:<path>/jmockit.jar=junit4) mocking of static, ﬁnal, or private methods and constructors possible no other tool necessary Dr. Beat Fluri © 2009 81

uDoo <<interface>> <<interface>> ITodoListModel IObservable Todo * TodoList Dispatcher 1..* Project Goal: test TodoList in isolation Problems Project and Todo objects are necessary Dispatcher object is necessary to test notiﬁcation Dr. Beat Fluri © 2009 82

JMockit in uDoo Classical way: Use real objects if possible Project and Todo objects are normally used (container objects) Since Dispatcher can be replaced; test against IObserable Use expectations to test whether or not IObservable notiﬁes observers Use Hamcrest-Matchers to test whether or not the observers are notiﬁed with the correct Event Use mock for Project to test corrupted Project (i.e.,does not store Todos) Dr. Beat Fluri © 2009 83

Demo Unit Testing in uDoo

Integration Testing

Integration Testing Integration testing is a search for components faults that cause intercomponent failures Breaking up component isolation step-by-step Integration strategy Which components are the focus of the integration test? In what sequence should component interfaces be exercised? Integration testing in object-oriented development begins early Within a class (integrating methods) Within a class hierarchy (integrating superclasses) Between a client and its server Within a package (integrating classes) ... Dr. Beat Fluri © 2009 87

Dependency Analysis Integration strategy follows sequences in which components have to be tested Dependency analysis among components is used to deﬁne the sequences Dependency analysis: approach Take component diagram, e.g., UML class diagram Make topology (i.e., directed graph) of the dependencies Dr. Beat Fluri © 2009 88

Big Bang Integration Demonstrate stability by attempting to exercise an entire system with a few test runs Integrate all components at once Monolithic system Components tightly coupled No doubles are necessary Problems Fault locations are difﬁcult to ﬁnd Big bang test suites are usually to small to reveal all interface faults Try to avoid big bang integration! Dr. Beat Fluri © 2009 91

Bottom-up Integration Demonstrate stability by adding components to the system under test in uses-dependency order, beginning with components having the fewest dependencies Approach Test leaf components in dependency graph with drivers for each leaf. Use doubles when necessary Repeat: test components on the next level; until the entire system is tested Dr. Beat Fluri © 2009 93

Bottom-up Integration Advantages Early integration of leaf components Parallel development on components in subtrees of dependency graph Reduces the number of doubles (only for cycles) Disadvantages Many drivers have to be developed Interaction testing is limited to collaborations implemented in the component (root of subtree) Postpones checking critical control interfaces and collaborations until the end (root of dependency graph) Dr. Beat Fluri © 2009 96

Top-down Integration Demonstrate stability by adding components to the system under test in control hierarchy order, beginning with the top level control objects Approach Test component(s) at the highest level ﬁrst. Use doubles for lower level components Continue in a breath-ﬁrst order at each level in the dependency graph until all components are integrated Dr. Beat Fluri © 2009 97

Top-down Integration Advantages Early demonstration of end-to-end functionality Fewer drivers necessary than with bottom-up integration Test cases can be reused to drive lower-level tests Disadvantages Development and maintenance of doubles Test cases in high-level components may not be sufﬁcient for lower-level components Dr. Beat Fluri © 2009 100

Collaboration Integration Demonstrate stability by adding sets of components to the system under test that are required to support a particular collaboration Approach Map collaborations onto the dependency graph Choose a sequence in which to apply testing on collaborations Test collaboration in the chosen sequence Dr. Beat Fluri © 2009 101

Collaboration Integration Advantages Few drivers and doubles necessary Focus on end-to-end functionality; useful for system scope testing Reuse of collaboration integration test suite for system test Disadvantages Inter-collaboration may not be tested thoroughly Participants in a collaboration are not exercised separately (big bang) Probably many doubles for testing initial collaborations Dr. Beat Fluri © 2009 106

Backbone Integration Combination of top-down integration, bottom-up integration, and big bang integration to verify interoperability among tightly coupled subsystems Used, for instance, for embedded system applications Backbone provides services that are essential for running tests and the application Creating doubles for the entire backbone would be impractical Approach Test each component in backbone in isolation and do a bottom-up integration or big bang integration of the backbone Do a top-down integration on the application control components Dr. Beat Fluri © 2009 107

Integration Testing and Continuous Integration

Multi-Stage Builds Special test classes for integration testing john ferguson smart Say Maven to run integration tests only for goal “integration-test” mvn integration-test Make a separate job in Hudson for integration testing Rapid feedback on unit testing level is still satisﬁed Integration may and are allowed to run longer than unit tests Dr. Beat Fluri © 2009 111

Integration Testing for uDoo <<interface>> ITodoListController Integrating controller with model <<interface>> ITodoCommand * execute() TodoListController Integrating UI with controller and undo() model comes later AddTodo AddProject DueDateChange PriorityChange RemoveProject Bottom-up AddTodoToProject DescriptionChange MoveTodo ProjectTitleChange RemoveT ITodoCommands with TodoList TodoListController with * ITodoCommands and TodoList Todo 2 Project Top-down <<interface>> ITodoListModel TodoListController with TodoList (including ITodoCommands, i.e., no mocks for commands) Decision: top-down The two steps with bottom-up are more or less the same Dr. Beat Fluri © 2009 112

Demo Integration Testing for uDoo

GUI and System Testing

System and GUI Testing Testing at application level Why testing at system level scope? Individual veriﬁcation of components cannot guarantee a correctly functioning system. Demonstrate that the system implements all required capabilities. Answer the question: “Can we ship it yet?” Objectives for testing at system level scope Requirement documents (formal) Natural language documents (informal) Prototypes Use cases Dr. Beat Fluri © 2009 116

Two-step approach UI Integration test Verify if UI interaction works as intended with the subsystem User story or use case testing with UI Validate if system works as intended when simulating user stories or use cases with UI interaction Dr. Beat Fluri © 2009 117

GUI Integration Testing Manual GUI testing Testing script that is executed by a user Hard to repeat (e.g., in a CI environment) Semi-automated GUI testing Capture and replay Have to be adapted on changing UI Automated GUI testing with a framework FEST-swing: http://fest.easytesting.org Abbot: http://abbot.sourceforge.net/doc/overview.shtml Dr. Beat Fluri © 2009 118

FEST-Swing FEST-Swing features Simulation of user interaction with a GUI (e.g., mouse and keyboard input) Reliable GUI component lookup (by type, by name, or custom search criteria) Support for all Swing components included in the JDK Compact and powerful API for creation and maintenance of functional GUI tests Supports Applet testing Ability to embed screenshots of failed GUI tests in HTML test reports Can be used with either TestNG or JUnit Supports testing violations of Swing's threading rules Source: FEST-swing: http://fest.easytesting.org/swing/ Dr. Beat Fluri © 2009 119

uDoo and FEST-Swing uDoo UI integration with controller component For each functionality in the TodoController Trigger it via the UI Verify that the model is as expected (trigger via UI does not have side-effects in the model) Verify that the model is consistently displayed in the UI (trigger via UI does not have side-effects in the UI) Verify that the entire model is still accessible via UI Dr. Beat Fluri © 2009 120

User Story Testing Develop a system test suite by simulating a user story Difference to integration testing User stories cover a chain of actions Integration testing starts over for each test case Approach Write down a story that covers all features of the application Specify a test case that runs the story as a system test including UI interaction Verify consistency after each feature invocation Use coverage tool to verify that each method was called at least once Dr. Beat Fluri © 2009 121

uDoo User Story Testing Specify a user story that uses each feature of uDoo After executing each feature verify consistency Create a new todo and change its title: verify Move a todo and undo the action: verify etc. Implement the user story as a FEST-Swing test case Dr. Beat Fluri © 2009 122

Demo FEST-swing in uDoo UI Integration Testing User Story Testing

Regression Testing

Regression Testing A new or modiﬁed component can fail when used with unchanged components Failures by generating side-effects or feature interactions When a new or modiﬁed component causes a system failure, the system is said to regress A regression test case is a test case that the baseline version of a component has passed and which is expected to pass when run on a delta version Dr. Beat Fluri © 2009 125

Regression Testing Regression test suite is a set of existing test cases Does not contain tests for new or changed functionality Plays an important role in revealing bugs The software failure in the Ariane 5 rocket controller was in large part due to an assumption that previously tested code would work when it was reused, obviating the need for regression testing. Dr. Beat Fluri © 2009 126

Regression Testing - When and How When a new subclass has been developed Rerun the superclass test cases on the subclass, beside new test cases for the new subclass When a superclass is changed Rerun the superclass test cases on the superclass and on its subclasses. Rerun the subclass test cases When a server class is changed Rerun test cases for the clients of the changed class plus testing the changed class When a bug ﬁx has been completed Rerun test cases on any part of the software that depend on the changed code to reveal unintended side effects of the bug ﬁx Dr. Beat Fluri © 2009 127

Regression Faults Two components B and D pass individually their test suite, but fail when used together D has a side effect on B D changes an entity used by B D is a client of B D sends a msg that violates B’s invariant or a precondition D is a server of B B sends a msg to D. D’s postconditions have changed or are buggy. The return value may cause a failure of B Dr. Beat Fluri © 2009 128

Regression Faults D is a new subclass of a polymorphic hierarchy that is not LSP- compliant B is a client of D and expects the new subtype to be compatible D is incompatible with B Ariane 5 disaster resulted from an incorrect ﬂoating-point conversion. The requirements the algorithm , and the code did not change An undesirable feature interaction occurs between B and D Dr. Beat Fluri © 2009 129

Reducing a Test Suite Full regression test can be too time consuming (setup or run) Reduced regression test: Select a set of test cases for a speciﬁc test scenario In other words, which test cases can be skipped? Safe reduction (likelihood of ﬁnding regression faults is not reduced) Dr. Beat Fluri © 2009 130

Safe Reduction Inclusiveness Percentage of baseline test cases that may show regression faults; a safe regression test suite is 100 percent inclusive Precision Percentage of baseline test cases in a reduced test suite that cannot reveal regression faults and that are not selected for the reduced test suite Efﬁciency Cost of identifying a reduced regression test suite Generality The range of application for the selection strategy Dr. Beat Fluri © 2009 131

Regression Test Patterns Patterns describe which baseline test cases should be selected Each pattern has a “safe reduction” evaluation with the predicates inclusiveness, precision, efﬁciency, and generality Patterns Retest All Retest Risky Use Cases Retest Changed Code Dr. Beat Fluri © 2009 132

Rest All Rerun the entire baseline test suite on a delta version Just run all existing tests Safe reduction evaluation Inclusiveness: All baseline tests are selected; => is safe Precision: No tests that could be skipped are skipped; => the least precise Efﬁciency: Lowest cost of analysis and setup. Highest run cost Generality: Can always be applied. Easy to set up, no analysis or special development is necessary Dr. Beat Fluri © 2009 133

Retest Risky Use Cases Use risk-based heuristics to select a partial regression test suite. Run this subset on a delta version Suspicious use cases: Select use cases that depend on components that are individually unstable or unproven have not been shown to work together before implement complex business rules have a complex implementation were subject to high churn during development (many changes spread all over the system) Dr. Beat Fluri © 2009 134

Retest Risky Use Cases Critical use cases: Select tests for use cases that are necessary for safe, effective operation Safe reduction evaluation Inclusiveness: Unsafe; test cases are not selected by analysis of dependencies Precision: Some tests that could be skipped may be repeated Efﬁciency: Setup cost are low: no code analysis is necessary; run cost may be high Generality: Can always be applied. Easy to set up, no analysis or special development is necessary Dr. Beat Fluri © 2009 135

Retest Changed Code Use code change analysis to select a partial regression test suite. Run this subset on a delta version Procedure 1. Use Cobertura to check which test case cover which source code parts 2. Use CVS/SVN diff information to see which source code parts changed 3. Rerun test cases that cover deleted or changed source code parts Dr. Beat Fluri © 2009 136

Retest Changed Code Safe reduction evaluation Inclusiveness: All baseline test cases that can produce a different result are selected; => is safe Precision: Most precise of the white box regression test strategies Efﬁciency: If test case selection can be automated, the setup cost is low; in worst case (all baseline test case have to be selected) the total cost of this pattern can be higher than Retest All pattern Generality: Class scope; => analysis knowledge and special development is necessary Dr. Beat Fluri © 2009 137

Regression Testing and CI Recommendation: make a separate Maven module and Hudson job Retest all Copy all existing tests to the regression-testing module Retest risky use cases Create a test suite with the risky use cases and conﬁgure maven Retest changed code Write a script... Dr. Beat Fluri © 2009 138

Demo uDoo Regression Tests

CI Tools

Hudson Plugins Besides automated build and test Hudson offers a wealth of plugins Source code management, build triggers, build notiﬁers, build tools, build wrappers, build reports, Maven, etc. http://wiki.hudson-ci.org/display/HUDSON/Plugins Build report plugins for QA Checkstyle Cobertura (already shown) FindBugs PMD Dr. Beat Fluri © 2009 141

Checkstyle “Checkstyle is a development tool to help programmers write Java code that adheres to a coding standard. It automates the process of checking Java code to spare humans of this boring (but important) task. This makes it ideal for projects that want to enforce a coding standard.” http://checkstyle.sourceforge.net/ Maven plugin http://maven.apache.org/plugins/maven-checkstyle-plugin/ Conﬁgure via XML or wizard in Eclipse http://eclipse-cs.sourceforge.net/ Dr. Beat Fluri © 2009 142

FindBugs “[FindBugs] a program which uses static analysis to look for bugs in Java code.” http://findbugs.sourceforge.net/ Over 350 bug patterns http://findbugs.sourceforge.net/bugDescriptions.html Maven plugin http://mojo.codehaus.org/findbugs-maven-plugin/2.0.1/ Dr. Beat Fluri © 2009 143

PMD PMD scans Java source code and looks for potential problems like: Possible bugs - empty try/catch/ﬁnally/switch statements Dead code - unused local variables, parameters and private methods Suboptimal code - wasteful String/StringBuffer usage Overcomplicated expressions - unnecessary if statements, for loops that could be while loops Duplicate code - copied/pasted code means copied/pasted bugs http://pmd.sourceforge.net/ Over 280 rules http://pmd.sourceforge.net/rules/index.html Maven plugin http://maven.apache.org/plugins/maven-pmd-plugin/ Dr. Beat Fluri © 2009 144

Demo uDoo-QA

Seminar on Software Testing

0 comments

Notes on slide 1

1 Favorite