Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Software testing: an introduction - 2017

38 views

Published on

Invited talk to the INFOM110 Software engineering course, University of Namur

Published in: Software
  • Be the first to comment

  • Be the first to like this

Software testing: an introduction - 2017

  1. 1. www.unamur.be Software Testing: an Introduction Xavier Devroey <x.d.m.devroey@tudelft.nl> SERG, Delft University of Technology, The Netherlands INFO M110 : Ingénierie du logiciel 2017 - 2018
  2. 2. Error, fault, failure Developper 1. makes errors 2. failure 3. caused by a fault (bug)
  3. 3. Error, fault, failure, incident Developper User 4. reports the incident 5. assigned to Bug tracker
  4. 4. Why you should care about testing
  5. 5. Moth, found in one of the relays of Harvard's Mark II, taped to a log book (September 1947).
  6. 6. Vol 501 d'Ariane 5 (1996) • Carrying 370 millions $ • Flight for about 40 sec. • Integer overflow due to a 64-bit floating point number conversion into a 16-bit signed integer https://fr.wikipedia.org/wiki/Vol_501_d%27Ariane_5
  7. 7. Millennium bug (1999) • Date format problem https://en.wikipedia.org/wiki/Year_2000_problem 01 DATE. 05 DAY PIC 99. 05 MONTH PIC 99. 05 YEAR PIC 99. • Year declared on 2 digits
  8. 8. WoW Corrupted Blood Incident (2005) • Boss casts a highly contagious hit point-draining spell (Corrupted Blood) https://en.wikipedia.org/wiki/Corrupted_Blood_incident • Spell intended to last only seconds and functions only within the boss area • … but spreads through hunter pets and NPCs
  9. 9. Heartbleed (2014) • Security bug in OpenSSL • Buffer overread • Missing bounds check https://en.wikipedia.org/wiki/Heartbleed
  10. 10. Seville Airbus A400M Atlas crash (2015) https://en.wikipedia.org/wiki/2015_Seville_Airbus_A400M_Atlas_crash • Three engines stopped
 working • Engines software configuration issue « […] a weakness in the test procedure of planes before they fly, or a problem that results from the implementation of these procedures. » • Pre-delivery test flight
  11. 11. define: software testing
  12. 12. A test engineer walks into a bar and • orders a beer • orders 0 beers • orders 9999999 beers • orders a lizard • orders -1 beers • orders a "sfdeljknesv" — Bill Sempf (@sempf)
  13. 13. System
 under
 test
  14. 14. Menu System
 under
 testSpecification
  15. 15. Menu System
 under
 testSpecification Test case
  16. 16. Menu System
 under
 testSpecification Test case Output
  17. 17. Menu System
 under
 testSpecification Test case Output Pass or fail?
  18. 18. Software testing consists of the dynamic verification 
 that a program provides expected behaviours 
 on a finite set of test cases, suitably selected from the usually infinite execution domain. [SWEBOK v3]
  19. 19. Software testing consists of the dynamic verification 
 that a program provides expected behaviours 
 on a finite set of test cases, suitably selected from the usually infinite execution domain. vs static Formal, semiformal, or informal • compiler warnings (code analysis), • javadoc warnings (documentation analysis), • design inspection (model analysis), • theorem proving (e.g., Event-B), • code analysis • Code review • IDE warnings • FindBugs (http://findbugs.sourceforge.net)
  20. 20. Software testing consists of the dynamic verification 
 that a program provides expected behaviours 
 on a finite set of test cases, suitably selected from the usually infinite execution domain.The oracle problem “An oracle is any human or mechanical agent that decides whether a program behaved correctly in a given test and accordingly results in a verdict of “pass” or “fail”.” [SWEBOK v3]
  21. 21. Menu System
 under
 testSpecification Test case Output Pass or fail?
  22. 22. Software testing consists of the dynamic verification 
 that a program provides expected behaviours 
 on a finite set of test cases, suitably selected from the usually infinite execution domain. “Program testing can be used to show the presence of bugs, but never to show their absence” Dijkstra (1970) Exhaustive testing is not possible in practice !!! Trade-off between limited resources, schedules and unlimited test requirements Prioritization based on risk, usage, cost, …
  23. 23. Software testing consists of the dynamic verification 
 that a program provides expected behaviours 
 on a finite set of test cases, suitably selected from the usually infinite execution domain. “I have a lot of test cases, thus my program is bug free” WRONG !!! How to select test cases ? • Selection criteria • I/O coverage (black box testing) • No access to the source code • Code coverage (white box testing) • Access to source code • Test engineer expertise How to assess the quality of your test cases?
  24. 24. A software testing typology
  25. 25. Source of test generation • Black box testing • Based on specification documents • No access to the source code • White box testing • Access to the source code • Model-based testing Characteristic being tested • Functional • is the output correct for a given input ? • Non-functional • performance, robustness, security, usability, etc.
  26. 26. Purpose of the tests • Testing a single element • Unit testing (e.g., a single class) • Component testing • Integration of multiple elements • Integration testing • System testing • Test with users • Acceptance/Beta testing • After an evolution of the system • Regression testing
  27. 27. Requirements specification Design Coding and unit testing Integration/ System testing Acceptance testing Release Integration/ System test plan Acceptance test plan Maintenance and regression testing
  28. 28. Testing in Agile process Testing from the beginning • No commit without test! • Have a clear testing policy • Are your tests good enough? • Use dedicated tools Continuous Integration Dashboard Cobertura
  29. 29. Specify requirements as tests Testing in Agile process Testing from the beginning • State what system • should do • and should not do
  30. 30. Specify requirements as tests Testing in Agile process Testing from the beginning Testers and developers are allies • Testers are developers • developers are testers
  31. 31. Specify requirements as tests Testing in Agile process Testing from the beginning Testers and developers are allies Test often and in small chunks • Unit and integration tests are important • Code design matters!!!
  32. 32. Testing Driven Development (TDD) Red • Write automated tests (examples) from user stories • Tests fail Green • Developers design and write code to make the tests pass • Tests are run frequently Refactor • Refactor code • Without changing interfaces or behaviour • Tests must pass
  33. 33. The Telecommunication Software & Systems Group (TSSG) case study
  34. 34. Agile process implementation The Telecommunication Software & Systems Group (TSSG) case study From Dowling, P., & McGrath, K. (2015). Using free and open source tools to manage software quality. Communications of the ACM, 58(7), 51–55. doi:10.1145/2755503 • Open Source tools • Support team’s activities management • Language: Java
  35. 35. Agile process implementation The Telecommunication Software & Systems Group (TSSG) case study Project Management Requirements Management Issues Tracking Development environment Building Unit testing Code coverage Cobertura Continuous Integration Code Repository Test environment Functional Testing Test Case Management Load Testing Security Testing
  36. 36. Agile process implementation The Telecommunication Software & Systems Group (TSSG) case study Project Management Requirements Management Issues Tracking Development environment Building Unit testing Code coverage Cobertura Continuous Integration Code Repository Test environment Functional Testing Test Case Management Load Testing Security Testing
  37. 37. Agile process implementation The Telecommunication Software & Systems Group (TSSG) case study Project Management Requirements Management Issues Tracking Development environment Building Unit testing Code coverage Cobertura Continuous Integration Code Repository Test environment Functional Testing Test Case Management Load Testing Security Testing
  38. 38. (J)Unit testing
  39. 39. Class
 Under
 Test
  40. 40. public class Message {
     private long id = -1;     private String author;     private String message;
     public Message(String author, String message) {         this.author = author;         this.message = message;     }     // Gets and Sets ...     /**      * Appends the given text at the end of this' message.      * A space is added before the given text.      *      * @param text The test to append.      */     public void compose(String text) {         this.message = message + " " + text;     } }
  41. 41. • Collec'on of classes to perform unit tes'ng • Uses annota'ons (e.g., @Test)
  42. 42. public class MessageTest {     @BeforeClass     public static void setUpClass() {     // Executed once before all tests     }     @AfterClass     public static void tearDownClass() {     // Executed once after all tests     }
     @Before     public void setUp() {     // Executed before each test     }
     @After     public void tearDown() {     // Executed after each test     }
     @Test     public void testCompose() {         // Write test here     } }
  43. 43. @Test     public void testComposeString() {         Message msg = new Message("Me", "despicable");         msg.compose("me");     } Is the compose method covered by the test case? Is it enough? Yes No There are no assertions on the result
  44. 44. Software testing consists of the dynamic verification 
 that a program provides expected behaviours 
 on a finite set of test cases, suitably selected from the usually infinite execution domain.The oracle problem “An oracle is any human or mechanical agent that decides whether a program behaved correctly in a given test and accordingly results in a verdict of “pass” or “fail”.” [SWEBOK v3]
  45. 45. @Test     public void testComposeString() {         Message msg = new Message("Me", "despicable");         msg.compose("me");         assertThat(msg.getMessage(), equalTo("despicable me"));     } JUnit assertions Hamcrest matchers asserTrue(c);  assertFalse(c);  assertEquals(expected, actual);  assertNull(o);  assertNotNull(o);  fail(); assertThat(T actual, matcher); assertThat(T actual, equalTo(T operand)); assertThat(T actual, not(T operand)); assertThat(Iterable<? super T> c,
 hasItem(T item)); assertThat(T actual, nullValue()); assertThat(T actual, notNullValue());
  46. 46. Core • anything - always matches, useful if you don't care what the object under test is Logical • allOf - matches if all matchers match, short circuits (like Java &&) • anyOf - matches if any matchers match, short circuits (like Java ||) • not - matches if the wrapped matcher doesn't match and vice versa Object • equalTo - test object equality using Object.equals • hasToString - test Object.toString • instanceOf, isCompatibleType - test type • notNullValue, nullValue - test for null • sameInstance - test object identity Collections • hasEntry, hasKey, hasValue - test a map contains an entry, key or value • hasItem, hasItems - test a collection contains elements • hasItemInArray - test an array contains an element Number • closeTo - test floating point values are close to a given value • greaterThan, greaterThanOrEqualTo, lessThan, lessThanOrEqualTo - test ordering Text • equalToIgnoringCase - test string equality ignoring case • equalToIgnoringWhiteSpace - test string equality ignoring differences in runs of whitespace • containsString, endsWith, startsWith - test string matching
  47. 47. Fakes, Stubs, and Mocks
  48. 48. Class
 Under
 Test DAO Class Database
  49. 49. Fake Test implementation ≠ production implementation public class FakeMessageDAO extends MessageDAO{          private static long id = 0;     private final Map<Long, Message> messages;     public FakeMessageDAO(DAOFactory factory) {         messages = new HashMap<>();     }
     @Override     public Message create(Message value) throws DAOException {         value.setId(id); messages.put(id, value); id++;         return value;     }     @Override     public void delete(Message value) throws DAOException {…}
     @Override     public Message find(long id) throws DAOException {…}     … }
  50. 50. Fake Test database instance Design your code to • Externalize DB access information .properties • Manage DB connections BasicDataSource source = new BasicDataSource(); source.setUrl("jdbc:derby:memory:sample;create=true"); source.setDriverClassName("org.apache.derby.jdbc.EmbeddedDriver"); ... source.getConnection(); Connection pooling using commons-dbcp Connection management using commons-dbutils
  51. 51. Stub Test implementation holds predefined data public class StubMessageDAO extends MessageDAO {
     @Override     public Message create(Message value) throws DAOException {         value.setId(42l);         return value;     } 
     @Override     public Message find(long id) throws DAOException {         return new Message(id, "Me", "Despicable");     }     @Override     public List<Message> find(String author) throws DAOException {         List<Message> result = new ArrayList<>();         result.add(new Message(9, author, "One ring to rule them all"));         result.add(new Message(42, author, "Life, the Universe and Everything"));         return result;     }     … }
  52. 52. Mock Registers calls it receives @Test public void testMocking() throws Exception {     // Initialise mock object     MessageDAO dao = mock(MessageDAO.class);     when(dao.find(42)).thenReturn(new Message(42, "Me", "Despicable"));     // Check that method has been called     verify(dao).find(42); }     // Call the method     Message msg = dao.find(42);     assertThat(msg.getMessage(), equalTo("Despicable")); Should you always check method calls?
  53. 53. be.unamur:myfirstwebapp:0.0.1-SNAPSHOT MessageTest.java FakeMessageDAO.java StubMessageDAO.java MockMessageDAO.java PrintMessagesServletTest.java Simple example of JUnit test class Fake MessageDAO object with a Map implementation MessageDAO stub with predefined data Example of mock usage applied to MessageDAO class Complete example of a servlet unit test with mocking and a fake DAOFactory using a in memory database
  54. 54. Coverage
  55. 55. Requirements coverage Req.
 1 Req. 1.1 Req. 1.2 Req.
 2 Req. 2.1 Test1 x x Test2 x Test3 x Test4 x x • Coverage matrix
  56. 56. Input/output domain coverage public void compose(String text); public void compose(String t1, String t2); • Equivalence partitioning Normal case: "Despicable you" Limit cases: "" null "azrtyuiop^$ù m;21@98#3!" • t1 • t2 Normal case: "Despicable you" Limit cases: "" null "azrtyuiop^$ù m;21@98#3!" Normal case: "Despicable you" Limit cases: "" null "azrtyuiop^$ù m;21@98#3!" ☛ Combinatorial explosion with the number of parameters (based on pre/post conditions)
  57. 57. Combinatorial Interaction Testing (CIT) t1 t2 t3 "Despicable you" "Despicable you" "Despicable you" "" "" "" null null null "azrtyuiop^$ù" "azrtyuiop^$ù" "azrtyuiop^$ù" Hyp.: most faults are caused by interactions of at most two factors Pairwise testing: cover all pairs of values "Despicable you" "Despicable you" "Despicable you" "Despicable you" "" "" "Despicable you" null null "Despicable you" "azrtyuiop^$ù" "azrtyuiop^$ù" "" "Despicable you" "" "" "" "Despicable you" "" null "azrtyuiop^$ù" "" "azrtyuiop^$ù" null null "Despicable you" null null "" "azrtyuiop^$ù" null null "Despicable you" null "azrtyuiop^$ù" "" "azrtyuiop^$ù" "Despicable you" "azrtyuiop^$ù" "azrtyuiop^$ù" "" null "azrtyuiop^$ù" null "" "azrtyuiop^$ù" "azrtyuiop^$ù" "Despicable you" 16 test cases instead of 64
  58. 58. app. http://cse.unl.edu/~citportal/ http://ctweb.abstracta.com.uy/combinatorial.jsp Hyp.: most faults are caused by interactions of at most t factors T-wise testing: cover all t-uples of values Integration and System testing Which configuration should I use for my tests?
  59. 59. type Android version Screen Processor tablet 4.0.4 XLarge ARM Cortex smartphone 4.1 Large Intel x86 phablet 4.1.1 Normal 4.1.2 Small … or using a feature model http://familiar-project.github.io https://featureide.github.io http://www.skalup.com/ http://research.henard.net/SPL/PLEDGE/ http://martinfjohansen.com/models2011/spltool/ http://cse.unl.edu/~citportal/
  60. 60. Code coverage Line coverage: percentage of lines covered by tests https://github.com/cobertura/cobertura Achieving line coverage 
 = 
 executing each line at least once
  61. 61. Code coverage Branch coverage: percentage of lines covered by tests https://github.com/cobertura/cobertura Achieving branch coverage 
 = 
 executing each branch at least once Is it enough? No
  62. 62. @Test     public void testComposeString() {         Message msg = new Message("Me", "despicable");         msg.compose("me");     } There are no assertions on the result Cobertura mvn cobertura:cobertura be.unamur:myfirstwebapp:0.0.1-SNAPSHOT
  63. 63. Mutation testing
  64. 64. SUT System under test
  65. 65. SUT System under testTest cases executed on t1 t2 t3 t4 Mutation operators introduce small syntactic errors
  66. 66. SUT M1 Mutation operator Mutants t1 t2 t3 t4 Mutation operators introduce small syntactic errors
  67. 67. SUT M1 Mutation operator M2 Mutants t1 t2 t3 t4 Mutation operators introduce small syntactic errors
  68. 68. SUT M1 Mutation operator M2 M3 Mutants t1 t2 t3 t4 Mutation operators introduce small syntactic errors
  69. 69. Test cases M1 M2 M3 executed on Mutants t1 t2 t3 t4
  70. 70. M1 M2 M3 Mutation score = 2/3 MutantsTest cases t1 t2 t3 t4
  71. 71. Hypothesis Competent programmer hypothesis Programmers write programs that are almost perfect • Faults are syntactically small • Test catches artificial mutants test catches real mutants (faults) Coupling effect Big effects arising from bugs in the software are closely coupled to small and simple bugs. • Small mutations emulates real bugs in the software http://www0.cs.ucl.ac.uk/staff/M.Harman/exe10.html Javalanche µJava Major
  72. 72. CONDITIONALS_BOUNDARY Operators NEGATE_CONDITIONALS MATH INCREMENTS < → <= if(…) → if(true) REMOVE_CONDITIONALS == → != + → - i++ → i-- INVERT_NEGS -i → i INLINE_CONSTS true → false RETURN_VALS return o; → return null; ...
  73. 73. be.unamur:myfirstwebapp:0.0.1-SNAPSHOT mvn org.pitest:pitest-maven:mutationCoverage mvn clean test Results in target/pit-reports/
  74. 74. The symbolic way
  75. 75. Program execution using symbolic values {x : int} {x : int | x < 0} {x : int | x < 0} int foo(int x){ if(x < 0){ return -1/x; } return 1/x; }
  76. 76. int foo(int x){ if(x < 0){ return -1/x; } return 1/x; } Program execution using symbolic values {x : int} {x : int | x > 0} v {x : int | x = 0} test case 1 test case 2
  77. 77. The search-based way
  78. 78. Test case generation
 =
 a search optimization problem To find a (near) optimal solution Search guided 
 using a meta-heuristic 0 20 40 60 80 t1 t2 t3 t4 t5 t6 t7 Potential solutions Usefulness
  79. 79. Test case generation
 =
 a search optimization problem Want to minimize/maximize objective function Classic techniques: • linear programming 
 (e.g., simplex) • branch and bound • etc. Meta-heuristics: • genetic algorithms • hillclimbing • etc.
  80. 80. Search guided 
 using a meta-heuristic • Problem-independent • Approximates solution • Non deterministic • Requires to: • represent solutions • encoding/decoding • evaluate solutions • using a fitness function
  81. 81. • Pick a random candidate in the search space • Find the neighbours • Measure the fitness of the candidate and its neighbours • Either • Pick the best neighbour (if any), and iterate. • Or • If there is no better neighbour, (and the fitness is not ideal yet), re-start Hillclimbing
  82. 82. Hillclimbing Global maximum f(x) x Global minimum
  83. 83. Global maximum f(x) x Local maximum Hillclimbing
  84. 84. Evaluate Fitness Selection Crossover Initialize Population F = 0 || T Mutation Create the Next Generation Reinsertion Genetic algorithms
  85. 85. Random initial test suites Test suites evolution Minimize test suite with maximized coverage
  86. 86. • mvn compile evosuite:generate • mvn evosuite:export -DtargetFolder=target/ generated-test-sources/evosuite • mvn test Generates tests in .evosuite/ folder
  87. 87. System testing
  88. 88. Test environment Functional Testing Test Case Management Load Testing Security Testing Performance
 Testing Functional testing Security Testing NMap
  89. 89. Selenium UI testing • open: opens a page using a URL • click/clickAndWait: performs a click operation (and waits for a page to load) • verifyTitle/assertTitle: verifies an expected page title • verifyTextPresent: verifies expected text is somewhere on the page • verifyElementPresent: verifies an expected UI element, as defined by its HTML tag, is present on the page • verifyText: verifies expected text and its corresponding HTML tag are present on the page • verifyTable: verifies a table’s expected contents • waitForPageToLoad: pauses execution until an expected new page loads. Called automatically when clickAndWait is used • waitForElementPresent: pauses execution until an expected UI element, as defined by its HTML tag, is present on the page
  90. 90. Acceptance testing
  91. 91. Test with/by the user • Functionality • Performance • Security • … Acceptance test plan • Which features? • Which data? • Which KPI? • …
  92. 92. Take away message
  93. 93. Software testing consists of the dynamic verification 
 that a program provides expected behaviours 
 on a finite set of test cases, suitably selected from the usually infinite execution domain. White box Black box Model-based Functional Non-functional • performance • robustness • security • usability Unit Integration System Acceptance Regression
  94. 94. Testing is important • User satisfaction • Money • Life Software is developed for users Use proper tools … including QA tools
  95. 95. There is always one more bug…

×