Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Software testing: an introduction - 2016

17 views

Published on

Invited talk to the INFOM110 Software engineering course, University of Namur

Published in: Software
  • Be the first to comment

  • Be the first to like this

Software testing: an introduction - 2016

  1. 1. www.unamur.be Software Testing: an Introduction Xavier Devroey <xavier.devroey@unamur.be> INFO M110 Ingénierie du logiciel Namur, Belgique
  2. 2. Plan • Introduction • Software testing typology – Source of test generation – Characteristic being tested – Lifecycle phase • Agile process implementation: the TSSG case study • Unit testing using Maven and JUnit • Code review • Test-case assessment 21/10/2016 xavier.devroey@unamur.be 2
  3. 3. INTRODUCTION
  4. 4. Error, fault, failure, and incident Error 21/10/2016 xavier.devroey@unamur.be 4 Dev.
  5. 5. Error, fault, failure, and incident Error Fault Exec. Failure (bug) 21/10/2016 xavier.devroey@unamur.be 5 Dev.
  6. 6. Error, fault, failure, and incident Error Fault Exec. Failure (bug) Reports Incident Bug Tracker 21/10/2016 xavier.devroey@unamur.be 6 Dev. User
  7. 7. Software testing: definition [SWEBOK v3] “Software testing consists of the dynamic verification that a program provides expected behaviours on a finite set of test cases, suitably selected from the usually infinite execution domain.” 21/10/2016 xavier.devroey@unamur.be 7
  8. 8. Testing process 21/10/2016 xavier.devroey@unamur.be 8 Req. Test engineer Test case defines
  9. 9. Testing process 21/10/2016 xavier.devroey@unamur.be 9 Req. Test engineer Test case defines Pgm. executed on
  10. 10. Dynamic verification • Testing implies executing the program under test • Contrary to static analysis – Formal, semiformal, or informal – Examples: • compiler warnings (code analysis), • javadoc warnings (documentation analysis), • design inspection (model analysis), • theorem proving (e.g., Event-B), • code analysis – Code review – IDE warnings – FindBugs http://findbugs.sourceforge.net 21/10/2016 xavier.devroey@unamur.be 10
  11. 11. Expected behaviours • It must be possible to decide whether the observed outcomes for a given input is acceptable or not • The oracle problem – “An oracle is any human or mechanical agent that decides whether a program behaved correctly in a given test and accordingly results in a verdict of “pass” or “fail”.” [SWEBOK v3] 21/10/2016 xavier.devroey@unamur.be 11
  12. 12. Finite set of test cases • Exhaustive testing is not possible in practice – Infinite number of test cases – Select a subset of all possible tests – Trade-off between limited resources, schedules and unlimited test requirements • Prioritization based on risk, usage, cost, … “Program testing can be used to show the presence of bugs, but never to show their absence” Dijkstra (1970) 21/10/2016 xavier.devroey@unamur.be 12
  13. 13. Suitably selected • How to select test-cases ? • Selection criteria – Input/output domains coverage (black box testing) • Domain partitioning techniques • Based on specification documents • No access to the source code – Code coverage: statement coverage, decision coverage, path coverage, etc. (white box testing) • Access to source code – Test engineer expertise 21/10/2016 xavier.devroey@unamur.be 13
  14. 14. SOFTWARE TESTING TYPOLOGY
  15. 15. Types of testing • Source of test generation – Black box testing • Based on specification documents • No access to the source code – White box testing • Access to the source code – Model-based testing • Characteristic being tested – Functional testing: is the output correct for a given input ? – Non-functional testing: Performance, robustness, security, usability, etc. 21/10/2016 xavier.devroey@unamur.be 15
  16. 16. Types of testing • Lifecycle phase – Coding • Unit testing • Component testing – Integration • Integration testing – System integration • System testing – Release • Acceptance/Beta testing – Maintenance • Regression testing 21/10/2016 xavier.devroey@unamur.be 16
  17. 17. Testing in the waterfall model 21/10/2016 xavier.devroey@unamur.be 17 Requirements specification Design Coding and unit testing Integration testing System testing Acceptance testing Release Maintenance and regression testing
  18. 18. Testing in the V-model 21/10/2016 xavier.devroey@unamur.be 18 Requirements specification Design Coding and unit testing Integration/ System testing Acceptance testing Release Integration/ System test plan Acceptance test plan Maintenance and regression testing
  19. 19. Testing in an Agile process 1. Include testing activities from the beginning 2. Specify requirements in terms of tests 3. Testers and developers are not enemies 4. Test often and in small chunks 21/10/2016 xavier.devroey@unamur.be 19
  20. 20. Test Driven Development (TDD) • Red – Write automated tests (examples) from user stories – Tests fail (as the functionality is not implemented yet) • Green – Developers design and write code to make the tests pass – Tests are run frequently • Refactor – Refactor code without changing interfaces or behaviour (tests must pass) Red GreenRefactor 21/10/2016 xavier.devroey@unamur.be 20
  21. 21. AGILE PROCESS IMPLEMENTATION The Telecommunication Software & Systems Group (TSSG) case study
  22. 22. Agile process implementation: TSSG • From Dowling, P., & McGrath, K. (2015). Using free and open source tools to manage software quality. Communications of the ACM, 58(7), 51–55. doi:10.1145/2755503 • Open Source tools • Support team’s activities management • Language: Java 21/10/2016 xavier.devroey@unamur.be 22
  23. 23. 21/10/2016 xavier.devroey@unamur.be 23 Project Management Requirements Management Issues Tracking
  24. 24. 21/10/2016 xavier.devroey@unamur.be 24 Project Management Requirements Management Issues Tracking Development environment Building Unit testing Code coverage Cobertura Continuous Integration Code Repository
  25. 25. 21/10/2016 xavier.devroey@unamur.be 25 Project Management Requirements Management Issues Tracking Development environment Building Unit testing Code coverage Cobertura Continuous Integration Code Repository Test environment Functional Testing Test Case Management Load Testing Security Testing
  26. 26. Workflow • Developer forks the code base • Implements changes • Integrates possible changes from code repo. • Runs a complete build – Including Unit tests • Submits changes to code repo. • CI server performs integration tests 21/10/2016 xavier.devroey@unamur.be 26 Do NOT break the build!
  27. 27. 21/10/2016 xavier.devroey@unamur.be 27 Project Management Requirements Management Issues Tracking Development environment Building Unit testing Code coverage Cobertura Continuous Integration Code Repository Test environment Functional Testing Test Case Management Load Testing Security Testing
  28. 28. 21/10/2016 xavier.devroey@unamur.be 28 Project Management Requirements Management Issues Tracking Development environment Building Unit testing Code coverage Cobertura Continuous Integration Code Repository Test environment Functional Testing Test Case Management Load Testing Security Testing
  29. 29. 21/10/2016 xavier.devroey@unamur.be 29 Project Management Requirements Management Issues Tracking Development environment Building Unit testing Code coverage Cobertura Continuous Integration Code Repository Test environment Functional Testing Test Case Management Load Testing Security Testing
  30. 30. 21/10/2016 xavier.devroey@unamur.be 30 Project Management Requirements Management Issues Tracking Development environment Building Unit testing Code coverage Cobertura Continuous Integration Code Repository Test environment Functional Testing Test Case Management Load Testing Security Testing
  31. 31. Crash course
  32. 32. Maven 3: Introduction • Project Lifecycle management tool – Project description in Project Object Model (pom.xml) – Build automation (including unit testing phase) – Project modularization – Dependency management – Deployment automation – Reporting (code quality check, documentation generation, website generation, test coverage, etc.) • Convention over configuration (… like Java EE) “Systems, libraries, and frameworks should assume reasonable defaults. Without requiring unnecessary configuration, systems should “just work”.” [Maven by example] 21/10/2016 xavier.devroey@unamur.be 32
  33. 33. Installation • See https://maven.apache.org/install.html • Use command mvn –v to check installation • Local Maven files are in ~/.m2 • ~/.m2/settings.xml contains access configurations • ~/.m2/repository contains the local Maven repository 21/10/2016 xavier.devroey@unamur.be 33 $ mvn -version Apache Maven 3.1.1 Maven home: /usr/local/apache-maven-3.1.1 Java version: 1.8.0_31, vendor: Oracle Corporation Java home: /Library/Java/jdk1.8.0_31.jdk/Contents/Home/jre Default locale: fr_FR, platform encoding: UTF-8 OS name: "mac os x”
  34. 34. First Maven project (using NetBeans) 21/10/2016 xavier.devroey@unamur.be 34
  35. 35. First Maven project (using NetBeans) 21/10/2016 xavier.devroey@unamur.be 35
  36. 36. Maven project structure • myfirstmavenproject/ – pom.xml – src/ • main/ – java/ » Here goes your Java source code – resources/ » Here goes other resources (files, etc.) included in the .jar/.war • test/ – java/ » Here goes the JUnit tests – resources/ » Here goes other resources used by the tests – sub-module-1/ • pom.xml • src/ – sub-module-2/ • … 21/10/2016 xavier.devroey@unamur.be 36
  37. 37. Pom.xml 21/10/2016 xavier.devroey@unamur.be 37
  38. 38. Pom.xml 21/10/2016 xavier.devroey@unamur.be 38 Identifier • Maven identifiers have to be unique • Format: groupId:artifactId:version • GroupId should identify the organisation using Java package naming convention • ArtifactId should identify the project • Version should be encoded on 3 number XX.XX.XX • “-SNAPSHOT” is used to indicate the currently under development version
  39. 39. Pom.xml 21/10/2016 xavier.devroey@unamur.be 39 Type of the artefact • Packaging indicates to Maven the type of artifact produced during build • “jar” indicates a .jar file (executable or not) • “pom” indicates a maven project without compiled code (used by a parent project) • “war” indicates a .war file that will be deployed on a web server
  40. 40. Pom.xml 21/10/2016 xavier.devroey@unamur.be 40 Properties used by project and sub-projects • Properties are used as global constants • Implicit properties • ${basedir} current project root directory • ${project.version} current project version • ${project.groupId} current project groupId • Declared properties • Should contain dependencies versions
  41. 41. Sub-modules • Parent project (should have pom packaging) • Sub-module has to declare parent project • In this case: sub-module-1 version is inherited 21/10/2016 xavier.devroey@unamur.be 41 <project …> … <packaging>pom</packaging> <modules> <module>sub-module-1</module> </modules> </project> <project …> <parent> <groupId>be.unamur.info</groupId> <artifactId>myfirstmavenproject</artifactId> <version>0.0.1-SNAPSHOT</version> </parent> <artifactId>sub-module-1</artifactId> … </project>
  42. 42. Dependencies • Maven central repository – http://repo.maven.apache.org/maven2 – http://search.maven.org (to search an artifact) • Declare dependency in a Maven project 21/10/2016 xavier.devroey@unamur.be 42 <project …> … <dependencies> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>${junit.version}</version> <scope>test</scope> </dependency> </dependencies> </project>
  43. 43. mvn <phase> • compile – Compile source files • test – Launch JUnit tests in src/test/java/ • package – Create .jar or .war or … file in target/ • install – Install artefact in local Maven repository • deploy – Deploy artefact using deployment configuration 21/10/2016 xavier.devroey@unamur.be 43
  44. 44. Crash course
  45. 45. JUnit: introduction • Collection of classes to perform unit testing • Uses annotations (e.g., @Test) 21/10/2016 xavier.devroey@unamur.be 45 import static org.junit.Assert.assertEquals; import org.junit.Test; public class CalculatorTest { @Test public void evaluatesExpression() { Calculator calculator = new Calculator(); int sum = calculator.evaluate("1+2+3"); assertEquals(6, sum); } } java -cp .;junit-4.12.jar;hamcrest-core-1.3.jar org.junit.runner.JUnitCore CalculatorTest
  46. 46. Anatomy of a Junit test class import … public class MyClassTest{ private static final Logger logger = LoggerFactory.getLogger(JHConwayLifeCellTest.class); @Rule public TestRule watcher = new TestWatcher() { @Override protected void starting(Description description) { logger.info(String.format("Starting test: %s()...”, description.getMethodName()));};}; @BeforeClass public static void setUpClass() {logger.info("Setting up before class");} @AfterClass public static void tearDownClass() {logger.info("Tearing down after class");} @Before public void setUp() {logger.info("Setting up before test");} @After public void tearDown() {logger.info("Tearing down after test");} @Test public void testMyMethod() {assertTrue(true);} } 21/10/2016 xavier.devroey@unamur.be 46
  47. 47. Assertions • JUnit – asserTrue(c); assertFalse(c); assertEquals(expected, actual); assertNull(o); assertNotNull(o); fail() – assertThat(T actual, org.hamcrest.Matcher<T> matcher) • Hamcrest matchers (org.hamcrest.Matchers) – assertThat(T actual, equalTo(T operand)) – assertThat(T actual, not(T operand)) – assertThat(Iterable<? super T> c, hasItem(T item)) – assertThat(T actual, equalTo(T operand)) – assertThat(T actual, nullValue()) – assertThat(T actual, notNullValue()) 21/10/2016 xavier.devroey@unamur.be 47
  48. 48. Hamcrest common matchers • Core – anything - always matches, useful if you don't care what the object under test is • Logical – allOf - matches if all matchers match, short circuits (like Java &&) – anyOf - matches if any matchers match, short circuits (like Java ||) – not - matches if the wrapped matcher doesn't match and vice versa • Object – equalTo - test object equality using Object.equals – hasToString - test Object.toString – instanceOf, isCompatibleType - test type – notNullValue, nullValue - test for null – sameInstance - test object identity • Collections – hasEntry, hasKey, hasValue - test a map contains an entry, key or value – hasItem, hasItems - test a collection contains elements – hasItemInArray - test an array contains an element • Number – closeTo - test floating point values are close to a given value – greaterThan, greaterThanOrEqualTo, lessThan, lessThanOrEqualTo - test ordering • Text – equalToIgnoringCase - test string equality ignoring case – equalToIgnoringWhiteSpace - test string equality ignoring differences in runs of whitespace – containsString, endsWith, startsWith - test string matching 21/10/2016 xavier.devroey@unamur.be 48
  49. 49. Hands-on: Maven, JUnit and Hamcrest • be.unamur:myfirstwebapp • Class to test: be.unamur.myfirstwebapp.Message • Test coverage using Cobertura – mvn cobertura:cobertura – Report in target/site/cobertura/index.html 21/10/2016 xavier.devroey@unamur.be 49
  50. 50. Hands-on: test a data access object • Using a test database – Derby (https://db.apache.org/derby/) – H2 (http://www.h2database.com/) – DB server test instance (MySQL, Oracle, etc.) • Code designed to – Externalize DB access information (e.g., using a property file) – Manage DB connections (using a DataSource object) • Connection pooling using commonds-dbcp (https://commons.apache.org/proper/commons-dbcp/) • In this case SQL execution using commonds-dbutils (https://commons.apache.org/proper/commons-dbutils/) • Exercice: Add test to cover “if” branch in MessageDAO.update 21/10/2016 xavier.devroey@unamur.be 50
  51. 51. Hands-on: test a servlet • Using mock objects to emulate servlet environment • Code designed to – Strictly separate model (Beans + DAOs), view (JSPs), and controller (Servlets) • Mockito: tasty mocking framework for unit tests in Java – http://site.mockito.org • Create the mock – HttpServletRequest request = mock(HttpServletRequest.class); • Stub method calls – when(request.getParameter("author")).thenReturn("Albert"); • Check methods were called with given arguments – verify(context).getRequestDispatcher("/WEB-INF/messages.jsp"); 21/10/2016 xavier.devroey@unamur.be 51
  52. 52. CODE REVIEW A static analysis detour
  53. 53. Code review • Over-the-shoulder reviewing – sometimes called stepmother reviewing • Pair programming 21/10/2016 xavier.devroey@unamur.be 53
  54. 54. Code review • Over-the-shoulder reviewing – sometimes called stepmother reviewing • Pair programming • Pull request 21/10/2016 xavier.devroey@unamur.be 54 Dev. Team Project master Creates the feature in a dedicated branch in local repository Pushes the branch to central rep. and fills a pull request Reviews the code, discusses it, and alters it Merges the feature into the official rep. and closes the pull request
  55. 55. Code review • Over-the-shoulder reviewing – sometimes called stepmother reviewing • Pair programming • Pull request • Rubber duck debugging 21/10/2016 xavier.devroey@unamur.be 55
  56. 56. Code analysis: FindBugs • http://findbugs.sourceforge.net • Called after Maven test phase – mvn test findbugs:findbugs findbugs:gui 21/10/2016 xavier.devroey@unamur.be 56
  57. 57. UI web-app testing made easy
  58. 58. Selenium • User Interface (UI) testing framework for web application – Integration and acceptance – black-box – (mainly used for) regression testing 21/10/2016 xavier.devroey@unamur.be 58 Testing Environment Selenium WebDriver Selenium IDE 1. Design and/or execute test-cases 2. Export
  59. 59. Installation • See http://www.seleniumhq.org/docs/02_selenium_ide.jsp 21/10/2016 xavier.devroey@unamur.be 59
  60. 60. Selenium Commands • open – opens a page using a URL • click/clickAndWait – performs a click operation, and optionally waits for a new page to load • verifyTitle/assertTitle – verifies an expected page title • verifyTextPresent – verifies expected text is somewhere on the page • verifyElementPresent – verifies an expected UI element, as defined by its HTML tag, is present on the page • verifyText – verifies expected text and its corresponding HTML tag are present on the page • verifyTable – verifies a table’s expected contents • waitForPageToLoad – pauses execution until an expected new page loads. Called automatically when clickAndWait is used • waitForElementPresent – pauses execution until an expected UI element, as defined by its HTML tag, is present on the page 21/10/2016 xavier.devroey@unamur.be 60
  61. 61. EVALUATION OF THE TESTS PERFORMED Are my test cases suitably selected?
  62. 62. Testing process 21/10/2016 xavier.devroey@unamur.be 62 Req. Test engineer Test case defines Pgm. executed on Good enough?
  63. 63. Coverage • Requirements coverage • Input domain coverage – Equivalence partitioning – Combinatorial testing (e.g., pairwise testing) • Code coverage – Statement/block coverage – Decision/branch coverage – condition coverage – modified condition/decision coverage (MC/DC) – Path coverage – E.g., Cobertura, JaCoCo, Clover • mvn clean clover:setup test clover:aggregate clover:clover 21/10/2016 xavier.devroey@unamur.be 63
  64. 64. Mutation testing • Mimic typical errors by introducing small modifications in the program • Based on 2 hypothesis: – Competent programmer hypothesis – Coupling effect • Mutation operators introduce small syntactic errors 21/10/2016 xavier.devroey@unamur.be 64 Original program: if (a && x < 5) { foo(); } else { fee(); } Mutant 1: if (a || x < 5) { foo(); } else { fee(); } Mutant 2: if (a && x < 5) { fee(); } else { fee(); }
  65. 65. Mutation Testing 21/10/2016 xavier.devroey@unamur.be 65 SUT Mut. Op. Mut. Op.
  66. 66. Mutation Testing 21/10/2016 xavier.devroey@unamur.be 66 M1 M2 M3 M4
  67. 67. Mutation Testing 21/10/2016 xavier.devroey@unamur.be 67 M1 M2 M3 M4 Test case
  68. 68. Mutation Testing 21/10/2016 xavier.devroey@unamur.be 68 M1 M2 M3 M4 Test case Count( ) Mutation score = Total Nb. Mut.
  69. 69. Hands-on: test cases assessment • http://pitest.org • Called after Maven test phase – mvn org.pitest:pitest-maven:mutationCoverage 21/10/2016 xavier.devroey@unamur.be 69
  70. 70. FROM TEST SUITE EVALUATION … to test suite augmentation
  71. 71. Symbolic execution • Execute the program with symbolic values – Constraints on the variables to follow defined path • http://klee.github.io (C) • Pex (C#); CodeHunt (C# and Java) 21/10/2016 xavier.devroey@unamur.be 71 int foo(int x){ if(x < 0){ return -1/x; } return 1/x; }
  72. 72. Symbolic execution • Execute the program with symbolic values – Constraints on the variables to follow defined path • http://klee.github.io (C) • Pex (C#); CodeHunt (C# and Java) 21/10/2016 xavier.devroey@unamur.be 72 {x : int} {x : int | x < 0} {x : int | x < 0} int foo(int x){ if(x < 0){ return -1/x; } return 1/x; }
  73. 73. Symbolic execution • Execute the program with symbolic values – Constraints on the variables to follow defined path • http://klee.github.io (C) • Pex (C#); CodeHunt (C# and Java) 21/10/2016 xavier.devroey@unamur.be 73 {x : int} {x : int | x > 0} v {x : int | x = 0} int foo(int x){ if(x < 0){ return -1/x; } return 1/x; }
  74. 74. Search-based software testing • Use metaheuristics (e.g., evolutionary algorithms) to – generate test cases (or test data); – prioritize test cases; – minimize test suites; etc. • For instance, we want to generate input data for a given function, we need: – A search algorithm: hill climbing or genetic algorithm – A representation of the program and input values – Search operators: find the neighbours of the input values – A fitness function: decision coverage • This requires instrumentation of the program (to measure coverage when executing tests) 21/10/2016 xavier.devroey@unamur.be 74
  75. 75. Hands-on: test cases generation • http://www.evosuite.org/ • Generates tests in .evosuite/ folder • mvn compile evosuite:generate • mvn evosuite:export -DtargetFolder=target/generated- test-sources/evosuite • mvn test 21/10/2016 xavier.devroey@unamur.be 75
  76. 76. REFERENCES
  77. 77. • Mathur, A. P. (2008). Foundations of software testing. Pearson Education. • Utting, M., & Legeard, B. (2007). Practical Model-Based Testing: A Tools Approach. Morgan Kaufmann. • Bourque, P., & Fairley, R. E. (2014). Guide to the Software Engineering Body of Knowledge (SWEBOK(R)): Version 3.0. IEEE Computer Society. • Dowling, P., & McGrath, K. (2015). Using free and open source tools to manage software quality. Communications of the ACM, 58(7), 51–55. doi:10.1145/2755503 • Tim O'Brien et al. Maven by Example. Sonatype, http://www.sonatype.com/books/mvnex-book/reference/public-book.html • http://junit.org • http://hamcrest.org • http://site.mockito.org • http://www.seleniumhq.org • http://pitest.org • http://klee.github.io • https://www.microsoft.com/en-us/research/project/pex-and-moles-isolation-and- white-box-unit-testing-for-net/ • http://www.evosuite.org/ • https://openclassrooms.com/courses/creez-votre-application-web-avec-java-ee 21/10/2016 xavier.devroey@unamur.be 77
  78. 78. … last tricks • Google guava – https://github.com/google/guava/ • Apache commons – https://commons.apache.org • Simple Logging Facade for Java (SLF4J) – http://www.slf4j.org • Jetty – http://www.eclipse.org/jetty/ – mvn jetty:run 21/10/2016 xavier.devroey@unamur.be 78

×