Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Software testing: an introduction - 2015

17 views

Published on

Invited talk to the INFOM110 Software engineering course, University of Namur

Published in: Software
  • Be the first to comment

  • Be the first to like this

Software testing: an introduction - 2015

  1. 1.                                           www.unamur.be Software Testing: an introduction Xavier  Devroey  <xavier.devroey@unamur.be>     INFO  M110  Ingénierie  du  logiciel     Namur,  Belgique  
  2. 2. Plan •  Introduc:on   –  Error,  fault,  failure,  and  incident   –  So>ware  tes:ng:  defini:on   •  So>ware  tes:ng  typology   –  Source  of  test  genera:on   –  Characteris:c  being  tested   –  Lifecycle  phase   •  Agile  process  implementa:on:  the  TSSG  case  study   •  Hands-­‐on   –  Maven   –  JUnit   –  Selenium   2015  -­‐  2016   xavier.devroey@unamur.be   2  
  3. 3. INTRODUCTION
  4. 4. Error, fault, failure, and incident Error   Fault   Exec.   Failure  (bug)   Reports   Incident   Bug  Tracker   2015  -­‐  2016   xavier.devroey@unamur.be   4   Dev.   User  
  5. 5. Software Testing: definition [SWEBOK v3]     “So>ware  tes:ng  consists  of  the  dynamic  verifica:on     that  a  program  provides  expected  behaviours     on  a  finite  set  of  test  cases,  suitably  selected  from  the   usually  infinite  execu:on  domain.”   2015  -­‐  2016   xavier.devroey@unamur.be   5  
  6. 6. Testing 2015  -­‐  2016   xavier.devroey@unamur.be   6   Req.   Test  engineer   Test   case   defines   Pgm.   executed  on  
  7. 7. Dynamic verification •  Tes:ng  implies  execu6ng  the  program  under  test   •  Contrary  to  sta6c  analysis   –  Formal,  semiformal,  or  informal   –  Examples:     •  compiler  warnings  (code  analysis),     •  javadoc  warnings  (documenta:on  analysis),     •  design  inspec:on  (model  analysis),     •  theorem  proving  (e.g.,  Event-­‐B),     •  code  analysis   –  Code  review   –  IDE  warnings   –  FindBugs  h`p://findbugs.sourceforge.net   2015  -­‐  2016   xavier.devroey@unamur.be   7  
  8. 8. Expected behaviours •  It  must  be  possible  to  decide  whether  the  observed   outcomes  for  a  given  input  is  acceptable  or  not   •  The  oracle  problem   –  “An  oracle  is  any  human  or  mechanical  agent  that  decides   whether  a  program  behaved  correctly  in  a  given  test  and   accordingly  results  in  a  verdict  of  “pass”  or  “fail”.”  [SWEBOK  v3]   2015  -­‐  2016   xavier.devroey@unamur.be   8  
  9. 9. Finite set of test cases •  Exhaus:ve  tes:ng  is  not  possible  in  prac:ce   –  Infinite  number  of  test  cases   –  Select  a  subset  of  all  possible  tests   –  Trade-­‐off  between  limited  resources,  schedules  and  unlimited   test  requirements   •  Priori:za:on  based  on  risk,  usage,  cost,  …     “Program  tes6ng  can  be  used  to  show  the  presence  of  bugs,   but  never  to  show  their  absence!”  Dijkstra  (1970)   2015  -­‐  2016   xavier.devroey@unamur.be   9  
  10. 10. Suitably selected •  How  to  select  test-­‐cases  ?   •  Selec:on  criteria   –  Input/output  domains  coverage  (black  box  tes:ng)   •  Domain  par::oning  techniques   •  Based  on  specifica:on  documents   •  No  access  to  the  source  code   –  Code  coverage:  statement  coverage,  decision  coverage,  path   coverage,  etc.  (white  box  tes:ng)   •  Access  to  source  code   –  Test  engineer  exper:se   2015  -­‐  2016   xavier.devroey@unamur.be   10  
  11. 11. SOFTWARE TESTING TYPOLOGY
  12. 12. Types of testing •  Source  of  test  genera:on   –  Black  box  tes:ng   •  Based  on  specifica:on  documents   •  No  access  to  the  source  code   –  White  box  tes:ng   •  Access  to  the  source  code   –  Model-­‐based  tes:ng   •  Characteris:c  being  tested   –  Func:onal  tes:ng:  is  the  output  correct  for  a  given  input  ?   –  Non-­‐func:onal  tes:ng   •  Performance,  robustness,  security,  usability,  etc.   2015  -­‐  2016   xavier.devroey@unamur.be   12  
  13. 13. Types of testing •  Lifecycle  phase   –  Coding   •  Unit  tes:ng     •  Component  tes:ng   –  Integra:on   •  Integra:on  tes:ng   –  System  integra:on   •  System  tes:ng   –  Release   •  Acceptance/Beta  tes:ng   –  Maintenance     •  Regression  tes:ng     2015  -­‐  2016   xavier.devroey@unamur.be   13  
  14. 14. Testing in the waterfall model 2015  -­‐  2016   xavier.devroey@unamur.be   14   Requirements specification Design Coding and unit testing Integration testing System testing Acceptance testing Release Maintenance and regression testing
  15. 15. Testing in the V-model 2015  -­‐  2016   xavier.devroey@unamur.be   15   Requirements specification Design Coding and unit testing Integration/ System testing Acceptance testing Release Integration/ System test plan Acceptance test plan Maintenance and regression testing
  16. 16. Testing in an Agile process 1.  Include  tes:ng  ac:vi:es  from  the  beginning   2.  Specify  requirements  in  terms  of  tests   3.  Testers  and  developers  are  not  enemies     4.  Test  o>en  and  in  small  chunks   2015  -­‐  2016   xavier.devroey@unamur.be   16  
  17. 17. Test Driven Development (TDD) •  Red   –  Write  automated  tests   (examples)  from  user  stories   –  Tests  fail  (as  the  func:onality   is  not  implemented  yet)   •  Green   –  Developers  design  and  write   code  to  make  the  tests  pass   –  Tests  are  run  frequently   •  Refactor   –  Refactor  code  without   changing  interfaces  or   behaviour  (tests  must  pass)   Red   Green  Refactor   2015  -­‐  2016   xavier.devroey@unamur.be   17  
  18. 18. AGILE PROCESS IMPLEMENTATION The  Telecommunica:on  So>ware  &  Systems  Group   (TSSG)  case  study  
  19. 19. Agile process implementation: TSSG •  From  Dowling,  P.,  &  McGrath,  K.  (2015).  Using  free  and   open  source  tools  to  manage  soGware  quality.   Communica:ons  of  the  ACM,  58(7),  51–55.  doi: 10.1145/2755503   •  Open  Source  tools     •  Support  team’s  ac:vi:es  management     2015  -­‐  2016   xavier.devroey@unamur.be   19  
  20. 20. 2015  -­‐  2016   xavier.devroey@unamur.be   20   Development environment Project Management Requirements Management Issues Tracking Building Code Repository Unit testing Code coverage Cobertura Continuous Integration Test environment Functional Testing Test Case Management Load Testing Security Testing
  21. 21. 2015  -­‐  2016   xavier.devroey@unamur.be   21   Development environment Project Management Requirements Management Issues Tracking Building Code Repository Unit testing Code coverage Cobertura Continuous Integration Test environment Functional Testing Test Case Management Load Testing Security Testing
  22. 22. 2015  -­‐  2016   xavier.devroey@unamur.be   22   Development environment Project Management Requirements Management Issues Tracking Building Code Repository Unit testing Code coverage Cobertura Continuous Integration Test environment Functional Testing Test Case Management Load Testing Security Testing
  23. 23. Workflow •  Developer  forks  the  code   base   •  Implements  changes   •  Integrates  possible   changes  from  code  repo.   •  Runs  a  complete  build   –  Including  Unit  tests   •  Submits  changes  to  code   repo.   •  CI  server  performs   integra:on  tests   2015  -­‐  2016   xavier.devroey@unamur.be   23   h`ps://www.flickr.com/photos/b3c>/6713516627/   Do  NOT  break  the  build!  
  24. 24. 2015  -­‐  2016   xavier.devroey@unamur.be   24   Development environment Project Management Requirements Management Issues Tracking Building Code Repository Unit testing Code coverage Cobertura Continuous Integration Test environment Functional Testing Test Case Management Load Testing Security Testing
  25. 25. 2015  -­‐  2016   xavier.devroey@unamur.be   25   Development environment Project Management Requirements Management Issues Tracking Building Code Repository Unit testing Code coverage Cobertura Continuous Integration Test environment Functional Testing Test Case Management Load Testing Security Testing  Hudson  
  26. 26. 2015  -­‐  2016   xavier.devroey@unamur.be   26   Development environment Project Management Requirements Management Issues Tracking Building Code Repository Unit testing Code coverage Cobertura Continuous Integration Test environment Functional Testing Test Case Management Load Testing Security Testing Production Deployment
  27. 27. Crash  Course  
  28. 28. Maven 3: Introduction •  Project  Lifecycle  management  tool     –  Project  descrip:on  in  Project  Object  Model  (pom.xml)   –  Build  automa:on  (including  unit  tes9ng  phase)   –  Project  modulariza:on   –  Dependency  management   –  Deployment  automa:on   –  Repor:ng  (code  quality  check,  documenta:on  genera:on,   website  genera:on,  test  coverage,  etc.)   •  Conven:on  over  configura:on  (…  like  Java  EE)   “Systems,  libraries,  and  frameworks  should  assume  reasonable   defaults.  Without  requiring  unnecessary  configura6on,  systems   should  “just  work”.”  [Maven  by  example]   2015  -­‐  2016   xavier.devroey@unamur.be   28  
  29. 29. Installation •  See  h`ps://maven.apache.org/install.html   •  Use  command  mvn  –v    to  check  installa:on   •  Local  Maven  files  are  in  ~/.m2   •  ~/.m2/settings.xml  contains  access  configura:ons   •  ~/.m2/repository  contains  the  local  Maven   repository   2015  -­‐  2016   xavier.devroey@unamur.be   29   $  mvn  -­‐version   Apache  Maven  3.1.1     Maven  home:  /usr/local/apache-­‐maven-­‐3.1.1   Java  version:  1.8.0_31,  vendor:  Oracle  Corporation   Java  home:  /Library/Java/jdk1.8.0_31.jdk/Contents/Home/jre   Default  locale:  fr_FR,  platform  encoding:  UTF-­‐8   OS  name:  "mac  os  x”  
  30. 30. First Maven project (using NetBeans) 2015  -­‐  2016   xavier.devroey@unamur.be   30  
  31. 31. First Maven project (using NetBeans) 2015  -­‐  2016   xavier.devroey@unamur.be   31  
  32. 32. Maven project structure •  myfirstmavenproject/   –  pom.xml   –  src/   •  main/   –  java/       »  Here  goes  your  Java  source  code   –  resources/       »  Here  goes  other  resources  (files,  etc.)  included  in  the  .jar/.war   •  test/   –  java/       »  Here  goes  the  JUnit  tests   –  resources/       »  Here  goes  other  resources  used  by  the  tests   –  sub-­‐module-­‐1/   •  pom.xml   •  src/   –  sub-­‐module-­‐2/   •  …   2015  -­‐  2016   xavier.devroey@unamur.be   32  
  33. 33. Pom.xml 2015  -­‐  2016   xavier.devroey@unamur.be   33  
  34. 34. Pom.xml 2015  -­‐  2016   xavier.devroey@unamur.be   34   Iden9fier   •  Maven  iden:fiers  have  to  be  unique   •  Format:  groupId:ar6factId:version   •  GroupId  should  iden:fy  the  organisa:on  using  Java  package   naming  conven:on   •  Ar:factId  should  iden:fy  the  project   •  Version  should  be  encoded  on  3  number  XX.XX.XX   •  “-­‐SNAPSHOT”  is  used  to  indicate  the  currently  under   development  version  
  35. 35. Pom.xml 2015  -­‐  2016   xavier.devroey@unamur.be   35   Type  of  the  artefact   •  Packaging  indicates  to  Maven  the  type  of  ar:fact  produced  during   build   •  “jar”  indicates  a  .jar  file  (executable  or  not)   •  “pom”  indicates  a  maven  project  without  compiled  code  (used  by  a   parent  project)   •  “war”  indicates  a  .war  file  that  will  be  deployed  on  a  web  server  
  36. 36. Pom.xml 2015  -­‐  2016   xavier.devroey@unamur.be   36   Proper9es  used  by  project  and  sub-­‐projects   •  Proper:es  are  used  as  global  constants     •  Implicit  proper:es   •  ${basedir}  current  project  root  directory   •  ${project.version}  current  project  version   •  ${project.groupId}  current  project  groupId   •  Declared  proper:es   •  Should  contain  dependencies  versions  
  37. 37. Sub-modules •  Parent  project  (should  have  pom  packaging)   •  Sub-­‐module  has  to  declare  parent  project   •  In  this  case:  sub-­‐module-­‐1  version  is  inherited     2015  -­‐  2016   xavier.devroey@unamur.be   37   <project  …>        …        <packaging>pom</packaging>        <modules>                  <module>sub-­‐module-­‐1</module>        </modules>   </project>   <project  …>        <parent>                  <groupId>be.unamur.info</groupId>                  <artifactId>myfirstmavenproject</artifactId>                  <version>0.0.1-­‐SNAPSHOT</version>        </parent>        <artifactId>sub-­‐module-­‐1</artifactId>        …   </project>  
  38. 38. Dependencies •  Maven  central  repository       –  h`p://repo.maven.apache.org/maven2   –  h`p://search.maven.org  (to  search  an  ar:fact)     •  Declare  dependcy  in  a  Maven  project   2015  -­‐  2016   xavier.devroey@unamur.be   38   <project  …>        …        <dependencies>                  <dependency>                          <groupId>junit</groupId>                          <artifactId>junit</artifactId>                          <version>${junit.version}</version>                          <scope>test</scope>                  </dependency>    </dependencies>   </project>  
  39. 39. mvn <phase> •  compile   –  Compile  source  files   •  test   –  Launch  JUnit  tests  in     src/test/java/   •  package   –  Create  .jar  or  .war  or  …  file  in   target/   •  install   –  Install  artefact  in  local  Maven   repository   •  deploy   –  Deploy  artefact  using   deployment  configura:on   2015  -­‐  2016   xavier.devroey@unamur.be   39  
  40. 40. Crash  Course  
  41. 41. JUnit: Introduction •  Collec:on  of  classes  to  perform  unit  tes:ng   •  Uses  annota:ons  (e.g.,  @Test)     2015  -­‐  2016   xavier.devroey@unamur.be   41   import  static  org.junit.Assert.assertEquals;   import  org.junit.Test;     public  class  CalculatorTest  {      @Test      public  void  evaluatesExpression()  {          Calculator  calculator  =  new  Calculator();          int  sum  =  calculator.evaluate("1+2+3");          assertEquals(6,  sum);      }   }   java  -­‐cp  .;junit-­‐4.12.jar;hamcrest-­‐core-­‐1.3.jar  org.junit.runner.JUnitCore  CalculatorTest  
  42. 42. Anatomy of a Junit test class import    …   public  class  MyClassTest{          private  sta:c  final  Logger  logger  =  LoggerFactory.getLogger(JHConwayLifeCellTest.class);            @Rule          public  TestRule  watcher  =  new  TestWatcher()  {    @Override  protected  void  star:ng(Descrip:on  descrip:on)  {      logger.info(String.format("Star:ng  test:  %s()...”,  descrip:on.getMethodName()));};};            @BeforeClass          public  sta:c  void  setUpClass()  {logger.info("Se|ng  up  before  class");}            @A>erClass          public  sta:c  void  tearDownClass()  {logger.info("Tearing  down  a>er  class");}            @Before          public  void  setUp()  {logger.info("Se|ng  up  before  test");}            @A>er          public  void  tearDown()  {logger.info("Tearing  down  a>er  test");}            @Test          public  void  testMyMethod()  {assertTrue(true);}   }   2015  -­‐  2016   xavier.devroey@unamur.be   42  
  43. 43. Assertions •  JUnit   –  asserTrue(c);  assertFalse(c);  assertEquals(expected,  actual);   assertNull(o);  assertNotNull(o);  fail()   –  assertThat(T  actual,  org.hamcrest.Matcher<T>  matcher)   •  Hamcrest  matchers  (org.hamcrest.Matchers)   –  assertThat(T  actual,  equalTo(T  operand))   –  assertThat(T  actual,  not(T  operand))   –  assertThat(Iterable<?  super  T>  c,  hasItem(T  item))   –  assertThat(T  actual,  equalTo(T  operand))   –  assertThat(T  actual,  nullValue())   –  assertThat(T  actual,  notNullValue())   2015  -­‐  2016   xavier.devroey@unamur.be   43  
  44. 44. Hamcrest common matchers •  Core   –  anything  -­‐  always  matches,  useful  if  you  don't  care  what  the  object  under  test  is   •  Logical   –  allOf  -­‐  matches  if  all  matchers  match,  short  circuits  (like  Java  &&)   –  anyOf  -­‐  matches  if  any  matchers  match,  short  circuits  (like  Java  ||)   –  not  -­‐  matches  if  the  wrapped  matcher  doesn't  match  and  vice  versa   •  Object   –  equalTo  -­‐  test  object  equality  using  Object.equals   –  hasToString  -­‐  test  Object.toString   –  instanceOf,  isCompa:bleType  -­‐  test  type   –  notNullValue,  nullValue  -­‐  test  for  null   –  sameInstance  -­‐  test  object  iden:ty   •  Collec:ons   –  hasEntry,  hasKey,  hasValue  -­‐  test  a  map  contains  an  entry,  key  or  value   –  hasItem,  hasItems  -­‐  test  a  collec:on  contains  elements   –  hasItemInArray  -­‐  test  an  array  contains  an  element   •  Number   –  closeTo  -­‐  test  floa:ng  point  values  are  close  to  a  given  value   –  greaterThan,  greaterThanOrEqualTo,  lessThan,  lessThanOrEqualTo  -­‐  test  ordering   •  Text   –  equalToIgnoringCase  -­‐  test  string  equality  ignoring  case   –  equalToIgnoringWhiteSpace  -­‐  test  string  equality  ignoring  differences  in  runs  of  whitespace   –  containsString,  endsWith,  startsWith  -­‐  test  string  matching   2015  -­‐  2016   xavier.devroey@unamur.be   44  
  45. 45. Hands-on: Maven, JUnit and Hamcrest •  be.unamur.info:mavencrashcourse-­‐cellularautomata   –  From  “Travaux  pra:ques  de  concep:on  et  programma:on  orientée  objet  :  Automate  cellulaire”  G.  Saval,  2008   •  Classes  to  test   –  JHConwayLifeCell   –  Grid   –  CellGrid   •  Test  coverage  using  Cobertura   –  mvn  cobertura:cobertura   •  Code  analysis  using  FindBugs   –  mvn  findbugs:findbugs   –  mvn  findbugs:gui   2015  -­‐  2016   xavier.devroey@unamur.be   45  
  46. 46. UI  web-­‐app  tes:ng  made  easy  
  47. 47. Selenium •  User  Interface  (UI)  tes:ng  framework  for  web  applica:on   –  Integra9on  and  acceptance     –  black-­‐box   –  (mainly  used  for)  regression  tes:ng     2015  -­‐  2016   xavier.devroey@unamur.be   47   Testing Environment Selenium WebDriver Selenium IDE 1. Design and/or execute test-cases 2. Export
  48. 48. Installation •  See  h`p://www.seleniumhq.org/docs/02_selenium_ide.jsp   2015  -­‐  2016   xavier.devroey@unamur.be   48  
  49. 49. Selenium Commands •  open   –  opens  a  page  using  a  URL   •  click/clickAndWait   –  performs  a  click  opera:on,  and  op:onally  waits  for  a  new  page  to  load   •  verifyTitle/assertTitle   –  verifies  an  expected  page  :tle   •  verifyTextPresent   –  verifies  expected  text  is  somewhere  on  the  page   •  verifyElementPresent   –  verifies  an  expected  UI  element,  as  defined  by  its  HTML  tag,  is  present  on  the  page   •  verifyText   –  verifies  expected  text  and  its  corresponding  HTML  tag  are  present  on  the  page   •  verifyTable   –  verifies  a  table’s  expected  contents   •  waitForPageToLoad   –  pauses  execu:on  un:l  an  expected  new  page  loads.  Called  automa:cally  when  clickAndWait  is  used   •  waitForElementPresent   –  pauses  execu:on  un:l  an  expected  UI  element,  as  defined  by  its  HTML  tag,  is  present  on  the  page   2015  -­‐  2016   xavier.devroey@unamur.be   49  
  50. 50. Hands-on: Selenium IDE •  Let’s  Chat  applica:on     –  h`ps://sdelements.github.io/lets-­‐chat/   •  Test  server   –  h`p://rollier.info.fundp.ac.be:5000   •  Test  Cases   –  Create  chat  room,  post  some  message,  exit  room   –  Edit  profile,  change  display  name,  first  name  and  last  name   –  Create/Enter  room,  edit  room  name  and  descrip:on   2015  -­‐  2016   xavier.devroey@unamur.be   50  
  51. 51. References •  Mathur,  A.  P.  (2008).  Founda6ons  of  soGware  tes6ng.  Pearson  Educa:on.   •  U|ng,  M.,  &  Legeard,  B.  (2007).  Prac6cal  Model-­‐Based  Tes6ng:  A  Tools   Approach.  Morgan  Kaufmann.   •  Bourque,  P.,  &  Fairley,  R.  E.  (2014).  Guide  to  the  SoGware  Engineering  Body  of   Knowledge  (SWEBOK(R)):  Version  3.0.  IEEE  Computer  Society.   •  Dowling,  P.,  &  McGrath,  K.  (2015).  Using  free  and  open  source  tools  to  manage   soGware  quality.  Communica:ons  of  the  ACM,  58(7),  51–55.  doi: 10.1145/2755503   •  Tim  O'Brien  et  al.  Maven  by  Example.  Sonatype,   h`p://www.sonatype.com/books/mvnex-­‐book/reference/public-­‐book.html   •  h`p://junit.org   •  h`p://hamcrest.org   •  h`p://www.seleniumhq.org   2015  -­‐  2016   xavier.devroey@unamur.be   51   Images  credits:  h`p://hedbonstudios.deviantart.com;  h`p://commons.wikimedia.org/;  h`p://wh1pla5h.deviantart.com;  h`p://wh1pla5h.deviantart.com;  h`p:// darkangelaw1986.deviantart.com;  h`p://phantomcameron.deviantart.com;  h`p://newyorkkid618.deviantart.com;  h`p://doloresmine`e.deviantart.com;  h`ps:// www.flickr.com/photos/tbuser  ;  h`p://fullmetaldevil.deviantart.com/  ;  h`ps://www.flickr.com/photos/fastjack/  ;  h`ps://www.flickr.com/photos/eli:stczar/    
  52. 52. … last tricks •  Google  guava   –  h`ps://github.com/google/guava/   •  Apache  commons   –  h`ps://commons.apache.org   –  Database  connec:on  pooling  services     •  h`ps://commons.apache.org/proper/commons-­‐dbcp/   –  JDBC  helper   •  h`ps://commons.apache.org/proper/commons-­‐dbu:ls/   •  Simple  Logging  Facade  for  Java  (SLF4J)     –  h`p://www.slf4j.org   2015  -­‐  2016   xavier.devroey@unamur.be   52  

×