• Like
Experiences in Software Testing (lecture slides)
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Experiences in Software Testing (lecture slides)

  • 263 views
Published

Online lecture at the School of Computer Science, University of Hertfordshire, Hatfield, UK, as part of the 10th Europe Week from 3rd to 7th March 2014.

Online lecture at the School of Computer Science, University of Hertfordshire, Hatfield, UK, as part of the 10th Europe Week from 3rd to 7th March 2014.

Published in Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
263
On SlideShare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
8
Comments
0
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield Experiences in Software Testing Prof. Dr. Dagmar Monett Díaz Computer Science Dept. Faculty of Cooperative Studies Berlin School of Economics and Law dagmar@monettdiaz.com Europe Week, 3rd – 7th March 2014
  • 2. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 2 The waterfall Software Development Life-Cycle in theory and practice
  • 3. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 3 ProgrammingGeeks at Google+
  • 4. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 4 Agenda
  • 5. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 5 Agenda  Where does the major content come from?  Why is Software Testing of utter importance?  What is Software Testing?  Testing principles and methods  General testing guidelines  What to do with software errors that cause failures?  Test-Driven Development  Current trends in Software Testing  Further reading, sources of inspiration, and more…
  • 6. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 6 ©
  • 7. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield Software Engineering Ian Sommerville 9th Edition, 792 pp. Addison-Wesley, March 2011 ISBN-13: 978-0-137-03515-1 What I also use in my lectures at the HWR… 7
  • 8. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield Softwaretechnik. Mit Fallbeispielen aus realen Entwicklungsprojekten (Software Engineering. With case studies from real-world development projects) Thomas Grechenig, Mario Bernhart, Roland Breiteneder, Karin Kappel In German, 688 pp. Pearson Studium, 2010 ISBN-13: 978-3-86894-007-7 What I also use in my lectures at the HWR… 8
  • 9. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield Further reading  I. Sommerville (2011): “Chapter 8: Software Testing”. With material for instructors available at http://www.softwareengineering-9.com/  L. Crispin, T. House (2012): “Testing extreme programming”  R. Patton (2005): “Software Testing”  ISTQB Exam Certification website available at http://istqbexamcertification.com/ 9
  • 10. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 10 Why is Software Testing of utter importance? Image © renjith krishnan at http://www.freedigitalphotos.net/
  • 11. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield When I searched the Web on January 5th 2013… 11
  • 12. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield When I searched the Web on January 5th 2013… 12 I tried again about 30 seconds later and could continue searching the Web…
  • 13. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 13 But what about…
  • 14. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 14http://news.bbc.co.uk/2/hi/3772077.stm
  • 15. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 15 What is Software Testing?
  • 16. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield Definition according to IEEE 610.12, 1990 „Software testing is a formal process carried out by a specialised testing team in which a software unit, several integrated software units or an entire software package are examined by running the programs on a computer. All the associated tests are performed according to approved test procedures on approved test cases.“ 16 Interesting issues: - Software testing as formal process - Testing by a specialised team - Approved test procedures and test cases
  • 17. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 17 Testing principles
  • 18. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 18 Testing principles (ISTQB) ISTQB: International Software Testing Qualifications Board
  • 19. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 19 Testing principles (ISTQB) Principle 1: Testing shows presence of defects ISTQB: International Software Testing Qualifications Board
  • 20. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 20 Testing principles (ISTQB) Principle 1: Testing shows presence of defects Principle 2: Exhaustive testing is impossible ISTQB: International Software Testing Qualifications Board
  • 21. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 21 Testing principles (ISTQB) Principle 1: Testing shows presence of defects Principle 2: Exhaustive testing is impossible Principle 3: Early testing ISTQB: International Software Testing Qualifications Board
  • 22. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 22 Testing principles (ISTQB) Principle 1: Testing shows presence of defects Principle 2: Exhaustive testing is impossible Principle 3: Early testing Principle 4: Defect clustering ISTQB: International Software Testing Qualifications Board
  • 23. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 23 Testing principles (ISTQB) Principle 1: Testing shows presence of defects Principle 2: Exhaustive testing is impossible Principle 3: Early testing Principle 4: Defect clustering Principle 5: Pesticide paradox ISTQB: International Software Testing Qualifications Board
  • 24. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 24 Testing principles (ISTQB) Principle 1: Testing shows presence of defects Principle 2: Exhaustive testing is impossible Principle 3: Early testing Principle 4: Defect clustering Principle 5: Pesticide paradox Principle 6: Testing is context dependent ISTQB: International Software Testing Qualifications Board
  • 25. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 25 Testing principles (ISTQB) Principle 1: Testing shows presence of defects Principle 2: Exhaustive testing is impossible Principle 3: Early testing Principle 4: Defect clustering Principle 5: Pesticide paradox Principle 6: Testing is context dependent Principle 7: Absence-of-errors fallacy ISTQB: International Software Testing Qualifications Board
  • 26. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 26 Active learning exercise: “Mention some concrete examples of testing principles’ application!” Image © renjith krishnan at http://www.freedigitalphotos.net/
  • 27. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 27 Software testing methods
  • 28. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield - Functional testing (Black-box testing) - Structural testing (White-box testing) - Combinations of both (Grey-box testing) 28 Software testing methods Images © Master isolated images and Serge Bertasius Photography @ http://www.freedigitalphotos.net/
  • 29. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 29 Examples of Black-box testing
  • 30. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 30 Partition testing  Also: “Equivalence class partitioning”  Input data and output results often fall into different classes where all members of a class are related.
  • 31. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 31 Partition testing  Also: “Equivalence class partitioning”  Input data and output results often fall into different classes where all members of a class are related.  Each of these classes is an equivalence partition or domain where the program behaves in an equivalent way for each class member.
  • 32. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 32 Partition testing  Also: “Equivalence class partitioning”  Input data and output results often fall into different classes where all members of a class are related.  Each of these classes is an equivalence partition or domain where the program behaves in an equivalent way for each class member.  Test cases should be chosen from each partition.
  • 33. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 33  Example: In a system requirements documentation it is specified that only product prices starting from 0,01 € and up to 500 € must be entered by the user. It can be assumed that products with a price equal to 100 € will be properly validated and processed. Furthermore, it can be assumed that all prices between 0,01 € and 500 € are also correct. Thus, it is sufficient to test only one arbitrary value from each equivalence class in order to identify possible improper behaviour of the system. Partition testing
  • 34. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 34 It would be sufficient to consider three partitions: And then to check only one (arbitrary) representative of each equivalence class in a test case. Partition testing  Example (cont.): invalid partition 1 invalid partition 2valid partition x 0 0 < x 500 x > 500 0 500
  • 35. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 35 Boundary-value analysis  Special case of the partitioning testing  Boundary values are common locations for errors that result in software failures
  • 36. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 36 Boundary-value analysis  Special case of the partitioning testing  Boundary values are common locations for errors that result in software failures  Possible boundary values for the previous example: invalid partition 1 invalid partition 2valid partition 0 500 0 0,01 500 500,01
  • 37. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 37 Boundary-value analysis  Or even better…
  • 38. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 38 Boundary-value analysis  Or even better… invalid partition 1 valid partition 0 500 0 0,01 500 500,01 215 215 215+1 216 216 216+1 231 231 231+1 invalid partition 2 232 232 232+1 … …
  • 39. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 39 Examples of White-box testing
  • 40. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 40 Structure-based methods (i)  Function coverage  Statement coverage  Decision (or branch) coverage  Condition (or predicate) coverage  Multiple condition coverage  …and many other code coverage metrics that are used in structural software testing!
  • 41. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 41 Structure-based methods (ii) int rst(int r, int s, int t){ if (r > 0 || s > 0) { while (r != s) { while (r > s) { r = r - s; } r = s; } } else { r = t; } return r; }
  • 42. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 42 Structure-based methods (iii) int rst(int r, int s, int t){ //n0 if (r > 0 || s > 0) { //n1 while (r != s) { //n2 while (r > s) { //n3 r = r - s; //n4 } r = s; //n5 } } else { r = t; } //n6 return r; //n7 } //n8
  • 43. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 43 Structure-based methods (iv) n0 n1 n6 n2 n7 n8 n3 n4n5 Control flow graph int rst(int r, int s, int t){ //n0 if (r > 0 || s > 0) { //n1 while (r != s) { //n2 while (r > s) { //n3 r = r - s; //n4 } r = s; //n5 } } else { r = t; } //n6 return r; //n7 } //n8
  • 44. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 44 Structure-based methods (v) Test case: {r = 1, s = 2, t = 4} n0 n1 n6 n2 n7 n8 n3 n4n5 Control flow graph int rst(int r, int s, int t){ //n0 if (r > 0 || s > 0) { //n1 while (r != s) { //n2 while (r > s) { //n3 r = r - s; //n4 } r = s; //n5 } } else { r = t; } //n6 return r; //n7 } //n8
  • 45. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 45 Structure-based methods (vi) int rst(int r, int s, int t){ //n0 if (r > 0 || s > 0) { //n1 while (r != s) { //n2 while (r > s) { //n3 r = r - s; //n4 } r = s; //n5 } } else { r = t; } //n6 return r; //n7 } //n8 Test case: {r = 1, s = 2, t = 4} n0 n1 n6 n2 n7 n8 n3 n4n5 Control flow graph 78% Statement coverage 63% Decision coverage
  • 46. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 46 General testing guidelines
  • 47. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield General testing guidelines  Choose inputs that force the system to generate all error messages  Design inputs that cause input buffers to overflow  Repeat the same input or series of inputs numerous times  Force invalid outputs to be generated  Force computation results to be too large or too small. 47
  • 48. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield Test ideas and heuristics  “Test Heuristics Cheat Sheet: Data Type Attacks & Web Tests” © Quality Tree Software, Inc. (2006) Available at http://testobsessed.com/wp- content/uploads/2011/04/testheuristicscheatsheetv1.pdf  “How To Generate Test Ideas (A checklist). Everything you need to trigger thoughts, discussions and actions in the projects you are working on” © Smart Bear and Ministry of Testing (2013) Available at http://www.ministryoftesting.com/2013/12/checklist -how-to-generate-test-ideas/ 48
  • 49. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 49 Active learning exercise: “Which is then the main goal of software testing?” Image © renjith krishnan at http://www.freedigitalphotos.net/
  • 50. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield Press the Bug key! “For when you're having a bad day at work and you can't find any bugs” Andy Glover (@cartoontester) http://cartoontester.blogspot.de/2012/11/press-bug-key.html 50
  • 51. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 51 What to do with software errors that cause failures?
  • 52. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield
  • 53. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 53 Who should fix the failures? How does the tester report them?
  • 54. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 54 Managing software errors report Tester re-test rate assign correct deliver terminate Test lead Developer Change management
  • 55. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 55 How should tests be conducted? Which goals and conditions are important?
  • 56. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 56 Testing process (i)  What should be considered when conducting tests: (1) Criteria for the success or failure of a test run. (2) Criteria to start and finish a test run. (3) Criteria to interrupt or continue a test. (4) Documentation of results and communication channels. (5) Testing methods and depth. (6) Definition of the test environment. (cont.)
  • 57. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 57 Testing process (ii)  What should be considered when conducting tests: (7) Definition of the test basis (information and stakeholders needed). (8) Problems found and responsibilities for their elimination. (9) Objects to be tested. (10) Properties to be tested. (11) Properties that should not be tested. (12) Economic objectives of the test.
  • 58. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 58 Test-Driven Development (TDD)
  • 59. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield Test-driven development  Approach to program development in which you inter-leave testing and code development.  TDD was introduced as part of agile methods such as Extreme Programming.  Tests are written before code and ‘passing’ the tests is the critical driver of development. 59
  • 60. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield Test-driven development  You develop code incrementally, along with a test for that increment. You don’t move on to the next increment until the code that you have developed passes its test. Identify new functionality Write test Run test Implement Functionality and refactor fail pass 60
  • 61. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 61 All code is guilty… Image © http://www.printfection.com/codesmack/All-Code-Is-Guilty/_s_58243
  • 62. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 62 Current trends in Software Testing
  • 63. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield World Quality Report • >1500 Respondents: Chief financial officers (CFO), chief information officers (CIO), IT directors, and quality assurance (QA) directors worldwide (>25 countries) • Detailed telephone interviews undertaken during April-May 2012 and April-May 2013 4th Edition 2012-2013 5th Edition 2013-2014 Available at http://www.capgemini.com/insights-and-resources/ by-publication/world-quality-report-2012--2013/ Available at http://www.capgemini.com/thought-leadership/ world-quality-report-2013-14 63
  • 64. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield Change in testing budgets Figure 1, pp. 13 (2012), Gapgemini, Sogeti and HP 64 2012-13
  • 65. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield Greatest increase in spend 65 Figure 3, pp. 16 (2012), Gapgemini, Sogeti and HP 2012-13
  • 66. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield Increase in mobile testing Figure 10, pp. 25 (2013), Gapgemini, Sogeti and HP 66 2013-14
  • 67. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield Challenges to mobile testing Figure 7, pp. 25 (2012), Gapgemini, Sogeti and HP 67 2012-13
  • 68. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield Challenges to mobile testing Figure 11, pp. 27 (2013), Gapgemini, Sogeti and HP 68 2013-14
  • 69. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield Germany – Top trends pp. 64 (2012), Gapgemini, Sogeti and HP 69  „There is a significant increase in move towards development of a Testing Centre of Excellence (TCOE).“  „A cautious migration of applications towards the cloud has become evident.“  „More German organizations are testing mobile applications and devices than in any other country surveyed.“ 2012-13
  • 70. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield UK – Top trends pp. 72 (2012), Gapgemini, Sogeti and HP 70  „19% of UK organizations currently test mobile applications or devices.“  „More than 83% of organizations intend to use Testing as a Service (TaaS) offered by a third party.“  „Mobile application and device testing is a growing challenge.“ 2012-13
  • 71. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 71 Homework: “Search for testing tools and use them in your current course projects!” Image © renjith krishnan at http://www.freedigitalphotos.net/
  • 72. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 72 To take away…
  • 73. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 73 Fehlermanagement (i)
  • 74. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 74 Assessment Image © renjith krishnan at http://www.freedigitalphotos.net/
  • 75. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield Question 75 (Taken from the public Practice Exam © ISTQB) Which of the following statements BEST describes one of the seven key principles of software testing? (A) Automated tests are better than manual tests for avoiding the Exhaustive Testing. (B) Exhaustive testing is, with sufficient effort and tool support, feasible for all software. (C) It is normally impossible to test all input / output combinations for a software system. (D) The purpose of testing is to demonstrate the absence of defects. https://docs.google.com/forms/d/1r338sJJzg_DwpWTouEiEeD7KUBI0VD6k8AdbNI7LcfU/viewform PLEASE ANSWER AT:
  • 76. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 76 Last active learning exercise Image © renjith krishnan at http://www.freedigitalphotos.net/
  • 77. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 77 The content
  • 78. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 78 References
  • 79. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield Others… 79
  • 80. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield Further reading and sites…  ISTQB - International Software Testing Qualifications Board http://www.istqb.org/  Testing Experience magazine http://www.testingexperience.com/  Agile Testing Days 2014, “Europe’s greatest testing event”, Potsdam, Germany http://www.agiletestingdays.com/call_for_proposals.php  IEEE International Conference on Software Testing, Verification and Validation (ICST’14), Cleveland, Ohio https://sites.google.com/site/icst2014/ 80
  • 81. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield With special thanks to… 81
  • 82. D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 82 Slides of the talk per request: dagmar@monettdiaz.com Prof. Dr. Dagmar Monett Díaz monettdiaz @dmonett http://monettdiaz.com