Introduction to Software Testing


Published on

The main introductory concepts on software testing.

Published in: Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Introduction to Software Testing

  1. 1. Università degli Studi dell’AquilaL19: Introduction to Software Testing Henry Muccini DISIM, University of L’Aquila,
  2. 2. The material in these slides may be freely reproduced anddistributed, partially or totally, as far as an explicitreference or acknowledge to the material author ispreserved.With very special thanks to Antonia Bertolino and Debra J.Richardson which collaborated in previous versions ofthese lecture notes
  3. 3. AGENDASoftware Failures: examples Verification and ValidationSoftware Testing: Intro Software Testing: Basics Testing Process Type of Testing
  4. 4. The Skypepervasivefailures:•Black out for two hours•Europe, Japan, Australia,Afghanistan, Sud Africa,Malesia and Brasil havebeen affected
  5. 5. Therac-25 safety failure: Factors: •approximately 100 times the intended dose of •Overconfidence in radiation Software •3 people died, and 6 got injured •Confusing reliability with safety •Lack of defensive Design •Failure to eliminate fault causes •Inadequate software engineering practices •…see article at:
  6. 6. Ash Cloud–relatedstress softwarefailures
  7. 7. Trains in the Netherlands (march 22, 2012)Tens of thousands of people around the large cities weren’t able to travel by trainThursday morning. No trains from and to Amsterdam and Airport Schiphol fromearly morning until after the morning rush hour. A failure in the back-up systemwas the cause. ProRail said that there was a fault in the ‘switch software’. Thesystem therefore didn’t start. And then the signals and switches could not beoperated.Checking some other articles, it simply tries to say that both primary and backupfailed, hence no operations.Links: impact on people:
  8. 8. The Poste Italiane business failure:
  9. 9. Il sistema del MIUR per le prove di maturità:“La maturità 2.0 parte con un flop. Il sistema «commissione web», la novità dell’esame di Stato2012, non ha funzionato. Il software, messo a punto per consentire alle commissioni dicomunicare in tempo reale col cervellone centrale del Miur tutte le attività connesse con gliesami, è andato in tilt ancora prima di partire. Nelle scuole di Firenze le commissioni non sonoriuscite ad inserire online i verbali delle riunioni di insediamento che si sono tenute questamattina. ” Trenitalia:“Il nuovo sistema di Ferrovie dello Stato è un disastro: cè chi non riesce più a usare il propriocodice ma non può cancellarsi perché per farlo occorre usare il codice.Dalle 1 alle 3 di notte non funziona, perché fanno la manutenzione, ma mica te lo dicono …”, pagamento IMU F24:Con la detrazione prima casa, il mio imponibile va sotto zero e il sistema va in tilt.
  10. 10.
  11. 11. [Fatal Defect, Ivars Peterson, 1995]Half book is about failures insoftware development
  12. 12.
  13. 13.
  14. 14. NIST (National Institute of Standards andTechnology) study in 2002 [NIST], →software errors cost the U.S. economy $59.5 billion every year.Standish Chaos report [Standish] →a clear statement of requirements is one of the three main reasons that lead to project success, as well as incomplete requirements are one of the main reason of projects deletion.[NIST] The economic impacts of inadequate infrastructure for software testing. InNIST Planning Report 02-3. 2002.[Standish] The standish group report: Chaos. 1995.
  15. 15. Validation:does the software system meets the users real needs? are we building the right software? (valid with respect to users’ needs)Verification:does the software system meets the requirementsspecifications? are we building the software right? (valid with respect to the system specification)
  16. 16. Software Inspection (static analysis technique)Debugging (to locate and fix bugs)Theorem provingModel checking (to prove a propertycorrectness)Software Testing (None is the absolute perfect solution)
  17. 17. Completeness & Correctness • Correctness properties are undecidable • False positive and False negativeTimeliness • The V&V process stops (most of the time) where there is no more time • Time is one of the stopping rulesCost-effectiveness • “Select the less that gives you the most” • V&V is justified especially when failures are expensive
  19. 19. An all-inclusive definition IMP Software testing consists of:the dynamic verification of the behavior of a program on a finite set of test cases suitably selected from the (in practice infinite) input domain against the specified expected behavior [A. Bertolino]
  20. 20. is not IMP (citation from Hamlet, 1994):Ive searched hard for defects in this program, found a lot of them, and repaired them. I cant find any more, so Im confident there arent any. Testing is NOT exhaustive ⇒ What to test? ⇒ When to stop? Testing is NOT cheap ⇒ test less and best!! ⇒ When to stop?
  21. 21. (1) Testing Process: Glossary Systematic vs. Ad Hoc Test Selection (Category partition) Test Execution Oracle vs. Ad Hoc Regression Testing(2) Type of Testing: Black Box and White Box Unit, Integration, System White Box
  22. 22. IMPTesting involves several demanding tasks: →Test selection ─ how to identify a suitable finite set of test cases →Test execution ─ how to translate test cases into executable runs →Test oracle ─ Deciding wheter the test outcome is acceptable or not ─ If not, evaluating the impact of the failure and its direct cause (the fault) →Testing adequacy ─ Judging wheter the test campaign is sufficient →Test coverage
  23. 23. Test selection consists in the identification of a “suitable” and finite set of test cases.The test selection activity provides guidelines on how toselect test cases. It is driven by a ‘‘test criterion’’ and hasto produce ‘‘suitable’’ test cases
  24. 24. Slide taken from Alex Orso
  25. 25. Test Criterion: A test criterion provides the guidelines, rules, and strategy by which test cases are selected. In general, a test criterion is a means of deciding which shall be a ‘‘good’’ set of test cases (Reference 117 of [Muccini08]) .Suitability: A test case is suitable if it contributes to discovering as many failures as possible, according to a test criterion.[Muccini08] Henry Muccini, Software Testing: Testing New Software Paradigms andNew Artifacts, in: Wiley Encyclopedia of Computer Science and Engineering, John Wiley &Sons, Inc., 2008
  26. 26. Test Case: A test case is a set of inputs, execution conditions, and a pass/fail criterion (Ref. 116 of [Muccini08]) . A test case thus includes not only input data but also any relevant execution conditions and procedures, and includes a way of determining whether the program has passed or failed the test on a particular execution (Ref. 8 of [Muccini08]).Test Suite: A test suite is a collection of test cases.
  27. 27. The EasyLine system iscomposed of three sub-systems: The SP system The Mobile app The server-sideapplication
  28. 28. How to select test cases? (test selection technique)How many test cases? (when to stop --stopping rule?)Which artefacts to use for selecting test cases?(code, spec?)Ad hoc or Systematic testing?
  29. 29. Tester’s intuition and expertise • “Ad hoc testing” (sometime quite effective) • Special cases Fault-based Specifications • Error guessing/special cases • Mutation • Equivalence partitioning • Boundary-value analysis Usage • Decision table • SRET • Field testing • Automated derivation from formal specs (conformance t.) • .... Nature of application, e.g.: Code • Object Oriented • Web • Control-flow based • GUI • Data-flow based • Real-time, embedded • Scientific • .....No one is the best technique, but a combination of different criteria has empiricallyshown to be the most effective approach
  30. 30. (Test) InputsCode-based: (code graphs) Source →Structural/White Box Testing Code ─ Test cases selected based on structure of code ─ Views program /component as white box (also called glass box testing) Output Internal behavior (Test) InputsSpecification-based: (Input-output) →Functional/Black Box Testing Bynary ─ Test cases selected based on specification Code or ─ Views program/component as black box Spec Output
  31. 31. We focus on “systematic” testing: →Repeatable →Measurable IMP ─ best tester ─ coverage →Based on sampling: ─ Infinite input domain, but finite set of test cases
  32. 32. Two are the main sub-activities to be performed: →B1) identify those “inputs" which force the execution of the selected test case, →B2) put the system in a state from which the specified test can be launched.B1 -- Forcing the execution of the test cases derivedaccording to one criterion might not be obvious →In code-based testing, we have entry-exit paths over the graph model and test inputs that execute the corresponding program paths need be found
  33. 33. B2 -- put the system in a state from which the specifiedtest can be launched →Also called, Test Pre-condition →In Synchronous Systems: ─ Several runs in sequence are required to put the system in the test pre-condition →In Concurrent Systems: ─ Non Determinism problem Replay problem
  34. 34. The EasyLine system iscomposed of :Web servicesSensorsMobile applications routing algorithms…
  35. 35. A test oracle is a mechanism for verifying the behavior of test execution → extremely costly and error prone to verify → oracle design is a critical part of test planningSources of oracles → input/outcome oracle → tester decision → regression test suites → standardized test suites and oracles → gold or existing program → formal specification BACK
  36. 36. The Expected Output is f*(d) YES, given input d f(d) = f*(d)» In some cases easier (e.g., an existing version, existing formal specification), but generally very difficult (e.g., operational testing)» Not enough emphasized research problem
  37. 37. Theoretical notions of test adequacy are usually defined in terms of adequacy criteria → Coverage metrics (sufficient percentage of the program structure has been exercised) → Empirical assurance (failures/test curve flatten out) → Error seeding (percentage of seeded faults found is proportional to the percentage of real faults found) → Independent testing (faults found in common are representative of total population of faults)Adequacy criteria are evaluated with respect to a test suite and a program under test BACK
  38. 38. (1) Testing Process: Glossary Systematic vs. Ad Hoc Test Selection (Category partition) Test Execution Oracle vs. Ad Hoc Regression Testing(2) Type of Testing: Black Box and White Box Unit, Integration, System White Box
  39. 39. Black box vs White box [in next lectures]Unit, Integration, SystemPerformance, StressRegression Testing [in next lectures]…
  40. 40. Unit: →The Unit test purpose is to ensure that the unit satisfies its functional specification and/or that its implemented structure matches the intended design structure →Unit tests can also be applied for test interface or local data structure.Integration: →Integration testing is specifically aimed at exposing the problems that arise from the combination of components →Communicating interfaces among integrated components need to be tested →Type: big-bang or incremental (top-down, bottom-up, mixed)System: →Itattempts to reveal bugs which depends on the environment →Recovery testing, security testing, stress testing and performance testing
  41. 41. levels of abstraction User Acceptance Requirements Testing plan & validate Software /verify Requirements System Specification Testing Architecture Design Integration Specification Testing Component Component integrate design & Specifications Testing & test analyze Unit Unit Implementations Testing time
  42. 42. The EasyLine system iscomposed of three sub-systems: The SP system The Mobile app The server-sideapplication
  43. 43. Stress testing: designed to test the software with abnormalsituations. →Stress testing attempts to find the limits at which the system will fail through abnormal quantity or frequency of inputs. →The test is expected to succeed when the system is stressed with higher rates of inputs, maximum use of memory or system resources.Performance testing is usually applied to real-time, embeddedsystems in which low performances may have serious impact onthe normal execution. →Performance testing checks the run-time performance of the system and may be coupled with stress testing. →Performance is not strictly related to functional requirements: functional tests may fail, while performance ones may succeed.