The document discusses the importance of testability in software systems, outlining factors that affect both white box and black box testability. It emphasizes the need for controllability and observability in test design, as well as strategies to improve testability through structured code and automation. Additionally, it addresses challenges such as system complexity, resource constraints, and the significance of understanding system requirements for effective testing.

1. Testability: Factors and StrategyRobert V. Binderrvbinder@gmail.comGoogle Test Automation Conference Hyderabad October 28, 20101 © 2010, Robert V. Binder

2. OverviewWhy testability mattersWhite box testabilityBlack box testabilityTest automationStrategyQ&A2

3. Why Testability Matters3

4. Testability EconomicsAssumptionsSooner is betterEscapes are badFewer tests means more escapesFixed budget – how to optimizeEfficiency: average tests per unit of effortEffectiveness: average probability of killing a bug per unit of effortHigher testability: more better tests, same costLower testability: fewer weaker tests, same cost Testability defines the limits on producing complex systems with an acceptable risk of costly or dangerous defects.4

5. What Makes an SUT Testable?ControllabilityWhat do we have to do to run a test case?How hard (expensive) is it to do this?Does the SUT make it impractical to run some kinds of tests ? Given a testing goal, do we know enough to produce an adequate test suite?How much tooling can we afford?ObservabilityWhat do we have to do to determine test pass/fail?How hard (expensive) is it to do this?Does the SUT make it impractical to fully determine test results ?Do we know enough to decide pass/fail/DNF ?5

6. Testability Factors6

7. Testability Factors7

8. Examples8

9. White BOX Testability9

10. White Box Testability10

11. The Anatomy of a Successful TestTo reveal a bug, a test mustReach the buggy codeTrigger the bugPropagate the incorrect resultIncorrect result must be observedThe incorrect result must be recognized as suchint scale(int j) { j = j - 1; j = j/30000; return j;}// should be j = j + 1Out of 65,536 possible values for j, only six will produce an incorrect result: -30001, -30000, -1, 0, 29999, and 30000. 11

12. What Diminishes Testability?Non-deterministic DependenciesRace conditionsMessage latencyThreadingCRUD shared and unprotected data12

13. What Diminishes Testability?ComplexityEssentialAccidental13

15. What Improves Testability?Points of Control and Observation (PCOs)State test helpersBuilt-in TestWell-structured code15

16. PCOsPoint of Control or ObservationAbstraction of any kind of interfaceSee TTCN-3 for a complete discussionWhat does a tester/test harness have to do activate SUT components and aspects? Components easierAspects more interesting, but typically not directly controllableWhat does a tester/test harness have to do inspect SUT state or traces to evaluate a test? Traces easier, but often not sufficient or noisyEmbedded state observers effective, but expensive or polluting Aspects often critical, but typically not directly observableDesign for testability: Determine requirements for aspect-oriented PCOs16

17. State Test HelpersSet state Get current stateLogical State Invariant Functions (LSIFs)ResetAll actions controllableAll events observable17

18. Implementation and Test Models18

19. Test Plan and Test SizeK eventsN statesWith LSIFsKN testsNo LSIFsK × N3 tests19

20. Built-in TestAssertsLoggingDesign by ContractNo Code Left Behind patternPercolation pattern20

21. No Code Left Behind ForcesHow to have extensive BIT without code bloat?How to have consistent, controllableLoggingFrequency of evaluationOptions for handling detectionDefine once, reuse globally?SolutionDev adds Invariant function to each classInvariant calls InvariantCheck evaluates necessary stateInvariantCheck function with global settingsSelective run time evaluation constand inline idiom for no object code in production21

22. Percolation PatternEnforces LiskovSubsitutabilityImplement with No Code Left Behind22

23. Well-structured CodeMany well-established principlesSignificant for TestabilityNo cyclic dependenciesDon’t allow build dependencies to leak over structural and functional boundaries (levelization)Partition classes and packages to minimize interface complexity23

24. Black Box Testability24

25. Black Box Testability25

26. System SizeHow big is the essential system?Feature pointsUse casesSingly invocable menu itemsCommand/sub commandsComputational strategyTransaction processingSimulationVideo gamesStorageNumber of tables/views26

27. Network ExtentHow many independent nodes?Client/serverTiersPeer to peerMinimum Spanning TreeMust have one at least of each onlineMS-TPSO, Transaction Processing System Overview27

28. VariantsHow many configuration options?How many platforms supported?How many localization variantsHow many “editions”? Combinational coverageTry each option with every other at least once (pair-wise)Pair-wise worst case product of option sizesMay be reduced with good tools28

29. Weather – EnvironmentalThe SUT has elements that cannot be practically established in the labCellular base stationExpensive server farmCompetitor/aggressor capabilitiesInternet – you can never step in the same river twiceTest environment not feasibleMust use production/field system for test Cannot stress production/field system29

30. More is LessOther things being equal, a larger system is less testableSame budget spread thinnerReducing system scope improves testabilityTry to partition into independent subsystems: additive versus multiplicative extent of test suiteFor example10000 Feature Points6 Network Node types20 options, five variables eachMust test when financial markets are open How big is that?30

31. M6695000 Light Years1 Pixel ≈ 400 Light YearsSolar System ≈ 0.0025 LY

32. Understanding Improves TestabilityPrimary source of system knowledgeDocumented, validated?Intuited, guessed?IEEE 830Test ModelDifferent strokesTest-ready or hints?OracleComputable or judgment?Indeterminate?32

33. Test Automation33

34. Automation Improves Testability Bigger systems need many more testsIntellectual leverageHuge coverage spaceRepeatableScale up functional suites for load testWhat kind of automation?HarnessDeveloper harnessSystem harnessCapture-replayModel-basedGenerationEvaluation34

35. Test Automation Envelope35Model-BasedVisionBespokeModel-Based5 Nines4 Nines3 Nines2 Nines1 NineManual1101001,00010,000ScriptingReliability (Effectiveness)Productivity: Tests/Hour (Efficiency)

36. Strategy36

37. minimizemaximizeHow to Improve Testability?minimizemaximizeWBT = BIT + State Helpers +PCOs - Structure - Complexity– NDDsBBT = Model + Oracle + Harness – Size – Nodes – Variants - Weather37

38. Who Owns Testability Drivers?Testers typically don’t own or control the work that drives testability38

39. StrategyEmphasize Functional TestingIncentivize RepeatersHighBlack BoxTestabilityEmphasize Code CoverageManageExpectationsLowLowHighWhite BoxTestability39

40. Q & A40

41. Media CreditsRobert V. Binder. Design for testability in object-oriented systems. Communications of the ACM, Sept 1994. Diomidis D. Spinellis. Internet Explorer Call Tree.Unknown. Dancing Hamsters.Jackson Pollock, Autumn Rhythms.Brian Ferneyhough. Plötzlichkeitfor Large Orchestra.Robert V. Binder. Testing Object-Oriented Systems: Models, Patterns, and Tools.Microsoft, MS-TPSO, Transaction Processing System Overview.NASA, Hubble Space Telescope, M66 Galaxy.Holst - The Planets – Uranus. Peabody Concert Orchestra,Hajime Teri Murai, Music DirectorTest Automation Envelope. mVerify Corporation.41