Testing Object-Oriented Systems: Lessons Learned

2,054 views

Published on

Software Quality Week. San Francisco, June 15, 2000
Report on the state of the art and practice of testing object-oriented software.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,054
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
53
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Testing Object-Oriented Systems: Lessons Learned

  1. 1. Testing Object-OrientedSystems:Lessons LearnedRobert V. BinderRBSC Corporation www.rbsc.comJune 15, 2000
  2. 2. Overview • Lessons Learned – Design, automation, and process • The State of the Art – Design, automation, and process • The State of the Practice – Three levels – Testing can achieve world class quality© 2000 Robert V. Binder, all rights reserved 2
  3. 3. Lessons Learned • Class/Cluster test design – Super/subclass interaction must be tested: test at flattened scope. – Design subclass test suites to re-run on superclasses. – Design superclass test suites to re-run on subclasses. – Test polymorphic servers for LSP compliance.© 2000 Robert V. Binder, all rights reserved 3
  4. 4. Lessons Learned • Class/Cluster test design – Exercise each binding of a polymorphic server message – Test all parameters for generics – Test interface data flow of non-modal classes© 2000 Robert V. Binder, all rights reserved 4
  5. 5. Lessons Learned • Subsystem/system test design – Control easily obscured or accidental • Complex dependencies between concrete state and message sequence • Hierarchic control in state-based subclasses • Mosaic modularity at larger scope – Model behavior with state machines; achieve transition cover or better© 2000 Robert V. Binder, all rights reserved 5
  6. 6. Lessons Learned • Subsystem/system test design – Objects don’t compose (easily) – Producer’s framework should not be in consumer’s test scope – Minimum system/subsystem test includes • Testing exceptions • Testing class associations • Testing use cases (requires testable content)© 2000 Robert V. Binder, all rights reserved 6
  7. 7. Lessons Learned • Test Automation – Encapsulation and mosaic modularity decrease controllability and observability – Design-by-contract/assertions is the only practical counter-measure for inherent non- determinism and loss of testability© 2000 Robert V. Binder, all rights reserved 7
  8. 8. Lessons Learned • Test Automation – Avoid stubs: increase scope of the IUT or test in bottom-up order – Design test harness to exploit the structure and particulars of the system under test – Complete app = app components + test components under CM control© 2000 Robert V. Binder, all rights reserved 8
  9. 9. Lessons Learned • Process – Inspect for omissions and inconsistencies, test for everything else – Design for testability • Implement hierarchic architecture patterns • Eliminate or encapsulate cyclic dependencies • Assert class invariants, at least – Support reuse with complementary producer/consumer testing strategies© 2000 Robert V. Binder, all rights reserved 9
  10. 10. Lessons Learned • Process – 3 to 6 development increments • Developer class/cluster test -- Run tests locally, design tests globally: “unigration” • XP: “Continuous integration, relentless testing” • Independent build/integration group tests completed increment • Test suites must be regress-able – System testing on final increment© 2000 Robert V. Binder, all rights reserved 10
  11. 11. State of the Art • Representation • Design for Testability • Test Design • Test Automation© 2000 Robert V. Binder, all rights reserved 11
  12. 12. SOA: Representation • Best Practices – Syntropy, Design by Contract – UML/OCL 1.0 – Design Patterns • Challenges – Architecture – Limits of cartoons – Test design as software engineering© 2000 Robert V. Binder, all rights reserved 12
  13. 13. SOA: Design for Testability • Best Practices – Frameworks/libraries with assertions – Lakos’ levelizable architecture – Percolation pattern – OS/400 test framework© 2000 Robert V. Binder, all rights reserved 13
  14. 14. SOA: Design for Testability • Challenges – Seamless language support – OO testability an oxymoron? – Entropy horizon about 24 months© 2000 Robert V. Binder, all rights reserved 14
  15. 15. SOA: Test Design • Best Practices – Test design patterns • Challenges – Intra-class coverage – Polymorphic paths – Validated failure metrics/fault models© 2000 Robert V. Binder, all rights reserved 15
  16. 16. Test Design Pattern • New pattern schema for test design Name/Intent Context Fault Model Test Model Strategy Test Procedure Entry Criteria Oracle Exit Criteria Automation Consequences Known Uses Testing Object-Oriented Systems: Models, Patterns, and Tools. Addison-Wesley.© 2000 Robert V. Binder, all rights reserved 16
  17. 17. Test Design Patterns • Method Scope • Class/Cluster Scope – Category-Partition – Invariant Boundaries – Combinational Function – Modal Class – Recursive Function – Quasi-Modal Class – Polymorphic Message – Polymorphic Server – Modal Hierarchy© 2000 Robert V. Binder, all rights reserved 17
  18. 18. Test Design Patterns • Subsystem Scope • Reusable Components – Class Associations – Abstract Class – Round-Trip Scenarios – Generic Class – Mode Machine – New Framework – Controlled Exceptions – Popular Framework© 2000 Robert V. Binder, all rights reserved 18
  19. 19. Test Design Patterns • Intra-class Integration • Integration Strategy – Small Pop – Big Bang – Alpha-Omega Cycle – Bottom up – Top Down – Collaborations – Backbone – Layers – Client/Server – Distributed Services – High Frequency© 2000 Robert V. Binder, all rights reserved 19
  20. 20. Test Design Patterns • System Scope • Regression Testing – Extended Use Cases – Retest All – Covered in CRUD – Retest Risky Use Cases – Allocate by Profile – Retest Profile – Retest Changed Code – Retest Within Firewall© 2000 Robert V. Binder, all rights reserved 20
  21. 21. SOA: Test Automation • Best Practices – Design patterns for test automation – Automatic driver generation – Simple coverage analyzers© 2000 Robert V. Binder, all rights reserved 21
  22. 22. SOA: Test Harness Patterns • Test Case • Test Drivers Implementation – TestDriver Super Class – Test Case/Test Suite – Percolate the Object Method Under Test – Test Case /Test Suite – Symmetric Driver Class – Subclass Driver – Catch All Exceptions – Private Access Driver • Test Control – Test Control Interface – Server Stub – Drone – Server Proxy – Built-in Test Driver© 2000 Robert V. Binder, all rights reserved 22
  23. 23. SOA: Test Harness Patterns • Test Execution • Built-in Test – Command Line Test – Coherence idiom Bundle – Percolation – Incremental Testing – Built-in Test Driver Framework (e.g. Junit) – Fresh Objects© 2000 Robert V. Binder, all rights reserved 23
  24. 24. SOA: Test Automation • Challenges – Validated failure metrics/fault models – Tool capability gaps • No support for OO-specific coverage • Very weak specification-based test generation • Weak support for test harness generation© 2000 Robert V. Binder, all rights reserved 24
  25. 25. State of the Practice • Best Practices – Testing by scope (about 10%) – Many embedded/real-time shops – Extreme Programming • Challenges – High-frequency/short cycle development – Naïve test design – Tool capability gaps© 2000 Robert V. Binder, all rights reserved 25
  26. 26. SOP: Testing by Poking Around • About 70% of all organizations • Characteristics – Testing done at developer discretion – No test entry/exit criteria – High tolerance for low quality© 2000 Robert V. Binder, all rights reserved 26
  27. 27. SOP: Testing by Poking Around • Improvement Strategy – Assess limits of improvability – Train developers in basic test design – Install basic tool set: • Coverage analyzer • Memory leak detector • Test harness framework/generator (e.g. Junit)© 2000 Robert V. Binder, all rights reserved 27
  28. 28. SOP: Testing by Use Cases • About 20% of all organizations • Complies with “Unified Process” test approach • Characteristics – Assumes objects “just work” – System test from use cases – Frustrated with chronic bugginess© 2000 Robert V. Binder, all rights reserved 28
  29. 29. SOP: Testing by Use Cases • Improvement Strategy – Achieve exit criteria for indicated class/cluster test patterns – Use appropriate component/subsystem test design patterns. – Develop testable use cases – Implement test automation to support regression testing© 2000 Robert V. Binder, all rights reserved 29
  30. 30. SOP: Testing by Scope • About one in ten • Characteristics – Test design corresponds to scope – Scope-specific test entry/exit criteria – Appropriate testing at all scopes – Effective test automation – Stable, repeatable process© 2000 Robert V. Binder, all rights reserved 30
  31. 31. SOP: Testing by Scope • Improvement Strategy – Internal test design pattern-mining – Design for testability – Advanced test automation – Quantified closed loop feedback© 2000 Robert V. Binder, all rights reserved 31
  32. 32. Best Practice Examples • Stepstone Corporation • Ericsson CEE Project • Testing was the primary quality technique© 2000 Robert V. Binder, all rights reserved 32
  33. 33. Stepstone Corporation • ICpack 201 -- Objective-C class library • Inspections for all classes • Extensive automated test harness developed for each complex class • No systematic test design© 2000 Robert V. Binder, all rights reserved 33
  34. 34. Ericsson CEE • 75 KLOC C++ cellular support application • Systematic testing at class, cluster, and system scope • No other verification techniques used© 2000 Robert V. Binder, all rights reserved 34
  35. 35. Achieving World Class OO Quality • Best-in-Class level: – An average of “less than 0.025 user-reported defects per function point” in the first year after release • World Class = 10x Best in Class Capers Jones. Software Quality: Analysis and Guidelines for Success. (London: International Thompson Computer Press, 1997) p. 44© 2000 Robert V. Binder, all rights reserved 35
  36. 36. Achieving World Class OO Quality Organization/ KLOC FP Major Bugs/FP Language Post Release Bugs Stepstone 12 414 5 0.0121 Objective-C Ericsson 75 1364 7 0.0051 C++© 2000 Robert V. Binder, all rights reserved 36
  37. 37. Summary • Lessons learned: OO testing requires unique test design approaches • State of the art: expressed in patterns • State of the practice: world class quality can be achieved through testing© 2000 Robert V. Binder, all rights reserved 37

×