0
Testing Object-OrientedSystems:Lessons LearnedRobert V. BinderRBSC Corporation www.rbsc.comJune 15, 2000
Overview • Lessons Learned           – Design, automation, and process • The State of the Art           – Design, automati...
Lessons Learned • Class/Cluster test design           – Super/subclass interaction must be tested:             test at fla...
Lessons Learned • Class/Cluster test design           – Exercise each binding of a polymorphic             server message ...
Lessons Learned • Subsystem/system test design           – Control easily obscured or accidental                     • Com...
Lessons Learned • Subsystem/system test design           – Objects don’t compose (easily)           – Producer’s framework...
Lessons Learned • Test Automation           – Encapsulation and mosaic modularity             decrease controllability and...
Lessons Learned • Test Automation           – Avoid stubs: increase scope of the IUT or             test in bottom-up orde...
Lessons Learned • Process           – Inspect for omissions and inconsistencies,             test for everything else     ...
Lessons Learned • Process           – 3 to 6 development increments                     • Developer class/cluster test -- ...
State of the Art • Representation • Design for Testability • Test Design • Test Automation© 2000 Robert V. Binder, all rig...
SOA: Representation • Best Practices           – Syntropy, Design by Contract           – UML/OCL 1.0           – Design P...
SOA: Design for Testability • Best Practices           – Frameworks/libraries with assertions           – Lakos’ levelizab...
SOA: Design for Testability • Challenges           – Seamless language support           – OO testability an oxymoron?    ...
SOA: Test Design • Best Practices           – Test design patterns • Challenges           – Intra-class coverage          ...
Test Design Pattern • New pattern schema for test design             Name/Intent             Context             Fault Mod...
Test Design Patterns • Method Scope                                • Class/Cluster Scope           – Category-Partition   ...
Test Design Patterns • Subsystem Scope                             • Reusable Components           – Class Associations   ...
Test Design Patterns • Intra-class Integration • Integration Strategy           – Small Pop                         – Big ...
Test Design Patterns • System Scope                                • Regression Testing           – Extended Use Cases    ...
SOA: Test Automation • Best Practices           – Design patterns for test automation           – Automatic driver generat...
SOA: Test Harness Patterns • Test Case                                   • Test Drivers   Implementation                  ...
SOA: Test Harness Patterns • Test Execution                              • Built-in Test           – Command Line Test    ...
SOA: Test Automation • Challenges           – Validated failure metrics/fault models           – Tool capability gaps     ...
State of the Practice • Best Practices           – Testing by scope (about 10%)           – Many embedded/real-time shops ...
SOP: Testing by Poking Around • About 70% of all organizations • Characteristics           – Testing done at developer dis...
SOP: Testing by Poking Around • Improvement Strategy           – Assess limits of improvability           – Train develope...
SOP: Testing by Use Cases • About 20% of all organizations • Complies with “Unified Process” test   approach • Characteris...
SOP: Testing by Use Cases • Improvement Strategy           – Achieve exit criteria for indicated class/cluster            ...
SOP: Testing by Scope • About one in ten • Characteristics           – Test design corresponds to scope           – Scope-...
SOP: Testing by Scope • Improvement Strategy           – Internal test design pattern-mining           – Design for testab...
Best Practice Examples • Stepstone Corporation • Ericsson CEE Project • Testing was the primary quality technique© 2000 Ro...
Stepstone Corporation • ICpack 201 -- Objective-C class library • Inspections for all classes • Extensive automated test h...
Ericsson CEE • 75 KLOC C++ cellular support application • Systematic testing at class, cluster, and   system scope • No ot...
Achieving World Class OO Quality • Best-in-Class level:           – An average of “less than 0.025 user-reported          ...
Achieving World Class OO Quality Organization/ KLOC FP                                     Major   Bugs/FP Language       ...
Summary • Lessons learned: OO testing requires   unique test design approaches • State of the art: expressed in patterns •...
Upcoming SlideShare
Loading in...5
×

Testing Object-Oriented Systems: Lessons Learned

844

Published on

Software Quality Week. San Francisco, June 15, 2000
Report on the state of the art and practice of testing object-oriented software.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
844
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
29
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Testing Object-Oriented Systems: Lessons Learned"

  1. 1. Testing Object-OrientedSystems:Lessons LearnedRobert V. BinderRBSC Corporation www.rbsc.comJune 15, 2000
  2. 2. Overview • Lessons Learned – Design, automation, and process • The State of the Art – Design, automation, and process • The State of the Practice – Three levels – Testing can achieve world class quality© 2000 Robert V. Binder, all rights reserved 2
  3. 3. Lessons Learned • Class/Cluster test design – Super/subclass interaction must be tested: test at flattened scope. – Design subclass test suites to re-run on superclasses. – Design superclass test suites to re-run on subclasses. – Test polymorphic servers for LSP compliance.© 2000 Robert V. Binder, all rights reserved 3
  4. 4. Lessons Learned • Class/Cluster test design – Exercise each binding of a polymorphic server message – Test all parameters for generics – Test interface data flow of non-modal classes© 2000 Robert V. Binder, all rights reserved 4
  5. 5. Lessons Learned • Subsystem/system test design – Control easily obscured or accidental • Complex dependencies between concrete state and message sequence • Hierarchic control in state-based subclasses • Mosaic modularity at larger scope – Model behavior with state machines; achieve transition cover or better© 2000 Robert V. Binder, all rights reserved 5
  6. 6. Lessons Learned • Subsystem/system test design – Objects don’t compose (easily) – Producer’s framework should not be in consumer’s test scope – Minimum system/subsystem test includes • Testing exceptions • Testing class associations • Testing use cases (requires testable content)© 2000 Robert V. Binder, all rights reserved 6
  7. 7. Lessons Learned • Test Automation – Encapsulation and mosaic modularity decrease controllability and observability – Design-by-contract/assertions is the only practical counter-measure for inherent non- determinism and loss of testability© 2000 Robert V. Binder, all rights reserved 7
  8. 8. Lessons Learned • Test Automation – Avoid stubs: increase scope of the IUT or test in bottom-up order – Design test harness to exploit the structure and particulars of the system under test – Complete app = app components + test components under CM control© 2000 Robert V. Binder, all rights reserved 8
  9. 9. Lessons Learned • Process – Inspect for omissions and inconsistencies, test for everything else – Design for testability • Implement hierarchic architecture patterns • Eliminate or encapsulate cyclic dependencies • Assert class invariants, at least – Support reuse with complementary producer/consumer testing strategies© 2000 Robert V. Binder, all rights reserved 9
  10. 10. Lessons Learned • Process – 3 to 6 development increments • Developer class/cluster test -- Run tests locally, design tests globally: “unigration” • XP: “Continuous integration, relentless testing” • Independent build/integration group tests completed increment • Test suites must be regress-able – System testing on final increment© 2000 Robert V. Binder, all rights reserved 10
  11. 11. State of the Art • Representation • Design for Testability • Test Design • Test Automation© 2000 Robert V. Binder, all rights reserved 11
  12. 12. SOA: Representation • Best Practices – Syntropy, Design by Contract – UML/OCL 1.0 – Design Patterns • Challenges – Architecture – Limits of cartoons – Test design as software engineering© 2000 Robert V. Binder, all rights reserved 12
  13. 13. SOA: Design for Testability • Best Practices – Frameworks/libraries with assertions – Lakos’ levelizable architecture – Percolation pattern – OS/400 test framework© 2000 Robert V. Binder, all rights reserved 13
  14. 14. SOA: Design for Testability • Challenges – Seamless language support – OO testability an oxymoron? – Entropy horizon about 24 months© 2000 Robert V. Binder, all rights reserved 14
  15. 15. SOA: Test Design • Best Practices – Test design patterns • Challenges – Intra-class coverage – Polymorphic paths – Validated failure metrics/fault models© 2000 Robert V. Binder, all rights reserved 15
  16. 16. Test Design Pattern • New pattern schema for test design Name/Intent Context Fault Model Test Model Strategy Test Procedure Entry Criteria Oracle Exit Criteria Automation Consequences Known Uses Testing Object-Oriented Systems: Models, Patterns, and Tools. Addison-Wesley.© 2000 Robert V. Binder, all rights reserved 16
  17. 17. Test Design Patterns • Method Scope • Class/Cluster Scope – Category-Partition – Invariant Boundaries – Combinational Function – Modal Class – Recursive Function – Quasi-Modal Class – Polymorphic Message – Polymorphic Server – Modal Hierarchy© 2000 Robert V. Binder, all rights reserved 17
  18. 18. Test Design Patterns • Subsystem Scope • Reusable Components – Class Associations – Abstract Class – Round-Trip Scenarios – Generic Class – Mode Machine – New Framework – Controlled Exceptions – Popular Framework© 2000 Robert V. Binder, all rights reserved 18
  19. 19. Test Design Patterns • Intra-class Integration • Integration Strategy – Small Pop – Big Bang – Alpha-Omega Cycle – Bottom up – Top Down – Collaborations – Backbone – Layers – Client/Server – Distributed Services – High Frequency© 2000 Robert V. Binder, all rights reserved 19
  20. 20. Test Design Patterns • System Scope • Regression Testing – Extended Use Cases – Retest All – Covered in CRUD – Retest Risky Use Cases – Allocate by Profile – Retest Profile – Retest Changed Code – Retest Within Firewall© 2000 Robert V. Binder, all rights reserved 20
  21. 21. SOA: Test Automation • Best Practices – Design patterns for test automation – Automatic driver generation – Simple coverage analyzers© 2000 Robert V. Binder, all rights reserved 21
  22. 22. SOA: Test Harness Patterns • Test Case • Test Drivers Implementation – TestDriver Super Class – Test Case/Test Suite – Percolate the Object Method Under Test – Test Case /Test Suite – Symmetric Driver Class – Subclass Driver – Catch All Exceptions – Private Access Driver • Test Control – Test Control Interface – Server Stub – Drone – Server Proxy – Built-in Test Driver© 2000 Robert V. Binder, all rights reserved 22
  23. 23. SOA: Test Harness Patterns • Test Execution • Built-in Test – Command Line Test – Coherence idiom Bundle – Percolation – Incremental Testing – Built-in Test Driver Framework (e.g. Junit) – Fresh Objects© 2000 Robert V. Binder, all rights reserved 23
  24. 24. SOA: Test Automation • Challenges – Validated failure metrics/fault models – Tool capability gaps • No support for OO-specific coverage • Very weak specification-based test generation • Weak support for test harness generation© 2000 Robert V. Binder, all rights reserved 24
  25. 25. State of the Practice • Best Practices – Testing by scope (about 10%) – Many embedded/real-time shops – Extreme Programming • Challenges – High-frequency/short cycle development – Naïve test design – Tool capability gaps© 2000 Robert V. Binder, all rights reserved 25
  26. 26. SOP: Testing by Poking Around • About 70% of all organizations • Characteristics – Testing done at developer discretion – No test entry/exit criteria – High tolerance for low quality© 2000 Robert V. Binder, all rights reserved 26
  27. 27. SOP: Testing by Poking Around • Improvement Strategy – Assess limits of improvability – Train developers in basic test design – Install basic tool set: • Coverage analyzer • Memory leak detector • Test harness framework/generator (e.g. Junit)© 2000 Robert V. Binder, all rights reserved 27
  28. 28. SOP: Testing by Use Cases • About 20% of all organizations • Complies with “Unified Process” test approach • Characteristics – Assumes objects “just work” – System test from use cases – Frustrated with chronic bugginess© 2000 Robert V. Binder, all rights reserved 28
  29. 29. SOP: Testing by Use Cases • Improvement Strategy – Achieve exit criteria for indicated class/cluster test patterns – Use appropriate component/subsystem test design patterns. – Develop testable use cases – Implement test automation to support regression testing© 2000 Robert V. Binder, all rights reserved 29
  30. 30. SOP: Testing by Scope • About one in ten • Characteristics – Test design corresponds to scope – Scope-specific test entry/exit criteria – Appropriate testing at all scopes – Effective test automation – Stable, repeatable process© 2000 Robert V. Binder, all rights reserved 30
  31. 31. SOP: Testing by Scope • Improvement Strategy – Internal test design pattern-mining – Design for testability – Advanced test automation – Quantified closed loop feedback© 2000 Robert V. Binder, all rights reserved 31
  32. 32. Best Practice Examples • Stepstone Corporation • Ericsson CEE Project • Testing was the primary quality technique© 2000 Robert V. Binder, all rights reserved 32
  33. 33. Stepstone Corporation • ICpack 201 -- Objective-C class library • Inspections for all classes • Extensive automated test harness developed for each complex class • No systematic test design© 2000 Robert V. Binder, all rights reserved 33
  34. 34. Ericsson CEE • 75 KLOC C++ cellular support application • Systematic testing at class, cluster, and system scope • No other verification techniques used© 2000 Robert V. Binder, all rights reserved 34
  35. 35. Achieving World Class OO Quality • Best-in-Class level: – An average of “less than 0.025 user-reported defects per function point” in the first year after release • World Class = 10x Best in Class Capers Jones. Software Quality: Analysis and Guidelines for Success. (London: International Thompson Computer Press, 1997) p. 44© 2000 Robert V. Binder, all rights reserved 35
  36. 36. Achieving World Class OO Quality Organization/ KLOC FP Major Bugs/FP Language Post Release Bugs Stepstone 12 414 5 0.0121 Objective-C Ericsson 75 1364 7 0.0051 C++© 2000 Robert V. Binder, all rights reserved 36
  37. 37. Summary • Lessons learned: OO testing requires unique test design approaches • State of the art: expressed in patterns • State of the practice: world class quality can be achieved through testing© 2000 Robert V. Binder, all rights reserved 37
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×