Demand-Driven Structural Testing with Dynamic Instrumentation (ICSE 2005)

435 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
435
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Demand-Driven Structural Testing with Dynamic Instrumentation (ICSE 2005)

  1. 1. Jonathan Misurda (jmisurda@cs.pitt.edu)James A. Clause, Juliya L. Reed, Bruce R. ChildersDepartment of Computer ScienceUniversity of PittsburghPittsburgh, Pennsylvania 15260 USAMary Lou SoffaDepartment of Computer ScienceUniversity of VirginiaCharlottesville, Virginia 22904 USADemand-Driven Structural Testingwith Dynamic Instrumentation
  2. 2. Software Testingn  What is Software Testing?q  Gather information about the behavior of the software beingdeveloped or modified.n  Why test software?q  Gain confidence and insights into software correctnessq  Create quality, robust programsn  Why is testing hard?q  Testing is expensiven  50-60% of the total cost of software developmentq  Adds complexity to development processn  One testing technique is not enough
  3. 3. Structural Software Testingn  Structural Testingq  The process of discovering run-time properties of aprogram specifically pertaining to control-flowq  Demonstrate adequacy of the test casesq  Different granularities of structures:n  Node coverage records statements (basic blocks)n  Branch coverage records control-flow edgesn  Def-Use coverage records pairs of variable definitionsand usesq  Over multiple test cases until coverage criteria
  4. 4. Current Testing Toolsn  E.g.: JCover, Clover, IBM Rational PurifyPlusn  Static instrumentation with test probesq  Probes are injected instrumentation to gather coverageinformationn  Limitations:q  Expensive in both time and spaceq  Only one type of testq  Limited code regions and scopesq  Cannot define their own ways to test
  5. 5. Our Approachn  Demand-driven structural testingq  Specification driven: User written testsq  Test plans: Recipe of how & where to testq  Path specific: Instrument only what is neededq  Dynamic: Insert & remove instrumentationn  Jazz: A scalable & flexible framework for testingq  Automatically apply multiple test strategiesq  Handle large programsq  Allows user customization of tests
  6. 6. Jazz OverviewApplicationIDE & Test GUITest SpecificationTest PlannerTest PlanTest Analyzer and Result ReporterTest Dynamic InstrumenterTest ResultsSelect the regions and test typesAllows user customizationAutomatically generate where andhow to instrumentRun the program with demand-driveninstrumentationCollect the results
  7. 7. Test Specifier1a1b32Specify the regions to test1a. Highlight line regions1b. Select classes and methodsList of desired testsClick to create a test
  8. 8. Test Plannern  Test plan: Where and how to test programn  Test storage, probe location table,instrumentation payloadn  Can combine payloads to create custom strategiesApplicationTestPlannerTestSpecificationTest InstrumentationPayloadTest PlanGlobal Storage & Probe Location Table
  9. 9. Test Plannern  Challenges:q  When to insert test probesq  Where to test/instrumentq  What to do at each proben  A planner determines three things:q  Seed Probe Set – probes which are initially inserted in atest region.q  Coverage Probe Set – probes that can be inserted andremoved on demand along a path.q  Probe Lifetime – whether a probe must be re-insertedafter being hit by its “control flow successor” basic blocks.
  10. 10. Branch Coverage Example Planpublic class simpleDemo {public static void main(String[] args) {int evenSum = 0;int oddSum = 0;for(int i = 0; i < 100; i++) {if(i%2==0) {evenSum += i;}else {oddSum += i;}}System.out.println("The sum of the even numbers from 0 to 100 is: " + evenSum);System.out.println("The sum of the odd numbers from 0 to 100 is: " + oddSum);}}DEFINITIONS {NAME: d_method, REGION_D,LOCATION: FILE simpleDemo.java {CLASS simpleDemo, METHOD main}}BODY {DO BRANCH_TEST ON REGION d_method UNTIL: 85%}simpleDemo.javasimpleDemo.testspec
  11. 11. Branch Coverage Plannern  Record which edges are executedq  Determine (source, sink) edge pairs hit at run-timeq  Source is a branchq  Sink can be taken & not-taken target of branchProbe Set MembersSeed Probe First block in testing regionCoverage Probe Sinks of currently instrumented blockProbe Lifetime Until all incoming edges have beencovered
  12. 12. Branch Coverage ExampleProbe Loc Tab StorageBlock Next Hit1 2,32 43 44 1Test PayloadMark edge hitInsert at next pointRemove instrumentation12 34Coverage probeHitN, NNNNY, YYYYSeed probe
  13. 13. Demon  Branch coverage of music playerq  Test region is whole program (JOrbis)q  Test input is a songn  Comparedq  Traditional approach with static instrumentationq  Demand-driven approach with dynamic instrumentationWith DynamicInstrumentationWith StaticInstrumentation
  14. 14. Demon  Branch coverage of music playerq  Test region is whole program (JOrbis)q  Test input is a songn  Comparedq  Traditional approach with static instrumentationq  Demand-driven approach with dynamic instrumentationWith DynamicInstrumentationWith StaticInstrumentation
  15. 15. Demon  Branch coverage of music playerq  Test region is whole program (JOrbis)q  Test input is a songn  Comparedq  Traditional approach with static instrumentationq  Demand-driven approach with dynamic instrumentationWith DynamicInstrumentationWith StaticInstrumentation
  16. 16. Other Plannersn  Node Coverage Plannerq  Record which statements are executedProbe Set MembersSeed Probe First statement in the testing regionCoverage Probe Next reachable statementProbe Lifetime Removed as soon as statement is executedn  Def-Use Coverage Plannerq  Record which variable definition reaches an executed useProbe Set MembersSeed Probe All variable definitionsCoverage Probe Uses associated with all definitionsProbe Lifetime Defs removed after all reachable uses arecovered. Uses are removed as executed
  17. 17. Dynamic Instrumentationn  Test plan targets an instrumentation APIn  FIST instrumentation engine [WOSS’04]q  Retargetable & reconfigurableq  Dynamic insertion & removal of instrumentationq  Binary level instrumentation (post JIT)n  Uses fast breakpoints [Kessler]: Replace existinginstruction with a jump to instrumentation
  18. 18. Experimental Methodologyn  Test Specificationq  Implemented as an Eclipse 3.0 pluginn  Test Planner & Dynamic Instrumenterq  Implemented using the Jikes RVM version 2.1.1q  On the x86 platformn  SPECjvm98 benchmarks test inputq  unloaded 2.4 Ghz Pentium IV, 1GB of memoryq  RedHat Linux 7.3n  Traditional vs. Jazz (in same implementation)q  Compared coverage, run-time, memory
  19. 19. Coverage Resultsn  Coverage – same reported by both toolsBranch Node Def-Usecompress 58% 90.6% 89.8%jess 46.8% 80.3% 71.8%db 44.4% 76.9% 75.0%javac 38.9% 75.0% 66.9%mpegaudio 60.9% 88.7% 90.5%mtrt 50.6% 90.3% 87.3%jack 55.6% 82.2% 73.4%
  20. 20. Run-time PerformanceStatic BranchDynamic BranchStatic Node Static Def-UseDynamic Node Dynamic Def-Use
  21. 21. Memory RequirementsBranch Node Def-UseDemand Static Demand Static Demand Staticcompress 7.9 7.5 4.4 3.7 8.8 5.6jess 50.2 60.3 34.1 29.7 70.3 44.8db 9.7 8.9 5.0 4.1 9.8 5.1javac 178.9 186 107.3 85.3 202.0 125.3mpegaudio 24.7 29.5 15.0 14.7 34.8 22.3mtrt 22.4 23 12.5 11.0 26.1 16.1jack 73.4 78 46.5 37.6 89.0 46.8• In Kilobytes
  22. 22. Summaryn  New demand-driven approachq  A tool (Jazz) for structural testingq  Dynamic instrumentation guided by a program’sexecutionq  Minimally intrusiveq  User configurable and flexible testingn  Very low overheadq  E.g., branch coverage tool is 3-4x faster thantraditional approaches
  23. 23. For Further Information…jmisurda@cs.pitt.eduhttp://www.cs.pitt.edu/coco
  24. 24. Test Results1 Click to run test suite2 Results
  25. 25. Static Instrumentationn  Shortcomings of static instrumentation:q  Not scalable: Instrumentation left in place causesunnecessary increase in overheadq  Inflexible: Only certain tests, languages, andplatformsq  Cumbersome: Requires recompilations fordedicated testingq  Intrusive: Addition of testing code may alter ormask defects
  26. 26. Run-time Performance-10%30%70%110%150%190%230%270%compress jess db javac mpeg mtrt jack avg%slowdownoverbaselineStatic Branch Dynamic Branch Static NodeDynamic Node Static Def-Use Dynamic Def-Use
  27. 27. Branch Testing0102030405060708090100Time(seconds)compress jess db javac mpeg mtrt jack avgDynamic Static None
  28. 28. Node Testing0102030405060708090100Time(seconds)compress jess db javac mpeg mtrt jack avgDynamic Static None
  29. 29. Def-Use Testing0102030405060708090100Time(seconds)compress jess db javac mpeg mtrt jack avgDynamic Static None

×