Automating Regression Testing  For Evolving GUI Software       Zheng-Wen Shen         2005/10/19                          ...
ReferencesAutomating regression testing for evolvingGUI software– Atif Memon, Adithya Nagarajan and Qing Xie– JOURNAL OF S...
Outline1. Introduction2. The DART process3. Design of DART– 3.1. GUI Representation– 3.2 Modules of DART4. Experiments5. C...
1. IntroductionToday’s competitive software developmentmarket demands that several developers,who are perhaps geographical...
DART(Daily Automated Regression Tester) A new framework that address the needs of re- testing frequent builds of GUI softw...
2. The DART Process                6
7
Roles of the developer/tester and DARTMaintenance   cycle                               8
M    M’         9
3. Design of DART Driving philosophy behind the design 1. Automated 2. Efficient 3. Robust 4. Portable 5. General        ...
Runs all                   test casesExpected results                      How much                       testing         ...
3.1 GUI RepresentationA formal model of the AUT’s GUI               GUI Representation       1            Objects & Proper...
3.1.1 Objects and propertiesobjects O = {o1, o2, . . . ,om}properties P = {p1, p2, . . . ,pl }A GUI = O + PThe state of a ...
3.1.2 EventsThe events E = {e1, e2, . . . , en} associatedwith a GUI are functions from one state toanother state of the G...
3.1.3 ComponentA GUI component C is an ordered pair (RF, UF)– A modal window + a set of modeless windowsRF represents a mo...
3.1.4 Event-flow graphsAll possible interactions among theevents in a component.An event-flow graph for a component C isa ...
EFG for part of MS WordPad.“main” component                       17
3.1.5 Integration Tree (IT)To identify interactions amongcomponents.Invocations – Component Cx invokes component Cy iff   ...
Definition of ITAn IT is a triple <N, R, B>– N : components in the GUI.– R ∈ N: a designated component called the  Main co...
IT for part of MS WordPad.                     20
3.1.6 Event Classification1. Restricted-focus events2. Unrestricted-focus events3. Termination events4. Menu-open events5....
1 Restricted-focus event   open modal windows                        22
2 Unrestricted-focus events     open modeless windows                        23
3 Termination events  close modal windows                        24
4 Menu-open events    Open menus                 25
5 System-interaction eventsInteract with the underlying softwareto perform some action                                26
3.2 Modules of DART                                 2      1                 Test Case Generator                          ...
3.2.1 GUI RipperStep1: GUI traversal and extraction (DFS)Step2: Manual inspection by TesterStep3: Generating the event-flo...
3.2.2 Test-case generatorA GUI test case T is a pair <S0, e1; e2; . . . ;e n>Generate test case by traversing theevent-flo...
3.2.3 Test-oracle generatorTest oracles are used to determinewhether or not the software executedcorrectly during testing....
Approach: execution extractionA test case is executed on an existing,presumably correct version of the softwareand its sta...
Level of Oracle Information            (LOI)Complete (LO1)– {(w, p, o), ∀w ∈ Windows, ∀o = objects ∈ w, ∀p =  properties ∈...
3.2.4 Coverage evaluatorCode coverage and event coverageEvent-based coverage criteria– Event coverage (length-1)– Event-in...
3.2.5 Event instrumenterAll event sequences that are executed onthe GUI be collected.The key Idea: To detect the existingl...
35
3.2.6 Test executorExecuting an entire test suiteautomatically on the AUT.– perform all the events in each test caseWhat p...
Possibilities available to the test designer for level            of detail of oracle information.             10 differen...
4. ExperimentsThe subject applications: TerpOffice  http://www.cs.umd.edu/~atif/newsite/terpoffice.htm                    ...
39
40
41
42
43
44
4.1 ResultsSpace RequirementsTime requirementsCode coverage                        45
Space requirements       500mb                46
Time requirementsAll times are reported for a 2.2 GHzPentium 4 machine with 256 MB of RAM.The results show that the smoke ...
TerpPaint                 51.9 Hours            48
TerpPad           4.7 Hours          49
TerpCalc                11 Hours           50
TerpSpreadSheet                       18.6 Hours                  51
TerpDraw                8 Hours           52
TerpManager                   2.7 Hours              53
Code coverageWe instrumented the application beforerunning all of the smoke test cases.We recorded the statements that wer...
TerpPaint            55
TerpPad          56
TerpCalc           57
TerpSpreadSheet                  58
TerpDraw           59
TerpManager              60
5. ConclusionsWe define a formal model of a GUIderived from specifications that isuseful for smoke testing.We develop a ne...
In the futureWe will study the effectiveness of the DARTprocess by analyzing the number of faultsdetected.We will also int...
Upcoming SlideShare
Loading in …5
×

20051019 automating regression testing for evolving gui software

948 views
800 views

Published on

reference "Automating Regression Testing For Evolving GUI Software"

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
948
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • 對 GUI 的一種 modeling ,是不是一種 specification ??
  • TerpCalc - a calculator program TerpManager- an explorer program for TerpOffice Suite, customizable to any set of programs TerpPaint - a paint/drawing program TerpPresent - a presentation program TerpSpreadsheet - a spreadsheet program TerpWord - a word processor/html editor
  • 20051019 automating regression testing for evolving gui software

    1. 1. Automating Regression Testing For Evolving GUI Software Zheng-Wen Shen 2005/10/19 1
    2. 2. ReferencesAutomating regression testing for evolvingGUI software– Atif Memon, Adithya Nagarajan and Qing Xie– JOURNAL OF SOFTWARE MAINTENANCE AND EVOLUTION: RESEARCH AND PRACTICE 2005; 17:27–64. 2
    3. 3. Outline1. Introduction2. The DART process3. Design of DART– 3.1. GUI Representation– 3.2 Modules of DART4. Experiments5. Conclusions 3
    4. 4. 1. IntroductionToday’s competitive software developmentmarket demands that several developers,who are perhaps geographicallydistributed, work simultaneously on largeparts of the code during maintenance.Rapid quality assurance feedback– nightly/daily building and smoke testing.– To build the product and test it every day 4
    5. 5. DART(Daily Automated Regression Tester) A new framework that address the needs of re- testing frequent builds of GUI software. Automation!!– structural GUI analysis– test-case generation– test-oracle creation– code instrumentation to test execution coverage evaluation, regeneration of test cases Test cases re-execution. 5
    6. 6. 2. The DART Process 6
    7. 7. 7
    8. 8. Roles of the developer/tester and DARTMaintenance cycle 8
    9. 9. M M’ 9
    10. 10. 3. Design of DART Driving philosophy behind the design 1. Automated 2. Efficient 3. Robust 4. Portable 5. General 10
    11. 11. Runs all test casesExpected results How much testing “glue” Coverageinformation Reverse EngineeringTest Cases 11
    12. 12. 3.1 GUI RepresentationA formal model of the AUT’s GUI GUI Representation 1 Objects & Properties 2 Events 3 Components 4 Event-flow Graphs 5 Integration Tree 12
    13. 13. 3.1.1 Objects and propertiesobjects O = {o1, o2, . . . ,om}properties P = {p1, p2, . . . ,pl }A GUI = O + PThe state of a GUI = a set P at time t.valid initial state set = iff the GUI may be inany state Si ∈ SI when it is first invoked. 13
    14. 14. 3.1.2 EventsThe events E = {e1, e2, . . . , en} associatedwith a GUI are functions from one state toanother state of the GUI.legal event sequence = e1;e2;e3; . . . ; en 14
    15. 15. 3.1.3 ComponentA GUI component C is an ordered pair (RF, UF)– A modal window + a set of modeless windowsRF represents a modal window in terms of itsevents.UF is a set whose elements represent modelesswindows also in terms of their events.Each element of UF is invoked either by anevent in UF or RF. 15
    16. 16. 3.1.4 Event-flow graphsAll possible interactions among theevents in a component.An event-flow graph for a component C isa 4-tuple <V, E, B, I> where:– V : all the events in the component– E ⊆ V × V : directed edges– B ⊆ V : available to the user when the component is first invoked– I ⊆ V : invoke other components. 16
    17. 17. EFG for part of MS WordPad.“main” component 17
    18. 18. 3.1.5 Integration Tree (IT)To identify interactions amongcomponents.Invocations – Component Cx invokes component Cy iff Cx contains an event ex that invokes Cy . 18
    19. 19. Definition of ITAn IT is a triple <N, R, B>– N : components in the GUI.– R ∈ N: a designated component called the Main component.– B : directed edges , the invokes relation between components 19
    20. 20. IT for part of MS WordPad. 20
    21. 21. 3.1.6 Event Classification1. Restricted-focus events2. Unrestricted-focus events3. Termination events4. Menu-open events5. System-interaction events 21
    22. 22. 1 Restricted-focus event open modal windows 22
    23. 23. 2 Unrestricted-focus events open modeless windows 23
    24. 24. 3 Termination events close modal windows 24
    25. 25. 4 Menu-open events Open menus 25
    26. 26. 5 System-interaction eventsInteract with the underlying softwareto perform some action 26
    27. 27. 3.2 Modules of DART 2 1 Test Case Generator 3 GUI Ripper Test Oracle Generator GUI Representation 4 6 5 Test ExecutorCoverage Evaluator Code/Event Instrumenter 27
    28. 28. 3.2.1 GUI RipperStep1: GUI traversal and extraction (DFS)Step2: Manual inspection by TesterStep3: Generating the event-flow graph and Integration Tree 28
    29. 29. 3.2.2 Test-case generatorA GUI test case T is a pair <S0, e1; e2; . . . ;e n>Generate test case by traversing theevent-flow graphs 29
    30. 30. 3.2.3 Test-oracle generatorTest oracles are used to determinewhether or not the software executedcorrectly during testing.<S0, e1; e2; . . . ; en>  S1;S2;…;Sn expected state or not 30
    31. 31. Approach: execution extractionA test case is executed on an existing,presumably correct version of the softwareand its state is extracted and stored asoracle information. 31
    32. 32. Level of Oracle Information (LOI)Complete (LO1)– {(w, p, o), ∀w ∈ Windows, ∀o = objects ∈ w, ∀p = properties ∈ o}Complete visible (LO2)– {(w, p, o), ∀(w ∈ Windows)&(w is visible), ∀o = objects ∈ w, ∀p = properties ∈ o}Active window (LO3)– {(w, p, o), (w = active Window), ∀o = objects ∈ w, ∀p = properties ∈ o},Widget (LO4)– {(w, p, o), (w = active Window), o = current object, ∀p = properties ∈ o} 32
    33. 33. 3.2.4 Coverage evaluatorCode coverage and event coverageEvent-based coverage criteria– Event coverage (length-1)– Event-interaction coverage (length-2)– Length-n event-sequence coverage 33
    34. 34. 3.2.5 Event instrumenterAll event sequences that are executed onthe GUI be collected.The key Idea: To detect the existinglisteners and attach our own listeners. 34
    35. 35. 35
    36. 36. 3.2.6 Test executorExecuting an entire test suiteautomatically on the AUT.– perform all the events in each test caseWhat properties should be compared?– level of testing (LOT1 - LOT4)– The test designer may choose to employ partial oracle information. 36
    37. 37. Possibilities available to the test designer for level of detail of oracle information. 10 different combinations 37
    38. 38. 4. ExperimentsThe subject applications: TerpOffice http://www.cs.umd.edu/~atif/newsite/terpoffice.htm 38
    39. 39. 39
    40. 40. 40
    41. 41. 41
    42. 42. 42
    43. 43. 43
    44. 44. 44
    45. 45. 4.1 ResultsSpace RequirementsTime requirementsCode coverage 45
    46. 46. Space requirements 500mb 46
    47. 47. Time requirementsAll times are reported for a 2.2 GHzPentium 4 machine with 256 MB of RAM.The results show that the smoke testingprocess is practical, in that it can beperformed in one night. 47
    48. 48. TerpPaint 51.9 Hours 48
    49. 49. TerpPad 4.7 Hours 49
    50. 50. TerpCalc 11 Hours 50
    51. 51. TerpSpreadSheet 18.6 Hours 51
    52. 52. TerpDraw 8 Hours 52
    53. 53. TerpManager 2.7 Hours 53
    54. 54. Code coverageWe instrumented the application beforerunning all of the smoke test cases.We recorded the statements that wereexecuted for each user-implemented classduring test-case execution. 54
    55. 55. TerpPaint 55
    56. 56. TerpPad 56
    57. 57. TerpCalc 57
    58. 58. TerpSpreadSheet 58
    59. 59. TerpDraw 59
    60. 60. TerpManager 60
    61. 61. 5. ConclusionsWe define a formal model of a GUIderived from specifications that isuseful for smoke testing.We develop a new process for re-testing nightly builds of GUI software.Our regression testing processcannot only be used for nightly builds,but also for general GUI re-testing. 61
    62. 62. In the futureWe will study the effectiveness of the DARTprocess by analyzing the number of faultsdetected.We will also integrate DART in a higher-levelprocess that involves executing other types non-GUI) of smoke tests on the software.We will also investigate the application of DARTto other software systems that take events asinput. (e.g. Web applications) 62

    ×