Automated Model-Based Testing of Community-Driven Open-Source GUI      Applications       Zheng-Wen Shen         2006/07/1...
Reference• “Automated Model-Based Testing of  Community-Driven Open-Source GUI  Applications”  – Qing Xie and Atif Memon  ...
Outline• 1.   Introduction• 2.   Testing Loops• 3.   Overview of GUI Model• 4.   Experiment• 5.   Conclusions             ...
1. Introduction            (1/3)• Open-source software (OSS) development  on world-wide web. (WWW)  – Communities of progr...
Problems on OSS              (2/3)• Little direct inter-developer  communication  – CVS commit log message, bug reports,  ...
Key idea         (3/3) • To create concentric testing loopsSpecific GUI testing goals,   Comprehensive GUI testingresource...
2. Testing Loops• CR Tools Test cases  fragile  – Input event sequence can no longer    execute on the GUI  – The expecte...
8
3. GUI Model              (1/5)• An event-flow graph (EFG) model  represents all possible event sequences  that may be exe...
Event-Interaction Graph                (2/5)• Test interactions between loosely-coupled  parts of an OSS• System-interacti...
Path in an EFG and EIG   (3/5)                   11
Test Case Generation                (4/5)1. Test cases are short  generate and   execute very quickly2. Only consists of ...
Test Oracle Creation                    (5/5)• Oracles for Crash Tests  – Crashes during test execution may be used to    ...
4. Experiment• Do popular web based community-  driven GUI–based OSS have  problems that can be detected by  our automated...
Notice• Execute fully-automatic crash testing  process on applications and report  problems  – A crash  An uncaught excep...
Subject Applications•   FreeMind, a premier mind-mapping software    –   0.0.2, 0.1.0, 0.4, 0.7.1, 0.8.0RC5, 0.8.0•   Gant...
FreeMind Bugs1. NullPointerException when trying to open a non-existent file (0.0.2, 0.1.0)2. FileNotFoundException when t...
Gantt Project Bugs1. NumberFormatException when setting non-numeric inputs while   expecting a number in “New task” window...
JMSN Bugs1. InvocationTargetException when trying to   refresh the buddy list (0.9a, 0.9.2)2. FileNotFoundException when t...
Crossword Sage Bugs• NullPointerException in Crossword Builder when  trying to delete a word (0.3.0, 0.3.1)• NullPointerEx...
4.1 Results              21
Results• The some bug existed across  applications  shared open-source  GUI components (FileSave)   sanitize inputs• Man...
the reasons for crash1. Invalid text input: validity, size2. Widget enabled when it should be   disabled3. Object declared...
5. Conclusions             (1/2)• Recognition the nature of the WWW  – enables the separation of GUI testing steps by    l...
Conclusions           (2/2)• A more detailed study of the overall  benefits of this technique• Extended subject applicatio...
Upcoming SlideShare
Loading in …5
×

20060712 automated model based testing of community-driven open-source gui applications

326 views

Published on

reference: Automated Model-Based Testing of Community-Driven Open-Source GUI Applications

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
326
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

20060712 automated model based testing of community-driven open-source gui applications

  1. 1. Automated Model-Based Testing of Community-Driven Open-Source GUI Applications Zheng-Wen Shen 2006/07/12 1
  2. 2. Reference• “Automated Model-Based Testing of Community-Driven Open-Source GUI Applications” – Qing Xie and Atif Memon – 22nd International Conference on Software Maintenance (ICSM 2006), Philadelphia, PA, USA, Sep. 25-27, 2006. 2
  3. 3. Outline• 1. Introduction• 2. Testing Loops• 3. Overview of GUI Model• 4. Experiment• 5. Conclusions 3
  4. 4. 1. Introduction (1/3)• Open-source software (OSS) development on world-wide web. (WWW) – Communities of programmers distributed world-wide – Unprecedented OSS code churn rates 4
  5. 5. Problems on OSS (2/3)• Little direct inter-developer communication – CVS commit log message, bug reports, change-requests, and comments• Developers work on loosely coupled parts of the application code. – Local change has inadvertently broken other parts of the overall software code 5
  6. 6. Key idea (3/3) • To create concentric testing loopsSpecific GUI testing goals, Comprehensive GUI testingresource usage, and targetdeveloper sub-groups for Smoke testingerror reports Crash testing 6
  7. 7. 2. Testing Loops• CR Tools Test cases  fragile – Input event sequence can no longer execute on the GUI – The expected output stored with the test case becomes obsolete.• Model-based techniques – Generate and maintain test cases automatically during OSS evolution – Employs GUI models to generate test cases. 7
  8. 8. 8
  9. 9. 3. GUI Model (1/5)• An event-flow graph (EFG) model represents all possible event sequences that may be executed on a GUI. – All executable paths of the software• Microsoft Word (total 4210 events) – 80 events open menus, 346 events open windows, 196 events close windows, and the remaining 3588 events interact with the underlying code. 9
  10. 10. Event-Interaction Graph (2/5)• Test interactions between loosely-coupled parts of an OSS• System-interaction events = Non- structural events + close windows events• Test cases consist of event-flow-paths that start and end with system-interaction events, without any intermediate system- interaction events. 10
  11. 11. Path in an EFG and EIG (3/5) 11
  12. 12. Test Case Generation (4/5)1. Test cases are short  generate and execute very quickly2. Only consists of system-interaction events3. The expected state is stored only for system-interaction events4. All system-interaction events are executed; most of the GUI’s functionality is covered.5. Each test case is independent and the suite can be distributed. 12
  13. 13. Test Oracle Creation (5/5)• Oracles for Crash Tests – Crashes during test execution may be used to identify serious problems in the software.• Oracles for Smoke Tests – Software does what it was doing before modifications – Reference Testing• Oracles for Comprehensive Testing – Specifications-based approach – Precondition + Effect 13
  14. 14. 4. Experiment• Do popular web based community- driven GUI–based OSS have problems that can be detected by our automated techniques?• Do these problems persist across multiple versions of the OSS? 14
  15. 15. Notice• Execute fully-automatic crash testing process on applications and report problems – A crash  An uncaught exception• Determine how long these problems have been in the application code• We overall process executed without any human intervention in 5-8 hours. 15
  16. 16. Subject Applications• FreeMind, a premier mind-mapping software – 0.0.2, 0.1.0, 0.4, 0.7.1, 0.8.0RC5, 0.8.0• Gantt Project, a project scheduling application – 1.6, 19.11, 1.10.3, 1.11, 1.11.1, 2.pre1• JMSN, a pure java Microsoft MSN messenger clone – 0.9a, 0.9.2, 0.9.7, 0.9.8b7, 0.9.9b1• Crossword Sage, a tool for creating (and solving) professional looking crosswords with powerful word suggestion capabilities – 0.1, 0.2, 0.3.0, 0.3.1, 0.3.2, and 0.3.5 16
  17. 17. FreeMind Bugs1. NullPointerException when trying to open a non-existent file (0.0.2, 0.1.0)2. FileNotFoundException when trying to save a file with a very long le name (0.0.2, 0.1.0, 0.4)3. NullPointerException when clicking on some buttons on the main toolbar when no le is open (0.1.0);4. NullPointerException when clicking on some menu items if no le is open (0.1.0, 0.4, 0.7.1, 0.8.0RC5)5. NullPointerException when trying to save a “blank” file (0.1.0)6. NullPointerException when adding a new node after toggling folded node (0.4)7. FileNotFoundException when trying to import a nonexistent file (0.4, 0.7.1, 0.8.0RC5, 0.8.0)8. FileNotFoundException when trying to export a file with a very long le name (0.7.1, 0.8.0RC5, 0.8.0)9. NullPointerException when trying to split a node in “Edit a long node” window (0.7.1, 0.8.0RC5, 0.8.0)10.NumberFormatException when setting non-numeric input while expecting a number in “preferences setting” window (0.8.0RC5, 0.8.0) 17
  18. 18. Gantt Project Bugs1. NumberFormatException when setting non-numeric inputs while expecting a number in “New task” window (1.6)2. FileNotFoundException when trying to open a nonexistent file (1.6)3. FileNotFoundException when trying to save a file with a very long le name (1.6, 1.9.11, 1.10.3, 1.11, 1.11.1, 2.pre1)4. NullPointerException after confirming any preferences setting (1.9.11)5. NullPointerException when trying to save the content to a server (1.9.11)6. NullPointerException when trying to import a nonexistent file (1.9.11, 1.10.3, 1.11, 1.11.1, 2.pre1)7. InterruptedException when trying to open a new window (1.10.3)8. Runtime error when trying to send e-mail (1.11, 1.11.1, 2.pre1) 18
  19. 19. JMSN Bugs1. InvocationTargetException when trying to refresh the buddy list (0.9a, 0.9.2)2. FileNotFoundException when trying to submit a bug/request report because the submission page doesnt exist (0.9a, 0.9.2, 0.9.5, 0.9.7, 0.9.8b7, 0.9.9b2);3. NullPointerException when trying to check the validity of the login data (0.9.7, 0.9.8b7, 0.9.9b2)4. SocketException and NullPointerException when stopping a socket that has been started (0.9.8b7, 0.9.9b2) 19
  20. 20. Crossword Sage Bugs• NullPointerException in Crossword Builder when trying to delete a word (0.3.0, 0.3.1)• NullPointerException in Crossword Builder when trying to suggest a new word (0.3.0, 0.3.1, 0.3.2, 0.3.5)• NullPointerException in Crossword Builder when trying to write a clue for a word (0.3.0, 0.3.1, 0.3.2, 0.3.5)• NullPointerException when loading a new crossword file (0.3.5)• NullPointerException when splitting a word (0.3.5)• NullPointerException when publishing the crossword (0.3.5) 20
  21. 21. 4.1 Results 21
  22. 22. Results• The some bug existed across applications  shared open-source GUI components (FileSave)  sanitize inputs• Many bugs are persistent across versions• There are fewer bugs in the first version than in later versions  consistent with our experience 22
  23. 23. the reasons for crash1. Invalid text input: validity, size2. Widget enabled when it should be disabled3. Object declared but not initialized4. Obsolete external resources 23
  24. 24. 5. Conclusions (1/2)• Recognition the nature of the WWW – enables the separation of GUI testing steps by level of automation, feedback, and resource utilization.• Demonstration that resources may be better utilized by defining a concentric loop-based GUI testing approach.• Demonstration that popular GUI-based OSS developed on the WWW have flaws that can be detected by our fully automated approach. 24
  25. 25. Conclusions (2/2)• A more detailed study of the overall benefits of this technique• Extended subject application  test TerpOffice incrementally• Web application that have complex back- ends• The interaction between three test loops – Whether one loop can benefit from the execution of the inner loops – Study the need of additional loops 25

×