0
Automated ExploratoryTests for Web ApplicationZbyszek Moćkun                   © 2010 Cognifide Limited. In commercial con...
ObjectiveHow Cognifide has developed automatedtests and where we are going                             © 2010 Cognifide Li...
IntroductionIn short about process and environment                               © 2010 Cognifide Limited. In commercial c...
Methodologies, projects …  Methodologies                   Projects                              • Web application• Agile:...
Common standards forautomationFunctional tests                   © 2010 Cognifide Limited. In commercial confidence only.
Test Automation Lifecycle                    Write TC          Write        automated          script                     ...
Advantages and disadvantages                             • Time consuming                             • Cost • Coverage by...
Can you believe your tests?• Incorrect Xpath (locator) used in tests• Not all data were verified (unable to write verifica...
Migration (upgrade) projects• No existing automated test for regression test• Few thousands of pages• Quick response neede...
Automated ExploratoryTestsNew approach, theory part                            © 2010 Cognifide Limited. In commercial con...
We need automated testsbecause ...                Source: http://www.testfacebook.com                                     ...
What should we test?         Two perspectives:      User              Client                                 © 2010 Cognif...
AspectsCI Environment Client              User  Analytics                 Layout   Functionality   Performance            ...
Looking for a symptom„Symptom is a departure from normal function or feelingwhich is noticed by a patient, indicating the ...
Symptoms                                 Layout                   Performance               Content    URL strategy       ...
Oracle HeuristicsIt is used by comparing the output(s) of the system under test,for a given test case input, to the output...
Oracle Heuristic for GUI                           © 2010 Cognifide Limited. In commercial confidence only.
Reference symptoms• Previous version of application• Reference data saved on disc• Other version of the same application (...
Robots  • Scripts that are looking for symptoms  • The same script for all projects  • Open Source toolsJavaScript   Serve...
Exploratory testing„Exploratory testing is an approach to software testing that is concisely described as simultaneous lea...
What are Automated ExploratoryTests?Meny different definition:• Record and play manual exploratory test sessions• Combined...
What are Automated ExploratoryTests?„Tests supporting exploratory testing, not necessarily based on  the test scenario, se...
Automated Exploratory Tests -advantages• One tool −   Gather all symptoms −   Use in all projects −   Quick to introduce t...
Track the propagation ofchanges in the code- See what pages/components are infected by change- Positive – false (code chan...
Automated ExploratoryTestsPractical part                 © 2010 Cognifide Limited. In commercial confidence only.
Report – example of use                          © 2010 Cognifide Limited. In commercial confidence only.
URL management• Sitemap• Link crawler• CSV file with url list• Symptoms as parameter (on, off)• Comments (bug id, ...)• Hi...
Pattern management• Easy to use (overwrite pattern)• Checkbox (check all)• Commit to svn                                  ...
Dynamic content• Reference symptoms – need processing   • Use regexp   • Hide element   • Write generic function• Robots –...
Mobile version• Required (not nice to have), can’t avoid testing• Mobile pages are more static than desktop version• Do no...
Narzędzia• Selenium II/WebDriver• Komparatory (ImageMagick, Pretty Diff)• Walidatory (W3C)• Performance (JMeter, Proxy)• S...
Q&A      © 2010 Cognifide Limited. In commercial confidence only.
Upcoming SlideShare
Loading in...5
×

Automated Exploratory Tests

1,879

Published on

automated exploratory test tools for web application. Make your test agile. Do not overpay for your automation. Test all aspects (performance, layout, analytic) not only concentrate on one specific thing.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,879
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
21
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Automated Exploratory Tests"

  1. 1. Automated ExploratoryTests for Web ApplicationZbyszek Moćkun © 2010 Cognifide Limited. In commercial confidence only.
  2. 2. ObjectiveHow Cognifide has developed automatedtests and where we are going © 2010 Cognifide Limited. In commercial confidence only.
  3. 3. IntroductionIn short about process and environment © 2010 Cognifide Limited. In commercial confidence only.
  4. 4. Methodologies, projects … Methodologies Projects • Web application• Agile: SCRUM, Kanban • Migration• Frequently releases • Content Management• No time for manual Systems (CMS) regression • Short or medium term• Lot of work at the end of projects sprint • Automation investment, no• CI Environment time for money return © 2010 Cognifide Limited. In commercial confidence only.
  5. 5. Common standards forautomationFunctional tests © 2010 Cognifide Limited. In commercial confidence only.
  6. 6. Test Automation Lifecycle Write TC Write automated script Run Manage Update © 2010 Cognifide Limited. In commercial confidence only.
  7. 7. Advantages and disadvantages • Time consuming • Cost • Coverage by automated tests • Invest and wait for return • No duplications during • Test dependency regression test • Only coded functionality is • Functionality is tested tested © 2010 Cognifide Limited. In commercial confidence only.
  8. 8. Can you believe your tests?• Incorrect Xpath (locator) used in tests• Not all data were verified (unable to write verifications for all data)• CSS/images or other files wasn’t load• Frequently design changes require xpath (locator) update• Incorrect CSS fixes (designs are not reflected by page layout)• Analytics doesn’t work• Performance decrease• Few thousands of pages (components variation – can’t test all of them © 2010 Cognifide Limited. In commercial confidence only.
  9. 9. Migration (upgrade) projects• No existing automated test for regression test• Few thousands of pages• Quick response needed• Iterative development © 2010 Cognifide Limited. In commercial confidence only.
  10. 10. Automated ExploratoryTestsNew approach, theory part © 2010 Cognifide Limited. In commercial confidence only.
  11. 11. We need automated testsbecause ... Source: http://www.testfacebook.com © 2010 Cognifide Limited. In commercial confidence only.
  12. 12. What should we test? Two perspectives: User Client © 2010 Cognifide Limited. In commercial confidence only.
  13. 13. AspectsCI Environment Client User Analytics Layout Functionality Performance © 2010 Cognifide Limited. In commercial confidence only.
  14. 14. Looking for a symptom„Symptom is a departure from normal function or feelingwhich is noticed by a patient, indicating the presence of disease or abnormality. A symptom is subjective, observed by the patient, and not measured.” http://wikipedia.org © 2010 Cognifide Limited. In commercial confidence only.
  15. 15. Symptoms Layout Performance Content URL strategy HTML code Bug JavaScript Link checker errors Server Validators response (W3C) (404) Analytics © 2010 Cognifide Limited. In commercial confidence only.
  16. 16. Oracle HeuristicsIt is used by comparing the output(s) of the system under test,for a given test case input, to the outputs that the oracledetermines that product should have.− other products (second program that uses a different algorithm to evaluate the same mathematical expression as the product under test)− a consistency oracle that compares the results of one test execution to another for similarity,− a model-based oracle that uses the same model to generate and verify system behavior,− or a human beings judgment (i.e. does the program "seem" to the user to do the correct thing?) © 2010 Cognifide Limited. In commercial confidence only.
  17. 17. Oracle Heuristic for GUI © 2010 Cognifide Limited. In commercial confidence only.
  18. 18. Reference symptoms• Previous version of application• Reference data saved on disc• Other version of the same application (different servers)• The same script takes pattern and compare• One tool for different projects Performance Layout Content HTML code monitoring © 2010 Cognifide Limited. In commercial confidence only.
  19. 19. Robots • Scripts that are looking for symptoms • The same script for all projects • Open Source toolsJavaScript Server response Validators Analytics Link checker URL Strategy errors (404) (HTLM, CSS) © 2010 Cognifide Limited. In commercial confidence only.
  20. 20. Exploratory testing„Exploratory testing is an approach to software testing that is concisely described as simultaneous learning, test design, test execution and test result interpretation” Cem Kaner © 2010 Cognifide Limited. In commercial confidence only.
  21. 21. What are Automated ExploratoryTests?Meny different definition:• Record and play manual exploratory test sessions• Combined manual and automated tests• Test steps automation• Blackbox tests with input/output data generator• Not at all possible due to human factor © 2010 Cognifide Limited. In commercial confidence only.
  22. 22. What are Automated ExploratoryTests?„Tests supporting exploratory testing, not necessarily based on the test scenario, searching areas where there may be errors or a change in the application, allowing the examination of applications faster and more deeply or not to test sites available to man.” Zbyszek Moćkun„Automated Tests do not provide complete information. They provide partial information that might be useful” Cem Kaner © 2010 Cognifide Limited. In commercial confidence only.
  23. 23. Automated Exploratory Tests -advantages• One tool − Gather all symptoms − Use in all projects − Quick to introduce to project (half a day) − Resistant to appliciaton changes• Do not require deep knowledge about application, support the learning process• Integrated with Continuous Intergration env• Strong solution if combined with automated functional tests• Migration or upgrade tests• Iterative development © 2010 Cognifide Limited. In commercial confidence only.
  24. 24. Track the propagation ofchanges in the code- See what pages/components are infected by change- Positive – false (code change affect all necessary places?) © 2010 Cognifide Limited. In commercial confidence only.
  25. 25. Automated ExploratoryTestsPractical part © 2010 Cognifide Limited. In commercial confidence only.
  26. 26. Report – example of use © 2010 Cognifide Limited. In commercial confidence only.
  27. 27. URL management• Sitemap• Link crawler• CSV file with url list• Symptoms as parameter (on, off)• Comments (bug id, ...)• Hide element• Other actions• Keep in SVN © 2010 Cognifide Limited. In commercial confidence only.
  28. 28. Pattern management• Easy to use (overwrite pattern)• Checkbox (check all)• Commit to svn © 2010 Cognifide Limited. In commercial confidence only.
  29. 29. Dynamic content• Reference symptoms – need processing • Use regexp • Hide element • Write generic function• Robots – do not need any additional actions• Dynamic components (counters, date, random numbers)• Flash/video files• Embedded content (twitter, facebook,...)• Require functional test• Static components – about 75% © 2010 Cognifide Limited. In commercial confidence only.
  30. 30. Mobile version• Required (not nice to have), can’t avoid testing• Mobile pages are more static than desktop version• Do not use flash• Not all symptoms are easy to check © 2010 Cognifide Limited. In commercial confidence only.
  31. 31. Narzędzia• Selenium II/WebDriver• Komparatory (ImageMagick, Pretty Diff)• Walidatory (W3C)• Performance (JMeter, Proxy)• Serwery Proxy (BrowserMob)• Firefox/Firebug (NetExport, ConsoleExport, JS Console output redirector)• HAR Viewer (http://www.softwareishard.com/har/viewer/) © 2010 Cognifide Limited. In commercial confidence only.
  32. 32. Q&A © 2010 Cognifide Limited. In commercial confidence only.
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×