automated exploratory test tools for web application. Make your test agile. Do not overpay for your automation. Test all aspects (performance, layout, analytic) not only concentrate on one specific thing.

1. Automated Exploratory
Tests for Web Application
Zbyszek Moćkun
© 2010 Cognifide Limited. In commercial confidence only.

2. Objective
How Cognifide has developed automated
tests and where we are going
© 2010 Cognifide Limited. In commercial confidence only.

3. Introduction
In short about process and environment
© 2010 Cognifide Limited. In commercial confidence only.

4. Methodologies, projects …
Methodologies Projects
• Web application
• Agile: SCRUM, Kanban • Migration
• Frequently releases • Content Management
• No time for manual Systems (CMS)
regression • Short or medium term
• Lot of work at the end of projects
sprint • Automation investment, no
• CI Environment time for money return
© 2010 Cognifide Limited. In commercial confidence only.

5. Common standards for
automation
Functional tests
© 2010 Cognifide Limited. In commercial confidence only.

6. Test Automation Lifecycle
Write TC
Write
automated
script
Run
Manage
Update
© 2010 Cognifide Limited. In commercial confidence only.

7. Advantages and disadvantages
• Time consuming
• Cost
• Coverage by
automated tests • Invest and wait for return
• No duplications during • Test dependency
regression test • Only coded functionality is
• Functionality is tested tested
© 2010 Cognifide Limited. In commercial confidence only.

8. Can you believe your tests?
• Incorrect Xpath (locator) used in tests
• Not all data were verified (unable to write verifications for all
data)
• CSS/images or other files wasn’t load
• Frequently design changes require xpath (locator) update
• Incorrect CSS fixes (designs are not reflected by page
layout)
• Analytics doesn’t work
• Performance decrease
• Few thousands of pages (components variation – can’t test
all of them
© 2010 Cognifide Limited. In commercial confidence only.

9. Migration (upgrade) projects
• No existing automated test for regression test
• Few thousands of pages
• Quick response needed
• Iterative development
© 2010 Cognifide Limited. In commercial confidence only.

10. Automated Exploratory
Tests
New approach, theory part
© 2010 Cognifide Limited. In commercial confidence only.

11. We need automated tests
because ...
Source: http://www.testfacebook.com
© 2010 Cognifide Limited. In commercial confidence only.

12. What should we test?
Two perspectives:
User Client
© 2010 Cognifide Limited. In commercial confidence only.

13. Aspects
CI Environment
Client
User
Analytics
Layout Functionality Performance
© 2010 Cognifide Limited. In commercial confidence only.

14. Looking for a symptom
„Symptom is a departure from normal function or feeling
which is noticed by a patient, indicating the presence of
disease or abnormality. A symptom is subjective,
observed by the patient, and not measured.”
http://wikipedia.org
© 2010 Cognifide Limited. In commercial confidence only.

15. Symptoms
Layout
Performance Content
URL strategy HTML code
Bug
JavaScript
Link checker
errors
Server
Validators
response
(W3C)
(404)
Analytics
© 2010 Cognifide Limited. In commercial confidence only.

16. Oracle Heuristics
It is used by comparing the output(s) of the system under test,
for a given test case input, to the outputs that the oracle
determines that product should have.
− other products (second program that uses a different algorithm to
evaluate the same mathematical expression as the product
under test)
− a consistency oracle that compares the results of one test
execution to another for similarity,
− a model-based oracle that uses the same model to generate and
verify system behavior,
− or a human being's judgment (i.e. does the program "seem" to
the user to do the correct thing?)
© 2010 Cognifide Limited. In commercial confidence only.

17. Oracle Heuristic for GUI
© 2010 Cognifide Limited. In commercial confidence only.

18. Reference symptoms
• Previous version of application
• Reference data saved on disc
• Other version of the same application (different servers)
• The same script takes pattern and compare
• One tool for different projects
Performance
Layout Content HTML code
monitoring
© 2010 Cognifide Limited. In commercial confidence only.

19. Robots
• Scripts that are looking for symptoms
• The same script for all projects
• Open Source tools
JavaScript Server response Validators
Analytics Link checker URL Strategy
errors (404) (HTLM, CSS)
© 2010 Cognifide Limited. In commercial confidence only.

20. Exploratory testing
„Exploratory testing is an approach to software testing
that is concisely described as simultaneous learning, test
design, test execution and test result interpretation”
Cem Kaner
© 2010 Cognifide Limited. In commercial confidence only.

21. What are Automated Exploratory
Tests?
Meny different definition:
• Record and play manual exploratory test sessions
• Combined manual and automated tests
• Test steps automation
• Blackbox tests with input/output data generator
• Not at all possible due to human factor
© 2010 Cognifide Limited. In commercial confidence only.

22. What are Automated Exploratory
Tests?
„Tests supporting exploratory testing, not necessarily based on
the test scenario, searching areas where there may be errors
or a change in the application, allowing the examination of
applications faster and more deeply or not to test sites
available to man.”
Zbyszek Moćkun
„Automated Tests do not provide complete information. They
provide partial information that might be useful”
Cem Kaner
© 2010 Cognifide Limited. In commercial confidence only.

23. Automated Exploratory Tests -
advantages
• One tool
− Gather all symptoms
− Use in all projects
− Quick to introduce to project (half a day)
− Resistant to appliciaton changes
• Do not require deep knowledge about application,
support the learning process
• Integrated with Continuous Intergration env
• Strong solution if combined with automated
functional tests
• Migration or upgrade tests
• Iterative development
© 2010 Cognifide Limited. In commercial confidence only.

24. Track the propagation of
changes in the code
- See what pages/components are infected by change
- Positive – false (code change affect all necessary places?)
© 2010 Cognifide Limited. In commercial confidence only.

25. Automated Exploratory
Tests
Practical part
© 2010 Cognifide Limited. In commercial confidence only.

26. Report – example of use
© 2010 Cognifide Limited. In commercial confidence only.

27. URL management
• Sitemap
• Link crawler
• CSV file with url list
• Symptoms as parameter (on, off)
• Comments (bug id, ...)
• Hide element
• Other actions
• Keep in SVN
© 2010 Cognifide Limited. In commercial confidence only.

28. Pattern management
• Easy to use (overwrite pattern)
• Checkbox (check all)
• Commit to svn
© 2010 Cognifide Limited. In commercial confidence only.

29. Dynamic content
• Reference symptoms – need processing
• Use regexp
• Hide element
• Write generic function
• Robots – do not need any additional actions
• Dynamic components (counters, date, random numbers)
• Flash/video files
• Embedded content (twitter, facebook,...)
• Require functional test
• Static components – about 75%
© 2010 Cognifide Limited. In commercial confidence only.

30. Mobile version
• Required (not nice to have), can’t avoid testing
• Mobile pages are more static than desktop version
• Do not use flash
• Not all symptoms are easy to check
© 2010 Cognifide Limited. In commercial confidence only.

31. Narzędzia
• Selenium II/WebDriver
• Komparatory (ImageMagick, Pretty Diff)
• Walidatory (W3C)
• Performance (JMeter, Proxy)
• Serwery Proxy (BrowserMob)
• Firefox/Firebug (NetExport, ConsoleExport, JS Console
output redirector)
• HAR Viewer (http://www.softwareishard.com/har/viewer/)
© 2010 Cognifide Limited. In commercial confidence only.

32. Q&A
© 2010 Cognifide Limited. In commercial confidence only.