SlideShare a Scribd company logo
1 of 32
Download to read offline
Automated Exploratory
Tests for Web Application
Zbyszek Moćkun

                   © 2010 Cognifide Limited. In commercial confidence only.
Objective

How Cognifide has developed automated
tests and where we are going




                             © 2010 Cognifide Limited. In commercial confidence only.
Introduction

In short about process and environment




                               © 2010 Cognifide Limited. In commercial confidence only.
Methodologies, projects …
  Methodologies                   Projects

                              • Web application
• Agile: SCRUM, Kanban        • Migration
• Frequently releases         • Content Management
• No time for manual          Systems (CMS)
  regression                  • Short or medium term
• Lot of work at the end of   projects
  sprint                      • Automation investment, no
• CI Environment              time for money return




                                                  © 2010 Cognifide Limited. In commercial confidence only.
Common standards for
automation
Functional tests




                   © 2010 Cognifide Limited. In commercial confidence only.
Test Automation Lifecycle


                    Write TC




          Write
        automated
          script

                            Run
                           Manage
                           Update


                                    © 2010 Cognifide Limited. In commercial confidence only.
Advantages and disadvantages



                             • Time consuming
                             • Cost
 • Coverage by
   automated tests           • Invest and wait for return
 • No duplications during    • Test dependency
   regression test           • Only coded functionality is
 • Functionality is tested     tested




                                                 © 2010 Cognifide Limited. In commercial confidence only.
Can you believe your tests?

• Incorrect Xpath (locator) used in tests
• Not all data were verified (unable to write verifications for all
  data)
• CSS/images or other files wasn’t load
• Frequently design changes require xpath (locator) update
• Incorrect CSS fixes (designs are not reflected by page
  layout)
• Analytics doesn’t work
• Performance decrease
• Few thousands of pages (components variation – can’t test
  all of them


                                                        © 2010 Cognifide Limited. In commercial confidence only.
Migration (upgrade) projects

• No existing automated test for regression test

• Few thousands of pages
• Quick response needed
• Iterative development




                                                   © 2010 Cognifide Limited. In commercial confidence only.
Automated Exploratory
Tests
New approach, theory part




                            © 2010 Cognifide Limited. In commercial confidence only.
We need automated tests
because ...




                Source: http://www.testfacebook.com

                                        © 2010 Cognifide Limited. In commercial confidence only.
What should we test?

         Two perspectives:




      User              Client




                                 © 2010 Cognifide Limited. In commercial confidence only.
Aspects


CI Environment

 Client

              User
  Analytics
                 Layout   Functionality   Performance




                                               © 2010 Cognifide Limited. In commercial confidence only.
Looking for a symptom



„Symptom is a departure from normal function or feeling
which is noticed by a patient, indicating the presence of
   disease or abnormality. A symptom is subjective,
     observed by the patient, and not measured.”
                                           http://wikipedia.org




                                                 © 2010 Cognifide Limited. In commercial confidence only.
Symptoms
                                 Layout


                   Performance               Content




    URL strategy                                        HTML code




                                 Bug
                                                        JavaScript
    Link checker
                                                          errors




                                              Server
                    Validators
                                             response
                     (W3C)
                                               (404)

                                 Analytics




                                                                     © 2010 Cognifide Limited. In commercial confidence only.
Oracle Heuristics

It is used by comparing the output(s) of the system under test,
for a given test case input, to the outputs that the oracle
determines that product should have.
− other products (second program that uses a different algorithm to
  evaluate the same mathematical expression as the product
  under test)
− a consistency oracle that compares the results of one test
  execution to another for similarity,
− a model-based oracle that uses the same model to generate and
  verify system behavior,
− or a human being's judgment (i.e. does the program "seem" to
  the user to do the correct thing?)



                                                         © 2010 Cognifide Limited. In commercial confidence only.
Oracle Heuristic for GUI




                           © 2010 Cognifide Limited. In commercial confidence only.
Reference symptoms

• Previous version of application
• Reference data saved on disc
• Other version of the same application (different servers)
• The same script takes pattern and compare
• One tool for different projects


                                              Performance
          Layout      Content     HTML code
                                               monitoring




                                                       © 2010 Cognifide Limited. In commercial confidence only.
Robots


  • Scripts that are looking for symptoms
  • The same script for all projects
  • Open Source tools




JavaScript   Server response    Validators
                                             Analytics   Link checker                    URL Strategy
  errors          (404)        (HTLM, CSS)




                                                                        © 2010 Cognifide Limited. In commercial confidence only.
Exploratory testing

„Exploratory testing is an approach to software testing
 that is concisely described as simultaneous learning, test
 design, test execution and test result interpretation”



                                       Cem Kaner




                                                 © 2010 Cognifide Limited. In commercial confidence only.
What are Automated Exploratory
Tests?
Meny different definition:
• Record and play manual exploratory test sessions
• Combined manual and automated tests
• Test steps automation
• Blackbox tests with input/output data generator
• Not at all possible due to human factor




                                                     © 2010 Cognifide Limited. In commercial confidence only.
What are Automated Exploratory
Tests?

„Tests supporting exploratory testing, not necessarily based on
  the test scenario, searching areas where there may be errors
  or a change in the application, allowing the examination of
  applications faster and more deeply or not to test sites
  available to man.”
                                             Zbyszek Moćkun



„Automated Tests do not provide complete information. They
  provide partial information that might be useful”
                                                   Cem Kaner

                                                    © 2010 Cognifide Limited. In commercial confidence only.
Automated Exploratory Tests -
advantages
• One tool
 −   Gather all symptoms
 −   Use in all projects
 −   Quick to introduce to project (half a day)
 −   Resistant to appliciaton changes
• Do not require deep knowledge about application,
  support the learning process
• Integrated with Continuous Intergration env
• Strong solution if combined with automated
  functional tests
• Migration or upgrade tests
• Iterative development


                                                     © 2010 Cognifide Limited. In commercial confidence only.
Track the propagation of
changes in the code
- See what pages/components are infected by change
- Positive – false (code change affect all necessary places?)




                                                    © 2010 Cognifide Limited. In commercial confidence only.
Automated Exploratory
Tests
Practical part




                 © 2010 Cognifide Limited. In commercial confidence only.
Report – example of use




                          © 2010 Cognifide Limited. In commercial confidence only.
URL management

• Sitemap
• Link crawler
• CSV file with url list
• Symptoms as parameter (on, off)
• Comments (bug id, ...)
• Hide element
• Other actions
• Keep in SVN




                                    © 2010 Cognifide Limited. In commercial confidence only.
Pattern management

• Easy to use (overwrite pattern)
• Checkbox (check all)
• Commit to svn




                                    © 2010 Cognifide Limited. In commercial confidence only.
Dynamic content
• Reference symptoms – need processing
   • Use regexp
   • Hide element
   • Write generic function
• Robots – do not need any additional actions
• Dynamic components (counters, date, random numbers)
• Flash/video files
• Embedded content (twitter, facebook,...)
• Require functional test
• Static components – about 75%




                                                © 2010 Cognifide Limited. In commercial confidence only.
Mobile version


• Required (not nice to have), can’t avoid testing
• Mobile pages are more static than desktop version
• Do not use flash
• Not all symptoms are easy to check




                                                     © 2010 Cognifide Limited. In commercial confidence only.
Narzędzia

• Selenium II/WebDriver
• Komparatory (ImageMagick, Pretty Diff)
• Walidatory (W3C)
• Performance (JMeter, Proxy)
• Serwery Proxy (BrowserMob)
• Firefox/Firebug (NetExport, ConsoleExport, JS Console
  output redirector)
• HAR Viewer (http://www.softwareishard.com/har/viewer/)




                                                 © 2010 Cognifide Limited. In commercial confidence only.
Q&A




      © 2010 Cognifide Limited. In commercial confidence only.

More Related Content

Viewers also liked

Diarrhea:Myths and facts, Precaution
Diarrhea:Myths and facts, Precaution Diarrhea:Myths and facts, Precaution
Diarrhea:Myths and facts, Precaution
Wuzna Haroon
 
Energy Strategy Group_Report 2012 efficienza energetica
Energy Strategy Group_Report 2012 efficienza energeticaEnergy Strategy Group_Report 2012 efficienza energetica
Energy Strategy Group_Report 2012 efficienza energetica
Eugenio Bacile di Castiglione
 
Alta White Paper D2C eCommerce Case Study 2016
Alta White Paper D2C eCommerce Case Study 2016Alta White Paper D2C eCommerce Case Study 2016
Alta White Paper D2C eCommerce Case Study 2016
Patrick Nicholson
 

Viewers also liked (12)

Basics of Coding in Pediatrics Medical Billing
Basics of Coding in Pediatrics Medical BillingBasics of Coding in Pediatrics Medical Billing
Basics of Coding in Pediatrics Medical Billing
 
cathy resume
cathy resumecathy resume
cathy resume
 
Enterprise workspaces - Extending SAP NetWeaver Portal capabilities
Enterprise workspaces - Extending SAP NetWeaver Portal capabilities Enterprise workspaces - Extending SAP NetWeaver Portal capabilities
Enterprise workspaces - Extending SAP NetWeaver Portal capabilities
 
Information från Läkemedelsverket #5 2013
Information från Läkemedelsverket #5 2013Information från Läkemedelsverket #5 2013
Information från Läkemedelsverket #5 2013
 
Secure PIN Management How to Issue and Change PINs Securely over the Web
Secure PIN Management How to Issue and Change PINs Securely over the WebSecure PIN Management How to Issue and Change PINs Securely over the Web
Secure PIN Management How to Issue and Change PINs Securely over the Web
 
Diarrhea:Myths and facts, Precaution
Diarrhea:Myths and facts, Precaution Diarrhea:Myths and facts, Precaution
Diarrhea:Myths and facts, Precaution
 
Credit cards
Credit cardsCredit cards
Credit cards
 
mpx Replay, Expedite Your Catch-Up and C3 Workflow 2 of 2
mpx Replay, Expedite Your Catch-Up and C3 Workflow 2 of 2mpx Replay, Expedite Your Catch-Up and C3 Workflow 2 of 2
mpx Replay, Expedite Your Catch-Up and C3 Workflow 2 of 2
 
"15 Business Story Ideas to Jump on Now"
"15 Business Story Ideas to Jump on Now""15 Business Story Ideas to Jump on Now"
"15 Business Story Ideas to Jump on Now"
 
Context Based Authentication
Context Based AuthenticationContext Based Authentication
Context Based Authentication
 
Energy Strategy Group_Report 2012 efficienza energetica
Energy Strategy Group_Report 2012 efficienza energeticaEnergy Strategy Group_Report 2012 efficienza energetica
Energy Strategy Group_Report 2012 efficienza energetica
 
Alta White Paper D2C eCommerce Case Study 2016
Alta White Paper D2C eCommerce Case Study 2016Alta White Paper D2C eCommerce Case Study 2016
Alta White Paper D2C eCommerce Case Study 2016
 

Similar to Automated Exploratory Tests

Sucheta_kale_4.8years_QA
Sucheta_kale_4.8years_QASucheta_kale_4.8years_QA
Sucheta_kale_4.8years_QA
Sucheta Kale
 
Model-Based Testing for ALM Octane: Better tests, built faster
Model-Based Testing for ALM Octane: Better tests, built faster Model-Based Testing for ALM Octane: Better tests, built faster
Model-Based Testing for ALM Octane: Better tests, built faster
Curiosity Software Ireland
 

Similar to Automated Exploratory Tests (20)

Automation as a project - lessons learnt
Automation as a project - lessons learntAutomation as a project - lessons learnt
Automation as a project - lessons learnt
 
Managing Continuous Delivery of Mobile Apps - for the Enterprise
Managing Continuous Delivery of Mobile Apps - for the EnterpriseManaging Continuous Delivery of Mobile Apps - for the Enterprise
Managing Continuous Delivery of Mobile Apps - for the Enterprise
 
Live Webinar- Making Test Automation 10x Faster for Continuous Delivery- By R...
Live Webinar- Making Test Automation 10x Faster for Continuous Delivery- By R...Live Webinar- Making Test Automation 10x Faster for Continuous Delivery- By R...
Live Webinar- Making Test Automation 10x Faster for Continuous Delivery- By R...
 
Microsoft power point automation-opensourcetestingtools_matrix-1
Microsoft power point   automation-opensourcetestingtools_matrix-1Microsoft power point   automation-opensourcetestingtools_matrix-1
Microsoft power point automation-opensourcetestingtools_matrix-1
 
Microsoft power point automation-opensourcetestingtools_matrix-1
Microsoft power point   automation-opensourcetestingtools_matrix-1Microsoft power point   automation-opensourcetestingtools_matrix-1
Microsoft power point automation-opensourcetestingtools_matrix-1
 
Exploratory testing in practice, short story how approach influenced on strategy
Exploratory testing in practice, short story how approach influenced on strategyExploratory testing in practice, short story how approach influenced on strategy
Exploratory testing in practice, short story how approach influenced on strategy
 
Continuous Integration for Mobile App Testing
Continuous Integration for Mobile App TestingContinuous Integration for Mobile App Testing
Continuous Integration for Mobile App Testing
 
Continuous Integration for Mobile App Testing
Continuous Integration for Mobile App TestingContinuous Integration for Mobile App Testing
Continuous Integration for Mobile App Testing
 
Test automation lessons from WebSphere Application Server
Test automation lessons from WebSphere Application ServerTest automation lessons from WebSphere Application Server
Test automation lessons from WebSphere Application Server
 
Introductie Visual Studio ALM 2012
Introductie Visual Studio ALM 2012Introductie Visual Studio ALM 2012
Introductie Visual Studio ALM 2012
 
Introductie Visual Studio ALM 2012
Introductie Visual Studio ALM 2012Introductie Visual Studio ALM 2012
Introductie Visual Studio ALM 2012
 
Sucheta_kale_4.8years_QA
Sucheta_kale_4.8years_QASucheta_kale_4.8years_QA
Sucheta_kale_4.8years_QA
 
Jgayatri-QA-Profile
Jgayatri-QA-ProfileJgayatri-QA-Profile
Jgayatri-QA-Profile
 
Top 5 automation testing tools to gear up website development
Top 5 automation testing tools to gear up website developmentTop 5 automation testing tools to gear up website development
Top 5 automation testing tools to gear up website development
 
Resume
ResumeResume
Resume
 
Test Strategy
Test StrategyTest Strategy
Test Strategy
 
Using BladeRunnerJS to Build Front-End Apps that Scale - Fluent 2014
Using BladeRunnerJS to Build Front-End Apps that Scale - Fluent 2014Using BladeRunnerJS to Build Front-End Apps that Scale - Fluent 2014
Using BladeRunnerJS to Build Front-End Apps that Scale - Fluent 2014
 
Model-Based Testing for ALM Octane: Better tests, built faster
Model-Based Testing for ALM Octane: Better tests, built faster Model-Based Testing for ALM Octane: Better tests, built faster
Model-Based Testing for ALM Octane: Better tests, built faster
 
Neev QA Offering
Neev QA OfferingNeev QA Offering
Neev QA Offering
 
Between Scrum and Kanban - define a test process for Agile methodologies
Between Scrum and Kanban - define a test process for Agile methodologiesBetween Scrum and Kanban - define a test process for Agile methodologies
Between Scrum and Kanban - define a test process for Agile methodologies
 

More from Zbyszek Mockun

Selenium - The Way Of Success
Selenium - The Way Of SuccessSelenium - The Way Of Success
Selenium - The Way Of Success
Zbyszek Mockun
 

More from Zbyszek Mockun (19)

Freestyle your leadership
Freestyle your leadershipFreestyle your leadership
Freestyle your leadership
 
How to develop diverse team? v2.0
How to develop diverse team? v2.0How to develop diverse team? v2.0
How to develop diverse team? v2.0
 
How to develop diverse team?
How to develop diverse team?How to develop diverse team?
How to develop diverse team?
 
Automation in the world of project
Automation  in the world of projectAutomation  in the world of project
Automation in the world of project
 
The little thing
The little thingThe little thing
The little thing
 
Ptaq day one intro presentation
Ptaq day one intro presentationPtaq day one intro presentation
Ptaq day one intro presentation
 
We are all engineers
We are all engineers We are all engineers
We are all engineers
 
Automation in the world of projects - A few thoughts from business perspective
Automation in the world of projects - A few thoughts from business perspectiveAutomation in the world of projects - A few thoughts from business perspective
Automation in the world of projects - A few thoughts from business perspective
 
PTaQ in numbers after 50 meetups.
PTaQ in numbers after 50 meetups.PTaQ in numbers after 50 meetups.
PTaQ in numbers after 50 meetups.
 
Shift left as first transformation step into Quality Assurance
Shift left as first transformation step into Quality AssuranceShift left as first transformation step into Quality Assurance
Shift left as first transformation step into Quality Assurance
 
There and back again, Our journey with QA Reports and metrics
There and back again, Our journey with QA Reports and metricsThere and back again, Our journey with QA Reports and metrics
There and back again, Our journey with QA Reports and metrics
 
Project quality (and test process) metrics
Project quality (and test process) metricsProject quality (and test process) metrics
Project quality (and test process) metrics
 
User groups - share your knowledge
User groups - share your knowledgeUser groups - share your knowledge
User groups - share your knowledge
 
Knowledge sharing for QA teams
Knowledge sharing for QA teamsKnowledge sharing for QA teams
Knowledge sharing for QA teams
 
Automated Exploratory Tests
Automated Exploratory TestsAutomated Exploratory Tests
Automated Exploratory Tests
 
Selenium - The Way Of Success
Selenium - The Way Of SuccessSelenium - The Way Of Success
Selenium - The Way Of Success
 
Automation Responsibility
Automation ResponsibilityAutomation Responsibility
Automation Responsibility
 
Continuous Integration testing based on Selenium and Hudson
Continuous Integration testing based on Selenium and HudsonContinuous Integration testing based on Selenium and Hudson
Continuous Integration testing based on Selenium and Hudson
 
Agile testing
Agile testingAgile testing
Agile testing
 

Recently uploaded

Recently uploaded (20)

How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
 
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone KomSalesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
 
What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024
 
The UX of Automation by AJ King, Senior UX Researcher, Ocado
The UX of Automation by AJ King, Senior UX Researcher, OcadoThe UX of Automation by AJ King, Senior UX Researcher, Ocado
The UX of Automation by AJ King, Senior UX Researcher, Ocado
 
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfSimplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
 
Optimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityOptimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through Observability
 
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxWSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
 
UiPath Test Automation using UiPath Test Suite series, part 1
UiPath Test Automation using UiPath Test Suite series, part 1UiPath Test Automation using UiPath Test Suite series, part 1
UiPath Test Automation using UiPath Test Suite series, part 1
 
Connecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAKConnecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAK
 
UiPath Test Automation using UiPath Test Suite series, part 2
UiPath Test Automation using UiPath Test Suite series, part 2UiPath Test Automation using UiPath Test Suite series, part 2
UiPath Test Automation using UiPath Test Suite series, part 2
 
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi IbrahimzadeFree and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
 
Buy Epson EcoTank L3210 Colour Printer Online.pptx
Buy Epson EcoTank L3210 Colour Printer Online.pptxBuy Epson EcoTank L3210 Colour Printer Online.pptx
Buy Epson EcoTank L3210 Colour Printer Online.pptx
 
AI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří KarpíšekAI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří Karpíšek
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara Laskowska
 
Syngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdfSyngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdf
 
Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutes
 
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
 
Designing for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastDesigning for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at Comcast
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
 

Automated Exploratory Tests

  • 1. Automated Exploratory Tests for Web Application Zbyszek Moćkun © 2010 Cognifide Limited. In commercial confidence only.
  • 2. Objective How Cognifide has developed automated tests and where we are going © 2010 Cognifide Limited. In commercial confidence only.
  • 3. Introduction In short about process and environment © 2010 Cognifide Limited. In commercial confidence only.
  • 4. Methodologies, projects … Methodologies Projects • Web application • Agile: SCRUM, Kanban • Migration • Frequently releases • Content Management • No time for manual Systems (CMS) regression • Short or medium term • Lot of work at the end of projects sprint • Automation investment, no • CI Environment time for money return © 2010 Cognifide Limited. In commercial confidence only.
  • 5. Common standards for automation Functional tests © 2010 Cognifide Limited. In commercial confidence only.
  • 6. Test Automation Lifecycle Write TC Write automated script Run Manage Update © 2010 Cognifide Limited. In commercial confidence only.
  • 7. Advantages and disadvantages • Time consuming • Cost • Coverage by automated tests • Invest and wait for return • No duplications during • Test dependency regression test • Only coded functionality is • Functionality is tested tested © 2010 Cognifide Limited. In commercial confidence only.
  • 8. Can you believe your tests? • Incorrect Xpath (locator) used in tests • Not all data were verified (unable to write verifications for all data) • CSS/images or other files wasn’t load • Frequently design changes require xpath (locator) update • Incorrect CSS fixes (designs are not reflected by page layout) • Analytics doesn’t work • Performance decrease • Few thousands of pages (components variation – can’t test all of them © 2010 Cognifide Limited. In commercial confidence only.
  • 9. Migration (upgrade) projects • No existing automated test for regression test • Few thousands of pages • Quick response needed • Iterative development © 2010 Cognifide Limited. In commercial confidence only.
  • 10. Automated Exploratory Tests New approach, theory part © 2010 Cognifide Limited. In commercial confidence only.
  • 11. We need automated tests because ... Source: http://www.testfacebook.com © 2010 Cognifide Limited. In commercial confidence only.
  • 12. What should we test? Two perspectives: User Client © 2010 Cognifide Limited. In commercial confidence only.
  • 13. Aspects CI Environment Client User Analytics Layout Functionality Performance © 2010 Cognifide Limited. In commercial confidence only.
  • 14. Looking for a symptom „Symptom is a departure from normal function or feeling which is noticed by a patient, indicating the presence of disease or abnormality. A symptom is subjective, observed by the patient, and not measured.” http://wikipedia.org © 2010 Cognifide Limited. In commercial confidence only.
  • 15. Symptoms Layout Performance Content URL strategy HTML code Bug JavaScript Link checker errors Server Validators response (W3C) (404) Analytics © 2010 Cognifide Limited. In commercial confidence only.
  • 16. Oracle Heuristics It is used by comparing the output(s) of the system under test, for a given test case input, to the outputs that the oracle determines that product should have. − other products (second program that uses a different algorithm to evaluate the same mathematical expression as the product under test) − a consistency oracle that compares the results of one test execution to another for similarity, − a model-based oracle that uses the same model to generate and verify system behavior, − or a human being's judgment (i.e. does the program "seem" to the user to do the correct thing?) © 2010 Cognifide Limited. In commercial confidence only.
  • 17. Oracle Heuristic for GUI © 2010 Cognifide Limited. In commercial confidence only.
  • 18. Reference symptoms • Previous version of application • Reference data saved on disc • Other version of the same application (different servers) • The same script takes pattern and compare • One tool for different projects Performance Layout Content HTML code monitoring © 2010 Cognifide Limited. In commercial confidence only.
  • 19. Robots • Scripts that are looking for symptoms • The same script for all projects • Open Source tools JavaScript Server response Validators Analytics Link checker URL Strategy errors (404) (HTLM, CSS) © 2010 Cognifide Limited. In commercial confidence only.
  • 20. Exploratory testing „Exploratory testing is an approach to software testing that is concisely described as simultaneous learning, test design, test execution and test result interpretation” Cem Kaner © 2010 Cognifide Limited. In commercial confidence only.
  • 21. What are Automated Exploratory Tests? Meny different definition: • Record and play manual exploratory test sessions • Combined manual and automated tests • Test steps automation • Blackbox tests with input/output data generator • Not at all possible due to human factor © 2010 Cognifide Limited. In commercial confidence only.
  • 22. What are Automated Exploratory Tests? „Tests supporting exploratory testing, not necessarily based on the test scenario, searching areas where there may be errors or a change in the application, allowing the examination of applications faster and more deeply or not to test sites available to man.” Zbyszek Moćkun „Automated Tests do not provide complete information. They provide partial information that might be useful” Cem Kaner © 2010 Cognifide Limited. In commercial confidence only.
  • 23. Automated Exploratory Tests - advantages • One tool − Gather all symptoms − Use in all projects − Quick to introduce to project (half a day) − Resistant to appliciaton changes • Do not require deep knowledge about application, support the learning process • Integrated with Continuous Intergration env • Strong solution if combined with automated functional tests • Migration or upgrade tests • Iterative development © 2010 Cognifide Limited. In commercial confidence only.
  • 24. Track the propagation of changes in the code - See what pages/components are infected by change - Positive – false (code change affect all necessary places?) © 2010 Cognifide Limited. In commercial confidence only.
  • 25. Automated Exploratory Tests Practical part © 2010 Cognifide Limited. In commercial confidence only.
  • 26. Report – example of use © 2010 Cognifide Limited. In commercial confidence only.
  • 27. URL management • Sitemap • Link crawler • CSV file with url list • Symptoms as parameter (on, off) • Comments (bug id, ...) • Hide element • Other actions • Keep in SVN © 2010 Cognifide Limited. In commercial confidence only.
  • 28. Pattern management • Easy to use (overwrite pattern) • Checkbox (check all) • Commit to svn © 2010 Cognifide Limited. In commercial confidence only.
  • 29. Dynamic content • Reference symptoms – need processing • Use regexp • Hide element • Write generic function • Robots – do not need any additional actions • Dynamic components (counters, date, random numbers) • Flash/video files • Embedded content (twitter, facebook,...) • Require functional test • Static components – about 75% © 2010 Cognifide Limited. In commercial confidence only.
  • 30. Mobile version • Required (not nice to have), can’t avoid testing • Mobile pages are more static than desktop version • Do not use flash • Not all symptoms are easy to check © 2010 Cognifide Limited. In commercial confidence only.
  • 31. Narzędzia • Selenium II/WebDriver • Komparatory (ImageMagick, Pretty Diff) • Walidatory (W3C) • Performance (JMeter, Proxy) • Serwery Proxy (BrowserMob) • Firefox/Firebug (NetExport, ConsoleExport, JS Console output redirector) • HAR Viewer (http://www.softwareishard.com/har/viewer/) © 2010 Cognifide Limited. In commercial confidence only.
  • 32. Q&A © 2010 Cognifide Limited. In commercial confidence only.