SlideShare a Scribd company logo
1 of 38
Download to read offline
Poznań Testing
                 and Quality Group




Automation as a project –
lessons learnt
Zbyszek Moćkun

                        © 2010 Cognifide Limited. In commercial confidence only.
Agenda


• Introduction
• Business
• Technology review
• Management
• Summary
• Discussion




                      © 2010 Cognifide Limited. In commercial confidence only.
Introduction

Few words about the author




                             © 2010 Cognifide Limited. In commercial confidence only.
Experience and domains
       Experience               Domains

                               • Mobile infrastructure
• 7 years in tests
                               • Billing and Roaming
• Three different companies
                               systems
• Several domains
                               • Business Assurance
• Product based and project    application
  based
                               • Medical
• Different methodologies
                               • Content Management
  (Waterfall, Scrum, Kanban)
                               Systems




                                                   © 2010 Cognifide Limited. In commercial confidence only.
Business

Automation vs. profitability




                               © 2010 Cognifide Limited. In commercial confidence only.
It’s all about the money




                      http://www.br-automation.com




                                              © 2010 Cognifide Limited. In commercial confidence only.
How to talk with stakeholders


• Time of money return
• Use statistics/metrics
• Remeber that statistics show incorrect data (only regression
  issues should be calculated)
• It’s not about how many issues were found
• Statistics fall down without continuous investement
• Code resistance on the same tests
• Decrease release time (time for manual tests)
• Number of issues found influence on time needed for
  regression


                                                    © 2010 Cognifide Limited. In commercial confidence only.
How to start?


• Show metrics
   • Why does we need it?
   • How much does it cost?
   • What does we want to achieve?
   • What are the goals?
• Small steps
   • First step should be cheap and quick
   • Show results for stakeholder
   • Plan and estimate next steps
• Gather metrics and statistics

                                            © 2010 Cognifide Limited. In commercial confidence only.
Approaches




   Product-   Project-
    based      based




                     © 2010 Cognifide Limited. In commercial confidence only.
Product against project
    Product-based                Project-based

                                 • No need of scalability
 • Scalability
                                 • Only basic modularity
 • Modularity
                                 • Coverage – main features
 • Coverage – as much as
   possible                      • Cheap - finite number of runs

 • Expensive - infinite number   • Reuse for other projects (not
   of runs                       always possible)

 • Reuse for other products      • No place for planning new
                                 features and improvements
 • Planing new features and
   improvements                  • Manual tests are cheaper at
                                 the most cases




                                                     © 2010 Cognifide Limited. In commercial confidence only.
Automation

Technology




             © 2010 Cognifide Limited. In commercial confidence only.
Continuous Integration

One place for automation management




                             © 2010 Cognifide Limited. In commercial confidence only.
Where should automated tests be run?

 • On developer machine if possible, before commit
 • As sanity check on other machines
 • Developer iterative work (migration as example)
 • Should we run autoamted tests on production (live)?
 • Independent test data



    Developer
                   Nightly       Iteration     Staging           Production
     machine




                               CI tool

                                                         © 2010 Cognifide Limited. In commercial confidence only.
Reports

• Nightly tests – preferred frequency
• Build results management (disc space)
• Reports improvements
  • Minimalize time spend on report (verification/debugging)
  • Display as much data as possible for failures
  • One issue failing several tess script

• Environment errors - stability
• Red reports effect
  • Issues are not fixed
  • A lof of failed tests cases
  • Noone look at the results




                                                               © 2010 Cognifide Limited. In commercial confidence only.
Reports




          © 2010 Cognifide Limited. In commercial confidence only.
Dependent or
independent?
How to build good test suite?




                                © 2010 Cognifide Limited. In commercial confidence only.
Independent and random

• All tests are independent
  • Pre – precondition (set require state)
  • Post – postcondition
  • Test – test script
  • Rec – recovery procedure after test failure (pre i post should have defined too)
  • LogData – gather all logs after failure, very important for „random” failures

• Random order
• Repeat counter


• Dependent only for CI and adult teams (founded issues are
  fixed as fast as possible)




                                                                                    © 2010 Cognifide Limited. In commercial confidence only.
Test data

How to manage test data? Load them or
create?




                               © 2010 Cognifide Limited. In commercial confidence only.
Record & Play

Does it work?




                © 2010 Cognifide Limited. In commercial confidence only.
Selenium IDE as example

 • Stability issues
 • Recorded values are not meaningfull
 • Require rewrite
 • User-extension.js (write own commands)
 • Quick and easy debug
 • Project-based – preferred as cheap and quick
 • Product-based – as starting point
 • More info link




                                                  © 2010 Cognifide Limited. In commercial confidence only.
Product against project
             HTML                       Java
• Quick and easy (+)
• Cost return (+)
• Coverage – max 10%-20% (-)    • Flexibility (+)
• Modularity using user-        • Scalability (+)
  extension.js (=/-)            • Modularity (+)
• No need to environment        • Coverage (+)
  setup (+)
                                • Development knowledge (+/-)
• No scalability (-)
                                • Environment setup (-)
• Cnvert to other languages
                                • No convert possibility (+/-)
  (+)
                                • Debug time (-)
• HTML Suites (works with CI)
                                • Reporting (JUnit, TestNG) (-)
  (+)
• Integrated reports


                                                     © 2010 Cognifide Limited. In commercial confidence only.
Survey

Two similar application and different technologies based on
 Selenium:
 • HTML based on Selenium IDE
 • Java based on Selenium RC/Eclipse


HTML – 50% more test cases written
Java – 20% more time spend on maintenance/debugging


Which technology would you like to use?
Java devs choose HTML/Selenium IDE as more friendly
 and faster

                                                    © 2010 Cognifide Limited. In commercial confidence only.
Automation

Management




             © 2010 Cognifide Limited. In commercial confidence only.
Coverage

Manage, monitor and plan




                           © 2010 Cognifide Limited. In commercial confidence only.
20/80 rule




             © 2010 Cognifide Limited. In commercial confidence only.
Coverage

Manage your coverage and use it for planning manual tests

 • Remember about 20/80 rule
 • Power function
 • 100% - is it possible?
 • Network architecture as example (100% achieved)
 • Human factor caused that 100% is not achievable




                                                     © 2010 Cognifide Limited. In commercial confidence only.
Defect management

 • Red report effect
 • Quick fixes for CI
 • Labels




                        © 2010 Cognifide Limited. In commercial confidence only.
Automation responsibility

Who should write and maintenance
automated tests?




                               © 2010 Cognifide Limited. In commercial confidence only.
Developer helping hand

Can developer deliver automated tests together with
 functionality?
 • Who should write test cases?
 • Scripts written using invalid test cases
 • Scripts written for not working functionality
 • Lack of time – functionality is more important
 • Control needed (sometimes devs forget)
 • Two unsuccessful attempts




                                                      © 2010 Cognifide Limited. In commercial confidence only.
Automation as a project

Triangle: Budget, Time, Scope (Aim)




                               © 2010 Cognifide Limited. In commercial confidence only.
There is no success without:

•   Continuous work
•   Gather statistics for stakeholders
•   Divide budget on manual and automated tests
•   Planning the future – where want to be in next year?
•   Control your costs
•   Remember – it’s all about the money




                                                      © 2010 Cognifide Limited. In commercial confidence only.
The future

Automated Exploratory Tests for Regression




                                 © 2010 Cognifide Limited. In commercial confidence only.
Aspects


 CI

  Client

                  User
      Analytics
                    Layout   Functionality   Performance




                                                 © 2010 Cognifide Limited. In commercial confidence only.
Symptoms


                                    Layout

                   URL
                                                    Content
                 strategy




         Link
                                                              Code
       checker

                                Bug

           W3C                                          JS errors



                                             Respons
                        Analitycs
                                              e (404)



                                                                     © 2010 Cognifide Limited. In commercial confidence only.
Heuristic Oracle

The balance between comparison and comparison not at all




      Layout       Content        Code       Performance




                                                 © 2010 Cognifide Limited. In commercial confidence only.
Robots

• Scripts
• Shared between projects
• Easy and quick to run




            Responses
JS errors               W3C   Analytics   Link checker                 Url strategy
              (404)




                                                         © 2010 Cognifide Limited. In commercial confidence only.
Q&A




      © 2010 Cognifide Limited. In commercial confidence only.
Discussion & beer

Poznań Testing and Qaulity Group




                              © 2010 Cognifide Limited. In commercial confidence only.

More Related Content

What's hot

Engineer Plus Company Profile
Engineer Plus Company ProfileEngineer Plus Company Profile
Engineer Plus Company Profile
DEEPAK BHANDARI
 
Introducing Jetbox
Introducing JetboxIntroducing Jetbox
Introducing Jetbox
lcraig333
 

What's hot (19)

TMAP Quality Engineering workshop on A4Q congress by Rik Marselis
TMAP Quality Engineering workshop on A4Q congress by Rik Marselis TMAP Quality Engineering workshop on A4Q congress by Rik Marselis
TMAP Quality Engineering workshop on A4Q congress by Rik Marselis
 
Quality engineering in DevOps... Why? How? (TestBusters Day&Night))
Quality engineering in DevOps... Why? How? (TestBusters Day&Night))Quality engineering in DevOps... Why? How? (TestBusters Day&Night))
Quality engineering in DevOps... Why? How? (TestBusters Day&Night))
 
Implementing BDD at scale for agile and DevOps teams
Implementing BDD at scale for agile and DevOps teamsImplementing BDD at scale for agile and DevOps teams
Implementing BDD at scale for agile and DevOps teams
 
A case study on VOLVO, Quest For Better Quality
A case study on VOLVO, Quest For Better QualityA case study on VOLVO, Quest For Better Quality
A case study on VOLVO, Quest For Better Quality
 
Company Profile
Company ProfileCompany Profile
Company Profile
 
Engineer Plus Company Profile
Engineer Plus Company ProfileEngineer Plus Company Profile
Engineer Plus Company Profile
 
COMPANY PROFILE - ICO Ltd.
COMPANY PROFILE - ICO Ltd.COMPANY PROFILE - ICO Ltd.
COMPANY PROFILE - ICO Ltd.
 
Quality Engineering in the New Era
Quality Engineering in the New EraQuality Engineering in the New Era
Quality Engineering in the New Era
 
Exploratory testing and Dev Ops - best friends?
Exploratory testing and Dev Ops - best friends?Exploratory testing and Dev Ops - best friends?
Exploratory testing and Dev Ops - best friends?
 
The Disciplined Approach to Change
The Disciplined Approach to ChangeThe Disciplined Approach to Change
The Disciplined Approach to Change
 
Introducing Jetbox
Introducing JetboxIntroducing Jetbox
Introducing Jetbox
 
Kanban Case Study
Kanban Case StudyKanban Case Study
Kanban Case Study
 
Agile Base Camp - Agile metrics
Agile Base Camp - Agile metricsAgile Base Camp - Agile metrics
Agile Base Camp - Agile metrics
 
Quality Index: A Composite Metric for the Voice of Testing
Quality Index: A Composite Metric for the Voice of TestingQuality Index: A Composite Metric for the Voice of Testing
Quality Index: A Composite Metric for the Voice of Testing
 
Upk for p6 testing system documentation training and job alerts ppt (1)
Upk for p6 testing system documentation training and job alerts ppt (1)Upk for p6 testing system documentation training and job alerts ppt (1)
Upk for p6 testing system documentation training and job alerts ppt (1)
 
DevOps at Impact Theater
DevOps at Impact TheaterDevOps at Impact Theater
DevOps at Impact Theater
 
Agile dashboard
Agile dashboardAgile dashboard
Agile dashboard
 
Internal Audits and Assessments with help of Enterprise SPiCE
Internal Audits and Assessments with help of Enterprise SPiCEInternal Audits and Assessments with help of Enterprise SPiCE
Internal Audits and Assessments with help of Enterprise SPiCE
 
Agile project management PMI-ACP
Agile project management PMI-ACPAgile project management PMI-ACP
Agile project management PMI-ACP
 

Similar to Automation as a project - lessons learnt

IDC & Gomez Webinar --Best Practices: Protect Your Online Revenue Through Web...
IDC & Gomez Webinar --Best Practices: Protect Your Online Revenue Through Web...IDC & Gomez Webinar --Best Practices: Protect Your Online Revenue Through Web...
IDC & Gomez Webinar --Best Practices: Protect Your Online Revenue Through Web...
Compuware APM
 

Similar to Automation as a project - lessons learnt (20)

Automated Exploratory Tests
Automated Exploratory TestsAutomated Exploratory Tests
Automated Exploratory Tests
 
Exploratory testing in practice, short story how approach influenced on strategy
Exploratory testing in practice, short story how approach influenced on strategyExploratory testing in practice, short story how approach influenced on strategy
Exploratory testing in practice, short story how approach influenced on strategy
 
Between Scrum and Kanban - define a test process for Agile methodologies
Between Scrum and Kanban - define a test process for Agile methodologiesBetween Scrum and Kanban - define a test process for Agile methodologies
Between Scrum and Kanban - define a test process for Agile methodologies
 
Between Scrum and Kanban - define test process for Agile methodologies
Between Scrum and Kanban - define test process for Agile methodologiesBetween Scrum and Kanban - define test process for Agile methodologies
Between Scrum and Kanban - define test process for Agile methodologies
 
Test Strategy
Test StrategyTest Strategy
Test Strategy
 
Test Design Essentials for Great Test Automation - Hans
Test Design Essentials for Great Test Automation - HansTest Design Essentials for Great Test Automation - Hans
Test Design Essentials for Great Test Automation - Hans
 
How to Balance System Speed and Risk for Multi-Platform Innovation
How to Balance System Speed and Risk for Multi-Platform InnovationHow to Balance System Speed and Risk for Multi-Platform Innovation
How to Balance System Speed and Risk for Multi-Platform Innovation
 
Project quality (and test process) metrics
Project quality (and test process) metricsProject quality (and test process) metrics
Project quality (and test process) metrics
 
IBM Cognos 10 Framework Manager in Action
IBM Cognos 10 Framework Manager in ActionIBM Cognos 10 Framework Manager in Action
IBM Cognos 10 Framework Manager in Action
 
IBM Cognos 10 Business Insight: Advanced Self-Service Authoring and Analysis
IBM Cognos 10 Business Insight: Advanced Self-Service Authoring and AnalysisIBM Cognos 10 Business Insight: Advanced Self-Service Authoring and Analysis
IBM Cognos 10 Business Insight: Advanced Self-Service Authoring and Analysis
 
How NBCUniversal Adopted DevOps
How NBCUniversal Adopted DevOpsHow NBCUniversal Adopted DevOps
How NBCUniversal Adopted DevOps
 
IBM Z for the Digital Enterprise 2018 - Automate Delivery Pipeline
IBM Z for the Digital Enterprise 2018 - Automate Delivery PipelineIBM Z for the Digital Enterprise 2018 - Automate Delivery Pipeline
IBM Z for the Digital Enterprise 2018 - Automate Delivery Pipeline
 
Measuring User Adoption of Your Cognos BI System
Measuring User Adoption of Your Cognos BI SystemMeasuring User Adoption of Your Cognos BI System
Measuring User Adoption of Your Cognos BI System
 
IBM Cognos 10.1.1 Active Reports: New Features Overview and Tips Demo
IBM Cognos 10.1.1 Active Reports: New Features Overview and Tips DemoIBM Cognos 10.1.1 Active Reports: New Features Overview and Tips Demo
IBM Cognos 10.1.1 Active Reports: New Features Overview and Tips Demo
 
LSU Nordic 2015 - Agile development on system z
LSU Nordic 2015 - Agile development on system zLSU Nordic 2015 - Agile development on system z
LSU Nordic 2015 - Agile development on system z
 
Test automation lessons from WebSphere Application Server
Test automation lessons from WebSphere Application ServerTest automation lessons from WebSphere Application Server
Test automation lessons from WebSphere Application Server
 
Developer want change Ops want control - devops
Developer want change Ops want control - devopsDeveloper want change Ops want control - devops
Developer want change Ops want control - devops
 
Oaksys Updated Presentation 2020
Oaksys Updated Presentation 2020Oaksys Updated Presentation 2020
Oaksys Updated Presentation 2020
 
IDC & Gomez Webinar --Best Practices: Protect Your Online Revenue Through Web...
IDC & Gomez Webinar --Best Practices: Protect Your Online Revenue Through Web...IDC & Gomez Webinar --Best Practices: Protect Your Online Revenue Through Web...
IDC & Gomez Webinar --Best Practices: Protect Your Online Revenue Through Web...
 
Refactoring Big Design Smells : Presented by Sanjay Kumar
Refactoring Big Design Smells : Presented by Sanjay KumarRefactoring Big Design Smells : Presented by Sanjay Kumar
Refactoring Big Design Smells : Presented by Sanjay Kumar
 

More from Zbyszek Mockun

Selenium - The Way Of Success
Selenium - The Way Of SuccessSelenium - The Way Of Success
Selenium - The Way Of Success
Zbyszek Mockun
 

More from Zbyszek Mockun (17)

Freestyle your leadership
Freestyle your leadershipFreestyle your leadership
Freestyle your leadership
 
How to develop diverse team? v2.0
How to develop diverse team? v2.0How to develop diverse team? v2.0
How to develop diverse team? v2.0
 
How to develop diverse team?
How to develop diverse team?How to develop diverse team?
How to develop diverse team?
 
Automation in the world of project
Automation  in the world of projectAutomation  in the world of project
Automation in the world of project
 
The little thing
The little thingThe little thing
The little thing
 
Ptaq day one intro presentation
Ptaq day one intro presentationPtaq day one intro presentation
Ptaq day one intro presentation
 
We are all engineers
We are all engineers We are all engineers
We are all engineers
 
Automation in the world of projects - A few thoughts from business perspective
Automation in the world of projects - A few thoughts from business perspectiveAutomation in the world of projects - A few thoughts from business perspective
Automation in the world of projects - A few thoughts from business perspective
 
PTaQ in numbers after 50 meetups.
PTaQ in numbers after 50 meetups.PTaQ in numbers after 50 meetups.
PTaQ in numbers after 50 meetups.
 
Shift left as first transformation step into Quality Assurance
Shift left as first transformation step into Quality AssuranceShift left as first transformation step into Quality Assurance
Shift left as first transformation step into Quality Assurance
 
There and back again, Our journey with QA Reports and metrics
There and back again, Our journey with QA Reports and metricsThere and back again, Our journey with QA Reports and metrics
There and back again, Our journey with QA Reports and metrics
 
User groups - share your knowledge
User groups - share your knowledgeUser groups - share your knowledge
User groups - share your knowledge
 
Automated Exploratory Tests
Automated Exploratory TestsAutomated Exploratory Tests
Automated Exploratory Tests
 
Selenium - The Way Of Success
Selenium - The Way Of SuccessSelenium - The Way Of Success
Selenium - The Way Of Success
 
Automation Responsibility
Automation ResponsibilityAutomation Responsibility
Automation Responsibility
 
Continuous Integration testing based on Selenium and Hudson
Continuous Integration testing based on Selenium and HudsonContinuous Integration testing based on Selenium and Hudson
Continuous Integration testing based on Selenium and Hudson
 
Agile testing
Agile testingAgile testing
Agile testing
 

Recently uploaded

The basics of sentences session 4pptx.pptx
The basics of sentences session 4pptx.pptxThe basics of sentences session 4pptx.pptx
The basics of sentences session 4pptx.pptx
heathfieldcps1
 
ppt your views.ppt your views of your college in your eyes
ppt your views.ppt your views of your college in your eyesppt your views.ppt your views of your college in your eyes
ppt your views.ppt your views of your college in your eyes
ashishpaul799
 

Recently uploaded (20)

philosophy and it's principles based on the life
philosophy and it's principles based on the lifephilosophy and it's principles based on the life
philosophy and it's principles based on the life
 
Open Educational Resources Primer PowerPoint
Open Educational Resources Primer PowerPointOpen Educational Resources Primer PowerPoint
Open Educational Resources Primer PowerPoint
 
Behavioral-sciences-dr-mowadat rana (1).pdf
Behavioral-sciences-dr-mowadat rana (1).pdfBehavioral-sciences-dr-mowadat rana (1).pdf
Behavioral-sciences-dr-mowadat rana (1).pdf
 
The Ultimate Guide to Social Media Marketing in 2024.pdf
The Ultimate Guide to Social Media Marketing in 2024.pdfThe Ultimate Guide to Social Media Marketing in 2024.pdf
The Ultimate Guide to Social Media Marketing in 2024.pdf
 
size separation d pharm 1st year pharmaceutics
size separation d pharm 1st year pharmaceuticssize separation d pharm 1st year pharmaceutics
size separation d pharm 1st year pharmaceutics
 
Capitol Tech Univ Doctoral Presentation -May 2024
Capitol Tech Univ Doctoral Presentation -May 2024Capitol Tech Univ Doctoral Presentation -May 2024
Capitol Tech Univ Doctoral Presentation -May 2024
 
Word Stress rules esl .pptx
Word Stress rules esl               .pptxWord Stress rules esl               .pptx
Word Stress rules esl .pptx
 
Danh sách HSG Bộ môn cấp trường - Cấp THPT.pdf
Danh sách HSG Bộ môn cấp trường - Cấp THPT.pdfDanh sách HSG Bộ môn cấp trường - Cấp THPT.pdf
Danh sách HSG Bộ môn cấp trường - Cấp THPT.pdf
 
The Last Leaf, a short story by O. Henry
The Last Leaf, a short story by O. HenryThe Last Leaf, a short story by O. Henry
The Last Leaf, a short story by O. Henry
 
UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...
UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...
UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...
 
Features of Video Calls in the Discuss Module in Odoo 17
Features of Video Calls in the Discuss Module in Odoo 17Features of Video Calls in the Discuss Module in Odoo 17
Features of Video Calls in the Discuss Module in Odoo 17
 
....................Muslim-Law notes.pdf
....................Muslim-Law notes.pdf....................Muslim-Law notes.pdf
....................Muslim-Law notes.pdf
 
The basics of sentences session 4pptx.pptx
The basics of sentences session 4pptx.pptxThe basics of sentences session 4pptx.pptx
The basics of sentences session 4pptx.pptx
 
Exploring Gemini AI and Integration with MuleSoft | MuleSoft Mysore Meetup #45
Exploring Gemini AI and Integration with MuleSoft | MuleSoft Mysore Meetup #45Exploring Gemini AI and Integration with MuleSoft | MuleSoft Mysore Meetup #45
Exploring Gemini AI and Integration with MuleSoft | MuleSoft Mysore Meetup #45
 
How to Manage Notification Preferences in the Odoo 17
How to Manage Notification Preferences in the Odoo 17How to Manage Notification Preferences in the Odoo 17
How to Manage Notification Preferences in the Odoo 17
 
ppt your views.ppt your views of your college in your eyes
ppt your views.ppt your views of your college in your eyesppt your views.ppt your views of your college in your eyes
ppt your views.ppt your views of your college in your eyes
 
Essential Safety precautions during monsoon season
Essential Safety precautions during monsoon seasonEssential Safety precautions during monsoon season
Essential Safety precautions during monsoon season
 
“O BEIJO” EM ARTE .
“O BEIJO” EM ARTE                       .“O BEIJO” EM ARTE                       .
“O BEIJO” EM ARTE .
 
Morse OER Some Benefits and Challenges.pptx
Morse OER Some Benefits and Challenges.pptxMorse OER Some Benefits and Challenges.pptx
Morse OER Some Benefits and Challenges.pptx
 
Matatag-Curriculum and the 21st Century Skills Presentation.pptx
Matatag-Curriculum and the 21st Century Skills Presentation.pptxMatatag-Curriculum and the 21st Century Skills Presentation.pptx
Matatag-Curriculum and the 21st Century Skills Presentation.pptx
 

Automation as a project - lessons learnt

  • 1. Poznań Testing and Quality Group Automation as a project – lessons learnt Zbyszek Moćkun © 2010 Cognifide Limited. In commercial confidence only.
  • 2. Agenda • Introduction • Business • Technology review • Management • Summary • Discussion © 2010 Cognifide Limited. In commercial confidence only.
  • 3. Introduction Few words about the author © 2010 Cognifide Limited. In commercial confidence only.
  • 4. Experience and domains Experience Domains • Mobile infrastructure • 7 years in tests • Billing and Roaming • Three different companies systems • Several domains • Business Assurance • Product based and project application based • Medical • Different methodologies • Content Management (Waterfall, Scrum, Kanban) Systems © 2010 Cognifide Limited. In commercial confidence only.
  • 5. Business Automation vs. profitability © 2010 Cognifide Limited. In commercial confidence only.
  • 6. It’s all about the money http://www.br-automation.com © 2010 Cognifide Limited. In commercial confidence only.
  • 7. How to talk with stakeholders • Time of money return • Use statistics/metrics • Remeber that statistics show incorrect data (only regression issues should be calculated) • It’s not about how many issues were found • Statistics fall down without continuous investement • Code resistance on the same tests • Decrease release time (time for manual tests) • Number of issues found influence on time needed for regression © 2010 Cognifide Limited. In commercial confidence only.
  • 8. How to start? • Show metrics • Why does we need it? • How much does it cost? • What does we want to achieve? • What are the goals? • Small steps • First step should be cheap and quick • Show results for stakeholder • Plan and estimate next steps • Gather metrics and statistics © 2010 Cognifide Limited. In commercial confidence only.
  • 9. Approaches Product- Project- based based © 2010 Cognifide Limited. In commercial confidence only.
  • 10. Product against project Product-based Project-based • No need of scalability • Scalability • Only basic modularity • Modularity • Coverage – main features • Coverage – as much as possible • Cheap - finite number of runs • Expensive - infinite number • Reuse for other projects (not of runs always possible) • Reuse for other products • No place for planning new features and improvements • Planing new features and improvements • Manual tests are cheaper at the most cases © 2010 Cognifide Limited. In commercial confidence only.
  • 11. Automation Technology © 2010 Cognifide Limited. In commercial confidence only.
  • 12. Continuous Integration One place for automation management © 2010 Cognifide Limited. In commercial confidence only.
  • 13. Where should automated tests be run? • On developer machine if possible, before commit • As sanity check on other machines • Developer iterative work (migration as example) • Should we run autoamted tests on production (live)? • Independent test data Developer Nightly Iteration Staging Production machine CI tool © 2010 Cognifide Limited. In commercial confidence only.
  • 14. Reports • Nightly tests – preferred frequency • Build results management (disc space) • Reports improvements • Minimalize time spend on report (verification/debugging) • Display as much data as possible for failures • One issue failing several tess script • Environment errors - stability • Red reports effect • Issues are not fixed • A lof of failed tests cases • Noone look at the results © 2010 Cognifide Limited. In commercial confidence only.
  • 15. Reports © 2010 Cognifide Limited. In commercial confidence only.
  • 16. Dependent or independent? How to build good test suite? © 2010 Cognifide Limited. In commercial confidence only.
  • 17. Independent and random • All tests are independent • Pre – precondition (set require state) • Post – postcondition • Test – test script • Rec – recovery procedure after test failure (pre i post should have defined too) • LogData – gather all logs after failure, very important for „random” failures • Random order • Repeat counter • Dependent only for CI and adult teams (founded issues are fixed as fast as possible) © 2010 Cognifide Limited. In commercial confidence only.
  • 18. Test data How to manage test data? Load them or create? © 2010 Cognifide Limited. In commercial confidence only.
  • 19. Record & Play Does it work? © 2010 Cognifide Limited. In commercial confidence only.
  • 20. Selenium IDE as example • Stability issues • Recorded values are not meaningfull • Require rewrite • User-extension.js (write own commands) • Quick and easy debug • Project-based – preferred as cheap and quick • Product-based – as starting point • More info link © 2010 Cognifide Limited. In commercial confidence only.
  • 21. Product against project HTML Java • Quick and easy (+) • Cost return (+) • Coverage – max 10%-20% (-) • Flexibility (+) • Modularity using user- • Scalability (+) extension.js (=/-) • Modularity (+) • No need to environment • Coverage (+) setup (+) • Development knowledge (+/-) • No scalability (-) • Environment setup (-) • Cnvert to other languages • No convert possibility (+/-) (+) • Debug time (-) • HTML Suites (works with CI) • Reporting (JUnit, TestNG) (-) (+) • Integrated reports © 2010 Cognifide Limited. In commercial confidence only.
  • 22. Survey Two similar application and different technologies based on Selenium: • HTML based on Selenium IDE • Java based on Selenium RC/Eclipse HTML – 50% more test cases written Java – 20% more time spend on maintenance/debugging Which technology would you like to use? Java devs choose HTML/Selenium IDE as more friendly and faster © 2010 Cognifide Limited. In commercial confidence only.
  • 23. Automation Management © 2010 Cognifide Limited. In commercial confidence only.
  • 24. Coverage Manage, monitor and plan © 2010 Cognifide Limited. In commercial confidence only.
  • 25. 20/80 rule © 2010 Cognifide Limited. In commercial confidence only.
  • 26. Coverage Manage your coverage and use it for planning manual tests • Remember about 20/80 rule • Power function • 100% - is it possible? • Network architecture as example (100% achieved) • Human factor caused that 100% is not achievable © 2010 Cognifide Limited. In commercial confidence only.
  • 27. Defect management • Red report effect • Quick fixes for CI • Labels © 2010 Cognifide Limited. In commercial confidence only.
  • 28. Automation responsibility Who should write and maintenance automated tests? © 2010 Cognifide Limited. In commercial confidence only.
  • 29. Developer helping hand Can developer deliver automated tests together with functionality? • Who should write test cases? • Scripts written using invalid test cases • Scripts written for not working functionality • Lack of time – functionality is more important • Control needed (sometimes devs forget) • Two unsuccessful attempts © 2010 Cognifide Limited. In commercial confidence only.
  • 30. Automation as a project Triangle: Budget, Time, Scope (Aim) © 2010 Cognifide Limited. In commercial confidence only.
  • 31. There is no success without: • Continuous work • Gather statistics for stakeholders • Divide budget on manual and automated tests • Planning the future – where want to be in next year? • Control your costs • Remember – it’s all about the money © 2010 Cognifide Limited. In commercial confidence only.
  • 32. The future Automated Exploratory Tests for Regression © 2010 Cognifide Limited. In commercial confidence only.
  • 33. Aspects CI Client User Analytics Layout Functionality Performance © 2010 Cognifide Limited. In commercial confidence only.
  • 34. Symptoms Layout URL Content strategy Link Code checker Bug W3C JS errors Respons Analitycs e (404) © 2010 Cognifide Limited. In commercial confidence only.
  • 35. Heuristic Oracle The balance between comparison and comparison not at all Layout Content Code Performance © 2010 Cognifide Limited. In commercial confidence only.
  • 36. Robots • Scripts • Shared between projects • Easy and quick to run Responses JS errors W3C Analytics Link checker Url strategy (404) © 2010 Cognifide Limited. In commercial confidence only.
  • 37. Q&A © 2010 Cognifide Limited. In commercial confidence only.
  • 38. Discussion & beer Poznań Testing and Qaulity Group © 2010 Cognifide Limited. In commercial confidence only.