Poznań Testing                 and Quality GroupAutomation as a project –lessons learntZbyszek Moćkun                     ...
Agenda• Introduction• Business• Technology review• Management• Summary• Discussion                      © 2010 Cognifide L...
IntroductionFew words about the author                             © 2010 Cognifide Limited. In commercial confidence only.
Experience and domains       Experience               Domains                               • Mobile infrastructure• 7 yea...
BusinessAutomation vs. profitability                               © 2010 Cognifide Limited. In commercial confidence only.
It’s all about the money                      http://www.br-automation.com                                              © ...
How to talk with stakeholders• Time of money return• Use statistics/metrics• Remeber that statistics show incorrect data (...
How to start?• Show metrics   • Why does we need it?   • How much does it cost?   • What does we want to achieve?   • What...
Approaches   Product-   Project-    based      based                     © 2010 Cognifide Limited. In commercial confidenc...
Product against project    Product-based                Project-based                                 • No need of scalabi...
AutomationTechnology             © 2010 Cognifide Limited. In commercial confidence only.
Continuous IntegrationOne place for automation management                             © 2010 Cognifide Limited. In commerc...
Where should automated tests be run? • On developer machine if possible, before commit • As sanity check on other machines...
Reports• Nightly tests – preferred frequency• Build results management (disc space)• Reports improvements  • Minimalize ti...
Reports          © 2010 Cognifide Limited. In commercial confidence only.
Dependent orindependent?How to build good test suite?                                © 2010 Cognifide Limited. In commerci...
Independent and random• All tests are independent  • Pre – precondition (set require state)  • Post – postcondition  • Tes...
Test dataHow to manage test data? Load them orcreate?                               © 2010 Cognifide Limited. In commercia...
Record & PlayDoes it work?                © 2010 Cognifide Limited. In commercial confidence only.
Selenium IDE as example • Stability issues • Recorded values are not meaningfull • Require rewrite • User-extension.js (wr...
Product against project             HTML                       Java• Quick and easy (+)• Cost return (+)• Coverage – max 1...
SurveyTwo similar application and different technologies based on Selenium: • HTML based on Selenium IDE • Java based on S...
AutomationManagement             © 2010 Cognifide Limited. In commercial confidence only.
CoverageManage, monitor and plan                           © 2010 Cognifide Limited. In commercial confidence only.
20/80 rule             © 2010 Cognifide Limited. In commercial confidence only.
CoverageManage your coverage and use it for planning manual tests • Remember about 20/80 rule • Power function • 100% - is...
Defect management • Red report effect • Quick fixes for CI • Labels                        © 2010 Cognifide Limited. In co...
Automation responsibilityWho should write and maintenanceautomated tests?                               © 2010 Cognifide L...
Developer helping handCan developer deliver automated tests together with functionality? • Who should write test cases? • ...
Automation as a projectTriangle: Budget, Time, Scope (Aim)                               © 2010 Cognifide Limited. In comm...
There is no success without:•   Continuous work•   Gather statistics for stakeholders•   Divide budget on manual and autom...
The futureAutomated Exploratory Tests for Regression                                 © 2010 Cognifide Limited. In commerci...
Aspects CI  Client                  User      Analytics                    Layout   Functionality   Performance           ...
Symptoms                                    Layout                   URL                                                  ...
Heuristic OracleThe balance between comparison and comparison not at all      Layout       Content        Code       Perfo...
Robots• Scripts• Shared between projects• Easy and quick to run            ResponsesJS errors               W3C   Analytic...
Q&A      © 2010 Cognifide Limited. In commercial confidence only.
Discussion & beerPoznań Testing and Qaulity Group                              © 2010 Cognifide Limited. In commercial con...
Upcoming SlideShare
Loading in …5
×

Automation as a project - lessons learnt

917 views

Published on

Presentation from second Poznań Testing and Quality group.
This presentation sum up my experience (about 5 years) with automation, as engineer or manager

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
917
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Automation as a project - lessons learnt

  1. 1. Poznań Testing and Quality GroupAutomation as a project –lessons learntZbyszek Moćkun © 2010 Cognifide Limited. In commercial confidence only.
  2. 2. Agenda• Introduction• Business• Technology review• Management• Summary• Discussion © 2010 Cognifide Limited. In commercial confidence only.
  3. 3. IntroductionFew words about the author © 2010 Cognifide Limited. In commercial confidence only.
  4. 4. Experience and domains Experience Domains • Mobile infrastructure• 7 years in tests • Billing and Roaming• Three different companies systems• Several domains • Business Assurance• Product based and project application based • Medical• Different methodologies • Content Management (Waterfall, Scrum, Kanban) Systems © 2010 Cognifide Limited. In commercial confidence only.
  5. 5. BusinessAutomation vs. profitability © 2010 Cognifide Limited. In commercial confidence only.
  6. 6. It’s all about the money http://www.br-automation.com © 2010 Cognifide Limited. In commercial confidence only.
  7. 7. How to talk with stakeholders• Time of money return• Use statistics/metrics• Remeber that statistics show incorrect data (only regression issues should be calculated)• It’s not about how many issues were found• Statistics fall down without continuous investement• Code resistance on the same tests• Decrease release time (time for manual tests)• Number of issues found influence on time needed for regression © 2010 Cognifide Limited. In commercial confidence only.
  8. 8. How to start?• Show metrics • Why does we need it? • How much does it cost? • What does we want to achieve? • What are the goals?• Small steps • First step should be cheap and quick • Show results for stakeholder • Plan and estimate next steps• Gather metrics and statistics © 2010 Cognifide Limited. In commercial confidence only.
  9. 9. Approaches Product- Project- based based © 2010 Cognifide Limited. In commercial confidence only.
  10. 10. Product against project Product-based Project-based • No need of scalability • Scalability • Only basic modularity • Modularity • Coverage – main features • Coverage – as much as possible • Cheap - finite number of runs • Expensive - infinite number • Reuse for other projects (not of runs always possible) • Reuse for other products • No place for planning new features and improvements • Planing new features and improvements • Manual tests are cheaper at the most cases © 2010 Cognifide Limited. In commercial confidence only.
  11. 11. AutomationTechnology © 2010 Cognifide Limited. In commercial confidence only.
  12. 12. Continuous IntegrationOne place for automation management © 2010 Cognifide Limited. In commercial confidence only.
  13. 13. Where should automated tests be run? • On developer machine if possible, before commit • As sanity check on other machines • Developer iterative work (migration as example) • Should we run autoamted tests on production (live)? • Independent test data Developer Nightly Iteration Staging Production machine CI tool © 2010 Cognifide Limited. In commercial confidence only.
  14. 14. Reports• Nightly tests – preferred frequency• Build results management (disc space)• Reports improvements • Minimalize time spend on report (verification/debugging) • Display as much data as possible for failures • One issue failing several tess script• Environment errors - stability• Red reports effect • Issues are not fixed • A lof of failed tests cases • Noone look at the results © 2010 Cognifide Limited. In commercial confidence only.
  15. 15. Reports © 2010 Cognifide Limited. In commercial confidence only.
  16. 16. Dependent orindependent?How to build good test suite? © 2010 Cognifide Limited. In commercial confidence only.
  17. 17. Independent and random• All tests are independent • Pre – precondition (set require state) • Post – postcondition • Test – test script • Rec – recovery procedure after test failure (pre i post should have defined too) • LogData – gather all logs after failure, very important for „random” failures• Random order• Repeat counter• Dependent only for CI and adult teams (founded issues are fixed as fast as possible) © 2010 Cognifide Limited. In commercial confidence only.
  18. 18. Test dataHow to manage test data? Load them orcreate? © 2010 Cognifide Limited. In commercial confidence only.
  19. 19. Record & PlayDoes it work? © 2010 Cognifide Limited. In commercial confidence only.
  20. 20. Selenium IDE as example • Stability issues • Recorded values are not meaningfull • Require rewrite • User-extension.js (write own commands) • Quick and easy debug • Project-based – preferred as cheap and quick • Product-based – as starting point • More info link © 2010 Cognifide Limited. In commercial confidence only.
  21. 21. Product against project HTML Java• Quick and easy (+)• Cost return (+)• Coverage – max 10%-20% (-) • Flexibility (+)• Modularity using user- • Scalability (+) extension.js (=/-) • Modularity (+)• No need to environment • Coverage (+) setup (+) • Development knowledge (+/-)• No scalability (-) • Environment setup (-)• Cnvert to other languages • No convert possibility (+/-) (+) • Debug time (-)• HTML Suites (works with CI) • Reporting (JUnit, TestNG) (-) (+)• Integrated reports © 2010 Cognifide Limited. In commercial confidence only.
  22. 22. SurveyTwo similar application and different technologies based on Selenium: • HTML based on Selenium IDE • Java based on Selenium RC/EclipseHTML – 50% more test cases writtenJava – 20% more time spend on maintenance/debuggingWhich technology would you like to use?Java devs choose HTML/Selenium IDE as more friendly and faster © 2010 Cognifide Limited. In commercial confidence only.
  23. 23. AutomationManagement © 2010 Cognifide Limited. In commercial confidence only.
  24. 24. CoverageManage, monitor and plan © 2010 Cognifide Limited. In commercial confidence only.
  25. 25. 20/80 rule © 2010 Cognifide Limited. In commercial confidence only.
  26. 26. CoverageManage your coverage and use it for planning manual tests • Remember about 20/80 rule • Power function • 100% - is it possible? • Network architecture as example (100% achieved) • Human factor caused that 100% is not achievable © 2010 Cognifide Limited. In commercial confidence only.
  27. 27. Defect management • Red report effect • Quick fixes for CI • Labels © 2010 Cognifide Limited. In commercial confidence only.
  28. 28. Automation responsibilityWho should write and maintenanceautomated tests? © 2010 Cognifide Limited. In commercial confidence only.
  29. 29. Developer helping handCan developer deliver automated tests together with functionality? • Who should write test cases? • Scripts written using invalid test cases • Scripts written for not working functionality • Lack of time – functionality is more important • Control needed (sometimes devs forget) • Two unsuccessful attempts © 2010 Cognifide Limited. In commercial confidence only.
  30. 30. Automation as a projectTriangle: Budget, Time, Scope (Aim) © 2010 Cognifide Limited. In commercial confidence only.
  31. 31. There is no success without:• Continuous work• Gather statistics for stakeholders• Divide budget on manual and automated tests• Planning the future – where want to be in next year?• Control your costs• Remember – it’s all about the money © 2010 Cognifide Limited. In commercial confidence only.
  32. 32. The futureAutomated Exploratory Tests for Regression © 2010 Cognifide Limited. In commercial confidence only.
  33. 33. Aspects CI Client User Analytics Layout Functionality Performance © 2010 Cognifide Limited. In commercial confidence only.
  34. 34. Symptoms Layout URL Content strategy Link Code checker Bug W3C JS errors Respons Analitycs e (404) © 2010 Cognifide Limited. In commercial confidence only.
  35. 35. Heuristic OracleThe balance between comparison and comparison not at all Layout Content Code Performance © 2010 Cognifide Limited. In commercial confidence only.
  36. 36. Robots• Scripts• Shared between projects• Easy and quick to run ResponsesJS errors W3C Analytics Link checker Url strategy (404) © 2010 Cognifide Limited. In commercial confidence only.
  37. 37. Q&A © 2010 Cognifide Limited. In commercial confidence only.
  38. 38. Discussion & beerPoznań Testing and Qaulity Group © 2010 Cognifide Limited. In commercial confidence only.

×