Download as PDF, PPTX
Uploaded byZbyszek Mockun
Automation as a project - lessons learnt
The document discusses automation in testing and quality assurance, outlining lessons learned and strategies for effective implementation. It emphasizes the importance of metrics, cost considerations, and stakeholder communication in achieving successful automation projects. Key topics include the differentiation between product-based and project-based approaches, continuous integration, and the management of automated test suites.
More Related Content
Similar to Automation as a project - lessons learnt
Automation as a project - lessons learnt
- 1. Poznań Testing and Quality Group Automation as a project – lessons learnt Zbyszek Moćkun © 2010 Cognifide Limited. In commercial confidence only.
- 2. Agenda • Introduction • Business •Technology review • Management • Summary • Discussion © 2010 Cognifide Limited. In commercial confidence only.
- 3. Introduction Few words aboutthe author © 2010 Cognifide Limited. In commercial confidence only.
- 4. Experience and domains Experience Domains • Mobile infrastructure • 7 years in tests • Billing and Roaming • Three different companies systems • Several domains • Business Assurance • Product based and project application based • Medical • Different methodologies • Content Management (Waterfall, Scrum, Kanban) Systems © 2010 Cognifide Limited. In commercial confidence only.
- 5. Business Automation vs. profitability © 2010 Cognifide Limited. In commercial confidence only.
- 6. It’s all aboutthe money http://www.br-automation.com © 2010 Cognifide Limited. In commercial confidence only.
- 7. How to talkwith stakeholders • Time of money return • Use statistics/metrics • Remeber that statistics show incorrect data (only regression issues should be calculated) • It’s not about how many issues were found • Statistics fall down without continuous investement • Code resistance on the same tests • Decrease release time (time for manual tests) • Number of issues found influence on time needed for regression © 2010 Cognifide Limited. In commercial confidence only.
- 8. How to start? •Show metrics • Why does we need it? • How much does it cost? • What does we want to achieve? • What are the goals? • Small steps • First step should be cheap and quick • Show results for stakeholder • Plan and estimate next steps • Gather metrics and statistics © 2010 Cognifide Limited. In commercial confidence only.
- 9. Approaches Product- Project- based based © 2010 Cognifide Limited. In commercial confidence only.
- 10. Product against project Product-based Project-based • No need of scalability • Scalability • Only basic modularity • Modularity • Coverage – main features • Coverage – as much as possible • Cheap - finite number of runs • Expensive - infinite number • Reuse for other projects (not of runs always possible) • Reuse for other products • No place for planning new features and improvements • Planing new features and improvements • Manual tests are cheaper at the most cases © 2010 Cognifide Limited. In commercial confidence only.
- 11. Automation Technology © 2010 Cognifide Limited. In commercial confidence only.
- 12. Continuous Integration One placefor automation management © 2010 Cognifide Limited. In commercial confidence only.
- 13. Where should automatedtests be run? • On developer machine if possible, before commit • As sanity check on other machines • Developer iterative work (migration as example) • Should we run autoamted tests on production (live)? • Independent test data Developer Nightly Iteration Staging Production machine CI tool © 2010 Cognifide Limited. In commercial confidence only.
- 14. Reports • Nightly tests– preferred frequency • Build results management (disc space) • Reports improvements • Minimalize time spend on report (verification/debugging) • Display as much data as possible for failures • One issue failing several tess script • Environment errors - stability • Red reports effect • Issues are not fixed • A lof of failed tests cases • Noone look at the results © 2010 Cognifide Limited. In commercial confidence only.
- 15. Reports © 2010 Cognifide Limited. In commercial confidence only.
- 16. Dependent or independent? How tobuild good test suite? © 2010 Cognifide Limited. In commercial confidence only.
- 17. Independent and random •All tests are independent • Pre – precondition (set require state) • Post – postcondition • Test – test script • Rec – recovery procedure after test failure (pre i post should have defined too) • LogData – gather all logs after failure, very important for „random” failures • Random order • Repeat counter • Dependent only for CI and adult teams (founded issues are fixed as fast as possible) © 2010 Cognifide Limited. In commercial confidence only.
- 18. Test data How tomanage test data? Load them or create? © 2010 Cognifide Limited. In commercial confidence only.
- 19. Record & Play Doesit work? © 2010 Cognifide Limited. In commercial confidence only.
- 20. Selenium IDE asexample • Stability issues • Recorded values are not meaningfull • Require rewrite • User-extension.js (write own commands) • Quick and easy debug • Project-based – preferred as cheap and quick • Product-based – as starting point • More info link © 2010 Cognifide Limited. In commercial confidence only.
- 21. Product against project HTML Java • Quick and easy (+) • Cost return (+) • Coverage – max 10%-20% (-) • Flexibility (+) • Modularity using user- • Scalability (+) extension.js (=/-) • Modularity (+) • No need to environment • Coverage (+) setup (+) • Development knowledge (+/-) • No scalability (-) • Environment setup (-) • Cnvert to other languages • No convert possibility (+/-) (+) • Debug time (-) • HTML Suites (works with CI) • Reporting (JUnit, TestNG) (-) (+) • Integrated reports © 2010 Cognifide Limited. In commercial confidence only.
- 22. Survey Two similar applicationand different technologies based on Selenium: • HTML based on Selenium IDE • Java based on Selenium RC/Eclipse HTML – 50% more test cases written Java – 20% more time spend on maintenance/debugging Which technology would you like to use? Java devs choose HTML/Selenium IDE as more friendly and faster © 2010 Cognifide Limited. In commercial confidence only.
- 23. Automation Management © 2010 Cognifide Limited. In commercial confidence only.
- 24. Coverage Manage, monitor andplan © 2010 Cognifide Limited. In commercial confidence only.
- 25. 20/80 rule © 2010 Cognifide Limited. In commercial confidence only.
- 26. Coverage Manage your coverageand use it for planning manual tests • Remember about 20/80 rule • Power function • 100% - is it possible? • Network architecture as example (100% achieved) • Human factor caused that 100% is not achievable © 2010 Cognifide Limited. In commercial confidence only.
- 27. Defect management •Red report effect • Quick fixes for CI • Labels © 2010 Cognifide Limited. In commercial confidence only.
- 28. Automation responsibility Who shouldwrite and maintenance automated tests? © 2010 Cognifide Limited. In commercial confidence only.
- 29. Developer helping hand Candeveloper deliver automated tests together with functionality? • Who should write test cases? • Scripts written using invalid test cases • Scripts written for not working functionality • Lack of time – functionality is more important • Control needed (sometimes devs forget) • Two unsuccessful attempts © 2010 Cognifide Limited. In commercial confidence only.
- 30. Automation as aproject Triangle: Budget, Time, Scope (Aim) © 2010 Cognifide Limited. In commercial confidence only.
- 31. There is nosuccess without: • Continuous work • Gather statistics for stakeholders • Divide budget on manual and automated tests • Planning the future – where want to be in next year? • Control your costs • Remember – it’s all about the money © 2010 Cognifide Limited. In commercial confidence only.
- 32. The future Automated ExploratoryTests for Regression © 2010 Cognifide Limited. In commercial confidence only.
- 33. Aspects CI Client User Analytics Layout Functionality Performance © 2010 Cognifide Limited. In commercial confidence only.
- 34. Symptoms Layout URL Content strategy Link Code checker Bug W3C JS errors Respons Analitycs e (404) © 2010 Cognifide Limited. In commercial confidence only.
- 35. Heuristic Oracle The balancebetween comparison and comparison not at all Layout Content Code Performance © 2010 Cognifide Limited. In commercial confidence only.
- 36. Robots • Scripts • Sharedbetween projects • Easy and quick to run Responses JS errors W3C Analytics Link checker Url strategy (404) © 2010 Cognifide Limited. In commercial confidence only.
- 37. Q&A © 2010 Cognifide Limited. In commercial confidence only.
- 38. Discussion & beer PoznańTesting and Qaulity Group © 2010 Cognifide Limited. In commercial confidence only.