Your SlideShare is downloading. ×
It Seemed a Good Idea at the Time: Intelligent Mistakes in Test Automation
It Seemed a Good Idea at the Time: Intelligent Mistakes in Test Automation
It Seemed a Good Idea at the Time: Intelligent Mistakes in Test Automation
It Seemed a Good Idea at the Time: Intelligent Mistakes in Test Automation
It Seemed a Good Idea at the Time: Intelligent Mistakes in Test Automation
It Seemed a Good Idea at the Time: Intelligent Mistakes in Test Automation
It Seemed a Good Idea at the Time: Intelligent Mistakes in Test Automation
It Seemed a Good Idea at the Time: Intelligent Mistakes in Test Automation
It Seemed a Good Idea at the Time: Intelligent Mistakes in Test Automation
It Seemed a Good Idea at the Time: Intelligent Mistakes in Test Automation
It Seemed a Good Idea at the Time: Intelligent Mistakes in Test Automation
It Seemed a Good Idea at the Time: Intelligent Mistakes in Test Automation
It Seemed a Good Idea at the Time: Intelligent Mistakes in Test Automation
It Seemed a Good Idea at the Time: Intelligent Mistakes in Test Automation
It Seemed a Good Idea at the Time: Intelligent Mistakes in Test Automation
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

It Seemed a Good Idea at the Time: Intelligent Mistakes in Test Automation

215

Published on

Some test automation ideas seem very sensible at first glance but contain pitfalls and problems that can and should be avoided. Dot Graham describes five of these “intelligent mistakes”—1. Automated …

Some test automation ideas seem very sensible at first glance but contain pitfalls and problems that can and should be avoided. Dot Graham describes five of these “intelligent mistakes”—1. Automated tests will find more bugs quicker. (Automation doesn’t find bugs, tests do.) 2. Spending a lot on a tool must guarantee great benefits. (Good automation does not come “out of the box” and is not automatic.) 3. Let’s automate all of our manual tests. (This may not give you better or faster testing, and you will miss out on some benefits.) 4. Tools are expensive so we have to show a return on investment. (This is not only surprisingly difficult but may actually be harmful.) 5. Because they are called “testing tools,” they must be tools for testers to use. (Making testers become test automators may be damaging to both testing and automation.) Join Dot for a rousing discussion of “intelligent mistakes”—so you can be smart enough to avoid them.

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
215
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
2
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. It seemed a good idea at the time: Intelligent Mistakes in Test Automation Prepared and presented by Dorothy Graham email: info@dorothygraham.co.uk @DorothyGraham www.DorothyGraham.co.uk © Dorothy Graham 2013 1 Intelligent mistakes? •  mistake: –  an action resulting from defective judgment, deficient knowledge or carelessness, a misconception or misunderstanding •  intelligent –  exercising good judgment, showing mental resourcefulness, being well-informed •  intelligent mistake –  action based on a faulty premise, with the best of intentions, but with adverse consequences 2 info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  • 2. Contents •  •  •  •  •  Automation should find lots of bugs Automation comes out of a box (or download) Let’s automate all of our manual tests ROI is essential for automation Testing tools are testers’ tools 3 Good idea? Automation should find lots of bugs •  Seems very sensible –  we are automating tests –  tests find bugs –  automation is more thorough and faster –  therefore, automation should find more bugs - and quicker •  Problems –  what does automation actually give us? –  what tests are we automating? 4 info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  • 3. What is automated? most often automated likelihood of finding bugs regression tests exploratory testing 5 Automation success = find lots of bugs? •  tests find bugs, not automation •  automation is a mechanism for running tests •  the bug-finding ability of a test is not affected by the manner in which it is executed •  this can be a dangerous objective –  especially for regression automation! Automated tests Manual Scripted Exploratory Fix Verification 9.3% 24.0% 58.2% 8.4% Experiences of Test Automation, Ch 27, p 503, Ed Allen & Brian Newman info@dorothygraham.co.uk 6 © Dorothy Graham 2013 www.DorothyGraham.co.uk
  • 4. Efficiency and effectiveness better good slow testing Manual testing High good fast testing Automated Efficiency poor fast testing poor slow testing worst greatest benefit Effectiveness not good but common Low 7 When is “find more bugs” a good objective for automation? •  when the first run of a given test is automated –  Model-Based Testing (MBT), exploratory test automation, automated test design, monkey testing –  keyword-driven (e.g. users populate spreadsheet) •  find bugs in parts we wouldn’t have tested? –  indirect result of automation –  direct result of running more tests 8 info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  • 5. Contents •  •  •  •  •  Automation should find lots of bugs Automation comes out of a box (or download) Let’s automate all of our manual tests ROI is essential for automation Testing tools are testers’ tools 9 Good idea? Automation comes out of the box (or download) •  Seems very sensible –  automation gives great benefit, success stories –  spending a lot on the tool must guarantee great benefits –  all we need to do is “get the right tool” •  Problems –  technical issue: what testware architecture is right for your organisation? (hint: not the tool’s) –  management issues: staffing, support, resources 10 info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  • 6. Automation should fit your organisation •  your own testware architecture is critical –  a poor architecture gives high maintenance cost Testers&& write&tests&(in&DSTL)& •  most frequent cause of abandoned automation/ shelfware HL Keywords testware& architecture& –  two layers of abstraction – technical: for long life – human: for wide use –  using the tool’s architecture ties you to that tool [version] Structured Scripts Test&Execu+on&Tool& runs&scripts& 11 It’s not about the tool •  no such thing as “the right tool” or “best tool” –  what’s “the best car”? commercial tools? investment in good automation good benefits high cost poor benefits low cost benefits good benefits low cost cost moderate poor benefits high cost tool cost open source tools? info@dorothygraham.co.uk budget 12 © Dorothy Graham 2013 www.DorothyGraham.co.uk
  • 7. It takes time and effort to succeed •  management support is critical –  high level managers need to understand automation capability & limitations, and have realistic expectations –  building good automation is a learning process –  “people issues” – people use the automation, people develop the automation 13 Automated tests/automated testing Automated tests Select / identify test cases to run Set-up test environment: •  create test environment •  load test data Repeat for each test case: •  set-up test pre-requisites •  execute •  compare results •  log results •  analyse test failures •  report defect(s) •  clear-up after test case Clear-up test environment: •  delete unwanted data •  save important data Summarise results Manual process info@dorothygraham.co.uk Automated testing Select / identify test cases to run Set-up test environment: •  create test environment •  load test data Repeat for each test case: •  set-up test pre-requisites •  execute •  compare results •  log results •  clear-up after test case Clear-up test environment: •  delete unwanted data •  save important data Summarise results Analyse test failures Report defects Automated process 14 © Dorothy Graham 2013 www.DorothyGraham.co.uk
  • 8. Contents •  •  •  •  •  Automation should find lots of bugs Automation comes out of a box (or download) Let’s automate all of our manual tests ROI is essential for automation Testing tools are testers’ tools 15 Good idea? Let’s automate all of our tests •  Seems very sensible –  we already have lots of tests being done manually –  automation would be much quicker –  we can be completely automated, no need for manual tests [testers?!] •  Problems –  what tests are you automating? –  what are you missing by automating [only] existing tests? 16 info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  • 9. Automate manual tests? manual tests automated tests tests not automated yet tests not worth automating tests (& verification) not possible to do manually manual tests automated (% manual) new ways of automating, e.g. exploratory test automation 17 Tools will replace testers? •  “we can reduce the number of testers once we have the tool” –  what are your testers like? •  mindless morons, or •  intelligent investigators? –  need more skills, not fewer –  automation can free testers to do more test design, exploratory testing •  and find more bugs –  tools don’t replace testers, they support them 18 info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  • 10. Contents •  •  •  •  •  Automation should find lots of bugs Automation comes out of a box (or download) Let’s automate all of our manual tests ROI is essential for automation Testing tools are testers’ tools 19 Good idea? ROI is essential for automation •  Seems very sensible –  we will be spending time and possibly lots of money on tools – need to be sure it will be worth it –  a “water-tight” business case will convince our managers to invest in automation •  Problems –  ROI is not the whole story –  people are not convinced by numbers –  calculating ROI is difficult and can be dangerous 20 info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  • 11. Is this Return on Investment (ROI)? •  •  •  •  •  tests are run more often tests take less time to run it takes less human effort to run tests we can test (cover) more of the system we can run the equivalent of days / weeks of manual testing in a few minutes / hours •  faster time to market ROI = (benefit – cost) cost these are (good) benefits but are not ROI 21 How important is ROI? •  ROI can be dangerous –  may give impression that people are replaced by tools (see http://dorothygraham.blogspot.co.uk/) •  “automation is an enabler for success, not a cost reduction tool” – Yoram Mizrachi •  many achieve lasting success without measuring ROI (depends on your context) –  need to be aware of benefits (and publicize them) 22 info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  • 12. An example comparative benefits chart 80 70 60 50 40 man aut 30 20 10 0 exec speed 14 x faster times run data variety tester work 5 x more often 4 x more data 12 x less effort ROI spreadsheet – email me for a copy 23 Contents •  •  •  •  •  Automation should find lots of bugs Automation comes out of a box (or download) Let’s automate all of our manual tests ROI is essential for automation Testing tools are testers’ tools 24 info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  • 13. Good idea? Testing tools are testers’ tools •  Seems very sensible –  testers do testing – testing tools help with testing –  tools will make them better testers, because they will be able to run more tests, faster •  Problems –  tools are software packages using programming languages; tester skills are different to development skills –  spending time on / with tools means less testing, not more testing (initially at least) 25 Is it the tester’s job to automate tests? –  test automation is software development •  needs programming skills –  not all testers want to become developers •  or would be good at it –  automators need testing skills? –  if testers are automators  a conflict of interest •  do you run tests or do you automate tests? •  automation is better long-term, BUT •  deadline pressure pushes you back into manual testing 26 info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  • 14. Why are these intelligent mistakes? •  Automation should find bugs? –  tests find bugs; automated tests are still tests •  Automation comes “out of the box” –  you think you buy a solution that will just work •  Automate all manual tests? –  a starting point for tests to be automated •  Automation has to achieve ROI? –  can’t expect investment with showing some return •  Testing tools are tools for testers? –  testers write and run tests; tools run tests 27 Why are these intelligent mistakes? •  Automation should find bugs? –  tests find bugs, automation runs tests •  Automation comes “out of the box” –  effort and time, e.g. tailor your testware architecture •  Automate all manual tests? –  not all should be, automate more than manual •  Automation has to achieve ROI? –  may be dangerous, need to show benefits •  Testing tools are tools for testers? –  different skills, not all testers should code info@dorothygraham.co.uk 28 © Dorothy Graham 2013 www.DorothyGraham.co.uk
  • 15. Summary •  Many ideas seem sensible at first / on the surface, but have serious problems –  automation should find lots of bugs –  automation comes out of a box (or download) –  let’s automate all of our manual tests –  ROI is essential for automation –  testing tools are testers’ tools •  Recognise these “intelligent mistakes” and avoid them 29 More information •  downloads www.DorothyGraham.co.uk –  articles and papers •  email info@DorothyGraham.co.uk for –  Framework and test execution tool list –  ROI calculator –  my random newsletter •  blog http://dorothygraham.blogspot.com –  including automation, DDP, certification •  twitter –  @DorothyGraham 30 info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk

×