Lessons learned on software testing automation


Published on

lessoned learned from the field.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Lessons learned on software testing automation

  1. 1. Lessons Learned onAutomationLiang Gao (liangg@gmail.com)
  2. 2. Lesson -1 Automation Cost MoneyYou need people to write scriptsYou need testbed to run those scripts (lots ofequipments)You need people to monitor those test bedruns, and debug running failures.You need people to maintain those scripts toreflect product changeCan you justify all those to your uppermanagement?
  3. 3. You might if youShow some proof in the front.Can go to your boss and say: I can reducemajor release cycle from 8 month to 5month if I do this.Can go to your boss and say: I can reducecustomer found bugs down by 20% if I dothis.
  4. 4. Lesson 2: Automated Testing is DangerousOnce it is automated, chances are, it will notbe manually executed anymore.No exploratory testing can be doneIf your script has problem, and can not catchbugs (output is pass even it should be fail), itwill be going into darkness for everYou may lose your chance to catch bugs forever and you don’t know.
  5. 5. Lesson 3: No Framework, No AutomationIf you have one couple of hundred scripts inhand, you might be fine without a framework.You need one if you have thousands of scriptsto manage.How to know which test case passed/failed?Where to get a decent running report.How to group test cases easily.How to debugYou need one if there are many people developscripts in parallel.
  6. 6. Lesson 4: Use Standard Script LanguageVB/TCL/Perl/Python/RubyCustomized scripts need learning periodCustomized script language – Who maintainit? No community supportHard to communicate with others –developers, other test outside the company
  7. 7. Lesson 5: Separate writer and runnerEngineer should not develop script and thenexecute script. Script execution is a dedicatedjob.Debug takes timeTest bed problem?Script problem?Image regression bug?Script integration takes time.Script execution should be a 24/7 factory,should be a machine. Script is just a by-product, it is the full version report that youwant
  8. 8. Lesson 6: Test Bed Independent is very ImportantSeparate the writer and runner requires the scriptshould be testbed independent.Script should be able to run from testbed to testbedwith minimum change to the test frameworkconfiguration, not to script itself.Hard coded router/switch names, IP addresses,interface names are not good when switch testbedsA handover process is needed between the writerand runner.Develop a “script checker” tool to check the hardcoded values in the script as an acceptance criteria.
  9. 9. Lesson 7:Manage Your Scripts the Same as Your BugsScript need to have states like bugs(S) – Submitted: Script is submitted to theregression team(A) – Assigned: Script is assigned to a regressionengineer(I) – Integration: regression engineer is putting it onthe regression test bed.(P) – Production: Script is in the regression testbedand will be run on regression testing against release(F) - Feedback needed: Script has errors, morefeedback from writer needed.
  10. 10. Lesson 8: Design You Scripts As Data Driven.Script need to be data drivenDifferent data means different test cases.Test_case.tcl {router1 eth0} {router2 eth0}{router3 eth0 eth1} {router4 eth0 eth1 eth2}{mode 1} {phase 1} {traffic 1 speed}Can test all combinations of Mode, Phase andTraffic with one single script.Data generation can be automated too.Can catch more bugs when vary the data
  11. 11. Lesson 9: Log is More Important ThanScriptWhen fail, most of the time we only look atthe logs, not scripts for debugging.Read log like read a bookMore debugging info dumped when fail.
  12. 12. Lesson 10:test case designer and automator separate?Don’t use automator who doesn’t respecttestingDon’t use automator who doesn’tunderstand testingC company use same tester to design,manual execute and automate the testcases. And so is J company
  13. 13. Lesson 11: User Standard TestbedDUT2 DUT3 DUT4VLAN1VLAN2VLAN3VLAN4Four Ethernet on each of the Device Under Test1 2 3 4 1 2 3 4 1 2 3 4被测设备 2DUT11 2 3 4
  14. 14. Lesson 11: User Standard TestbedDUT1 DUT2 DUT3 DUT4Ethernet1 Open Open Shutdown ShutdownEthernet 2 Open Shutdown Open ShutdownEthernet 3 Shutdown Open Open ShutdownEthernet 4 Shutdown Shutdown Shutdown ShutdownLogical Topology:DUT2VLAN1VLAN2VLAN3DUT1DUT3
  15. 15. Lesson 12:Dynamic script allocationDifferent release has different feature setEach regression run need to pick differentscripts to runTricks for dynamic script allocationStatic file for version matchingDatabase
  16. 16. Lesson 13:Use Database to keep automation result historyDatabase for full version reportDatabase for regression bug debugging
  17. 17. Lesson 14:Script maintenance – version per branchCertify scripts for each new releaseKeep them in a separate code branch
  18. 18. Version controlVersion 1.0, Dec 24th, 2007, Liang GaoVersion 1.1 March 2008, Liang Gao