Your SlideShare is downloading. ×
0
Lessons learned on software testing automation
Lessons learned on software testing automation
Lessons learned on software testing automation
Lessons learned on software testing automation
Lessons learned on software testing automation
Lessons learned on software testing automation
Lessons learned on software testing automation
Lessons learned on software testing automation
Lessons learned on software testing automation
Lessons learned on software testing automation
Lessons learned on software testing automation
Lessons learned on software testing automation
Lessons learned on software testing automation
Lessons learned on software testing automation
Lessons learned on software testing automation
Lessons learned on software testing automation
Lessons learned on software testing automation
Lessons learned on software testing automation
Lessons learned on software testing automation
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Lessons learned on software testing automation

197

Published on

lessoned learned from the field.

lessoned learned from the field.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
197
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
18
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Lessons Learned onAutomationLiang Gao (liangg@gmail.com)
  • 2. Lesson -1 Automation Cost MoneyYou need people to write scriptsYou need testbed to run those scripts (lots ofequipments)You need people to monitor those test bedruns, and debug running failures.You need people to maintain those scripts toreflect product changeCan you justify all those to your uppermanagement?
  • 3. You might if youShow some proof in the front.Can go to your boss and say: I can reducemajor release cycle from 8 month to 5month if I do this.Can go to your boss and say: I can reducecustomer found bugs down by 20% if I dothis.
  • 4. Lesson 2: Automated Testing is DangerousOnce it is automated, chances are, it will notbe manually executed anymore.No exploratory testing can be doneIf your script has problem, and can not catchbugs (output is pass even it should be fail), itwill be going into darkness for everYou may lose your chance to catch bugs forever and you don’t know.
  • 5. Lesson 3: No Framework, No AutomationIf you have one couple of hundred scripts inhand, you might be fine without a framework.You need one if you have thousands of scriptsto manage.How to know which test case passed/failed?Where to get a decent running report.How to group test cases easily.How to debugYou need one if there are many people developscripts in parallel.
  • 6. Lesson 4: Use Standard Script LanguageVB/TCL/Perl/Python/RubyCustomized scripts need learning periodCustomized script language – Who maintainit? No community supportHard to communicate with others –developers, other test outside the company
  • 7. Lesson 5: Separate writer and runnerEngineer should not develop script and thenexecute script. Script execution is a dedicatedjob.Debug takes timeTest bed problem?Script problem?Image regression bug?Script integration takes time.Script execution should be a 24/7 factory,should be a machine. Script is just a by-product, it is the full version report that youwant
  • 8. Lesson 6: Test Bed Independent is very ImportantSeparate the writer and runner requires the scriptshould be testbed independent.Script should be able to run from testbed to testbedwith minimum change to the test frameworkconfiguration, not to script itself.Hard coded router/switch names, IP addresses,interface names are not good when switch testbedsA handover process is needed between the writerand runner.Develop a “script checker” tool to check the hardcoded values in the script as an acceptance criteria.
  • 9. Lesson 7:Manage Your Scripts the Same as Your BugsScript need to have states like bugs(S) – Submitted: Script is submitted to theregression team(A) – Assigned: Script is assigned to a regressionengineer(I) – Integration: regression engineer is putting it onthe regression test bed.(P) – Production: Script is in the regression testbedand will be run on regression testing against release(F) - Feedback needed: Script has errors, morefeedback from writer needed.
  • 10. Lesson 8: Design You Scripts As Data Driven.Script need to be data drivenDifferent data means different test cases.Test_case.tcl {router1 eth0} {router2 eth0}{router3 eth0 eth1} {router4 eth0 eth1 eth2}{mode 1} {phase 1} {traffic 1 speed}Can test all combinations of Mode, Phase andTraffic with one single script.Data generation can be automated too.Can catch more bugs when vary the data
  • 11. Lesson 9: Log is More Important ThanScriptWhen fail, most of the time we only look atthe logs, not scripts for debugging.Read log like read a bookMore debugging info dumped when fail.
  • 12. Lesson 10:test case designer and automator separate?Don’t use automator who doesn’t respecttestingDon’t use automator who doesn’tunderstand testingC company use same tester to design,manual execute and automate the testcases. And so is J company
  • 13. Lesson 11: User Standard TestbedDUT2 DUT3 DUT4VLAN1VLAN2VLAN3VLAN4Four Ethernet on each of the Device Under Test1 2 3 4 1 2 3 4 1 2 3 4被测设备 2DUT11 2 3 4
  • 14. Lesson 11: User Standard TestbedDUT1 DUT2 DUT3 DUT4Ethernet1 Open Open Shutdown ShutdownEthernet 2 Open Shutdown Open ShutdownEthernet 3 Shutdown Open Open ShutdownEthernet 4 Shutdown Shutdown Shutdown ShutdownLogical Topology:DUT2VLAN1VLAN2VLAN3DUT1DUT3
  • 15. Lesson 12:Dynamic script allocationDifferent release has different feature setEach regression run need to pick differentscripts to runTricks for dynamic script allocationStatic file for version matchingDatabase
  • 16. Lesson 13:Use Database to keep automation result historyDatabase for full version reportDatabase for regression bug debugging
  • 17. Lesson 14:Script maintenance – version per branchCertify scripts for each new releaseKeep them in a separate code branch
  • 18. Version controlVersion 1.0, Dec 24th, 2007, Liang GaoVersion 1.1 March 2008, Liang Gao

×