This document outlines 14 lessons learned about automation testing. Key points include that automation requires resources to develop, maintain and run scripts; automated tests can miss bugs if not run manually as well; a framework is needed to manage thousands of scripts; scripts should use standard languages and be data-driven and independent of testbeds; and logs are more important than scripts for debugging failures. Separating script writers from runners and using databases to store results are also advised.

1. Lessons Learned on
Automation
Liang Gao (liangg@gmail.com)

2. Lesson -1 Automation Cost Money

You need people to write scripts

You need testbed to run those scripts (lots of
equipments)

You need people to monitor those test bed
runs, and debug running failures.

You need people to maintain those scripts to
reflect product change

Can you justify all those to your upper
management?

3. You might if you

Show some proof in the front.

Can go to your boss and say: I can reduce
major release cycle from 8 month to 5
month if I do this.

Can go to your boss and say: I can reduce
customer found bugs down by 20% if I do
this.

4. Lesson 2: Automated Testing is Dangerous

Once it is automated, chances are, it will not
be manually executed anymore.

No exploratory testing can be done

If your script has problem, and can not catch
bugs (output is pass even it should be fail), it
will be going into darkness for ever

You may lose your chance to catch bugs for
ever and you don’t know.

5. Lesson 3: No Framework, No Automation

If you have one couple of hundred scripts in
hand, you might be fine without a framework.

You need one if you have thousands of scripts
to manage.

How to know which test case passed/failed?

Where to get a decent running report.

How to group test cases easily.

How to debug

You need one if there are many people develop
scripts in parallel.

6. Lesson 4: Use Standard Script Language

VB/TCL/Perl/Python/Ruby

Customized scripts need learning period

Customized script language – Who maintain
it? No community support

Hard to communicate with others –
developers, other test outside the company

7. Lesson 5: Separate writer and runner

Engineer should not develop script and then
execute script. Script execution is a dedicated
job.

Debug takes time

Test bed problem?

Script problem?

Image regression bug?

Script integration takes time.

Script execution should be a 24/7 factory,
should be a machine. Script is just a by-
product, it is the full version report that you
want

8. Lesson 6: Test Bed Independent is very Important

Separate the writer and runner requires the script
should be testbed independent.

Script should be able to run from testbed to testbed
with minimum change to the test framework
configuration, not to script itself.

Hard coded router/switch names, IP addresses,
interface names are not good when switch testbeds

A handover process is needed between the writer
and runner.

Develop a “script checker” tool to check the hard
coded values in the script as an acceptance criteria.

9. Lesson 7:
Manage Your Scripts the Same as Your Bugs

Script need to have states like bugs

(S) – Submitted: Script is submitted to the
regression team

(A) – Assigned: Script is assigned to a regression
engineer

(I) – Integration: regression engineer is putting it on
the regression test bed.

(P) – Production: Script is in the regression testbed
and will be run on regression testing against release

(F) - Feedback needed: Script has errors, more
feedback from writer needed.

10. Lesson 8: Design You Scripts As Data Driven.

Script need to be data driven

Different data means different test cases.
Test_case.tcl {router1 eth0} {router2 eth0}
{router3 eth0 eth1} {router4 eth0 eth1 eth2}
{mode 1} {phase 1} {traffic 1 speed}
Can test all combinations of Mode, Phase and
Traffic with one single script.

Data generation can be automated too.

Can catch more bugs when vary the data

11. Lesson 9: Log is More Important Than
Script

When fail, most of the time we only look at
the logs, not scripts for debugging.

Read log like read a book

More debugging info dumped when fail.

12. Lesson 10:
test case designer and automator separate?

Don’t use automator who doesn’t respect
testing

Don’t use automator who doesn’t
understand testing

C company use same tester to design,
manual execute and automate the test
cases. And so is J company

13. Lesson 11: User Standard Testbed
DUT2 DUT3 DUT4
VLAN1
VLAN2
VLAN3
VLAN4
Four Ethernet on each of the Device Under Test
1 2 3 4 1 2 3 4 1 2 3 4
被测设备 2
DUT1
1 2 3 4

14. Lesson 11: User Standard Testbed
DUT1 DUT2 DUT3 DUT4
Ethernet1 Open Open Shutdown Shutdown
Ethernet 2 Open Shutdown Open Shutdown
Ethernet 3 Shutdown Open Open Shutdown
Ethernet 4 Shutdown Shutdown Shutdown Shutdown
Logical Topology:
DUT2
VLAN1
VLAN2
VLAN3
DUT1
DUT3

15. Lesson 12:
Dynamic script allocation

Different release has different feature set

Each regression run need to pick different
scripts to run

Tricks for dynamic script allocation

Static file for version matching

Database

16. Lesson 13:
Use Database to keep automation result history

Database for full version report

Database for regression bug debugging

17. Lesson 14:
Script maintenance – version per branch

Certify scripts for each new release

Keep them in a separate code branch

18. Version control

Version 1.0, Dec 24th, 2007, Liang Gao

Version 1.1 March 2008, Liang Gao