Regression testing is important to ensure new software changes do not break existing functionality. Automating regression testing helps manage the large number of test cases needed and speeds up release cycles. Key aspects of managing regression include establishing a baseline, comparing new results to the baseline, debugging failures efficiently, and automating testing processes to reduce human effort and testing time.
2. Agenda
• Regression testing and Regression bugs.
• Regression’s When, What and How
• Importance of the automated regression
testing
• Managing regression testing
3. What is a regression testing
uncover regression bugs.
5. Why ？？
• No software design is 100% modulized.
• IP Network is layered (ISO 7 layers), break one
layer, rest of the layers won’t work
• The introduction of new code breaks the
existing code
• For Example:
– IPSec use Path MTU discovery
– ICMP can probe the MTU value
– IPSec use ICMP to change the MTU valule.
– If ICMP new code breaks the MTU function, part of
the IPSec won’t work
6. Regression bug is bad, very bad
• For a new product release, customer is ok that if the newly added
function has bugs, since the common understanding is new code is buggy.
And not all customer will use the newly added function anyway.
• But customer is not OK that the previously worked function is broken in
the new release, because that make things worse.
Don’t Piss Off the
Customer…..
7. How to Catch the Regression Bug
Use your analytical skills,
how to catch the
regression bug?
8. 3 Reasons A Regression Test Case Fail
• It is because of the testbed.
– Testbed instabilities.
– Cable connections. Switch issues. Bad hardware
– Improper testbed configurations.
• It is because of the script issues
– Version update of output changes.
– Bad quality scripts.
• It is a software problem
– Then it could be a regression bug.
9. Understand Clean Run
• “Clean Run” means after running all your
regression test cases against a release, all
failed test cases has been analyzed on the
reason of failure.
• Test cases result is either “PASS” or “FAIL”
with a defects ID or Issue ID (script or test
bed problem).
10. Understand “Baseline”
• Before running every regression against certain
release, always ask: What is the baseline of
this release?
• The baseline release is a “Clean Run” release
with each test bases be either “PASS” or “FAIL”
with a defects ID. All failed test cases have
been debugged and found the root cause.
• The current result will always be compared
with the baseline result for delta (differences)
• The best baseline is to have high percentage of
the PASS cases.
11. Compare with Baseline
• The current release regression result has
the same result as baseline
– Baseline PASS, current release PASS
– Baseline FAIL, current release FAIL
• Different than baseline
– Baseline PASS, current release FAIL
– Baseline FAIL, current release PASS
12. Benefit of the Baseline
• Identify regression bug quickly.
• Shorten your debugging time.
– Most of the case, if baseline also failed, it
should fail on the same reason.
– when you run 10K+ test cases and have 5%
failure rate, baseline result speed up your
debugging time greatly.
• Better test result management
– Have a traceable result history
13. When to Start the Regression testing
• Before or after the new feature testing?
• Before or after the system testing?
• Before or after the solution testing?
• Before or after the release?
• How many times you think a regression
testing should be run?
14. Regression is headache
• You have no choice but to do it. (don’t piss off the customer)
• You have to do it whenever to release a new version
• You may have to do it for every build
• As more features added, regression test cases keep growing and growing
and growing….
• What if your company release a new build every week, or every other
day? (Cisco does) Regression nightmare.
• Your engineers are bored of doing the same old thing again and again.
Bored low quality of execution can not find bugs piss off the
customer with regression bugs
15. Me and the Machine
• I am flexible and can execute beyond the test case
scope (explore and play a little to find more problems)
• I need to do different things, need to explore and to be
challenged.
• I need to sleep, rest, and play 斗地主，麻将 and go
to 青城山后山的 家农 乐 during the weekend.
• Machine will do exactly what I programmed him to do
• Machine can do it over and over without complaining
with exactly the same quality
• Machine can work overnight, over weekend and over
holidays.
16. Auto Regression is the Solution
• As regression test cases exploding, it can not
possibly be run by human (Cisco IOS regression
has 100,000 plus test cases, and need to be
finished in 2 days)
• It same time with quality (48 hours, for me,
means 6 days, for machine, means 2 days)
• It speeds up the release cycle: the faster the cycle
of a full regression, the more build can be
released (Cisco can release a build every 2 days,
Juniper can release a build every 4 days. Keep in
mind IOS has more than 6 million lines of code)
17. In Order to Run Auto Regression…..
• I need someone to write scripts, convert manual test cases in to scripts.
• I need an engine to execute thousands of scripts in batch mode, collect
pass and fail info.
• I need to have libraries. So when I write scripts, I can use APIs to speed up
my development.
• I need a database to store each regression cycles test report (running
results)
• I need a way to debug when scripts are failed (is it script problem? Test
bed problem? Or indeed the image problem)
• It would be nice if I can have a frame work that I can do all this.
18. Managing Regression Testing
• Schedule the run based on product release
schedule.
• Select suites
– If I can not cover them all, which set of features I need to pick
– How long can I finish it
– How much debugging time I need for failures.
• How many engineers I need to run (1 engineer
per 2 test beds?
• Regression engineers time
– Kick off the run
– Debugging (most of the time)
19. Find the Way to Reduce the Regression Cycle
• Look at the whole cycle of the regression testing, which part use most of the time
– Test bed management?
– Debugging and defect filing?
– Reporting?
• Lots of things can be automated (process automation) during a regression cycle)
• In Cisco, during a perfect run, a regression engineer don’t need to do anything
– Automatic copy new image after build team finish it
– Automatic load the image to router
– Automatic select the features need to be run based on the image name
(different image has different feature set)
– Automatic run
– Automatic reporting
20. Regression Bug Management
• Be careful whenever filing regression bugs,
need to be 100% Sure.
• How to be sure if it is a regression bug? Use
your analytical skills.
21. Make Regression As A Train
• Pull the regression cycle out of the release
cycle.
• Make regression cycle fixed (predictable).
• Make it an on-going, 24/7 operation.
• Benefit
– Any release can schedule a time to be on the
regression train, good to arrange regression
schedule.
– More regression, more stable of the product.
22. Regression Test Bed Management
• Standard topology test bed.
• Running in parallel (multiple identical
regression test beds)
• Fixed run time on each test bed on each
regression cycle
23. Don’t let regression team buy test bed
• Regression repository will keep expanding.
• More testbeds will be needed.
• Unlimited budget for regression team to buy
testbed? Probably not.
24. manage run time by minutes.
• Based on your release cycle, carefully control
total run time on a testbed.
• Each test bed has a maximum run time, if
exceed, consider build more testbeds (with in
budget)
25. Automat Regression - Engine
• If you have 1000 scripts
– How to run it in batch mode
– How to know which one is fail, which one is pass at any given time
– How to debug when a test case failed (pause on failure)
– How to move job across test bed
– Where to report the test results
– Where to keep the historical running logs
– How to run only certain modules
– How to select cases based on different release version
– How to run scripts even device is crashed
26. Automat Regression – Test Harness
• Need a place to store all test results
• Need to have version/release info
• Need to be able to compare from release to
release
