Understand regression testing


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Understand regression testing

  1. 1. Understand Regression Testing Testing Tiger Team (T Cube) Liang Gao
  2. 2. Agenda • Regression testing and Regression bugs. • Regression’s When, What and How • Importance of the automated regression testing • Managing regression testing
  3. 3. What is a regression testing uncover regression bugs.
  4. 4. Regression Bug? 很多功能以前好好的,加 入新功能以后就不 Work 了!
  5. 5. Why ?? • No software design is 100% modulized. • IP Network is layered (ISO 7 layers), break one layer, rest of the layers won’t work • The introduction of new code breaks the existing code • For Example: – IPSec use Path MTU discovery – ICMP can probe the MTU value – IPSec use ICMP to change the MTU valule. – If ICMP new code breaks the MTU function, part of the IPSec won’t work
  6. 6. Regression bug is bad, very bad • For a new product release, customer is ok that if the newly added function has bugs, since the common understanding is new code is buggy. And not all customer will use the newly added function anyway. • But customer is not OK that the previously worked function is broken in the new release, because that make things worse. Don’t Piss Off the Customer…..
  7. 7. How to Catch the Regression Bug Use your analytical skills, how to catch the regression bug?
  8. 8. 3 Reasons A Regression Test Case Fail • It is because of the testbed. – Testbed instabilities. – Cable connections. Switch issues. Bad hardware – Improper testbed configurations. • It is because of the script issues – Version update of output changes. – Bad quality scripts. • It is a software problem – Then it could be a regression bug.
  9. 9. Understand Clean Run • “Clean Run” means after running all your regression test cases against a release, all failed test cases has been analyzed on the reason of failure. • Test cases result is either “PASS” or “FAIL” with a defects ID or Issue ID (script or test bed problem).
  10. 10. Understand “Baseline” • Before running every regression against certain release, always ask: What is the baseline of this release? • The baseline release is a “Clean Run” release with each test bases be either “PASS” or “FAIL” with a defects ID. All failed test cases have been debugged and found the root cause. • The current result will always be compared with the baseline result for delta (differences) • The best baseline is to have high percentage of the PASS cases.
  11. 11. Compare with Baseline • The current release regression result has the same result as baseline – Baseline PASS, current release PASS – Baseline FAIL, current release FAIL • Different than baseline – Baseline PASS, current release FAIL – Baseline FAIL, current release PASS
  12. 12. Benefit of the Baseline • Identify regression bug quickly. • Shorten your debugging time. – Most of the case, if baseline also failed, it should fail on the same reason. – when you run 10K+ test cases and have 5% failure rate, baseline result speed up your debugging time greatly. • Better test result management – Have a traceable result history
  13. 13. When to Start the Regression testing • Before or after the new feature testing? • Before or after the system testing? • Before or after the solution testing? • Before or after the release? • How many times you think a regression testing should be run?
  14. 14. Regression is headache • You have no choice but to do it. (don’t piss off the customer) • You have to do it whenever to release a new version • You may have to do it for every build • As more features added, regression test cases keep growing and growing and growing…. • What if your company release a new build every week, or every other day? (Cisco does)  Regression nightmare. • Your engineers are bored of doing the same old thing again and again. Bored  low quality of execution can not find bugs  piss off the customer with regression bugs
  15. 15. Me and the Machine • I am flexible and can execute beyond the test case scope (explore and play a little to find more problems) • I need to do different things, need to explore and to be challenged. • I need to sleep, rest, and play 斗地主,麻将 and go to 青城山后山的 家农 乐 during the weekend. • Machine will do exactly what I programmed him to do • Machine can do it over and over without complaining with exactly the same quality • Machine can work overnight, over weekend and over holidays.
  16. 16. Auto Regression is the Solution • As regression test cases exploding, it can not possibly be run by human (Cisco IOS regression has 100,000 plus test cases, and need to be finished in 2 days) • It same time with quality (48 hours, for me, means 6 days, for machine, means 2 days) • It speeds up the release cycle: the faster the cycle of a full regression, the more build can be released (Cisco can release a build every 2 days, Juniper can release a build every 4 days. Keep in mind IOS has more than 6 million lines of code)
  17. 17. In Order to Run Auto Regression….. • I need someone to write scripts, convert manual test cases in to scripts. • I need an engine to execute thousands of scripts in batch mode, collect pass and fail info. • I need to have libraries. So when I write scripts, I can use APIs to speed up my development. • I need a database to store each regression cycles test report (running results) • I need a way to debug when scripts are failed (is it script problem? Test bed problem? Or indeed the image problem) • It would be nice if I can have a frame work that I can do all this.
  18. 18. Managing Regression Testing • Schedule the run based on product release schedule. • Select suites – If I can not cover them all, which set of features I need to pick – How long can I finish it – How much debugging time I need for failures. • How many engineers I need to run (1 engineer per 2 test beds? • Regression engineers time – Kick off the run – Debugging (most of the time)
  19. 19. Find the Way to Reduce the Regression Cycle • Look at the whole cycle of the regression testing, which part use most of the time – Test bed management? – Debugging and defect filing? – Reporting? • Lots of things can be automated (process automation) during a regression cycle) • In Cisco, during a perfect run, a regression engineer don’t need to do anything – Automatic copy new image after build team finish it – Automatic load the image to router – Automatic select the features need to be run based on the image name (different image has different feature set) – Automatic run – Automatic reporting
  20. 20. Regression Bug Management • Be careful whenever filing regression bugs, need to be 100% Sure. • How to be sure if it is a regression bug? Use your analytical skills.
  21. 21. Make Regression As A Train • Pull the regression cycle out of the release cycle. • Make regression cycle fixed (predictable). • Make it an on-going, 24/7 operation. • Benefit – Any release can schedule a time to be on the regression train, good to arrange regression schedule. – More regression, more stable of the product.
  22. 22. Regression Test Bed Management • Standard topology test bed. • Running in parallel (multiple identical regression test beds) • Fixed run time on each test bed on each regression cycle
  23. 23. Don’t let regression team buy test bed • Regression repository will keep expanding. • More testbeds will be needed. • Unlimited budget for regression team to buy testbed? Probably not.
  24. 24. manage run time by minutes. • Based on your release cycle, carefully control total run time on a testbed. • Each test bed has a maximum run time, if exceed, consider build more testbeds (with in budget)
  25. 25. Automat Regression - Engine • If you have 1000 scripts – How to run it in batch mode – How to know which one is fail, which one is pass at any given time – How to debug when a test case failed (pause on failure) – How to move job across test bed – Where to report the test results – Where to keep the historical running logs – How to run only certain modules – How to select cases based on different release version – How to run scripts even device is crashed
  26. 26. Automat Regression – Test Harness • Need a place to store all test results • Need to have version/release info • Need to be able to compare from release to release
  27. 27. Version control • 1st draft, Feb, 2008, Liang Gao