Your SlideShare is downloading. ×
  • Like
Continuous Automated Regression Testing to the Rescue
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Continuous Automated Regression Testing to the Rescue

  • 360 views
Published

A major concern when developing new software features is that another part of the code will be affected in unexpected ways. With a typical development processes, testers often do not run a full set of …

A major concern when developing new software features is that another part of the code will be affected in unexpected ways. With a typical development processes, testers often do not run a full set of product regression tests until late in the release when it is much more costly to fix and retest the product. Continuous automated regression testing to the rescue! Brenda Kise describes the team, project, and organization value and benefits of continuously performing automated regression tests throughout the development process. Brenda explains how this practice saves time and money in the long-run because the team and stakeholders gain an ongoing understanding of the quality of the code base every time a new build becomes available. Brenda describes the different approaches for introducing the practices of continuous automated regression testing into your organization. Find out how to create your immediate feedback mechanism to highlight the new code that creates regression defects.

Published in Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
360
On SlideShare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
5
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1.   BT4 Concurrent Session  11/14/2013 10:15 AM            "Continuous Automated Regression Testing to the Rescue"       Presented by: Brenda Kise ProtoLabs, Inc.             Brought to you by:        340 Corporate Way, Suite 300, Orange Park, FL 32073  888‐268‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
  • 2. Brenda Kise Proto Labs, Inc. Brenda Kise has fifteen years of experience in software, including development, test, test automation, management, and operations. Brenda has led teams that have automated testing from the ground up with a new code base and for Symantec’s NetBackup division, leading a team automating a legacy code base. In the first team, close to 100 percent automation was achieved using the JUnit framework in a tightly integrated Scrum/agile environment and went from a formal test cycle of six weeks to just eleven hours! At Symantec, with an 8 million line code base, the automation team focused on automating regression tests against the completed code base - going from 0% coverage to 14% coverage in one year. Brenda is currently managing a team of .NET developers working on internal applications and the Enterprise Data team doing business intelligence work for Proto Labs Inc.    
  • 3. 9/9/13   Continuous Automated Regression Testing to the Rescue! BRENDA KISE DEVELOPMENT MANAGER, PROTOLABS BRENDA.KISE@PROTOLABS.COM Brenda Kise - Proto Labs Inc. Who am I? —  Currently a development manager for Proto Labs Inc in Maple Plain MN —  Manage a team of C# developers —  Manage a team of Business Intelligence / Data Warehouse developers —  Proto Labs is a software company that does a little manufacturing on the side —  Hiring – growing by leaps and bounds Brenda Kise - Proto Labs Inc. 1  
  • 4. 9/9/13   Background —  16 years software experience —  Java / Database developer —  Managed operations, development, hardware buildouts, quality, test and automation teams —  Test team used Junit to automate 99% of tests ¡  Went from a 3-6 week test cycle to 8 hours —  Automation team gained code coverage ground against 200 developers Brenda Kise - Proto Labs Inc. Earlier is better —  The earlier after code check-in a defect is found, the easier it is to fix it —  Few people affected the earlier the defect is found —  Less paperwork involved when found earlier —  Often several layers of testing – if found early, less re-testing is needed —  If a defect makes it to customers, the repercussions can affect more than just the software Brenda Kise - Proto Labs Inc. 2  
  • 5. 9/9/13   Proof Earlier is Better Brenda Kise - Proto Labs Inc. Automated Testing —  Different levels for automation ¡  Component / Unit testing ÷ Generally done by developers ¡  Integration ¡  System level ¡  GUI Brenda Kise - Proto Labs Inc. 3  
  • 6. 9/9/13   How is the test automated? —  Run with each build ¡  Applications such as Hudson will run the unit tests and fail the build if the tests do not pass ¡  Immediate feedback, helps keep the build clean ¡  Can tell if someone is following process or not ÷ Broken build indicates coder did not run the build prior to checking into the main branch Brenda Kise - Proto Labs Inc. How is the test automated? —  Run on a schedule ¡  Generally need a test frame work to run the tests ¡  Developers can run the tests prior to check-in ¡  Process is dis-associated with the build ÷ Build is completed ÷ Regression tests kicked off through a chron job ÷ Need a check to ensure build does not fail prior to test kick-off ÷ Generally uses a separate system Brenda Kise - Proto Labs Inc. 4  
  • 7. 9/9/13   How is the test automated? —  Tester kicks off the test ¡  While this type of test is automated, it is not considered automated regression ¡  No schedule or automated kick-off, requires human intervention ¡  Still valuable as a time saving tool – allows the tester to work on something else while test is running ¡  Especially valuable for long scenarios Brenda Kise - Proto Labs Inc. How? —  Have a plan ¡  Are you working with a new code base? ¡  Are you focusing on unit / component testing? ¡  Is the code base stable? —  What level of testing? ¡  System ¡  Unit ¡  GUI Brenda Kise - Proto Labs Inc. 5  
  • 8. 9/9/13   How? —  Choose your tools ¡  Custom framework? ¡  Junit or Cunit? ¡  OTS software? ÷ Watir ÷ Selenium ÷ QuickTest Pro ÷ WinRunner Brenda Kise - Proto Labs Inc. How? —  Choose your approach ¡  Testers automate tests as they have time ÷ Like that ever happens ÷ Always seen as a secondary task ¡  Separate team of automaters ÷ Take tests that the testers use and automate ÷ Do not need to be experts in the areas of code ÷ Relies on well documented test scripts / scenarios ÷ Main focus of job Brenda Kise - Proto Labs Inc. 6  
  • 9. 9/9/13   How? —  What type of code are you testing? ¡  New code? ÷ Start with Unit tests – fastest and cheapest way to get a lot of code coverage ¡  Legacy Code? ÷ Do you have test hooks or command lines? ÷ Very difficult to add unit tests at this point ¡  Web site or program? ÷ Different approaches for each Brenda Kise - Proto Labs Inc. Case 1 —  FDA regulated website —  Written in Java, so used Junit —  Very specific requirements to test against —  Testing the values displayed on the website —  Did not include testing the webpage itself —  Tested against 6 languages —  Used drivers from Chrome, Firefox and IE Brenda Kise - Proto Labs Inc. 7  
  • 10. 9/9/13   Case 1 —  Previously used manual testing for majority of tests —  Took 6 weeks to complete a full regression run with no issues —  Testers were asked to automate tests as they had time —  Using custom written testing software – no off the shelf software Brenda Kise - Proto Labs Inc. Case 1 —  New code base started, done completely in Agile —  No previous code or tests were re-used —  Testers given training to use Junit – most had never used it prior —  Testers were not doing unit tests – considered component or integration tests —  Testers automated the tests as they created them Brenda Kise - Proto Labs Inc. 8  
  • 11. 9/9/13   Case 1 —  All integration, unit and component tests were written in the same sprint as the code —  Scrum teams were comprised of Product Owners, Business Analysts, Developer and Testers —  All tests had to run against all languages on Production Equivalent boxes —  4 levels of testing – subsystem, DAT, system and simulated use testing Brenda Kise - Proto Labs Inc. Case 1 —  Tests were created to specifically test each requirement (FDA requirement) —  Data was created and seeded into the database to test against ¡  Payload Data was actual production data that had been de-identified ¡  Very important to use as close to production data as possible —  Expected outcomes decided prior to tests being written Brenda Kise - Proto Labs Inc. 9  
  • 12. 9/9/13   Case 1 —  Junit tests / setups were created in a test project —  One piece of custom software used for testing was the ability to create patients / clinics / physicians with needed relationships ¡  Test frame work used the actual code to create the same as a human user would —  Setup file was run, then the test. —  Single failure would log, but test run would continue Brenda Kise - Proto Labs Inc. Case 1 —  Code was exercised in test using the interface ¡  Code was a client server base – published interfaces for all components —  XML being delivered to the web browser was intercepted and searched for the values expected —  Logged whether value found or not Brenda Kise - Proto Labs Inc. 10  
  • 13. 9/9/13   Case 1 —  Things to remember 1.  Does not test the page layout, only the content 2.  Works especially well if calculations are performed prior to display 3.  Can be used with graphics, although this is consuming to get done and every time the data changes the test has to be recaptured Brenda Kise - Proto Labs Inc. Case 1 —  Since the test code was created in it’s own project, it was compiled and then run on each localized environment —  All information was logged, including pass, fail, start and end time —  Logs were scraped and funneled to a status website used by management ¡  Thus relieving the testers of the inevitable question – are you done yet? —  Logs were archived as testing evidence for the release Brenda Kise - Proto Labs Inc. 11  
  • 14. 9/9/13   Case 1 —  Questions or comments about the first style of test automation? Brenda Kise - Proto Labs Inc. Case 2 —  Large software package —  Had a web interface, but not tested with automation —  Code was written in C++ / Java / XML —  Vague requirements to test against —  Testing to ensure functionality —  Tested against all supported versions of Linux, Windows, Solaris, HP Brenda Kise - Proto Labs Inc. 12  
  • 15. 9/9/13   Case 2 —  Still using manual testing for majority of tests —  Testers were asked to automate tests as they had time —  Manual tests were run through the different GUIs available to the customer (Java, MVC and Web) —  Test steps left to the individual tester, rarely written in detail Brenda Kise - Proto Labs Inc. Case 2 —  Single legacy code base, with modules added —  Very little component style coding —  Code had an extensive command line availability —  Very little code coverage from unit tests ¡  Since it was mostly one big pile of code, hard to test small areas —  Testers and automaters were separate skills Brenda Kise - Proto Labs Inc. 13  
  • 16. 9/9/13   Case 2 —  Test automation happened on it’s own schedule, without regard to releases —  Automation team was divided into three areas ¡  Test engine and tools ¡  Automation Leads ¡  Automaters —  All tests had to run against all OS flavors supported —  3 levels of testing – manual testing, automated regression and system level testing Brenda Kise - Proto Labs Inc. Case 2 —  Large test harness created to run the tests ¡  3 full time people working on it ¡  1 million + lines of Perl code ¡  Included log scrapers, setup code and failure processing —  Automation script would call setup routines from the test harness to do standard things ¡  Create storage unit, create back files etc —  Expected outcomes decided prior to automation scripts being written Brenda Kise - Proto Labs Inc. 14  
  • 17. 9/9/13   Case 2 —  Automaters are not testers – they are developers —  Automaters do not create the tests, test steps or decide what to test —  Are not experts on the code or the functionality since they works across a large area of code —  Allowed automation to continue no matter what the shape of the release —  Need a stable code base – attempt this type of automation on a moving target is immensely painful. Brenda Kise - Proto Labs Inc. Case 2 —  Code was exercised using the published command lines —  Looking for code coverage, not edge cases —  Covered the “vanilla” cases – allowed manual testers to concentrate on more complex cases not able to be tested in automation Brenda Kise - Proto Labs Inc. 15  
  • 18. 9/9/13   Case 2 —  Things to remember 1.  Need a stable code base 2.  Needs either command line or interface to call 3.  Often will need help from both development and test 1.  If the test case says something like “test X” with no steps, only tester knows what they mean. 2.  Amazing how often the command line information and error code details were incorrect Brenda Kise - Proto Labs Inc. Case 2 —  All automation was written in Perl —  All information was logged, including pass, fail, start and end time ¡  Timing was important in this case – when tests started running long it was usually an early issue indicator —  Logs were scraped and funneled to a storage area to be used if needed for debugging of issues found —  Pass / Failures were automatically uploaded to a webpage Brenda Kise - Proto Labs Inc. 16  
  • 19. 9/9/13   Case 2 —  Developers were expected to run specific tests against their code prior to check in. —  Unfortunately, any break in a regression test was automatically assumed to be a bad test ¡  Had an entire team in China that would analyze test results and create issue reports —  Regression score was used by management to show health of the release ¡  If < 80% of test passed, score was 0 ¡  Only a singe instance of a perfect regression score in the 6 years it has been running Brenda Kise - Proto Labs Inc. Case 2 —  While you can get massive amounts of regression testing, very expensive approach —  Team of 10 full time employees and 15 part time contractors, plus a team of 8 doing analysis —  Massive hardware usage – several hundred VMs on a nightly basis —  Had so many tests running took a full 24 hours to complete a full run against a single build Brenda Kise - Proto Labs Inc. 17  
  • 20. 9/9/13   Case 2 —  Advantages of using Perl to test C code: ¡  You can start small – you don’t need a full test harness, although you will like find yourself creating one to do basic setup tasks ÷ Watch out, this takes on a life of it’s own ¡  Can use a Chron job or Hudson to kick off the test scripts ¡  Logging is easy and straightforward in Perl Brenda Kise - Proto Labs Inc. Case 2 —  Questions or comments about the second style of test automation? Brenda Kise - Proto Labs Inc. 18  
  • 21. 9/9/13   Take Aways —  Have a plan! —  Know what type of testing you are trying to accomplish —  Know what type of results you are looking for ¡  Code coverage? ¡  Regression for found bugs? —  Know what type of team you are working with ¡  Automaters or testers? Brenda Kise - Proto Labs Inc. 19