Continuous Automated Regression Testing to the Rescue

BT4
Concurrent Session
11/14/2013 10:15 AM

"Continuous Automated
Regression Testing to the
Rescue"

Presented by:
Brenda Kise
ProtoLabs, Inc.

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073
888‐268‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Brenda Kise
Proto Labs, Inc.

Brenda Kise has fifteen years of experience in software, including
development, test, test automation, management, and operations. Brenda
has led teams that have automated testing from the ground up with a new
code base and for Symantec’s NetBackup division, leading a team
automating a legacy code base. In the first team, close to 100 percent
automation was achieved using the JUnit framework in a tightly integrated
Scrum/agile environment and went from a formal test cycle of six weeks to
just eleven hours! At Symantec, with an 8 million line code base, the
automation team focused on automating regression tests against the
completed code base - going from 0% coverage to 14% coverage in one
year. Brenda is currently managing a team of .NET developers working on
internal applications and the Enterprise Data team doing business
intelligence work for Proto Labs Inc.

9/9/13

Continuous Automated Regression
Testing to the Rescue!
BRENDA KISE
DEVELOPMENT MANAGER, PROTOLABS
BRENDA.KISE@PROTOLABS.COM

Brenda Kise - Proto Labs Inc.

Who am I?
—  Currently a development manager for Proto Labs

Inc in Maple Plain MN
—  Manage a team of C# developers
—  Manage a team of Business Intelligence / Data
Warehouse developers
—  Proto Labs is a software company that does a little
manufacturing on the side
—  Hiring – growing by leaps and bounds


1

9/9/13

Background
—  16 years software experience
—  Java / Database developer
—  Managed operations, development, hardware

buildouts, quality, test and automation teams
—  Test team used Junit to automate 99% of tests
¡  Went

from a 3-6 week test cycle to 8 hours

—  Automation team gained code coverage ground

against 200 developers


Earlier is better
—  The earlier after code check-in a defect is found,

the easier it is to fix it
—  Few people affected the earlier the defect is found
—  Less paperwork involved when found earlier
—  Often several layers of testing – if found early, less
re-testing is needed
—  If a defect makes it to customers, the repercussions
can affect more than just the software


2

9/9/13

Proof Earlier is Better


Automated Testing
—  Different levels for automation
¡  Component / Unit testing
÷ Generally done by developers
¡  Integration
¡  System

level

¡  GUI


3

9/9/13

How is the test automated?
—  Run with each build
¡  Applications such as Hudson will run the unit tests and
fail the build if the tests do not pass
¡  Immediate feedback, helps keep the build clean
¡  Can tell if someone is following process or not
÷ Broken build indicates coder did not run the build prior
to checking into the main branch


—  Run on a schedule
¡  Generally need a test frame work to run the tests
¡  Developers can run the tests prior to check-in
¡  Process is dis-associated with the build
÷ Build is completed
÷ Regression tests kicked off through a chron job
÷ Need a check to ensure build does not fail prior to test
kick-off
÷ Generally uses a separate system


4

9/9/13

—  Tester kicks off the test
¡  While

this type of test is automated, it is not considered
automated regression
¡  No schedule or automated kick-off, requires human
intervention
¡  Still valuable as a time saving tool – allows the tester to
work on something else while test is running
¡  Especially valuable for long scenarios


How?
—  Have a plan
¡  Are you working with a new code base?
¡  Are you focusing on unit / component testing?
¡  Is the code base stable?
—  What level of testing?
¡  System
¡  Unit
¡  GUI


5

9/9/13

How?
—  Choose your tools
¡  Custom framework?
¡  Junit or Cunit?
¡  OTS software?
÷ Watir
÷ Selenium
÷ QuickTest Pro
÷ WinRunner


How?
—  Choose your approach
¡  Testers automate tests as they have time
÷ Like that ever happens
÷ Always seen as a secondary task
¡  Separate team of automaters
÷ Take tests that the testers use and automate
÷ Do not need to be experts in the areas of code
÷ Relies on well documented test scripts / scenarios
÷ Main focus of job


6

9/9/13

How?
—  What type of code are you testing?
¡  New code?
÷ Start with Unit tests – fastest and cheapest way to get a
lot of code coverage
¡  Legacy Code?
÷ Do you have test hooks or command lines?
÷ Very difficult to add unit tests at this point
¡  Web site or program?
÷ Different approaches for each


Case 1
—  FDA regulated website
—  Written in Java, so used Junit
—  Very specific requirements to test against
—  Testing the values displayed on the website
—  Did not include testing the webpage itself
—  Tested against 6 languages
—  Used drivers from Chrome, Firefox and IE


7

9/9/13

Case 1
—  Previously used manual testing for majority of tests
—  Took 6 weeks to complete a full regression run

with no issues
—  Testers were asked to automate tests as they had
time
—  Using custom written testing software – no off the
shelf software


Case 1
—  New code base started, done completely in Agile
—  No previous code or tests were re-used
—  Testers given training to use Junit – most had

never used it prior
—  Testers were not doing unit tests – considered
component or integration tests
—  Testers automated the tests as they created them


8

9/9/13

Case 1
—  All integration, unit and component tests were

written in the same sprint as the code
—  Scrum teams were comprised of Product Owners,
Business Analysts, Developer and Testers
—  All tests had to run against all languages on
Production Equivalent boxes
—  4 levels of testing – subsystem, DAT, system and
simulated use testing


Case 1
—  Tests were created to specifically test each

requirement (FDA requirement)
—  Data was created and seeded into the database to
test against
¡  Payload

Data was actual production data that had been
de-identified
¡  Very important to use as close to production data as
possible
—  Expected outcomes decided prior to tests being

written

9

9/9/13

Case 1
—  Junit tests / setups were created in a test project
—  One piece of custom software used for testing was

the ability to create patients / clinics / physicians
with needed relationships
¡  Test

frame work used the actual code to create the same
as a human user would

—  Setup file was run, then the test.
—  Single failure would log, but test run would

continue

Case 1
—  Code was exercised in test using the interface
¡  Code was a client server base – published interfaces for
all components
—  XML being delivered to the web browser was

intercepted and searched for the values expected
—  Logged whether value found or not


10

9/9/13

Case 1
—  Things to remember
1. 
Does not test the page layout, only the content
2.  Works especially well if calculations are performed
prior to display
3.  Can be used with graphics, although this is consuming
to get done and every time the data changes the test
has to be recaptured


Case 1
—  Since the test code was created in it’s own project,

it was compiled and then run on each localized
environment
—  All information was logged, including pass, fail,
start and end time
—  Logs were scraped and funneled to a status website
used by management
¡  Thus

relieving the testers of the inevitable question – are
you done yet?

—  Logs were archived as testing evidence for the

release


11

9/9/13

Case 1
—  Questions or comments about the first style of test

automation?


Case 2
—  Large software package
—  Had a web interface, but not tested with

automation
—  Code was written in C++ / Java / XML
—  Vague requirements to test against
—  Testing to ensure functionality
—  Tested against all supported versions of Linux,
Windows, Solaris, HP


12

9/9/13

Case 2
—  Still using manual testing for majority of tests
—  Testers were asked to automate tests as they had

time
—  Manual tests were run through the different GUIs
available to the customer (Java, MVC and Web)
—  Test steps left to the individual tester, rarely
written in detail


Case 2
—  Single legacy code base, with modules added
—  Very little component style coding
—  Code had an extensive command line availability
—  Very little code coverage from unit tests
¡ 

Since it was mostly one big pile of code, hard to test small
areas

—  Testers and automaters were separate skills


13

9/9/13

Case 2
—  Test automation happened on it’s own schedule,

without regard to releases
—  Automation team was divided into three areas
¡  Test

engine and tools
¡  Automation Leads
¡  Automaters
—  All tests had to run against all OS flavors supported
—  3 levels of testing – manual testing, automated

regression and system level testing


Case 2
—  Large test harness created to run the tests
¡  3 full time people working on it
¡  1 million + lines of Perl code
¡  Included log scrapers, setup code and failure processing
—  Automation script would call setup routines from

the test harness to do standard things
¡  Create

storage unit, create back files etc

—  Expected outcomes decided prior to automation

scripts being written

14

9/9/13

Case 2
—  Automaters are not testers – they are developers
—  Automaters do not create the tests, test steps or

decide what to test
—  Are not experts on the code or the functionality
since they works across a large area of code
—  Allowed automation to continue no matter what
the shape of the release
—  Need a stable code base – attempt this type of
automation on a moving target is immensely
painful.

Case 2
—  Code was exercised using the published command

lines
—  Looking for code coverage, not edge cases
—  Covered the “vanilla” cases – allowed manual
testers to concentrate on more complex cases not
able to be tested in automation


15

9/9/13

Case 2
—  Things to remember
1. 
Need a stable code base
2.  Needs either command line or interface to call
3.  Often will need help from both development and test
1.  If the test case says something like “test X” with no
steps, only tester knows what they mean.
2.  Amazing how often the command line information
and error code details were incorrect


Case 2
—  All automation was written in Perl
—  All information was logged, including pass, fail,

start and end time
¡  Timing

was important in this case – when tests started
running long it was usually an early issue indicator

—  Logs were scraped and funneled to a storage area

to be used if needed for debugging of issues found
—  Pass / Failures were automatically uploaded to a
webpage

16

9/9/13

Case 2
—  Developers were expected to run specific tests

against their code prior to check in.
—  Unfortunately, any break in a regression test was
automatically assumed to be a bad test
¡  Had

an entire team in China that would analyze test
results and create issue reports

—  Regression score was used by management to show

health of the release
¡  If

< 80% of test passed, score was 0
¡  Only a singe instance of a perfect regression score in the 6
years it has been running

Case 2
—  While you can get massive amounts of regression

testing, very expensive approach
—  Team of 10 full time employees and 15 part time
contractors, plus a team of 8 doing analysis
—  Massive hardware usage – several hundred VMs
on a nightly basis
—  Had so many tests running took a full 24 hours to
complete a full run against a single build


17

9/9/13

Case 2
—  Advantages of using Perl to test C code:
¡  You can start small – you don’t need a full test harness,
although you will like find yourself creating one to do
basic setup tasks
÷ Watch out, this takes on a life of it’s own
¡  Can use a Chron job or Hudson to kick off the test scripts
¡  Logging is easy and straightforward in Perl


Case 2
—  Questions or comments about the second style of

test automation?


18

9/9/13

Take Aways
—  Have a plan!
—  Know what type of testing you are trying to

accomplish
—  Know what type of results you are looking for
¡  Code

coverage?
¡  Regression for found bugs?
—  Know what type of team you are working with
¡  Automaters

or testers?


19

Continuous Automated Regression Testing to the Rescue

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (17)

Similar to Continuous Automated Regression Testing to the Rescue

Similar to Continuous Automated Regression Testing to the Rescue (20)

More from TechWell

More from TechWell (20)

Recently uploaded

Recently uploaded (20)

Continuous Automated Regression Testing to the Rescue