Managing the Test Data Nightmare

Managing the Test Data Nightmare | QA Summit by Xpanxion | Andrew Knight | @AutomationPanda | AutomationPanda.com
Managing the
Test Data Nightmare
Andrew Knight
Lead Software Engineer in Test
PrecisionLender, a Q2 Company
AutomationPanda.com
@AutomationPanda

Scenario: Create a new loan application

Given the Chrome browser is open

And the page “https://www.myloanapp.com” is loaded

When the user creates a new loan application for “home mortgage”

test data

test data
Chrome browser
https://www.myloanapp.com
home mortgage
| amount | $200,000 |
reference number
loan application sent to the bank

How should we manage
test data?

Form a good strategy
to avoid a nightmare.

Product Data

Product Data
Everything in the database:

Product Data
● User accounts
● Administration settings
● Product customizations
● Records
● Files

Product Data
● User accounts
● Administration settings
● Product customizations
● Records
● Files
Our example loan web app:
user accounts, loan products, completed applications, bank data

How to set up product data?

Static Data Prep

Static Data Prep
● Created before tests run

Static Data Prep
● Good for slow or complicated data

Static Data Prep
● May make tests run faster

Static Data Prep
● May make tests brittle as data changes

Static Data Prep
● May turn stale over time

Static Data Prep
Dynamic Data Prep

Static Data Prep
Dynamic Data Prep
● Created when a test runs

Static Data Prep
Dynamic Data Prep
● Avoids brittle references

Static Data Prep
Dynamic Data Prep
● Exclusive use

Static Data Prep
Dynamic Data Prep
● Exclusive use
● Slows down individual tests

Static Data Prep
Dynamic Data Prep
● Exclusive use
● Slows down individual tests
● Requires cleanup

Static Data Prep Strategy Advantages Disadvantages

Manual configuration

Manual configuration Anyone can do it,
Extra tools are not needed

Slow and not scalable,
May fall into disrepair

Automated configuration

Automated configuration Create fresh data anytime,
Can also scrub data

Can also scrub data
Must maintain tools and scripts

Can also scrub data
Cloned databases

Can also scrub data
Cloned databases Easy to copy all data at once

Can also scrub data
Cloned databases Easy to copy all data at once Might be too much data,
Might need extra refinement

Can also scrub data
Mocked endpoints

Can also scrub data
Mocked endpoints Avoid dependencies,
Controls all data values

Can also scrub data
Mocked endpoints Avoid dependencies,
Controls all data values
Difficult to set up and maintain,
May overlook some variations

Factors for Static Data Prep

1. Size?

1. Size?
2. Freshness?

1. Size?
2. Freshness?
3. Update frequency?

1. Size?
2. Freshness?
4. Diﬃculty (especially for legacy systems)?

1. Size?
2. Freshness?
5. Bureaucracy?

1. Size?
2. Freshness?
5. Bureaucracy?
6. Skill level?

1. Size?
2. Freshness?
5. Bureaucracy?
6. Skill level?
7. Cost?

That’s product data.
What about test case data?

// inputs.json
{
“browser”: “Chrome”
}

// inputs.json
{
}
# bash variables
$ export TEST_BROWSER=Chrome

// inputs.json
{
“URL”: “https://www.myloanapp.com”
}

// inputs.json
{
“URL”: “https://www.myloanapp.com”
}
# bash variables
$ export TEST_BROWSER=Chrome
$ export TEST_URL=https://www.myloanapp.com

// inputs.json
{
“config”: “Staging”
}
// config.json
{
“Staging”: {
“URL”: “https://staging.myloanapp.com”
},
“Dev”: {
“URL”: “https://dev.myloanapp.com”
}
}

// inputs.json
{
}
// /some/api/endpoint/for/test/config
{
“Staging”: {
},
“Dev”: {
}
}

// inputs.json
{
}
// config.json
{
“Staging”: {
},
“Dev”: {
}
}

Given the MyLoanApp web page is loaded
// inputs.json
{
}
// config.json
{
“Staging”: {
},
“Dev”: {
}
}

When the user creates a new loan application for a home mortgage

When the user creates a new loan application for a home mortgage
// Automation pseudocode
request = MakeRestRequestForAvailableLoanTypes();
response = restClient.Get(request);
foreach loan in response.Data:
if loan.Type is home mortgage
return loan.SpecificName

Test Case Data

Test Case Data
Type Description
Test Control Inputs Direct how tests will be run, not what behavior is covered.
Should be supplied via ﬂat ﬁles or env vars.

Test Case Data
Type Description
Configuration Metadata Describe product configuration for the target environment.
Should be supplied via config files or service API calls.

Test Case Data
Type Description
Configuration Metadata Describe product configuration for the target environment.
Should be supplied via config files or service API calls.
Test Case Values Direct the behavior covered by the test.
May be literals, output references, or input references.
Input references may be hard-coded or discovered.

The nightmare isn’t over.

The nightmare isn’t over.
Collisions can happen!

Avoiding Collisions

Avoiding Collisions
1. Isolate the test environment(s)
a. Restrict access while testing?
b. Deploy new containers?
c. Make database clones?

Avoiding Collisions
2. Treat any shared data as immutable
a. Parallel tests should not alter shared data
b. If any tests must temporarily alter shared data, run them serially separately

Avoiding Collisions
2. Treat any shared data as immutable
a. Parallel tests should not alter shared data
b. If any tests must temporarily alter shared data, run them serially separately
3. Use dynamic data prep as much as possible
a. Tests cannot collide on data they don’t share

Major Takeaways

Major Takeaways
1. 2 types of test data: product data and test case data

Major Takeaways
2. Product data can be prepared statically or dynamically

Major Takeaways
3. Test case data either control how tests run or reﬂect product data

Major Takeaways
3. Test case data either control how tests run or reﬂect product data
4. Handle references and shared data carefully

Choose the best strategies
to defeat your nightmare!

Andrew “Pandy” Knight
Lead Software Engineer in Test
PrecisionLender, a Q2 Company
AutomationPanda.com
@AutomationPanda
Thanks!

Managing the Test Data Nightmare

Recommended

Recommended

More Related Content

Recently uploaded

Recently uploaded (20)

Featured

Featured (20)

Managing the Test Data Nightmare