Open Source Summit NA 2024: Open Source Cloud Costs - OpenCost's Impact on En...
Andrew Knight - Managing the Test Data Nightmare.pptx
1. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Pandy Knight
Automation Panda
Applitools Pr Developer Advocate
Test Automation University Director
Managing the Test Data Nightmare
2. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
suki_thefrenchie_
3. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
4. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
5. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
6. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
7. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
8. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
9. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Test data can be a nightmare.
10. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
11. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
12. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
13. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
14. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
15. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
16. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
17. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
18. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Scenario: Create a new loan application
19. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Scenario: Create a new loan application
Given the Chrome browser is open
20. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Scenario: Create a new loan application
Given the Chrome browser is open
And the page “https://www.myloanapp.com” is loaded
21. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Scenario: Create a new loan application
Given the Chrome browser is open
And the page “https://www.myloanapp.com” is loaded
When the user creates a new loan application for “home mortgage”
22. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Scenario: Create a new loan application
Given the Chrome browser is open
And the page “https://www.myloanapp.com” is loaded
When the user creates a new loan application for “home mortgage”
And the user enters the following information:
| field | value |
| name | Freddy Krueger |
| address | 1428 Elm Street |
| amount | $200,000 |
| maturity | 30 years |
| down payment | 20% |
23. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Scenario: Create a new loan application
Given the Chrome browser is open
And the page “https://www.myloanapp.com” is loaded
When the user creates a new loan application for “home mortgage”
And the user enters the following information:
| field | value |
| name | Freddy Krueger |
| address | 1428 Elm Street |
| amount | $200,000 |
| maturity | 30 years |
| down payment | 20% |
And the user submits the application
24. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Scenario: Create a new loan application
Given the Chrome browser is open
And the page “https://www.myloanapp.com” is loaded
When the user creates a new loan application for “home mortgage”
And the user enters the following information:
| field | value |
| name | Freddy Krueger |
| address | 1428 Elm Street |
| amount | $200,000 |
| maturity | 30 years |
| down payment | 20% |
And the user submits the application
Then the page displays a success message with a reference number
25. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Scenario: Create a new loan application
Given the Chrome browser is open
And the page “https://www.myloanapp.com” is loaded
When the user creates a new loan application for “home mortgage”
And the user enters the following information:
| field | value |
| name | Freddy Krueger |
| address | 1428 Elm Street |
| amount | $200,000 |
| maturity | 30 years |
| down payment | 20% |
And the user submits the application
Then the page displays a success message with a reference number
And the loan application is sent to the bank
26. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Scenario: Create a new loan application
Given the Chrome browser is open
And the page “https://www.myloanapp.com” is loaded
When the user creates a new loan application for “home mortgage”
And the user enters the following information:
| field | value |
| name | Freddy Krueger |
| address | 1428 Elm Street |
| amount | $200,000 |
| maturity | 30 years |
| down payment | 20% |
And the user submits the application
Then the page displays a success message with a reference number
And the loan application is sent to the bank
Personal Info
27. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Scenario: Create a new loan application
Given the Chrome browser is open
And the page “https://www.myloanapp.com” is loaded
When the user creates a new loan application for “home mortgage”
And the user enters the following information:
| field | value |
| name | Freddy Krueger |
| address | 1428 Elm Street |
| amount | $200,000 |
| maturity | 30 years |
| down payment | 20% |
And the user submits the application
Then the page displays a success message with a reference number
And the loan application is sent to the bank
Loan Type
28. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Scenario: Create a new loan application
Given the Chrome browser is open
And the page “https://www.myloanapp.com” is loaded
When the user creates a new loan application for “home mortgage”
And the user enters the following information:
| field | value |
| name | Freddy Krueger |
| address | 1428 Elm Street |
| amount | $200,000 |
| maturity | 30 years |
| down payment | 20% |
And the user submits the application
Then the page displays a success message with a reference number
And the loan application is sent to the bank
Loan App
Record
29. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Scenario: Create a new loan application
Given the Chrome browser is open
And the page “https://www.myloanapp.com” is loaded
When the user creates a new loan application for “home mortgage”
And the user enters the following information:
| field | value |
| name | Freddy Krueger |
| address | 1428 Elm Street |
| amount | $200,000 |
| maturity | 30 years |
| down payment | 20% |
And the user submits the application
Then the page displays a success message with a reference number
And the loan application is sent to the bank
Reference Number
30. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Scenario: Create a new loan application
Given the Chrome browser is open
And the page “https://www.myloanapp.com” is loaded
When the user creates a new loan application for “home mortgage”
And the user enters the following information:
| field | value |
| name | Freddy Krueger |
| address | 1428 Elm Street |
| amount | $200,000 |
| maturity | 30 years |
| down payment | 20% |
And the user submits the application
Then the page displays a success message with a reference number
And the loan application is sent to the bank
Website URL
31. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Scenario: Create a new loan application
Given the Chrome browser is open
And the page “https://www.myloanapp.com” is loaded
When the user creates a new loan application for “home mortgage”
And the user enters the following information:
| field | value |
| name | Freddy Krueger |
| address | 1428 Elm Street |
| amount | $200,000 |
| maturity | 30 years |
| down payment | 20% |
And the user submits the application
Then the page displays a success message with a reference number
And the loan application is sent to the bank
Browser Type
32. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Scenario: Create a new loan application
Given the Chrome browser is open
And the page “https://www.myloanapp.com” is loaded
When the user creates a new loan application for “home mortgage”
And the user enters the following information:
| field | value |
| name | Freddy Krueger |
| address | 1428 Elm Street |
| amount | $200,000 |
| maturity | 30 years |
| down payment | 20% |
And the user submits the application
Then the page displays a success message with a reference number
And the loan application is sent to the bank
33. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
34. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
test data
35. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
test data
36. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
test data
Chrome browser
https://www.myloanapp.com
home mortgage
| name | Freddy Krueger |
| address | 1428 Elm Street |
| amount | $200,000 |
| maturity | 30 years |
| down payment | 20% |
reference number
loan application sent to the bank
37. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
test data
Chrome browser
https://www.myloanapp.com
home mortgage
| name | Freddy Krueger |
| address | 1428 Elm Street |
| amount | $200,000 |
| maturity | 30 years |
| down payment | 20% |
reference number
loan application sent to the bank
38. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
39. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
When the user creates a new loan application for “home mortgage”
40. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
When the user creates a new loan application for “home mortgage”
41. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
When the user creates a new loan application for “home mortgage”
42. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
How should we manage
test data?
43. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Form a good strategy
to avoid a nightmare.
44. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Product Data
45. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Product Data
Everything in the database:
46. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Product Data
Everything in the database:
● User accounts
● Administration settings
● Product customizations
● Records
● Files
47. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Product Data
Everything in the database:
● User accounts
● Administration settings
● Product customizations
● Records
● Files
Our example loan web app:
user accounts, loan products, completed applications, bank data
48. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
How to set up product data?
49. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Static Data Prep
50. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Static Data Prep
● Created before tests run
51. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Static Data Prep
● Created before tests run
● Good for slow or complicated data
52. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Static Data Prep
● Created before tests run
● Good for slow or complicated data
● May make tests run faster
53. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Static Data Prep
● Created before tests run
● Good for slow or complicated data
● May make tests run faster
● May make tests brittle as data changes
54. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Static Data Prep
● Created before tests run
● Good for slow or complicated data
● May make tests run faster
● May make tests brittle as data changes
● May turn stale over time
55. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Static Data Prep
● Created before tests run
● Good for slow or complicated data
● May make tests run faster
● May make tests brittle as data changes
● May turn stale over time
Dynamic Data Prep
56. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Static Data Prep
● Created before tests run
● Good for slow or complicated data
● May make tests run faster
● May make tests brittle as data changes
● May turn stale over time
Dynamic Data Prep
● Created when a test runs
57. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Static Data Prep
● Created before tests run
● Good for slow or complicated data
● May make tests run faster
● May make tests brittle as data changes
● May turn stale over time
Dynamic Data Prep
● Created when a test runs
● Avoids brittle references
58. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Static Data Prep
● Created before tests run
● Good for slow or complicated data
● May make tests run faster
● May make tests brittle as data changes
● May turn stale over time
Dynamic Data Prep
● Created when a test runs
● Avoids brittle references
● Exclusive use
59. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Static Data Prep
● Created before tests run
● Good for slow or complicated data
● May make tests run faster
● May make tests brittle as data changes
● May turn stale over time
Dynamic Data Prep
● Created when a test runs
● Avoids brittle references
● Exclusive use
● Slows down individual tests
60. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Static Data Prep
● Created before tests run
● Good for slow or complicated data
● May make tests run faster
● May make tests brittle as data changes
● May turn stale over time
Dynamic Data Prep
● Created when a test runs
● Avoids brittle references
● Exclusive use
● Slows down individual tests
● Requires cleanup
61. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Static Data Prep
● Created before tests run
● Good for slow or complicated data
● May make tests run faster
● May make tests brittle as data changes
● May turn stale over time
Dynamic Data Prep
● Created when a test runs
● Avoids brittle references
● Exclusive use
● Slows down individual tests
● Requires cleanup
62. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Static Data Prep Strategy Advantages Disadvantages
63. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Static Data Prep Strategy Advantages Disadvantages
Manual configuration
64. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Static Data Prep Strategy Advantages Disadvantages
Manual configuration Anyone can do it,
Extra tools are not needed
65. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Static Data Prep Strategy Advantages Disadvantages
Manual configuration Anyone can do it,
Extra tools are not needed
Slow and not scalable,
May fall into disrepair
66. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Static Data Prep Strategy Advantages Disadvantages
Manual configuration Anyone can do it,
Extra tools are not needed
Slow and not scalable,
May fall into disrepair
Automated configuration
67. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Static Data Prep Strategy Advantages Disadvantages
Manual configuration Anyone can do it,
Extra tools are not needed
Slow and not scalable,
May fall into disrepair
Automated configuration Create fresh data anytime,
Can also scrub data
68. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Static Data Prep Strategy Advantages Disadvantages
Manual configuration Anyone can do it,
Extra tools are not needed
Slow and not scalable,
May fall into disrepair
Automated configuration Create fresh data anytime,
Can also scrub data
Must maintain tools and scripts
69. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Static Data Prep Strategy Advantages Disadvantages
Manual configuration Anyone can do it,
Extra tools are not needed
Slow and not scalable,
May fall into disrepair
Automated configuration Create fresh data anytime,
Can also scrub data
Must maintain tools and scripts
Cloned databases
70. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Static Data Prep Strategy Advantages Disadvantages
Manual configuration Anyone can do it,
Extra tools are not needed
Slow and not scalable,
May fall into disrepair
Automated configuration Create fresh data anytime,
Can also scrub data
Must maintain tools and scripts
Cloned databases Easy to copy all data at once
71. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Static Data Prep Strategy Advantages Disadvantages
Manual configuration Anyone can do it,
Extra tools are not needed
Slow and not scalable,
May fall into disrepair
Automated configuration Create fresh data anytime,
Can also scrub data
Must maintain tools and scripts
Cloned databases Easy to copy all data at once Might be too much data,
Might need extra refinement
72. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Static Data Prep Strategy Advantages Disadvantages
Manual configuration Anyone can do it,
Extra tools are not needed
Slow and not scalable,
May fall into disrepair
Automated configuration Create fresh data anytime,
Can also scrub data
Must maintain tools and scripts
Cloned databases Easy to copy all data at once Might be too much data,
Might need extra refinement
Mocked endpoints
73. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Static Data Prep Strategy Advantages Disadvantages
Manual configuration Anyone can do it,
Extra tools are not needed
Slow and not scalable,
May fall into disrepair
Automated configuration Create fresh data anytime,
Can also scrub data
Must maintain tools and scripts
Cloned databases Easy to copy all data at once Might be too much data,
Might need extra refinement
Mocked endpoints Avoid dependencies,
Controls all data values
74. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Static Data Prep Strategy Advantages Disadvantages
Manual configuration Anyone can do it,
Extra tools are not needed
Slow and not scalable,
May fall into disrepair
Automated configuration Create fresh data anytime,
Can also scrub data
Must maintain tools and scripts
Cloned databases Easy to copy all data at once Might be too much data,
Might need extra refinement
Mocked endpoints Avoid dependencies,
Controls all data values
Difficult to set up and maintain,
May overlook some variations
75. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Static Data Prep Strategy Advantages Disadvantages
Manual configuration Anyone can do it,
Extra tools are not needed
Slow and not scalable,
May fall into disrepair
Automated configuration Create fresh data anytime,
Can also scrub data
Must maintain tools and scripts
Cloned databases Easy to copy all data at once Might be too much data,
Might need extra refinement
Mocked endpoints Avoid dependencies,
Controls all data values
Difficult to set up and maintain,
May overlook some variations
76. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Static Data Prep Strategy Advantages Disadvantages
Manual configuration Anyone can do it,
Extra tools are not needed
Slow and not scalable,
May fall into disrepair
Automated configuration Create fresh data anytime,
Can also scrub data
Must maintain tools and scripts
Cloned databases Easy to copy all data at once Might be too much data,
Might need extra refinement
Mocked endpoints Avoid dependencies,
Controls all data values
Difficult to set up and maintain,
May overlook some variations
Production Data Synthetic Data
77. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Factors for Static Data Prep
78. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Factors for Static Data Prep
1. Size?
79. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Factors for Static Data Prep
1. Size?
2. Freshness?
80. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Factors for Static Data Prep
1. Size?
2. Freshness?
3. Update frequency?
81. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Factors for Static Data Prep
1. Size?
2. Freshness?
3. Update frequency?
4. Difficulty (especially for legacy systems)?
82. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Factors for Static Data Prep
1. Size?
2. Freshness?
3. Update frequency?
4. Difficulty (especially for legacy systems)?
5. Bureaucracy?
83. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Factors for Static Data Prep
1. Size?
2. Freshness?
3. Update frequency?
4. Difficulty (especially for legacy systems)?
5. Bureaucracy?
6. Skill level?
84. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Factors for Static Data Prep
1. Size?
2. Freshness?
3. Update frequency?
4. Difficulty (especially for legacy systems)?
5. Bureaucracy?
6. Skill level?
7. Cost?
85. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
That’s product data.
What about test case data?
86. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
87. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Scenario: Create a new loan application
Given the Chrome browser is open
And the page “https://www.myloanapp.com” is loaded
When the user creates a new loan application for “home mortgage”
And the user enters the following information:
| field | value |
| name | Freddy Krueger |
| address | 1428 Elm Street |
| amount | $200,000 |
| maturity | 30 years |
| down payment | 20% |
And the user submits the application
Then the page displays a success message with a reference number
And the loan application is sent to the bank
88. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Scenario: Create a new loan application
Given the Chrome browser is open
And the page “https://www.myloanapp.com” is loaded
When the user creates a new loan application for “home mortgage”
And the user enters the following information:
| field | value |
| name | Freddy Krueger |
| address | 1428 Elm Street |
| amount | $200,000 |
| maturity | 30 years |
| down payment | 20% |
And the user submits the application
Then the page displays a success message with a reference number
And the loan application is sent to the bank
89. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Scenario: Create a new loan application
Given the Chrome browser is open
And the page “https://www.myloanapp.com” is loaded
When the user creates a new loan application for “home mortgage”
And the user enters the following information:
| field | value |
| name | Freddy Krueger |
| address | 1428 Elm Street |
| amount | $200,000 |
| maturity | 30 years |
| down payment | 20% |
And the user submits the application
Then the page displays a success message with a reference number
And the loan application is sent to the bank
Test Control Input
90. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Given the Chrome browser is open
91. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Given the Chrome browser is open
// inputs.json
{
“browser”: “Chrome”
}
92. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Given the Chrome browser is open
// inputs.json
{
“browser”: “Chrome”
}
# bash variables
$ export TEST_BROWSER=Chrome
93. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Scenario: Create a new loan application
Given the page “https://www.myloanapp.com” is loaded
When the user creates a new loan application for “home mortgage”
And the user enters the following information:
| field | value |
| name | Freddy Krueger |
| address | 1428 Elm Street |
| amount | $200,000 |
| maturity | 30 years |
| down payment | 20% |
And the user submits the application
Then the page displays a success message with a reference number
And the loan application is sent to the bank
94. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Scenario: Create a new loan application
Given the page “https://www.myloanapp.com” is loaded
When the user creates a new loan application for “home mortgage”
And the user enters the following information:
| field | value |
| name | Freddy Krueger |
| address | 1428 Elm Street |
| amount | $200,000 |
| maturity | 30 years |
| down payment | 20% |
And the user submits the application
Then the page displays a success message with a reference number
And the loan application is sent to the bank
95. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Scenario: Create a new loan application
Given the page “https://www.myloanapp.com” is loaded
When the user creates a new loan application for “home mortgage”
And the user enters the following information:
| field | value |
| name | Freddy Krueger |
| address | 1428 Elm Street |
| amount | $200,000 |
| maturity | 30 years |
| down payment | 20% |
And the user submits the application
Then the page displays a success message with a reference number
And the loan application is sent to the bank
Config Metadata
96. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Given the page “https://www.myloanapp.com” is loaded
97. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Given the page “https://www.myloanapp.com” is loaded
// inputs.json
{
“browser”: “Chrome”
“URL”: “https://www.myloanapp.com”
}
98. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Given the page “https://www.myloanapp.com” is loaded
// inputs.json
{
“browser”: “Chrome”
“URL”: “https://www.myloanapp.com”
}
# bash variables
$ export TEST_BROWSER=Chrome
$ export TEST_URL=https://www.myloanapp.com
99. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Given the page “https://www.myloanapp.com” is loaded
// inputs.json
{
“browser”: “Chrome”
“config”: “Staging”
}
// config.json
{
“Staging”: {
“URL”: “https://staging.myloanapp.com”
},
“Dev”: {
“URL”: “https://dev.myloanapp.com”
}
}
100. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Given the page “https://www.myloanapp.com” is loaded
// inputs.json
{
“browser”: “Chrome”
“config”: “Staging”
}
// config.json
{
“Staging”: {
“URL”: “https://staging.myloanapp.com”
},
“Dev”: {
“URL”: “https://dev.myloanapp.com”
}
}
101. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Given the page “https://www.myloanapp.com” is loaded
// inputs.json
{
“browser”: “Chrome”
“config”: “Staging”
}
// /some/api/endpoint/for/test/config
{
“Staging”: {
“URL”: “https://staging.myloanapp.com”
},
“Dev”: {
“URL”: “https://dev.myloanapp.com”
}
}
102. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
// inputs.json
{
“browser”: “Chrome”
“config”: “Staging”
}
// config.json
{
“Staging”: {
“URL”: “https://staging.myloanapp.com”
},
“Dev”: {
“URL”: “https://dev.myloanapp.com”
}
}
Given the page “https://www.myloanapp.com” is loaded
103. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Given the MyLoanApp web page is loaded
// inputs.json
{
“browser”: “Chrome”
“config”: “Staging”
}
// config.json
{
“Staging”: {
“URL”: “https://staging.myloanapp.com”
},
“Dev”: {
“URL”: “https://dev.myloanapp.com”
}
}
104. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Scenario: Create a new loan application
Given the MyLoanApp web page is loaded
When the user creates a new loan application for “home mortgage”
And the user enters the following information:
| field | value |
| name | Freddy Krueger |
| address | 1428 Elm Street |
| amount | $200,000 |
| maturity | 30 years |
| down payment | 20% |
And the user submits the application
Then the page displays a success message with a reference number
And the loan application is sent to the bank
Test Case Values
105. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Scenario: Create a new loan application
Given the MyLoanApp web page is loaded
When the user creates a new loan application for “home mortgage”
And the user enters the following information:
| field | value |
| name | Freddy Krueger |
| address | 1428 Elm Street |
| amount | $200,000 |
| maturity | 30 years |
| down payment | 20% |
And the user submits the application
Then the page displays a success message with a reference number
And the loan application is sent to the bank
Literal Values
106. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Scenario: Create a new loan application
Given the MyLoanApp web page is loaded
When the user creates a new loan application for “home mortgage”
And the user enters the following information:
| field | value |
| name | Freddy Krueger |
| address | 1428 Elm Street |
| amount | $200,000 |
| maturity | 30 years |
| down payment | 20% |
And the user submits the application
Then the page displays a success message with a reference number
And the loan application is sent to the bank
Output References
107. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Scenario: Create a new loan application
Given the MyLoanApp web page is loaded
When the user creates a new loan application for “home mortgage”
And the user enters the following information:
| field | value |
| name | Freddy Krueger |
| address | 1428 Elm Street |
| amount | $200,000 |
| maturity | 30 years |
| down payment | 20% |
And the user submits the application
Then the page displays a success message with a reference number
And the loan application is sent to the bank
Created Record
108. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Scenario: Create a new loan application
Given the MyLoanApp web page is loaded
When the user creates a new loan application for “home mortgage”
And the user enters the following information:
| field | value |
| name | Freddy Krueger |
| address | 1428 Elm Street |
| amount | $200,000 |
| maturity | 30 years |
| down payment | 20% |
And the user submits the application
Then the page displays a success message with a reference number
And the loan application is sent to the bank
Input Reference
109. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
When the user creates a new loan application for a home mortgage
110. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
When the user creates a new loan application for a home mortgage
111. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
When the user creates a new loan application for a home mortgage
// Automation pseudocode
request = MakeRestRequestForAvailableLoanTypes();
response = restClient.Get(request);
foreach loan in response.Data:
if loan.Type is home mortgage
return loan.SpecificName
112. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Test Case Data
113. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Test Case Data
Type Description
Test Control Inputs Direct how tests will be run, not what behavior is covered.
Should be supplied via flat files or env vars.
114. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Test Case Data
Type Description
Test Control Inputs Direct how tests will be run, not what behavior is covered.
Should be supplied via flat files or env vars.
Configuration Metadata Describe product configuration for the target environment.
Should be supplied via config files or service API calls.
115. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Test Case Data
Type Description
Test Control Inputs Direct how tests will be run, not what behavior is covered.
Should be supplied via flat files or env vars.
Configuration Metadata Describe product configuration for the target environment.
Should be supplied via config files or service API calls.
Test Case Values Direct the behavior covered by the test.
May be literals, output references, or input references.
Input references may be hard-coded or discovered.
116. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
117. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
The nightmare isn’t over.
118. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
The nightmare isn’t over.
Collisions can happen!
119. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Avoiding Collisions
120. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Avoiding Collisions
1. Isolate the test environment(s)
a. Restrict access while testing?
b. Deploy new containers?
c. Make database clones?
121. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Avoiding Collisions
1. Isolate the test environment(s)
a. Restrict access while testing?
b. Deploy new containers?
c. Make database clones?
2. Treat any shared data as immutable
a. Parallel tests should not alter shared data
b. If any tests must temporarily alter shared data, run them serially separately
122. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Avoiding Collisions
1. Isolate the test environment(s)
a. Restrict access while testing?
b. Deploy new containers?
c. Make database clones?
2. Treat any shared data as immutable
a. Parallel tests should not alter shared data
b. If any tests must temporarily alter shared data, run them serially separately
3. Use dynamic data prep as much as possible
a. Tests cannot collide on data they don’t share
123. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Major Takeaways
124. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Major Takeaways
1. 2 types of test data: product data and test case data
125. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Major Takeaways
1. 2 types of test data: product data and test case data
2. Product data can be prepared statically or dynamically
126. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Major Takeaways
1. 2 types of test data: product data and test case data
2. Product data can be prepared statically or dynamically
3. Test case data either control how tests run or reflect product data
127. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Major Takeaways
1. 2 types of test data: product data and test case data
2. Product data can be prepared statically or dynamically
3. Test case data either control how tests run or reflect product data
4. Handle references and shared data carefully
128. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Choose the best strategies
to defeat your nightmare!
129. Managing the Test Data Nightmare | Pandy Knight | @AutomationPanda | AutomationPanda.com
Thanks!
Pandy Knight
Automation Panda
Applitools Developer Advocate
Test Automation University Director