Curiosity and Lemontree present - Data Breaks DevOps: Why you need automated test data in CI/CD

© Curiosity Software Ireland Ltd. 2021
Data Breaks DevOps
Why you need automated test data in CI/CD
Huw Price
Managing Director
Curiosity Software
Huw.Price@Curiosity.Software
@DataInventor

Short facts
• Established 1999
• Currently 40 consultants
• Owned by entrepreneurs
and employees
• Consultancy- or project
deliveries
• Customers in Sweden
and Norway
• Telecom, Banking and
Finance, Public, Energy
and Industry
• Various of partners

Questions & follow up:
patric.helje@lemontree.se
rickard.ridderstrom@lemontree.se
And now...
• 8:15 Welcome – a short introduction
Isabella Nyman, Lemontree
• 8:20-8:50 Huw Price, Curiosity
• 8:50-09:00 Q&A

Data Breaks DevOps: Why you need
automated test data in CI/CD
1. Problem statement: (Test) Data Breaks DevOps
2. Solution: Automate Data Allocation within CI/CD Pipelines
3. Demo: Test Data Automation at the speed of DevOps
Premise: Automating test data allows us to fully automate quality gates in CI/CD pipelines.
This eliminates a bottleneck that commonly derails DevOps, executing rigorous tests even
as systems and user behaviours change.

1. Test Data & DevOps
Slow, manual and siloed test data provisioning is the antithesis of
automation in testing. It’s therefore also the opposite of DevOps.

DevOps pipelines are only as fast as
their slowest part
• The average QA team spends
44% of their time searching
for, managing and generating
data.1
• Only 36% of organisations
deliver data on demand.2
• Half of developers encounter
the most delays in testing.3
Original Image: Khanargy, Wikimedia Commons, published under CC BY-SA
4.0 license.
[1] Capgemini, Sogeti (2020), The Continuous Testing Report 2020, 21
[2] Capgemini, Sogeti (2020), The Continuous Testing Report 2020, 14.
[3] GitLab (2019), 2019 Global Developer Report.
Test Data
Bottlenecks
Tank DevOps
Pipelines

Today’s test data requirement
To automate quality gates within CI/CD pipelines, we need data that is:
1. Complete, capable of executing every positive, negative, and unexpected scenario.
2. Up-to-date, including different combinations of system versions and unreleased logic.
3. Compliant, containing no personally identifiable or commercially sensitive data.
4. Consistent, seamlessly linking up across many different technologies.
5. Parallelised, available on demand to every automation framework, test, and tester.
6. As fast as API-fuelled developers who rip and replace containers full of reusable code.
Why is this not happening today?

Test Data: The traditional approach is
not fit for purpose.
Moving copies of production databases to test environments is too slow and
costly. Nor can it match the complexity of intricate, containerised systems.
• The data is historical, low variety and therefore low coverage.
• The data is unavailable in parallel – there’s never enough of it.
• The data contains personally and commercially sensitive information.
• The data is slow to move and requires vast amounts of costly infrastructure.
• By the time data is received, the combination of components have moved on.

2. Test Data Architectures

Databases
Files
Mainframes
APIs/Messages
Applications
Interrelated data sources
Simple
Configuration
Compliant and
complete data
for every test
Subset
Mask
Generate
Clone
DevOps-Enabled Test Data
Every process is fully parameterized, combinable.
They can all be triggered within automation and CI/CD
Windocks
Virtual data copies,
fully database cloning and
containerized deployment
in seconds
Databases
Files
Mainframes
Applications
Containerised test environments
reflect the latest system
Parallel test teams, CI/CD pipleines, automation frameworks
and source control trigger test data allocation jobs on demand

Model-based test automation for
Apache Solr and Kafka
Data from
multiple back-
end systems
Catalogue of
automated data
look ups
Drag-and-drop data
design
“Find and Make” Consistent
Data Journeys
Auto-generate rigorous test cases
Fire tests on demand to
Kafka Queues
One Click
Data-driven test
execution

DB2 VSAM QISAM IMS And More!
Portus
FTP x3270 Script and File Generation
CICS MQ
Add, Delete, Update
Files Data
Flowchart Models
Dynamic
Test Data
Optimised
Test Cases
Automated
Tests
Data Preparation
Catalogue
Automated test
script creation and
test data allocation
Transparency
layer
End-to-end test
execution
Run
Results
Automated Mainframe Testing
XML, EDI, JCL, IDCAM, XML, Flat Files

An Integrated
Development Chain
Recorded tests,
manual and
automated
Test Cases
BPMN
diagrams
Mainframe, ERP,
databases
System under Test
Mobile and
web
Thick
client
Back-end,
APIs,
Messages
Test Reports
and results
Auto-update HP Octane
Auto-update Trello boards, Git, etc.
Slack and
email alerts
Test Models
BDD
Scenarios
Business
Requirements
Test Data
Optimised Test Cases
Automated Tests
Development
Clear Specifications
New Iteration
Change
Request
Import New Requirements
Dependency Analysis
Automated Maintenance
Requirements, Tests, Automation and Test Data
must all be defined together to drive a state of the
art DevOps pipeline.

3. Test Data Problems

Today’s test data requirement
A myriad of different types of problems
1. Getting Valid and Invalid Data into a UI
2. Ditto for APIs
3. Ditto for flat files
4. Chained integration testing – data journeys – data
matching
5. Automated testing data provisioning – allocating data to
tests
6. Rule based engine testing - AI
7. BI and Report testing
8. Regulatory Reporting testing - XBRL
9. Cluster Testing - AI
10. Events and Time series testing - AI
11. Stability testing – same type of data – different code
coverage
12. Performance testing
13. ERP upgrade testing
14. Etc…

4. Think Differently

ETL – Dimensions and Fact Testing
Each Dimension needs an
intersection with data from
the other dimensions
Plus
Each Dimension needs an
intersection where data
does NOT intersect with
the other Dimensions
…
Select Age, Promotion,Profit
From Sales, Customer
Left Join …
Where SalesPeriod in
(select SalesPeriod from DimPeriods)
Group by Age, Promotion
Union
Select Distinct Age,Promotion,0
From DimPeriods, Customer
Where SalesPeriod not in
(select SalesPeriod from Sales)
Group by Age, Promotion
15
16 17
14
13
11
10
9
1
2 4
5
1
6
7
8
3
2
1
8

The Message – in play
 Data Dimensions
 Accounts Receivable
 Accounts Payable
 Rule Based Rating Analysis

The Message – Solving it
 Pin Total Income Range
 Pin Total Expense Range
 Pin Profit Range

The Message – Solving it
 Income Solved
 Expenditure Solved
 Rule Based Analysis Solved

Message Solving – Rules Rock and Role
• Adjust the constraints each time you filter or make messages
• Model your rule variations to predict results
• Create multi variate related data

Change the way you build transform requirements
Start with the Data
• Build a Requirement
• Look at the data
• Adjust the Requirement
• Deliver something
• Create some test data by hand
• Test it
• Look at the results
• Adjust the Requirements
• Deliver again
• Test it by hand
• Deliver again
• Rebuild the requirement
• …
• You’re bored
• Create some more test data
• Deliver again
• Model out the potential data variations
• Compare them with your existing data
• Generate all the test data
• AND the expected results
• Create the requirements
• Deliver

Demo
1. Expose test data utilities to a library of re-usable processes
2. Link test data processes to test generation or execution (A flow-based example)
3. Run on-the-fly data “Find and Makes” within CI/CD (A Jenkins example)

Analyse data patterns, trends and differences
• Scan Message Traffic – select and filter data dimensions
• Look for numeric data patterns, skewness, kurtosis (rareness)
• Compare environments for data density
• Compare message data over time
• Set it up to equalize your data characteristics automatically

5. Test Data Automation
“Reactive” and on demand test data automates quality gates within CI/CD
pipelines.

Reactive test data automation
• DevOps pipelines offer well-oiled machines for moving information from evolving
requirements, code changes, production monitoring and more.
• Each component reacts to changes across the pipeline – why not test data?
• If we parameterise up re-usable test data processes, we can auto-generate and
execute test data journeys within our CI/CD pipelines.
• This automates quality gates.
• The more information we expose, the better – test data should react to changing
user stories, Jiras, test cases, APM and more.

Test Data Automation
1. Parameterise and make re-usable
existing test data processes.
2. Methods for linking test data allocation
to changes across DevOps pipelines
Commodity Uncommon Cutting-edge
Masking
Subsetting
Provisioning
Generation
Cloning
Comparison
Test Data Allocation
Data Coverage, Data Aging
Version Compatibility
Virtual Endpoint provisioning
On-the-fly data “find and makes” performed during CI/CD
pipelines and DevOps toolchains.
Embed Test Data Automation jobs in automated tests, resolving
data “just in time” during test execution.
A self-service portal, providing testers and developers with
unconstrained data throughout each sprint.
Data flow modelling visually designs data journeys, executing
models as end-to-end test suites.
These can be in-house processes or third party
technologies. What’s key is that they are standardized,
combinable, and re-usable.
Test data is allocated (not provisioned) on-the-fly during
CI/CD. No siloes or bottlenecks, valid data for every test.
The missing piece in most
test data strategies today

Comprehensive Utilities
Standardized, Combinable, Re-usable
Subset Mask Make
Clone Find Compare
Allocate Analyse Age
Fully Connected
Easily Extendable
TC
1
TC
3
TC
2
TC
4
TC
5
Test Data Catalogue
Test Case Definitions
Parallel Test Teams
Parallelized Automation Frameworks
Direct to DBs & Mainframe / Via UIs & APIs
ALM Tools
 Continuous, “Just in Time” data delivery
 Unique combinations allocated to tests
 Full data coverage
 No false positives or negatives
 Self-service data, in parallel and on demand
 Continuous test execution as complex
systems change

Questions and Discussion

Next steps? Let’s chat!
Book us for a free 90 minute workshop – let’s team up to solve your test data
challenges.
− We’ll demo to your team & discuss their challenges with them.
− We’ll create a set of solutions and next steps together.
Visit TestDataAutomation.io and hit “Arrange Demo”.
Head to CuriositySoftware.ie for videos, webinars & eBooks.
Follow @CuriositySoft on Twitter and find us on LinkedIn.
Based in Sweden or Norway? Contact Isabella.Nyman@Lemontree.se

Thank you
Huw.Price@Curiosity.Software

Curiosity and Lemontree present - Data Breaks DevOps: Why you need automated test data in CI/CD

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Curiosity and Lemontree present - Data Breaks DevOps: Why you need automated test data in CI/CD

Similar to Curiosity and Lemontree present - Data Breaks DevOps: Why you need automated test data in CI/CD (20)

More from Curiosity Software Ireland

More from Curiosity Software Ireland (11)

Recently uploaded

Recently uploaded (20)

Curiosity and Lemontree present - Data Breaks DevOps: Why you need automated test data in CI/CD