Fundamentals of DevOps for Data Testers Course - Module 7

1. © 2019 Real-Time Technology Solutions, Inc.
22 West 38th Street FL 11, New York, NY 10018
www.rttsweb.com | (212) 240-9050
1 a software division of
QuerySurge™

2. © 2019 Real-Time Technology Solutions, Inc.
22 West 38th Street FL 11, New York, NY 10018
www.rttsweb.com | (212) 240-9050
2 a software division of
QuerySurge™
• DataOps Tests have a role in both the Value and Innovation Pipelines.
• In the Value Pipeline, tests monitor the data flowing through to catch
anomalies or flag data values that do not meet business requirements.
• In the Innovation Pipeline, tests validate new analytics and features
before deploying to production systems.
• In DataOps, tests target either data or code.
DataOps Tests
for
DATA

3. © 2019 Real-Time Technology Solutions, Inc.
22 West 38th Street FL 11, New York, NY 10018
www.rttsweb.com | (212) 240-9050
3 a software division of
QuerySurge™
• Data that flows through the Value Pipeline is variable and
subject to statistical process control and monitoring.
• Tests target the data which is continuously changing.
• The underlying code, in the Value Pipeline is fixed and
changed less frequently using a formal release process.
DataOps Tests (continued)

4. © 2019 Real-Time Technology Solutions, Inc.
22 West 38th Street FL 11, New York, NY 10018
www.rttsweb.com | (212) 240-9050
4 a software division of
QuerySurge™
• In the Innovation Pipeline code is variable and data is fixed. The code is
constantly revised and updated.
• The data doesn’t usually change frequently. In the Innovation Pipeline,
tests target the code, not the data.
• All tests must pass before promoting new code into production.
• A set of tests serves as an automated form of impact analysis that runs
on every code change before deployment.
DataOps Tests (continued)

5. © 2019 Real-Time Technology Solutions, Inc.
22 West 38th Street FL 11, New York, NY 10018
www.rttsweb.com | (212) 240-9050
5 a software division of
QuerySurge™
• Many tests can be designed for both the data and the code.
• For example, a test that makes sure that a database has the right
number of rows helps test data and code are working together.
• Eventually, both data tests and code tests need to come together in an
integrated pipeline as shown in a previous slide.
• The DataOps testing approach enables both code and data tests to
work together, ensuring all around quality remains high.
DataOps Tests (continued)

6. © 2019 Real-Time Technology Solutions, Inc.
22 West 38th Street FL 11, New York, NY 10018
www.rttsweb.com | (212) 240-9050
6 a software division of
QuerySurge™
• The main advantage of automated testing is that it’s easier to run so it’s
executed repeatedly and regularly.
• In the value pipeline the tests are often executed overnight after a data
migration or ETL job is complete.
• Manual testing is often too expensive and slow to run on a regular basis.
• To ensure high quality, you must be able to consistently and regularly
test your data and code.
• In the innovation pipeline, the code should be tested constantly.
The Automation Advantage

7. © 2019 Real-Time Technology Solutions, Inc.
22 West 38th Street FL 11, New York, NY 10018
www.rttsweb.com | (212) 240-9050
7 a software division of
QuerySurge™
• These concepts can be new to many data teams, but they are well
established in the software industry.
• The cycle time of software development releases has been reduced by
orders of magnitude through automation and process improvements (See
the next slide).
• The automation of testing in DevOps will have a positive effect on your
organization’s release cycle time.
• It is practically mandatory.
The Automation Advantage (continued)

8. © 2019 Real-Time Technology Solutions, Inc.
22 West 38th Street FL 11, New York, NY 10018
www.rttsweb.com | (212) 240-9050
8 a software division of
QuerySurge™
The Automation Advantage
The DataOps Cookbook — from DataKitchen

9. © 2019 Real-Time Technology Solutions, Inc.
22 West 38th Street FL 11, New York, NY 10018
www.rttsweb.com | (212) 240-9050
9 a software division of
QuerySurge™
Test Automation in the Value Pipeline
The DataOps Cookbook — from DataKitchen

10. © 2019 Real-Time Technology Solutions, Inc.
22 West 38th Street FL 11, New York, NY 10018
www.rttsweb.com | (212) 240-9050
10 a software division of
QuerySurge™
• The data that flows through the system needs continuous
testing as the data generally flows constantly.
• Data that moves through the pipeline is usually variable and
the test design needs to take this into account.
• As the previous slide shows, the data pipeline delivers value
to the end users.
Test Automation in
the Value Pipeline

11. © 2019 Real-Time Technology Solutions, Inc.
22 West 38th Street FL 11, New York, NY 10018
www.rttsweb.com | (212) 240-9050
11 a software division of
QuerySurge™
Test Automation in the
Innovation Pipeline
The DataOps Cookbook — from DataKitchen
Data Fixed Data Variable
Code Fixed Value Pipeline
Code Variable Innovation Pipeline

12. © 2019 Real-Time Technology Solutions, Inc.
22 West 38th Street FL 11, New York, NY 10018
www.rttsweb.com | (212) 240-9050
12 a software division of
QuerySurge™
• In the Innovation Pipeline code is variable, but data is fixed. Tests target
the code, not the data.
• The unit, integration, functional, performance and regression tests that
were mentioned earlier in the course is scrutinizing new code.
• All tests are run before promoting new code to production.
• A good test suite serves as an automated form of impact analysis that can
be run on every code change before deployment.
Test Automation in the Innovation Pipeline

13. © 2019 Real-Time Technology Solutions, Inc.
22 West 38th Street FL 11, New York, NY 10018
www.rttsweb.com | (212) 240-9050
13 a software division of
QuerySurge™
• In many instances, tests can be aimed at both data and code.
• For example, a test that makes sure that a database has the right
number of rows helps your data and code work together.
• Ultimately both data tests from the value pipeline and code tests
from the innovation pipeline can work together in DataOps,
keeping all around quality high.
• Robust and thorough automated testing removes or minimizes the
need to perform manual testing, avoiding a major bottleneck that
slows release cycle time.
More on Test Automation

14. © 2019 Real-Time Technology Solutions, Inc.
22 West 38th Street FL 11, New York, NY 10018
www.rttsweb.com | (212) 240-9050
14 a software division of
QuerySurge™
• Automated testing can work around the clock to ensure that bad
data never reaches users, and when an issue is found, new
automated tests can be created to make certain that the issue
never happens again.
• Automated testing and monitoring greatly multiplies the coverage
and quality that the data team provides.
More on Test Automation

15. © 2019 Real-Time Technology Solutions, Inc.
22 West 38th Street FL 11, New York, NY 10018
www.rttsweb.com | (212) 240-9050
15 a software division of
QuerySurge™
Test Automation: Data Flow
Example data flow: The DataOps Cookbook — from DataKitchen

16. © 2019 Real-Time Technology Solutions, Inc.
22 West 38th Street FL 11, New York, NY 10018
www.rttsweb.com | (212) 240-9050
16 a software division of
QuerySurge™
• We can think of data testing as a manufacturing pipeline.
• There are inputs (sources), processes (transformations) and outputs
(targets).
• A typical manufacturing process includes tests at every step in the pipeline
that attempt to identify issues as early as possible.
• It is much more efficient and less expensive to catch a problem at the
inputs opposed to the outputs.
Test Automation: Data Flow

17. © 2019 Real-Time Technology Solutions, Inc.
22 West 38th Street FL 11, New York, NY 10018
www.rttsweb.com | (212) 240-9050
17 a software division of
QuerySurge™
• Data input tests strive to prevent any bad data from being fed into the
pipeline.
• Allowing bad data to progress through the pipeline wastes processing
resources and increases the risk of never catching an issue.
• These tests also focuses the attention on the quality of data sources,
which must be actively validated.
• Automated tests should verify that the incoming data meets
requirements in order to be processed successfully.
Test Automation: Data Flow (continued)

18. © 2019 Real-Time Technology Solutions, Inc.
22 West 38th Street FL 11, New York, NY 10018
www.rttsweb.com | (212) 240-9050
18 a software division of
QuerySurge™
• Data output tests verify that a pipeline stage executed correctly.
• The outputs will need to be compared to the input to validate the
successful flow of data.
• Business logic tests validate data against true assumptions about the
business and requirements.
• For example, perhaps all European customers are assigned to a member of
the European sales team.
Test Automation: Data Flow (continued)

19. © 2019 Real-Time Technology Solutions, Inc.
22 West 38th Street FL 11, New York, NY 10018
www.rttsweb.com | (212) 240-9050
19 a software division of
QuerySurge™
About QuerySurge
QuerySurge is the smart Data Testing solution that automates the
data validation and ETL testing of:
o Big Data,
o Data Warehouses,
o Business Intelligence Reports and
o Enterprise Applications
with full DataOps functionality for continuous testing.

20. © 2019 Real-Time Technology Solutions, Inc.
22 West 38th Street FL 11, New York, NY 10018
www.rttsweb.com | (212) 240-9050
20 a software division of
QuerySurge™
Benefits of implementing
QuerySurge in DataOps pipeline
Testers can dynamically generate,
execute, and update tests and data
stores utilizing API calls.
QuerySurge’s RESTful API functions in
Swagger allowing for results to return
before implementation in code.
Teams have access to 60+
RESTful API calls with almost
100 different properties.
QuerySurge integrates with virtually
all DevOps solutions, including all Data
Integration/ETL solutions, Build
solutions, and QA solutions

21. © 2019 Real-Time Technology Solutions, Inc.
22 West 38th Street FL 11, New York, NY 10018
www.rttsweb.com | (212) 240-9050
21 a software division of
QuerySurge™
About QuerySurge and Swagger
• Swagger is an open-source Interface Description Language (IDL) for
describing RESTful APIs expressed using JSON.
• Swagger is used together with a set of open-source software tools to
design, build, document, and use RESTful web services.
• Swagger includes automated documentation, code generation, and test-
case generation.
• Swagger is embedded in QuerySurge! Labs included with this course will
give you hands-on knowledge of using the QuerySurge API.
RESTful API – Documented with Swagger

22. © 2019 Real-Time Technology Solutions, Inc.
22 West 38th Street FL 11, New York, NY 10018
www.rttsweb.com | (212) 240-9050
22 a software division of
QuerySurge™
• In the value pipeline automated tests are often executed
overnight after a data migration or ETL job is complete.
• In the innovation pipeline, the code should be tested frequently
and after every code change.
• Tests in the value pipeline tests the data.
• Tests in the innovation pipeline tests the code.
Testing in DataOps Highlights