Fast and easy. No Programming needed. The latest QuerySurge release introduces the new Query Wizards. The Wizards allow both novice and experienced team members to validate their organization's data quickly with no SQL programming required.
The Wizards provide an immediate ROI through their ease-of-use and ensure that minimal time and effort are required for developing tests and obtaining results. Even novice testers are productive as soon as they start using the Wizards!
According to a recent survey of Data Architects and other data experts on LinkedIn, approximately 80% of columns in a data warehouse have no transformations, meaning the Wizards can test all of these columns quickly & easily, (The columns with transformations can be tested using the QuerySurge Design library using custom SQL coding.)
There are 3 Types of automated Data Comparisons:
- Column-Level Comparison
- Table-Level Comparison
- Row Count Comparison
There are also automated features for filtering (‘Where’ clause) and sorting (‘Order By’ clause).
The Wizards provide both novices and non-technical team members with a fast & easy way to be productive immediately and speed up testing for team members skilled in SQL.
Trial our software either as a download or in the cloud at www.QuerySurge.com. The trial comes with a built-in tutorial and sample data.
Adobe Marketo Engage Deep Dives: Using Webhooks to Transfer Data
Query Wizards - data testing made easy - no programming
1. built by
Introducing the new
We just made data testing
REALLY EASY!
No programming needed
Automate your
Data Warehouse & Big Data Testing
and Reap the Benefits
*available for download on August 3, 2015
2. built by
QuerySurge™
About
FACTS
Founded:
1996
Locations:
New York (HQ), Atlanta,
Philadelphia, Phoenix
Strategic Partners:
IBM, Microsoft, HP,
Oracle, Teradata,
HortonWorks, Cloudera,
Amazon
Software:
QuerySurge
RTTS is the leading provider of software & data quality
for critical business systems
3. “70% of enterprises have either deployed or are planning to
deploy big data projects and programs this year”
– analyst firm IDG
“46% of companies cite data quality as a barrier for adopting
Business Intelligence products.”
- InformationWeek
“Poor data quality is a primary reason for 40% of all business
initiatives failing to achieve their targeted benefits.”
- analyst firm Gartner
Data Quality Issues
built by
QuerySurge™
5. DataTesting Strategy #1: Stare & Compare
built by
QuerySurge™
• Review Mapping Document (business rules, data flow mapping, data movement requirements)
• Write Tests in SQL editor
• Execute 2 Tests: 1 at Source & 1 at Target
• Dump results to 2 Excel files
• Compare results by eye (Stare & Compare)
Difficulty with Stare & Compare:
Impossible to visually compare millions/billions of data sets visually.
Example:
Current QuerySurge customer has:
• a single test with 100 million rows
• 200 columns
• = 20 billion data sets
• the client has > 7,000 total tests
Data Tester’s Current Process
6. built by
QuerySurge™
MINUS QUERIES subtract one result set from another result set to show difference
Comment: MINUS QUERIES need to be executed 2x (Source MINUS Target; Target MINUS Source)
Result sets may not be accurate when dealing with duplicate rows of data
No historical data from past testing – audit and regulatory issues
Processing of minus queries puts pressure on the servers
Double execution means 2x testing time and resource utilization
Potential for false positives (bad data could exist on both sides of an ETL leg)
DataTesting Strategy #2: Minus Queries
Minus Query #1: Table_1 MINUS Table_2
Minus Query #2: Table_2 MINUS Table_1
Result Set #1
Result Set #2
ISSUES with MINUS QUERIES
Write 2 MINUS queries
in SQL editor
Execute
MINUS queries 2x
8. What is QuerySurge™?
the collaborative
Data Testing solution that
finds bad data & provides
a holistic view of your
data’s health
built by
9. • Reduce your costs & risks
• Improve your data quality
• Accelerate your testing cycles
• Share information with your team
with QuerySurge™ you can:
built by
QuerySurge™
• Provides huge ROI (i.e. 1,300%)*
*based on client’s calculation of Return on Investment
10. the QuerySurge advantage
built by
QuerySurge™
Automate the entire testing cycle
Automate kickoff, tests, comparison, auto-emailed results
Create Tests easily with no SQL programming
ensures minimal time & effort to create tests / obtain results
Test across different platforms
data warehouse, Hadoop, NoSQL, database, flat file, XML
Collaborate with team
Data Health dashboard, shared tests & auto-emailed reports
Verify more data & do it quickly
verifies up to 100% of all data up to 1,000 x faster
Integrate for Continuous Delivery
Integrates with most Build, ETL & QA management software
11. Finding Bad Data
SQL
HQL
SQL
HQL
SQL
SQL
QS pulls data from data sources
QS pulls data from target data store
QS compares data quickly
QS generates reports, audit trails
How?
Reports, Data Health Dashboard
built by
QuerySurge™
Source Data
Target Data
13. Collaboration
Testers
- functional testing
- regression testing
- result analysis
Developers / DBAs
- unit testing
- result analysis
Data Analysts
- review, analyze data
- verify mapping failures
Operations teams
- monitoring
- result analysis
Managers
- oversight
- result analysis
Share information on the
built by
QuerySurge™
15. built by
From a recent poll1 of:
• Big Data Experts
• Data Warehouse Architects
• Solution Architects
• ETL Architects
Recent Survey: Data Experts
Consensus Answer:
80% of data columns have no transformation at all
Our Question: What % of columns in your Data Warehouse
have no transformations at all?
1Poll conducted by RTTS on targeted LinkedIn groups
Why is this important?
16. Fast and Easy.
No programming needed.
built by
QuerySurge™
QuerySurge™ Modules
Compare by Table, Column & Row
• Perform 80% of all data tests
•Automatically generates SQL code
• Opens up testing to novice & non-
technical team members
• Speeds up testing for skilled SQL coders
• provides a huge Return-On-Investment
17. built by
QuerySurge™
QuerySurge™ Modules
3 Types of Data Comparison Wizards:
The also provide you with automated features for:
o filtering (‘Where’ clause) and
o sorting (‘Order By’ clause)
Column-Level Comparison:
This is great for Big Data stores and Data Warehouses where tables will have some columns
containing transformations and some columns with no transformations. Many tables and
columns can be compared simultaneously and quickly.
Table-Level Comparison:
This comparator is great for Data Migrations and Database Upgrades with no
transformations at all. Many tables can be compared simultaneously and quickly.
Row Count Comparison:
Great for all - Big Data stores, Data Warehouses, Data Migrations and Database Upgrades.
Many tables and rows can be compared simultaneously and quickly.
18. Uses:
Tests the columns that have no
transformations, which means it tests
approximately 80% of your data store without
you writing any SQL code
Tests:
Big data, data warehouses
Value added:
novice or non-technical: no coding needed,
productive immediately
experienced user: saves time
built by
QuerySurge™
20. Uses:
Verifies data loads when no
transformation occurs
Tests:
data migrations, upgrades
Value added:
novice or non-technical:
no coding needed
experienced user:
saves time
built by
QuerySurge™
21. Use:
Verify that the amount of rows from the
source match the amount from the target
Tests:
Big data, data warehouse, data
migration, database upgrades, data
interfaces
Value added:
novice: no coding needed
experienced user: saves time
built by
QuerySurge™
_________
Total
23. Design Library
• Create custom Query Pairs (source & target SQLs)
• Great for team members skilled with SQL
QuerySurge™ Modules
Scheduling
Build groups of Query Pairs
Schedule Test Runs for:
• immediately
• at a specific date/time
• automatically after build or
ETL process
built by
QuerySurge™
24. Deep-Dive Reporting
Examine and automatically
email test results
Run Dashboard
View real-time execution
Analyze real-time results
QuerySurge™ Modules
built by
QuerySurge™
25. QuerySurge Test Management Connectors
built by
QuerySurge™
Drive QuerySurge execution from your Test Management Solution
Outcome results (Pass/Fail/etc.) are returned from QuerySurge to your Test Management Solution
Results are linked in your Test Management Solution so that you can click directly into detailed QuerySurge
results
• HP ALM (Quality Center)
• Microsoft Team Foundation Server
• IBM Rational Quality Manager
Integration with leading
Test Management Solutions
26. Licensing
License Types:
Full User
Participant User (i.e. read-only)
built by
QuerySurge™
License Model for:
Perpetual
own QuerySurgeTM, pay annual maintenance
Subscription
use QuerySurgeTM for set period (12 months)
QuerySurge™
27. 8/18/2015 27
built by
QuerySurge™
Training Courses
Data Warehouse Testing
• Data Warehouse & ETL Testing Fundamentals (1 day)
• Fundamentals of QuerySurge (1 day)
• Introduction to SQL for QuerySurge (1 day)
• Advanced SQL techniques for QuerySurge (1 day)
Big Data Testing
• Big Data And ETL Testing Fundamentals
• Introduction To Big Data Testing Using Hive And HQL
Consulting
RTTS, the software quality experts (and developer of QuerySurge), provides consulting
solutions to the challenges of Big Data & Data Warehouse / ETL Testing
• Jumpstart 2-week program – combines training courses, mentoring, consulting
• Staff Augmentation – add additional RTTS resources to your team
• Outsourcing - RTTS can perform all testing, including planning, design, execution
28. built by
QuerySurge™
Support
• Live Chat through QuerySurge or web site
• Email support through Zendesk
• Yammer Network
• Webex sessions
• Phone support
29. (1) Trial in the Cloud of QuerySurgeTM, including self-learning
tutorial that works with sample data for 3 days
(2) Downloaded Trial of QuerySurgeTM, including self-learning
tutorial with sample data or your data for 15 days
(3) Proof of Concept of QuerySurgeTM includes our team of experts
assisting you for 30 days
for more information on (1), (2) and (3),
Go to querysurge.com/compare-trial-options
TRIAL
IN THE CLOUD
built by
QuerySurge™
Free TrialsQuerySurge™
Proof
of
Concept
QuerySurge provides insight into the health of your data throughout your organization through BI dashboards and reporting at your fingertips. It is a collaborative tool that allows for distributed use of the tool throughout your organization and provides for a sharable, holistic view of your data’s health and your organization’s level of maturity of your data management.
QuerySurge helps your team coordinate your data quality initiatives while speeding up your development and testing cycles and finding your bad data. Why risk having your team identify trends and develop strategic initiatives when the underlying data is incorrect? QuerySurge reduces this risk.
QuerySurge finds bad data by natively connecting to:
any data source, whether it is any type of database, flat file or xml and
can connect to any data target, whether it is a db, file, xml, data warehouse or hadoop implementation.
QuerySurge pulls data from the source and the target and compares them very quickly (typically in a few minutes) and then produces reports that show every data difference, even if there are millions of rows and hundreds of columns in the test. These reports can be automatically emailed to your team.
You can pick from a multitude of reports or export the results so that you can build your own reports.
Your distributed team from around the world can use any of these web browsers: Internet Explorer, Chrome, Firefox and Safari.
Installs on operating systems: Windows & Linux.
QS connects to any JDBC-compliant data source. Even if it is not listed here.
QuerySurge can utilized by active practitioners such as testers & developers to create and launch tests, or by managers, analysts and operations to view data test results and the overall health of the data. QuerySurge facilitates this by providing 2 types of licenses: (1) full user & (2) participant user.
(1) Full User – This type of user has unlimited access to create QueryPairs, Suites, and Scenarios. This user can also schedule and run tests, see results, run and export reports, and export data. Perfect for anyone creating and/or running data tests while performing analysis of results.
(2) Participant User – This user cannot create or run tests, but has access to all other information - including viewing all query pairs, results, and reports, receiving email notifications, and exporting test results and reports. Perfect for managers, analysts, architects, DBAs, developers, and operations users who need to know the health of their data.