This document discusses data quality testing. It begins by defining data quality and listing its key dimensions such as accuracy, consistency, completeness and timeliness. It then notes common business problems caused by poor data quality and the benefits of improving data quality. Key aspects of data quality testing covered include planning, design, execution, monitoring and challenges. Best practices emphasized include understanding the business, planning for data quality early, being proactive about data growth and thoroughly understanding the data.

1. Data Quality - Testing
Vijaya Kokkili
Director of Quality
CommerceHub

2. MEEEE………  Overcome the fear!!!
Gardening
Adrenaline Junkie

3. MEEEE………  Still to come…….

4. Agenda:
Data Quality Data Quality Testing
World trending towards….. How to test data quality
Facts about data quality Data quality test management
Most common business problems Data quality testing challenges
Business benefits Data quality testing best practices
What is data quality?
Dimensions of data quality
Definitions of dimensions
Real time situation
Measuring data quality
Data profiling analysis
When and how to conduct data
profiling

5. • Operating systems
• Mobile platforms
• Software frameworks
• Hardware
• Software

7. Few facts about data quality:
● Cost of poor data quality in US - $600 Billion
● Poor Data/Lack of visibility cited as #1 reason for project cost overruns
● Poor data quality costs the US Economy $3.1 Trillion a year
● Implementing data quality best practices boosts revenue by 66%
● Median Fortune 1000 company could increase revenue by $2.01 Billion if they improved
usability of data by 10%

8. Most common business problems
• Billing and payment errors causing negative customer perceptions
• Operating expenses are inflated
• Regulatory fines are levied due to inaccurate reporting of data to government entities
• Customers and revenue are lost due to an inability to track customer interactions or to
recognize high-value customers
• Disruption of service
• Flawed analytics lead to poor tactical and strategic directions
• Extra time on IT projects to reconcile data
• Delays in deploying new systems

10. Business benefits:
• Customer satisfaction
• Strengthens trust and collaboration between trading partners
• Increases supply chain efficiencies and cuts costs by reducing errors
• Cuts delays at point-of-sale as a result of reduced measurement errors
• Increases reliability and efficiency
• Ensures better compliance

11. What is data quality?
Data quality is a perception or an assessment of data’s fitness to serve its
purpose in a given context

12. Dimensions of data quality:
● Consistency
● Accuracy
● Correctness
● Objectivity
● Timeliness
● Conciseness
● Precision
● Usefulness
● Unamiguous
● Usability
● Completeness
● Relevance
● Reliability
● Amount of data

13. Definitions of data quality dimensions:
•Correctness / Accuracy: Accuracy of data is the degree to which the
captured data correctly describes the real world entity.
•Consistency: This is about the single version of truth. Consistency means data
throughout the enterprise should be sync with each other.
•Completeness: It is the extent to which the expected attributes of data are
provided.
•Timeliness: Right data to the right person at the right time is important for
business.

14. Definitions of data quality dimensions:
•Correctness / Accuracy:
Accuracy of data is the degree to which the captured data correctly describes
the real world entity.
Ability to draw correct conclusions from data
Business process that match reality
Eg of data accuracy issues:
• An incident reported with $23M when the loss was $12k
• The amount invoiced does not represent the customer’s usage

15. Definitions of data quality dimensions:
•Consistency: This is about the single version of truth. Consistency means
throughout the enterprise should be sync with each other.
Ability to trust data regardless of source
Identical information available to all processes and units
Eg of data consistency issues:
• Mr.A defines “reprocessing” as cancel/total and Mr. B as Cancel/new.

16. Definitions of data quality dimensions:
•Completeness: It is the extent to which the expected attributes of data are
provided.
Data that does not leave any open questions
Ability to make a good decision based on available data
Closeness between “need to know” and what data tells you
Eg of data completeness issues:
• We cannot tell how many cell phone contracts Mr. X has
• A summary report includes projects that did not report status!

17. Definitions of data quality dimensions:
•Timeliness: Right data to the right person at the right time is important for
business.
Data that is available without delay
Ability to know what you need, when you need
Smooth information flow: “Data delayed is Data denied!”
Eg of data timeliness issues:
• Receiving a “budget exceeded” SMS after you went over the limit!

18. Real time situation
Many database professionals face situations like:
1. Several data inconsistencies in source, like missing records or NULL values.
2. column they chose to be the primary key column is not unique throughout the table.
3. Schema design is not coherent to the end user requirement.
4. Any other concern with the data, that must have been fixed right at the beginning

19. What does it mean to fix data quality issues?
Make changes in ETL data flow packages, cleaning identified inconsistencies etc..
Lot of re-work to be done
Added costs in terms of time and effort
So…..
What is the solution???

20. Solution
“PREVENTION IS BETTER THAN CURE”
Hence data profiling comes to the rescue

21. Measuring Data Quality
Profiling – Understand metadata
• Point of time shows what data looks like now
• Automating shows trends
o Alert to new/potential issues as they happen
o Potentially fix issues in near real time

22. Statistical process control
Automated inspection
Visibility shows process deviation

23. Data profiling analysis
Duplication
Pattern matching
Day of week
Character set
Reference data matching
Inter-data set comparisons

24. Master data management
Create a standard for data
Distribute data so that all sources are uniform
• Names
• Addresses
• Phone numbers
• Products
Can hook into 3rd party sources

25. Data Governance
Central authority for data quality control
Applies information collected from data profiling uniformly across the business
Communication channels between business and IT groups

26. Maintenance of data quality
Data quality results from the process of going through the data and scrubbing it,
standardizing it, and removing duplicate records, as well as doing some of the data
enrichment.
1. Maintain complete data
2. Clean up data by standardizing using rules
3. Using algorithms to detect duplicates
4. Avoid entry of duplicate leads and contacts
5. Merge existing duplicate records
6. Use roles for security

27. Inconsistent data before cleaning up
Bill no CustomerName SSN
101 Ms Vijaya Kokkili SSN100123
Bill no CustomerName SSN
204 Ms V Kokkili SSN100123
Bill no CustomerName SSN
354 Ms Kokkili Vijaya SSN100123
Bill no CustomerName SSN
467 Ms Vijaya K SSN100123
Invoice 1
Invoice 2
Invoice 3
Invoice 4

28. Consistent data after cleaning up
Bill no CustomerName SSN
101 Ms Vijaya Kokkili SSN100123
Bill no CustomerName SSN
204 Ms Vijaya Kokkili SSN100123
Bill no CustomerName SSN
354 Ms Vijaya Kokkili SSN100123
Bill no CustomerName SSN
467 Ms Vijaya Kokkili SSN100123
Invoice 1
Invoice 2
Invoice 3
Invoice 4

29. When and how to conduct data profiling?
Generally, data profiling is conducted in two ways:
1.Writing SQL queries on sample data extracts put into a database.
2.Using data profiling tools
When to conduct Data profiling?
At the discovery/requirements gathering phase

30. How to conduct data profiling?
Data profiling involves statistical analysis of the data at source and the data being loaded, as well as
analysis of metadata. These statistics may be used for various analysis purposes. Common examples
of analyses to be done are:
Data quality: Analyze the quality of data at the data source.
NULL values: Look out for the number of NULL values in an attribute
Candidate keys: Analysis of the extent to which certain columns are distinct will give developer
useful information w. r. t. selection of candidate keys.
Primary key selection: To check whether the candidate key column does not violate the basic
requirements of not having NULL values or duplicate values.
Empty string values: A string column may contain NULL or even empty sting values that may create
problems later.
String length: An analysis of largest and shortest possible length as well as the average string length
of a sting-type column can help us decide what data type would be most suitable for the said column

31. How to test for Data quality?
Discrepancy in
records count at
Source & target
When all data is at
source is present at
target
Ensure that source &
target don’t contain
conflicting facts
Degree of conformance
of data to its domain
and business values
Physical and logical
duplicates
Orphan records in
targets when no
corresponding parent
records
List of valid/invalid
values that are allowed
along with ranges, look
up etc
Degree to which
data reflects the
real world objects
Describes the
relevance &
meaning of data
Describes
availability of data
as per SLA
Row Count Completeness Consistency
Validity Redundancy Referential Integrity
Domain Integrity Accuracy Usability Timeliness

32. Data quality test management
Test planning Test design Test Execution Test monitoring
Requirements:
• BRD
• FSD
• Test Plan
Requirements:
• Test
scenarios
• Test cases
• Automated
Requirements:
• Executed in
test cycles
• Test
results/bugs
are shared
with
business
• Prioritize
Requirements:
• Collect
metrics
• Observe
trend

33. Data quality testing challenges
• Lack of tools
• Lack of domain knowledge
• Changing requirements
• Poor planning for data quality in initial phase of the application

34. Data quality testing best practices
• Understand user business
• Plan early in Design and testing phase
• Be proactive when it comes to data growth/trending
• Don’t assume! Understand data!

35. Q & A
@vkokkili
vkokkili@gmail.com