This document provides an overview of understanding data through data profiling. It discusses: 1. The five key steps to effective data profiling: defining how to analyze the data, what to review, what to look for, when to build rules, and what to communicate. 2. Common challenges with big data and new data types, and measurements for assessing data quality. 3. A case study of how British Airways leveraged data profiling and governance to ensure accurate customer data across multiple systems and improve analysis, marketing and service.

1. Understanding Your Data
Series: Foundational Strategies Trust in Big Data – Part 2

2. Webcast Audio
• Today’s webcast audio is streamed through your computer
speakers.
• If you need technical assistance with the web interface or audio,
please reach out to us using the Q&A box.
Questions Welcome
• Submit your questions at any time during the presentation using
the Q&A box.
• We will answer them during our Q&A session following the
presentation.
Recording and slides
• This webcast is being recorded. You will receive an email following
the webcast with a link to download both the recording and the
slides.
Housekeeping
Arianna Valentini
Product Marketing Manager

3. What You Will Learn Today
• Quick refresh on ingredients of successful Big Data
• Common challenges of Big Data and data profiling
• The top 5 steps needed for effective data profiling
• How another company saw success through data profiling
• What you can do in the next 90 days to take action on DI
Wrap up with:
• Q&A
3

4. 4
Ingredients of Successful Big Data
1. Clear Business Case 2. Extract Data 3. Understand Data 4. Trace Lineage
Data Governance

5. 80%of AI/ML projects are stalling
due to poor data quality
Dimensional Research, 2019
Big Data Needs
Data Quality
“Societal trust in business is
arguably at an all-time low
and, in a world increasingly
driven by data and
technology,
reputations and brands are
ever harder to protect.”
EY “Trust in Data and Why it Matters”, 2017.
The importance of data
quality in the enterprise:
• Decision making
• Customer centricity
• Compliance
• Machine learning & AI
5
64%of IT executives have
trouble finding and cleaning
the right data for strategic
data projects
Sierra Venture, 2020
90%of executives are concerned
about the how misused data
can impact corporate
reputation
PWC, 22nd Annual Global CEO Survey, 2019

6. Understanding Your Data
Data Profiling
The set of analytical techniques that
evaluate actual data content (vs.
metadata) to provide a complete view
of each data element in a data source.
Provides summarized inferences, and
details of value and pattern frequencies
to quickly gain data insights.
Business Rules
The data quality or validation rules that
help ensure that data is “fit for use” in
its intended operational and decision-
making contexts.
Covers the accuracy, completeness,
consistency, relevance, timeliness and
validity of data.
6

7. Five Key Steps to effective Data Profiling
These are not new, but good to reiterate in the
context of Big Data:
1. How you want to analyze the data?
2. What should you review? (there's a lot of stuff)
3. What should you look for? (based on data “type”)
4. When should you build rules? (laser-focus; CDE’s)
5. What needs to be communicated?
7

8. 1. How do you want to analyze the data?

9. “
”
Never lead with a data set;
lead with a question.
Anthony Scriffignano, Chief Data Scientist, Dun & Bradstreet
Forbes Insights, May 31, 2017, “The Data Differentiator”

10. Universal DQ best practices:
Understand the End Goal
• How does the business intend to
use the data (i.e. what’s the use
case)?
• Empower users (“Who”) to gain
new clarity into the core problem
(“Why”)
• What will the data be used for?
• What defines the Fitness for your
Purpose?
Establish Scope
• Ask the “right questions” about the
use case and the data (not just
“what” and “how”)
• What data is relevant to the effort?
• Big Data or other, you need to set
boundaries for the work
Understand Context
• How does the business define the
data?
• What are the important
characteristics and context of the
data?
• What are the Critical Data
Elements?
• What qualities will you need to
address, or leave alone?
• “High-quality data” definition will
vary by business problem“If you don’t know what you want to
get out of the data, how can you
know what data you need – and
what insight you’re looking for?”
Wolf Ruzicka, Chairman of the Board at EastBanc Technologies,
Blog post: June 1, 2017, “Grow A Data Tree Out Of The “Big Data”
Swamp”
10

11. To Sample or not to Sample?
Sampling helps with:
• Data Integration
• Source-to-target mapping
• Data Modeling
• Discovering Correlations
When the focus is on the structure of the data
• REMEMBER: your target is a
statistically valid sample!
• ~16k records gives you 99% confidence
with a margin of error of 1% for 100B
records
• ~66k records gives you 99% confidence
with a margin of error of .5% for same
Full Volume needed with:
• Data Quality
• Data Governance
• Regulatory Compliance
• Finding Outliers and Issues
with Content
• “Needles in the haystack”
When the focus is on the quality of or risks within
the data
• Focus on critical data elements and
leverage tools that scale to data volume
11

12. Big Data at scale distributes data across many
nodes – not necessarily with other relevant data!
• Processing routines must apply same approach and logic each
time
• Implications for profiling, joining, sorting, and matching data,
whether for enrichment, verification against trusted sources, or a
consolidated single view
Data Quality functions must be performed in a consistent manner,
no matter where actual processing takes place, how the data is
segmented, and what the data volume is.
Scaling Data Quality best practices:
Consistent processing at scale
Source: HP Analyst Briefing
12

13. 2. What do you want to review?

14. Common Data Quality Measurements
What measures can we take advantage of?
1. Completeness – Are the relevant fields populated?
2. Integrity – Does the data maintain an internal structural
integrity or a relational integrity across sources
3. Uniqueness – Are keys or records unique?
4. Validity – Does the data have the correct values?
• Code and reference values
• Valid ranges
• Valid value combinations
5. Consistency – Is the data at consistent levels of
aggregation or does it have consistent valid values
over time?
6. Timeliness – Did the data arrive in a time
period that makes it useful or usable?
14

15. New data, new data quality challenges
• 3rd Party and external data with unknown provenance or relevance
• Bias in the data – whether in collection, extraction, or other processing
• Data without standardized structure or formatting
• Continuously streaming data
• Disjointed data (e.g. gaps in receipt)
• Consistency and verification of data sources
• Changes and transformation applied to data (i.e. does it really
represent the original input)
New Data Quality Problems
“34 percent of bankers in our survey report that their organization
has been the target of adversarial AI at least once, and 78 percent
believe automated systems create new risks, such as fake data,
external data manipulation, and inherent bias.”
Accenture Banking Technology Vision 2018
15

16. • Contextual visualizations
• Value and pattern distributions
• Attribute summaries and metadata
• Sort and filter to quickly find data
of interest
• Detail drilldowns to any content
Let Data Profiling guide you
16

17. 3. What should you look for?

18. Common Data Types
What variances do you need awareness of?
1. Identifiers – data that uniquely identifies something
2. Indicators – data that flags a specific condition
3. Dates – data that identifies a point in time
4. Quantities – data that identifies an amount or value of something
5. Codes – data that segments other data
6. Text – data that describes or names something
18

19. 4. When do you build rules?

20. Focus on:
• Critical Data Elements (data quality dimensions)
• Policy-based conditions (e.g. regulatory
compliance)
• Correlated data conditions (e.g. If x, then y)
• Filtering and segmenting data (refining
evaluations; investigating root cause)
Build Rules for Defined Conditions
20

21. • Validate critical requirements within or
across data sources
• Build common rules that can be readily
tested and shared
• Evaluate and remediate issues
• Take action on incorrect data and defaults
• Create flags for subsequent use in marking
or remediating data
• Filter result sets and export for additional
use
Benefits of Business Rules
21

22. 5. What should you communicate?

23. 23
Communicate!
Culture of Data Literacy
• “Democratization of Data” requires cultural support
Program of Data Governance
• Provide the processes and practices necessary for
success
Center of Excellence/Knowledge Base
• Where do you go to find answers?
• Who can help show you how?

24. • Annotate what you’ve found
Annotate Results with Findings
24

25. British Airways
Leveraging Data as a Critical Asset
About
• World’s leading international
premium airlines
• 33M passengers every year
• 35,000+ employees
• Fleet of 240 aircraft
Goal
• Ensure accurate data to support
customer service, marketing,
retention and loyalty
• Implement enterprise-wide data
governance
Challenge
• Data from multiple
sources/systems, stored in many
different formats​
• No enterprise standard for data
quality
• Point solutions led to varying
levels of cleanliness, inefficiencies25

26. British Airways
Results: Trusted data for improved analysis
Solution
• Trillium Data Quality
Benefits Achieved
• Trusted data for faster, better
strategic and operational decision
making​
• More effective marketing and
better customer service
26

27. Looking at the Next 90 Days…
• Make profiling actionable
• You don’t know what you don’t know until you profile
• Keep the 5 key questions top of mind!
• Join us tomorrow for part 3 of our webinar series!
27

28. Questions?