We have seen vast improvements to data collection, storage, processing and transport in recent years. An increasing number of networked devices are emitting data and all of us are preparing to handle this wave of valuable data. ​Have we, as data professionals, been too focused on the technical challenges and analytical results? What about the data quality? Are we confident about it? How can we be sure we are making good decisions? We need to revisit methods of assessing data quality on our modernized data platforms. The quality of our decision making depends on it.

1. DOES MORE DATA MEAN BETTER DECISION MAKING?
(Assessing Data Quality with a Unified-Log and a bit of Stream-
Processing)
Scott Krueger
Data Architect

2. ATALE OF DATA-DRIVEN DECISION MAKING - ACT ONE
Me: "@StreamEngine - how many user sessions have we had from
Illinois in the last 4 hours from those #ChicagoRocks tweets?"
StreamEngine:" In the last 4 hours, we have had 43, 578 new user sessions
from Ohio as a result of the #ChicagoRocks tweets"
Me: "How confident are you about that?" (I need to make a
quick call here)
StreamEngine: "Sorry, what was that?"

3. WHY ARE WETALKING ABOUTTHIS
Analytical Sciences
Data Quality
Business and Culture
Tech
Talks at Big Data Week, London 2016

4. 2020

5. MOTIVATIONS

6. MOTIVATIONS
http://lemonly.com/work/the-cost-
of-bad-data/

7. MOTIVATIONS
"From now on, our cars will more deeply understand that
buses (and other large vehicles) are less likely to yield to
us than other types of vehicles, and we hope to handle
situations like this more gracefully in the future."
http://www.bbc.co.uk/news/technology-35692845

8. SO WHAT'S GOING ON HERE? WE ARE CREATORS OF POOR QUALITY DATA
Machines (we
make machines
that consume /
create data)
Software (we
create software
that consumes /
creates data sets)
photo: Faruk Ates, https://flic.kr/p/stxXK

9. COMMON DATA PROBLEMS
https://github.com/Quartz/bad-data-guide

10. Pillar 1: Data Integrity
Data completeness - is it there? is it in tact? are the ‘required-to-be-of-value’ fields
there
Data Interpretation - what is that thing? what does ‘cost’ mean?
Data change - we don't use this anymore so I’m not writing it anymore. Oh, you’re still
reading it?
Pillar 2: Data validity - what's in there?
Values make sense?
Values expected?
Data presence* - are the messages making it? Are there as many as there should be
when data was created? Is this an expected volume?
THE 2 (OR SO) PILLARS OF DATA QUALITY
http://www.newyorkinternationallimousines.com/

11. DATA INTEGRITY
A definition of what the data is so it can be turned into a meaningful
piece or set of information
Varies with approach and ‘structure’ of data
Event Schemas (not to be confused with the relational DB term)
Examples: protocol buffers, thrift, avro

12. DATA INTEGRITY:THE GREATTRADE-OFF
Somewhere between left and right something has to prepare data for
usage elsewhere.There is cost associated with every position.
Data-In Data-Out
Schema-on-Write Schema-on-Read
Schema-Inbetween

13. ATALE OF DATA-DRIVEN DECISION MAKING - ACTTWO
Me: "@StreamEngine - I would like to measure how effective all of our
data-driven decision making is. I need a measure of quality. I think you
can help me with this."
StreamEngine: "Are you from the future?"
Me: "I'm not from it, but I'm thinking about it..."

14. WHAT ARE WE DOING ABOUT IT?
This requires a brief understanding of our 'unified log' approach at
skyscanner

15. SKYSCANNER EVENT DATA PLATFORM

16. EXPLOITTHE PLATFORM - MAKE USE OF WHAT
YOU HAVE

17. DATA INTEGRITY (SCHEMAVALIDATION)
Data definition - a message that doesn't fit throws an exception
Try...Catch...Log To SchemaValidation Failure “stream/topic” with message

18. SchemaValidation

19. DATAVALIDATION: EVERYONE PLAYS A PART
Everywhere between left and right everything has a data validation
opportunity
Data-In Data-Out
Validation-on-Write Validation-on-Read
Validation-Inbetween

20. DataValidation ReferenceYAML configuration

21. DataValidation Flow

22. WHAT DOES IT ALL LOOK LIKE?
alert!
alert!

23. TO IMPROVE QUALITY….
… ISTO CLOSETHE LOOP

24. EVERYONE PLAYS A PART
A shared repository for event structure and validation rules
Any service that logs events runs automated tests that use this repository
A generic stream service that assesses data quality and gives the heads up to consumers
Data consumers who find new data quality mishaps commit back to the repository

25. IFYOU CAN'T MEASURE IT HOW DOYOU KNOW
YOU ARE IMPROVINGTHINGS?
Quality of decision making = 100 -
(((# of data issues detected + recent commits for improved detection)
/ # high quality events logged)))*100)
example: 99.8 %

26. TIPS FOR IMPROVEMENT
Master Data Management
Metadata Management
Handling Data Change
Culture

27. MASTER DATA MANAGEMENT
(“ONE SOURCE OF REFERENCE/LOOKUP DATA”)
Simple rules to maintain consistency of reference data
use of enums/constants in schemas for reference data sets you
don't provide in your systems
authoritative data sources (your internal data services; industry
standard sets e.g. "IATA" for travel, ISO geography/timezones etc.)
Bring this ref data as close to the processing as possible
API, csv / json, tables, trans logs -> Unified Log Topic

28. METADATA MANAGEMENT
(DATA PROVENANCE AND OTHER NICETIES)
Data Debugging
Data flow measurements - how long did it take for my message to go through this pipeline?
Historical records - transparency for everyone (you the business operator, and you the customer)
Governance and regulation - data quality laws? http://www.forbes.com/sites/forbestechcouncil/
2016/04/29/how-companies-can-leverage-real-time-platforms-and-metadata-to-improve-
healthcare-delivery/2/#53fbea89480b

29. + float device_diagonal_screen_size = 19;
+ float device_diagonal_screen_size = 19 [deprecated=true];
+ DisplayMeasurement diagonal_screen_size = 26;
DATA CHANGE:WE DON'T ALWAYS GET IT RIGHT
0
5
Time
Event Definition Changes
T1
T2
T3
+ // float device_diagonal_screen_size = 19 [deprecated=true];
+ DisplayMeasurement diagonal_screen_size = 26;
2 rules:
1. Maintain
backwards
compatibility.
2. Rebuild.

30. DATA QUALITY CULTURE
photo: scott krueger

31. WHAT ARE YOU GOINGTO DO ABOUT IT?
Understand the causes and details of data problems in your services
Unify and simplify - one source of truth for everything: reference data,
reports, archive, formulae, validation rules, data definitions, metdadata
Start measuring - this is your baseline and allows you to measure
confidence in decision making
Work (or evolve) your tech
Fix the system, stop moaning about it
It’s never too late

32. ATALE OF DATA-DRIVEN DECISION MAKING - ACTTHREE -THE FINALE
Me: "@StreamEngine - give me some decision quality numbers!"
StreamEngine: "Right now, decision quality is at 96.05%. This
time last week it was 92.4%. Well done! In 1 week sales are up 2
% and this is positively correlated to the decisions you made
with this data. Would you like me to predict sales uplift over the
next month if you improve decision quality by 1 %?
Me: "You bet I would..."

33. THANKS FOR LISTENING