MLSD18. Data Cleaning

1st edition
November 4-5, 2018
Machine Learning School in Doha
1

QCRI
Data Proﬁling, Data Cleaning,
Error Detection
Getting to ML Ready Data
Saravanan Thirumuruganathan
Scientist, QCRI
2

· @bigmlcom · @QatarComputing · #MLSD18 ·QCRI
Outline
• ML Ready data
• Data Preparation Workflow
• Error Detection and Repair
• Demo of Google Facets
3

ML Ready Data
4

ML Ready Data
• is one instance
• contains all information about this instance
Each row
5

ML Ready Data
• is a field that describes a property of the instance.
Each column
6

ML Ready Data
Features Instances Values
7

ML Ready Data : Don’t Dos
• Column headers are values, not variable names.
• Multiple variables are stored in one column.
• Variables are stored in both rows and columns.
8

Column headers are values
Source: Tidy Data by Hadley Wickham
9

10

Same data as previous slide. Columns are renamed to income and value to freq
11

12

Multiple variables in 1 column
13

14

15

16

17

18

19

20

Data Preparation
21

Data Preparation
Data Preparation
where is my “ML Ready” data?
22

Data Preparation
23

Source: http://visit.crowdﬂower.com/rs/416-ZBE-142/images/CrowdFlower_DataScienceReport_2016.pdf
Bad News
24

Day of a Data Scientist
– Mark Schreiber (Merck) reports that his data scientists
spend 98% of their time, i.e. 39 hours/week, in grunt work
and only 1 hour/week doing the job for which they were
hired
– For Big-Data Scientists, ‘Janitor Work’ Is Key Hurdle to
Insights (The New York Times https://www.nytimes.com/2014/08/18/technology/for-big-data-scientists-hurdle-to-insights-is-janitor-work.html)
– Reports vary between 60% and 80% of grunt work but
nobody reports less
25

Good News
Source: https://thodrek.github.io/di-ml/sigmod2018/slides/diml_sigmod2018.pdf
26

Good News
Data Preparation
Data Civilizer
(QCRI)
27

Data Preparation Workﬂow
28

Data Preparation Workﬂow
• Data Discovery
• Data Integration
• Data Profiling
• Error Detection
• Error Repair
29

Data Discovery
Source: https://tmrresearchblog.com/data-discovery-tools-provide-one-time-query-business-processes/
30

Data Discovery
• Merck has 4000 Oracle databases + countless other repositories
• MIT Data Warehouse has 2400 tables
• GE has 75 procurement systems and supplier databases
• Your organisation has many databases and many repositories!
31

Data Discovery
• the company stock variations in recent years
• the mentions of the company in social media channels
• the number of drugs about to be approved by the FDA
• the current productivity of the research department
Goal: predicting the change in stock price of a drug company
Source: Aurum: A Data Discovery System
Need: merge tables about drugs, stock performance, social media
mentions etc
32

Data Integration
Schema Alignment and Transformation
Source: Beaver: Towards a Declarative Schema Mapping
33

Data Integration
Source: Guoliang Li: Human-in-the-loop Data Integration
34

Data Integration
Protein Data Bank
(>400 tables, >3000 columns)
Source: Beaver: Towards a Declarative Schema Mapping
35

Data Proﬁling
• Single and multi-column statistics
• Common patterns
• Data types and analysis
• Column dependencies
• Data dependencies
• Unique column combinations
36

Data Proﬁling Tools in Industry
Source: Data Proﬁling Tutorial SIGMOD 2017 Abedjan et al
37

Data Proﬁling Tools in Research
Source: Data Proﬁling Tutorial SIGMOD 2017 Abedjan et al
38

Error Detection and Repair
39

Dirty Data
Source: Dilbert : Bad Data
• Garbage In Garbage Out!
40

Dirty Data
• GE : Normalization of 75 procurement systems could save $100M
a year!
• Dirty data costs the global banking industry over $400 billion
• $600 billion a year for US businesses
•12% lost revenue for most companies due to inaccurate,
inconsistent and incomplete data
Source: (a) Data Integration: The Current Status and the Way Forward (b) https://www.marklogic.com/blog/the-staggering-impact-of-dirty-
data/ (c ) The Data Warehousing Institute (TDWI) (d) Experian Quality Data
41

Error Detection
• Duplicates / Entity Resolution
• Pattern violations
• Integrity Constraints / Business rule violations
• Outliers
42

Entity Resolution
• Find records referring to the same real-world entity
• GE : 75 procurement systems, 2M records
• Your company: Missed opportunity for holistic picture of a client
43

Pattern Violations
44

Integrity Constraints
Source: Qualitative Data Cleaning : Xu Chu and Ihab Ilyas
45

46

47

Data Repair
48

Nadeef
Source: NADEEF: A Commodity Data Cleaning System
49

Data Cleaning
• Talk to us if you have any question on Data cleaning!
50

Data Proﬁling
Demo of Google Facets : https://pair-code.github.io/facets/
51

MLSD18. Data Cleaning

More Related Content

What's hot

Similar to MLSD18. Data Cleaning

More from BigML, Inc

Recently uploaded

MLSD18. Data Cleaning