Data Cleansing introduction (for BigClean Prague 2011)

Data Cleansing What about quality?Stefan Urbanekstefan.urbanek@gmail.com@Stiivi March 2011

Content■ Introduction■ What is data quality?■ E and T from ETL■ Summary

http://vestnik.transparency.sk

Brewery analytical data streams & Cubesonline analytical processing

Brewery analytical data streams & Cubesonline analytical processing github/bitbucket: Stiivi

Quality

What is data quality ?

Dimensions■ completeness – data provided■ accuracy – reﬂecting real world■ credibility – regarded as true■ timeliness – up-to-date■ consistency – matching facts across datasets■ integrity – valid references between datasets

completeness

all none better 0% 25% 50% 75% 100% 20 05 -3 20 05 -5 20 05 -7 20 05 -9 20 05 -1 1 20 06 -1 20 06 -3 20 06 -5 20 06 -7 20 06 -9 20 06 -1 1 20 07 -1 20 07 -3 20 07 -5 20 07 -7 20 07 -9 20 07 -1 1 20 08 -1 20 08 -3 20 08 -5 20 08 -7 20 08 -9 20 08 -1 1 20 09 -1 successfully processed? 20 20 09 09 -3 -5 20 09 -7 20 09 -9 20how many % of the ﬁeld is ﬁlled and 09 -1 1 20 10 -1 20 10 -3 20 10 -5 20 10 -7 20 10 -9 Quality measure completeness: 55%

type 1 type 2 +

all none better 0% 25% 50% 75% 100% 20 05 -3 20 05 -5 20 05 -7 20 05 -9 20 05 -1 0 20 05 -1 2 20 06 -3 20 06 -5 20 06 -7 20 06 -9 20 06 -1 1 20 07 -1 20 07 -3 20 07 -5 20 07 -7 20 07 -9 20 07 -1 0 20 07 -1 2 20 08 -3 20 08 -5 20 08 -7 20 08 -9 20 08 -1 1 20 09 -1 successfully processed? 20 20 09 09 -3 -5 20 09 -7 20 09 -9 20 09how many % of the ﬁeld is ﬁlled and -1 1 20 10 -1 20 10 -3 20 10 -5 20 10 -7 20 10 -9 Quality measure completeness: 88%

reconstruction: 5€ temperature: 32˚C accuracy

timeliness

Auto-measurable■ completeness – easily■ accuracy – somehow■ credibility – not-so■ timeliness – easily■ consistency – yes■ integrity – yes

What does that mean:“high quality data?” ?

appropriate for given purpose

attach quality report

Quality Measurement for accuracy and transparency

■ why to measure?■ when to measure?■ where to measure?

from staging to analytical datafrom source to staging data analytical modelsince 2009 description Download Parse Load source Cleanse Create cube staging clean data raw sources HTML files YAML files contracts table (staging)from source to staging data2005-2008 REGIS (SK "unknown" fact table organisations) suppliers map dimension tables Load source Download Parse 08 08 YAML files raw sources 2005-2008 search index Pre-process Create search index One HTML per Large HTML files Procurement dimension tables search index (one per year) Document dimension index keep intermediate results for auditability

from staging to analytical datafrom source to staging data analytical modelsince 2009 description Download Parse Load source Cleanse Create cube staging clean data raw sources HTML files YAML files contracts table (staging)from source to staging data2005-2008 REGIS (SK "unknown" fact table organisations) suppliers map dimension tables Load source Download Parse 08 08 YAML files raw sources 2005-2008 search index Pre-process Create search index One HTML per Large HTML files Procurement dimension tables search index (one per year) Document dimension index insert probes at appropriate places

like unit testing:1. write probes2. set data quality indicators3. pass data through

SQL PostgreSQL yml database tableYAML directory coalesce values {x:.2%} + 15.00% data audit threshold formatted printer

field nulls status distinct------------------------------------------------------------file 0.00% ok 100source_code 0.00% ok 6year 0.00% ok 6donor_code 0.00% ok 2receiver_name 1.25% fail 10363receiver_address 13.29% fail 9979receiver_ico 13.53% fail 5813project 0.01% ok 28370program 0.00% ok 29subprogram 11.60% fail 177project_budget 14.48% fail 9487requested_amount 88.73% fail 1356received_amount 9.32% fail 2179contract_number 13.29% fail 28627contract_date 57.88% fail 1425source_comment 99.93% fail 9source_id 89.52% fail 814

E and T from ETL E as Extraction

HTML Documents

Ceci ne sont pas des données

html body div id=#page div id=#page div id=#container div id=#main div id=#innerMain div (anonymous) div (anonymous) table tbody tr td tabletbody tr td table trtd tbody tabletd value √tr

Now: you parse! 3 seconds *non-technical explanation follows

More information

odkaz na projekt...

odkaz na projekt...here is a subtitleand it should be in upper-case:oAnd here is another subtitle:dkaz na (non-breaking space) projekt much better here is a label: Odkaz na projekt

“Structured”spreadsheets error prone more work needed

✓ structured ﬁle format

1 2 43 5 (1) image & title (2) repeating groups of columns (3) padding rows/columns (4) removed redundancy for readability (5) colored cells

1 2 3(1) header with row padding(2) multi-row logical cell(3) broken pattern

1 2(1) multi-row cell(2) more values in a row

why?source id itemid ﬁle format parser data extraction class id item amount class item amount class why not? amount “structured” ﬁle raw data

E and T from ETL T as Transformation

Basic pattern slightly more technical

source lists and maps ? + target ? diff ? target

SELECT ...EXCEPTSELECT ... *in PostgreSQL, not in MySQL

sta_vvo_vysledkysta_regis - - map_suppliers 1 unknown suppliers ? Slovensko + 2 + tmp_coalesced_suppliers_sk - sta_suppliers + 3 new suppliers

Script or manual? script

Script or manual?script■ recurrent processing (weekly, monthly,...)■ huge amount of data■ one-time processing■ small amount of data

appropriate tool for given task

balance

from staging to analytical datafrom source to staging data analytical modelsince 2009 description Download Parse Load source Cleanse Create cube staging clean data raw sources HTML files YAML files contracts table (staging)from source to staging data2005-2008 REGIS (SK "unknown" fact table organisations) suppliers map dimension tables Load source Download Parse 08 08 YAML files raw sources 2005-2008 search index Pre-process Create search index One HTML per Large HTML files Procurement dimension tables search index (one per year) Document dimension index

Brewery data streams

Data Sources Data Targets CSV ﬁle relational database data stream processing Google Spreadsheet report X remote Excel Spreadsheet URL processing streams

data row data row data rowdata source data target value value value value id id id item item item class class class amount amount amountdata source data target data record data record data record id value item value class value amount value

Sources X SQL CSV ﬁle XLS ﬁle SQL query mongo DB ymlGoogle spreadsheet YAML directory row list record list

Targets yml SQL CSV ﬁle SQL table mongo DB YAML directory {x:.2%} <html> 15.00%HTML table formatted printer row list record list

Record Operations+ !append distinct aggregate merge (join) !x ? ? nsample select set select data audit numerical statistics*

Field Operations A→B re + +ﬁeld map text substitute value threshold* derive* abc +string strip consolidate value histogram/bin* set to ﬂag* to type

+ SQL ? <html>SQL

yml nodes = { "source": CSVSourceNode(...), "clean": CoalesceValueToTypeNode(), "output": DatabaseTableTargetNode(...), "audit": AuditNode(...), "threshold": ValueThresholdNode(), "print": FormattedPrinterNode() } connections = [ ("source", "clean"), ("clean", "output"),SQL ("clean", "audit"), ("audit", "threshold"), ("threshold", "print") ] + ... # configure nodes here stream = Stream(nodes, connections) stream.initialize() {x:.2%} stream.run() 15.00%

Data Cleansing introduction (for BigClean Prague 2011)

by Stefan Urbanek

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 4

Statistics

Data Cleansing introduction (for BigClean Prague 2011) Presentation Transcript

http://coderwall.com	3
http://paper.li	1