Cleanliness is next to Godliness

Cleanliness is next to Godliness Deduplicating Your Customer Data

Parts of the talk Talking about Data Quality Techniques for Deduplication Processing, Timing and Mindset Timeline Part 1 Part 2 Part 3

Once upon a time... Age of information

Large amounts of data inputted by humans

Humans make mistakes...

Information is a significant raw material for businesses around the world.

Making data-based decisions Wrong information leads to wrong decisions Information as products Bad and unimpressive products Information for logistics Company may shut down

Gathering Data from Humans Paper forms • Spelling mistakes • Unclear questions • Bare minimum information • OCR Web forms • Bypassing filters

Tainting Existing Data Changes in procedures • Didn’t update older data • Different data structures • Different ways of handling data Importing sources of (bad) data

Some Industry Jargon Single View of Customer • Marketing Campaigns Single Version of the Truth • Strategy Getting Correct Reports

Consider this You start a direct mail marketing campaign And this happens...

Dear Mr ----- O’Brien, We are delighted to inform you that we have an amazing offer specifically for you..

Avoiding Embarrassing Mistakes • Marketing/PR • Accounting • Shipping • Strategy

How much is it worth? • 30% ROI (big consultancy) • 10-25% Loss of revenue for bad data quality • Competitive advantage • Avoid going out of business

MFI Group Founded 1964 Upgraded ERP systems early 2000’s Due to issues with data quality in 2004 • £46m in lost sales, £16m extra deliveries + technical costs and £20m for the actual system. Administration 2008 (Comeback 2010)

Recap Data Quality is a big subject Avoid embarrassing mistakes Keep company running efficiently Good for reports

What Deduplication is used for Increasing data quality Compressing data Pre-stage data cleansing needed

Matching Techniques • Address • Name • Fuzzy • DOB Business Rules Quality Matching Ask the Data

Address Matching Databases • Royal Mail (PAF) • Council Address Data • Do Your Own Fill in missing parts House Number, Building Number, House Name, Flat Number, Company Name, Street, Locality, Town, City, County, Country and Postcode

Name Matching Name, Full name Forename, Firstname, Lastname, Surname Initial Middle name(s) Title, Suffix Qualification Lord James Jonah William Smith 3rd

SQL example SELECT c1.*, c2.* FROM customers c1 INNER JOIN customers c2 ON c1.address_id = c2.address_id WHERE c1.surname = c2.surname AND c1.forename = c2.forename AND (c1.middlename = c2.middlename XOR (c1.middlename = ‘’ XOR c2.middle=name‘’));

Title Forename Middle Surname DOB MR MARK MADANES 05/10/1963 MR MARK MADANES 04/10/1963

Title Forename Middle Surname DOB MR CIARAN GERARD O’NEILL 26/07/1971 MR CIARAN M O’NEILL 26/07/1971

Title Forename Middle Surname DOB MS JAN PHILMORE 15/10/1954 MR JAN PHILMORE 00/00/0000

Title Forename Middle Surname DOB MR ALBERTO CARLOS 00/00/0000 MR ALBERT O CARLOS 00/00/0000

Fuzzy Matching Levenshtein select levenshtein(‘jonathan’,’jonathon’) -> 1 Download from: http://www.artfulsoftware.com/infotree/queries.php?&bw=1280#552

Fuzzy Matching Soundex select soundex('jonathan') -> J535 Metaphone echo metaphone('jonathan') -> JNON

Title Forename Middle Surname DOB SAMUEL JOHNSTONE 00/00/0000 MR SAMUEL JOHNSTON 00/00/0000

Business Rules Certain Level of Correctness Generic Rules and Source Specific Rules

Business Rules Example • Middle name: Adam Smith vs. Adam E Smith • Title: Miss vs. Ms vs. Lady • Initial: A Smith vs. Adam Smith (same address) • Surnames: O`Brien vs. O’Brien vs. O’Brien • More Surname: McDonald vs. Mc Donald vs. Mac Donald

Things to Watch Out for Same father/son or mother/daughter names Twins with same DOB Initial for a forename Mixing of forename with middle name Changing surname after marriage

Quality Matching Analyze data sources How recent the data is

Ask the Data Name popularity Number of sources • Example: 4 sources vs. 1 source say this spelling is right

Consider Using a Democratic System Opposite of hieratical (if-then-else) system If rules order is problematic Business Rules + Asking the Data

Recap Find address Find duplicates Try to make a decision for deduplication • Business Rules • Ask the Data

Processing CPU/Disk/Memory bound Sequential or parallel

Processing Data Extra data Result table Temp data

Timing On insert A few minutes after insert (events) Scheduled tasks Pre-fetch When user asks for it Points in Time New Data User Request

Using Your Team DBAs Database Developers/ETL experts Data Analysts Developers Testers

Benefits Mindset Never 100% Best Effort Cost Pareto Principle Continuous Improvement

Final Recap Continuous Improvements Which duplicate is the correct one? Combine business rules + ask the data

Questions & Answers

Contact Information: MySQL-related questions about presentation? Non-profit or Medical? contact@jonathanlevin.co.uk

Cleanliness is next to Godliness

by JonathanL on Nov 23, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

2 Embeds 72

Statistics

Cleanliness is next to Godliness — Presentation Transcript

http://www.jonathanlevin.co.uk	53
http://feeds.feedburner.com	19