Data Quality

954 views
824 views

Published on

A view of the importance of data quality and how to set about addressing this issue in your business

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
954
On SlideShare
0
From Embeds
0
Number of Embeds
11
Actions
Shares
0
Downloads
33
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Data Quality

  1. 1. “ Making your data a strategic asset” Data Quality - The Key to Successful Analytics and CRM Michael Collins BA(Hons), DipM, MCIM, FIDM Managing Consultant - Database Marketing Counsel Visiting University Lecturer in Database Marketing & CRM D ATABASE M ARKETING C OUNSEL
  2. 2. Our Agenda <ul><li>The importance of understanding your core systems </li></ul><ul><li>How clean should your data be? </li></ul><ul><li>The impact of a data audit </li></ul><ul><li>Where does poor data come from? </li></ul><ul><li>Methodology for improvement </li></ul><ul><li>Getting employees to ‘live and breath’ data quality </li></ul>
  3. 3. Data as a corporate asset <ul><li>Value to the business </li></ul><ul><li>Invest in the maintenance of an asset </li></ul><ul><li>Responsibility of all who use it, access it, are involved in its acquisition, storage or maintenance </li></ul><ul><li>Rules of management </li></ul><ul><li>Validation </li></ul><ul><li>Security </li></ul><ul><li>An appreciating asset – in everyone’s interest </li></ul>
  4. 4. <ul><li>Four kinds of quality issues </li></ul><ul><li>Common data-entry errors </li></ul><ul><li>Out of date – past its “use-by” date </li></ul><ul><li>Lack of consistency </li></ul><ul><li>Unreliable sources </li></ul><ul><li>Poor quality and integrity of data limits its value </li></ul>
  5. 5. Implications of Inaccurate Data <ul><li>There is no substitute for acquiring accurate data - analysis tools can’t compensate for lack of data </li></ul><ul><li>The more “real time” contact we have with customers, suppliers or employees, the more accurate the data needs to be and the more devastating can be the results of inaccuracy </li></ul><ul><li>Quality will determine how much of a guide or ‘black & white’ analysis can be reached </li></ul>
  6. 6. Implications of Inaccurate Data <ul><li>Skewed campaign planning </li></ul><ul><li>Improper selections for campaigns </li></ul><ul><li>Expensive product mistakes </li></ul><ul><li>Non-delivery of the message (esp. E-mail) </li></ul><ul><li>Reflection of your business </li></ul><ul><li>‘ Junk mail/spam’ tag </li></ul><ul><li>£££££ Wasted </li></ul>
  7. 7. Implications of Inaccurate Data <ul><li>Inaccuracy will </li></ul><ul><li>annoy customers, </li></ul><ul><li>suppliers and staff </li></ul>
  8. 8. Data Quality Quality Data 1 Profile Understanding 2 Audit Qualification 3 Integrate Consolidation 4 Enrich Improvement 5 Monitor Observation 6 Culture Compliance
  9. 9. Typical Framework Source A Source B Source C Sources Extract/Transform/Load Processes Operational CRM Campaign Management External Data BI & Visualisation Rules DATA QUALITY
  10. 10. Typical Framework Source A Source B Source C Sources Extract/Transform/Load Processes Operational CRM Campaign Management External Data BI & Visualisation Rules QUALITY DATA
  11. 11. Profiling Your Sources <ul><li>Current business processes </li></ul><ul><li>Tactical activity </li></ul><ul><li>Enhancement from external sources </li></ul><ul><li>Business information vendors </li></ul><ul><li>Purchased lists </li></ul><ul><li>Marketing partners </li></ul>
  12. 12. Sources WARRANTY SURVEYS - Behavioural ENQUIRIES/HELP LINE SALES COMPLAINTS BRANCHES/CHANNELS ACCOUNTS OTHER TOUCH POINTS SMS Social Networking EXTERNAL DATABASE
  13. 13. Scoring the Sources <ul><li>Score the data as part of your data strategy </li></ul><ul><li>Build a model that provides a level of confidence </li></ul><ul><li>Base the model on known factors </li></ul><ul><ul><li>Source </li></ul></ul><ul><ul><li>Recency of update </li></ul></ul><ul><ul><li>Testing </li></ul></ul><ul><li>Use the score to determine priorities for enhancement and to inform the business of the level of confidence </li></ul><ul><li>Strive to improve the level of confidence </li></ul>
  14. 14. Review Your Scores Business Services Company
  15. 15. Compare Your Scores Business Services Company – No. of Employees
  16. 16. Typical Framework Source A Source B Source C Sources Extract/Transform/Load Processes Operational CRM Campaign Management External Data BI & Visualisation Rules HIERARCHY
  17. 17. Data Quality Process <ul><li>Data Audit -technology </li></ul><ul><li>What needs fixing </li></ul><ul><li>What needs summarizing </li></ul><ul><li>What derived data is required </li></ul><ul><li>Attrition - data use by date! </li></ul><ul><li>How do you fix and improve it </li></ul><ul><ul><li>external enhancement, internal technology </li></ul></ul><ul><li>Data business rules </li></ul><ul><li>What to do while you are fixing it! </li></ul><ul><li>Keeping it fixed – monitor and enhance! </li></ul>
  18. 18. Data Audit <ul><li>Appraise the data </li></ul><ul><ul><li>Technology for auditing the data </li></ul></ul><ul><ul><li>Do fields hold what they claim to hold? </li></ul></ul><ul><ul><li>Is it in a usable format </li></ul></ul><ul><ul><ul><li>For operations? </li></ul></ul></ul><ul><ul><ul><li>For analytics? </li></ul></ul></ul><ul><ul><li>How extensively populated are the fields? </li></ul></ul><ul><li>Ascertain the age of the data – has it passed its ‘USE BY date’? </li></ul><ul><li>What needs to be done to make this data usable/valuable </li></ul>
  19. 19. Data Audit Technology <ul><li>Tools to report on the quality of data - attention is drawn to those fields that require analysis. </li></ul><ul><li>Against each column name </li></ul><ul><li>Minimum Value and Maximum Value </li></ul><ul><li>Mean, Median, Mode </li></ul><ul><li>Minimum Length and Maximum Length </li></ul><ul><li>Mean and Mode Length </li></ul><ul><li>Defined Data Type and number/% records that conflict </li></ul><ul><li>% populated with valid characters (excluding spaces) </li></ul><ul><li>Number of unique values </li></ul>
  20. 20. Data Report – Logistics Company Example of some of the data irregularities identified – addresses in the name field, addresses and postcodes in the Town field, lower case characters, invalid postcodes etc What lies underneath?
  21. 21. Drill Down to Format Data Format No of Records Sample of Data XX## #XX 1203 AB12 3AB XX##X #XX 63 AB12A 3AB XX# #XX 2014 AB1 3AB XXXXX#XXX 1203 ABFDA1ABC Postcode Data Format No of Records Sample of Data ##### ###### 21003 01932 124689 #### ### #### 1095 0115 236 1236 ##### ###### XXXX### 2014 01892 226819 ext.354 XX XXX XXXX 54 Do Not Call Telephone Number
  22. 22. How to Fix what needs Fixing! <ul><li>Internal processes </li></ul><ul><ul><li>Data cleansing </li></ul></ul><ul><ul><li>Corrections </li></ul></ul><ul><ul><li>Use of address enhancement software </li></ul></ul><ul><ul><li>Use of touch-points </li></ul></ul><ul><ul><li>Use of people in the business who know </li></ul></ul><ul><ul><li>Source evaluation </li></ul></ul>
  23. 23. Internal processes <ul><li>Common functions of data cleansing technology </li></ul><ul><li>Find and Replace </li></ul><ul><li>Standardisation: compare values from the given column with a column in a compiled Knowledge Base e.g. list of Titles </li></ul>
  24. 24. Internal processes <ul><li>Common functions of data cleansing technology </li></ul><ul><li>Find and Replace </li></ul><ul><li>Standardisation: compare values from the given column with a column in a compiled Knowledge Base e.g. Job Titles </li></ul>
  25. 25. <ul><li>Common functions of data cleansing technology </li></ul><ul><li>Find and Replace </li></ul><ul><li>Standardisation: compare values from the given column with a column in a compiled Knowledge Base e.g. Job Titles </li></ul><ul><li>Data split: Divide data in a single field into multiple fields e.g. Mr John Smith to be divided into three fields of Title, First Name and Surname </li></ul><ul><li>De-duplication and Merge/purge </li></ul><ul><li>Case conversion </li></ul><ul><li>Address technology – correction, replacement, batch and interactive </li></ul>Internal processes
  26. 26. External Processes <ul><li>Bureau services </li></ul><ul><ul><li>Name and address enhancement </li></ul></ul><ul><ul><li>Verification, insertion, correction </li></ul></ul><ul><ul><li>Data augmentation </li></ul></ul><ul><ul><li>Telephone services – calling to correct details </li></ul></ul><ul><ul><li>Dynamics – B2C </li></ul></ul><ul><ul><ul><li>National Change of Address </li></ul></ul></ul><ul><ul><ul><li>Gone Away Suppression </li></ul></ul></ul><ul><ul><ul><li>Mortascreen Plus (Grey market) </li></ul></ul></ul><ul><ul><ul><li>Mortascreen </li></ul></ul></ul><ul><ul><ul><li>Bereavement register </li></ul></ul></ul><ul><ul><li>Dynamics – B2B </li></ul></ul><ul><ul><ul><li>Mergers & acquisitions </li></ul></ul></ul><ul><ul><ul><li>Job changes </li></ul></ul></ul><ul><ul><ul><li>Status change </li></ul></ul></ul><ul><ul><ul><li>Purchasing strategy (central/local) </li></ul></ul></ul><ul><ul><ul><li>Official name/colloquial name </li></ul></ul></ul><ul><ul><ul><li>Business demographics </li></ul></ul></ul>
  27. 27. External Data Example Company Name Postcode Business Demographics Sec tor Registration Code Advertising spend Job Function Job Title Turnover Product/Service 1. Business demographics: Enhancement /verification 2. PAF data (UK & Foreign) Address verification & formatting 3. Weather/Travel Info Exhibitions organiser 4. Advertising Monitoring Market share, expenditure comparison 5. Sector performance
  28. 28. Multi-source: The strength of “blended” data Source A Source B No. of Employees in the company
  29. 29. Data Business Rules <ul><li>Business rules manage the validation process and the ongoing protection of data quality </li></ul><ul><li>Make your rules as stringent as you can to begin, then assess the volumes of rejects and adjust accordingly </li></ul><ul><li>Quarantine offenders </li></ul><ul><li>Impose rules on internal data acquisition </li></ul><ul><li>Ensure they are included in the brief for any external data capture resources </li></ul>
  30. 30. Example of Business Rules
  31. 31. Data Strategy <ul><li>Data is volatile </li></ul><ul><li>A data strategy is required for keeping it up to date </li></ul><ul><li>Documented </li></ul><ul><li>Maintained </li></ul><ul><li>Reviewed </li></ul><ul><li>Internal & external data </li></ul>
  32. 32. Enterprise Data Maturity Model Local Global Local Collectively Local Global Global Global Undisciplined Reactive Proactive Governed Think Act Benefit High Low Risk Low High Data Governance Direct Marketing Database Marketing & Sales Force Automation <ul><li>Data Warehousing </li></ul><ul><li>Enterprise </li></ul><ul><li>Project </li></ul><ul><li>Explorer </li></ul><ul><li>Marts </li></ul>ERP <ul><li>CRM </li></ul><ul><li>Operational </li></ul><ul><li>Analytical </li></ul><ul><li>Collaborative </li></ul>Customer Data Integration Product Data Integration Master Data Management Business Process Management Business Intelligence Service Orientated Architecture
  33. 33. Remember the Real World Acquisition Retention Utilisation What is most What is most What is reliable? useful? 80/20 easily available/ done? Costs You cannot do it all overnight Any enhancement to the data must be driven by commercial benefit
  34. 34. Importance Accepted in the Business <ul><li>Lip service to business as usual </li></ul><ul><li>Incentives or penalties </li></ul><ul><li>Demonstration of the implications of poor data and/or how it makes them more effective at their job </li></ul><ul><li>Ensure they know how important it is that they comply </li></ul><ul><li>Make it easy for them to adhere to the rules </li></ul><ul><li>Listen to them and address their problems – their view of poor quality data may be different to yours </li></ul><ul><li>Be prepared to change </li></ul><ul><ul><li>Software amendments </li></ul></ul><ul><ul><li>Business processes </li></ul></ul><ul><ul><li>Forum or reporting channel for data issues </li></ul></ul>
  35. 35. Touch Points <ul><li>Encountering customers as part of regular business processes - the touch points </li></ul><ul><li>Opportunities for </li></ul><ul><ul><li>Acquiring new data </li></ul></ul><ul><ul><li>Qualifying existing contacts </li></ul></ul><ul><ul><li>Verifying or updating existing data </li></ul></ul><ul><ul><li>Testing the relationship (Jenkinson 1995) </li></ul></ul><ul><li>Consider all of these opportunities within your business processes </li></ul>
  36. 36. And finally…. <ul><li>Know your data </li></ul><ul><li>Document your data strategy </li></ul><ul><li>Score the data based on level of confidence </li></ul><ul><li>Determine internal & external solutions </li></ul><ul><li>Create rules to apply at all data collection points – don’t forget your external data capture bureaux, partners and sales channels </li></ul><ul><li>Regular review in the light of on-going data quality </li></ul><ul><li>Learn from your experience </li></ul><ul><li>Be prepared for change to processes and software </li></ul><ul><li>Achieve quality and maintain quality – get it right, keep it right! </li></ul>
  37. 37. Thank you [email_address]

×