1. It's Time to Start Caring About
Data Quality
Data Quality at Scale

2. Ignacio Elola

3. Everyone is talking about how
useful data is

4. data can save your business

5. data can save your life

7. but...

8. all that is only true if you have the
right data

9. data tend to be dirty and
unstructured

10. specially web data!

12. Let’s start simple

13. I’ve created an extractor

15. I’ve pass a bunch of queries (bulk)

18. and got a dataset

20. How can you QA this data?

21. eyeballing

24. eyeballing we can find anomalies
without having domain expertise

25. Quick summary:
- create extractors
- combine extractors
- schedule data extraction

26. What if we need to scale up?

27. if you have:
- more than ~3 datasources
- more than ~2 extractors per ds
- big volume of queries
- pre or post processing

28. you will need:
- people to create and maintain
extractors
- process to clean and validate
data

29. Data Quality
think about it pre and post data
extraction!

30. tips and tricks to increase data
quality

31. XPaths

34. //div[@id="priceBlock"]/table/tbody/tr/td
[b/@class="priceLarge"]/b
better than
//*[@id="priceBlock"]/table/tbody/tr[2]/td[2]/b[1]

35. Regex

38. More at:
http://support.import.
io/knowledgebase/articles/341182-xpaths-regex
http://www.w3schools.com/xsl/xpath_intro.asp

39. Required column

44. measuring data quality

46. completeness

47. coverage

49. post extraction data quality
improvements?

51. how we do it

54. Smart automation

55. anomaly detection

56. variance, variability, noise

57. normalization

58. confidence score

59. Human input

60. Transparency

61. summary

62. Data Quality is essential

63. think about it from the very
beginning

64. develop a process to measure
data quality before scaling up

65. if you don’t want to reinvent the
wheel - contact us!

66. Thank you
ignacio.elola@import.io