The task of “data profiling”—assessing the overall content and quality of a data set—is a core aspect of the analytic experience. Traditionally, profiling was a fairly cut-and-dried task: load the raw numbers into a stat package, run some basic descriptive statistics, and report the output in a summary file or perhaps a simple data visualization. However, data volumes can be so large today that traditional tools and methods for computing descriptive statistics become intractable; even with scalable infrastructure like Hadoop, aggressive optimization and statistical approximation techniques must be used. In this talk Sean will cover technical challenges in keeping data profiling agile in the Big Data era. He will discuss both research results and real-world best practices used by analysts in the field, including methods for sampling, summarizing and sketching data, and the pros and cons of using these various approaches. Sean is Trifacta’s Chief Technical Officer. He completed his Ph.D. at Stanford University, where his research focused on user interfaces for database systems. At Stanford, Sean led development of new tools for data transformation and discovery, such as Data Wrangler. He previously worked as a data analyst at Citadel Investment Group.

1. Agile Data Profiling
Sean Kandel

2. What’s in your data?

3. Opening Questions
in the Data Lifecycle…
Unboxing What’s in this data?
Can I make use of it?

4. … Become Persistent Questions
in the Data Lifecycle
What’s in this data?
Can I make use of it?
Unboxing Transformation Analysis Visualization Productization

5. Unboxing Transformation Analysis Visualization Productization

6. Unboxing Transformation Analysis Visualization Productization
STRUCTURING CLEANING
ENRICHMENT DISTILLATION

7. “Its easy to just think you know what you
are doing and not look at data at every
intermediary step.
An analysis has 30 different steps. Its
tempting to just do this then that and then
this. You have no idea in which ways you
are wrong and what data is wrong.”

8. What’s in the data?
• The Expected: Models, Densities, Constraints
• The Unexpected: Residuals, Outlier, Anomalies

9. Average Movie Ratings

10. Expected

11. Unexpected

15. Overview of all variables

16. Show relevant perspectives

17. What to compute?
• Densities and descriptive statistics
• Identify anomalies and outliers

18. How often to compute it?
Unboxing Transformation Analysis Visualization Productization

19. Challenge: Agility
• Profiling throughout the lifecycle
• Particularly important as you manipulate data

20. Design Space and Tradeoffs

21. Mapping out the Design Space
How much data to examine?
How accurate are the results?
How fast can you get them?

22. Mapping out the Design Space
Decide how your requirements fall on these axes
Find a strategy (if one exists) that fits the requirements
Accuracy
Urgency
Data Volume

23. Accuracy
Urgency
Data Volume
Strategy vs Cost
Head of file
Good EnoughAnomaliesBig PictureUnbox

24. Strategy vs Cost
Random Sample
Accuracy
Urgency
Data Volume
Good EnoughAnomaliesBig PictureUnbox

25. Strategy vs Cost
Scan, summarize, collect samples
Accuracy
Urgency
Data Volume
Good EnoughAnomaliesBig PictureUnbox

26. Far better an approximate answer
to the right question, which is often
vague, than the exact answer to
the wrong question, which can
always be made precise.
Data Analysis & Statistics, Tukey & Wilk 1966

27. Technical Methods

28. Sanity Check: Is this really expensive?
• Computers are fast
• In-memory, column stores, OLAP, …
• Still, “Big Data” can be hard
• Big is sometimes really big
• Big data can be raw: no indexes or precomputed summaries
• Agility remains critical to harness the “informed human mind”

29. Two Useful Techniques
Sampling
• A variety of techniques available
Sketches
• One-pass memory-efficient structures for capturing distributions
Accuracy
Urgency
Data Volume

30. Technique I: Sampling

31. Approaches to Sampling
• Scan-based access
• Head-of-file
• Bernoulli
• Reservoir
• Random I/O Sampling
• Block-level sampling

32. Head-of-File
• Pros:
• Very fast: small data, no disk seeks
• Absolutely required when unboxing raw data
• Nested data (JSON/XML), Text (logs, database dumps, etc.)
• Cons:
• Correlation of position and value

33. Bernoulli
• Take a full pass, flip a (weighted) coin for each record
• Pros:
• trivial to implement
• trivial to parallelize
• almost no memory required
• Cons:
• requires a full scan of the data
• output size proportional to input size, and random
filter(lambda x : random() < 0.01, data)

34. Reservoir
• Fix “reservoir”. For each item, with probability eject old for new
• Pros:
• trivial to implement
• easy to parallelize
• constant memory required
• fixed-size output — need not know input size in advance
• Cons:
• Requires a full scan of the data
… 61141217 139
res = data [0:k] //initialize: first k items
counter = k
for x in data [k:]:
if random () < k/float(counter+1):
res[randint(0,len(res)-1)] = x
counter += 1

35. 1141217
Reservoir … 6 133
• Fix “reservoir”. For each item, with probability eject old for new
• Pros:
• trivial to implement
• easy to parallelize
• constant memory required
• fixed-size output — need not know input size in advance
• Cons:
• Requires a full scan of the data
res = data [0:k] //initialize: first k items
counter = k
for x in data [k:]:
if random () < k/float(counter+1):
res[randint(0,len(res)-1)] = x
counter += 1

36. 41217
Reservoir … 6 137 3
• Fix “reservoir”. For each item, with probability eject old for new
• Pros:
• trivial to implement
• easy to parallelize
• constant memory required
• fixed-size output — need not know input size in advance
• Cons:
• Requires a full scan of the data
res = data [0:k] //initialize: first k items
counter = k
for x in data [k:]:
if random () < k/float(counter+1):
res[randint(0,len(res)-1)] = x
counter += 1

37. Meta-Strategy: Stratified Sampling
• Sometimes you need representative samples from each “group”
• Coverage: e.g., displaying examples for every state in a map
• Robustness: e.g., consider average income
• if you miss the rare top tax bracket, estimate is way off

38. Stratification: the GroupBy / Agg pattern
• Given:
• A group-partitioning key for stratification
• Sizes for each stratum
• Easy to implement: partition, and construct sample per partition
• your favorite sampling technique applies
SELECT D.group_key, reservoir(D.value)
FROM data D
GROUP BY D.group_key;

39. Record Sampling
• Randomly sample records?
• r the % items sampled; p #rows/block
• 20x random I/O penalty => read fewer than 5% of blocks!

40. Record Sampling
• Randomly sample records?
• r the % items sampled; p #rows/block
• 20x random I/O penalty => read fewer than 5% of blocks!
• Pretty inefficient: touches 1-(1-r)p blocks

41. Record Sampling
% items sampled
%blockstouched(expected)
1-(1-r)p with p = 100

42. Block Sampling
• Randomly sample blocks of records from disk
• Concern: clustering bias.
• Techniques from database literature: assess bias and correct
• Beware: even block sampling needs to be well below 5%.

43. Sampling in Hadoop
• Larger unit of access: HDFS blocks (128MB vs. 64KB)
• HDFS buffering makes forward seeking within block cheaper
• But CPU costs may encourage sampling within the block.
• …and Hadoop makes it easy to sample across nodes
• Each worker only processes one block
• Must find record boundaries
• Tougher when dealing with quote escaping

44. Technique II: Sketching

45. Sketching
• Family of algorithms for estimating contents of a data stream
• Constant-sized memory footprint
• Computed in 1 pass over the data
• Classic Examples
• Bloom filter: existence testing
• HyperLogLog Sketches (FM): distinct values
• CountMin (CM): a surprisingly versatile sketch for frequencies

46. CountMin Sketch: Initialization
0
dhashfunctions
w hash buckets
Count-Min Sketch
0 0 0 0
0 0 0 0 0
0 0 0 0 0

47. CountMin Sketch: Insertion
dhashfunctions
w hash buckets
Count-Min Sketch
Insert(7)
h1
h2
hw

48. CountMin Sketch: Insertion
dhashfunctions
w hash buckets
Count-Min Sketch
1
h1(7)
h2(7)
hw(7)
1
1

49. CountMin Sketch: Insertion
dhashfunctions
w hash buckets
Count-Min Sketch
Insert(4)
h1
h2
hw

50. CountMin Sketch: Insertion
dhashfunctions
w hash buckets
Count-Min Sketch
1
h1(4)
h2(4)
hw(4)
2
1

51. CountMin Sketch: Insertion
dhashfunctions
w hash buckets
Count-Min Sketch
1
h1(4)
h2(4)
hw(4)
2
1

52. CountMin Sketch: Query
dhashfunctions
w hash buckets
Count-Min Sketch
Count(7)?
h1
h2
hw

53. CountMin Sketch: Query
dhashfunctions
w hash buckets
Count-Min Sketch
1
h1(7)
h2(7)
hw(7)
2
1
Count(7)?

54. CountMin Sketch: Query
dhashfunctions
w hash buckets
Count-Min Sketch
1
h1(7)
h2(7)
hw(7)
2
1
min
Count(7)

55. CountMin Sketch: Theorem & Tuning
— Cormode/Muthukrishnan, J Algorithm 55(1) (2005).
dhashfunctions
w hash buckets
Count-Min Sketch

56. CountMin Sketch: Theorem & Tuning
— Cormode/Muthukrishnan, J Algorithm 55(1) (2005).
dhashfunctions
w hash buckets
Count-Min Sketch
an over-estimate

57. CountMin Sketch: Theorem & Tuning
— Cormode/Muthukrishnan, J Algorithm 55(1) (2005).
dhashfunctions
w hash buckets
Count-Min Sketch
w controls expected error amount
d controls probability of error
Suppose we want:
0.1% error, 99.9% probability.
w = 2000
d = 10

58. CountMeanMin Sketch
dhashfunctions
w hash buckets
Count-Mean-Min Sketch Idea: subtract out expected
overage.
i.e. mean of other cells

59. CountMeanMin Sketch
dhashfunctions
w hash buckets
Count-Mean-Min Sketch mean
—

60. CountMeanMin Sketch
dhashfunctions
w hash buckets
Count-Mean-Min Sketch mean
—
mean
—
median

61. CountMeanMin Sketch
dhashfunctions
w hash buckets
Count-Mean-Min Sketch mean
—
mean
—
mean
—
median
Count(7)

62. CountMin (and CountMeanMin) answer “point frequency queries”.
Surprisingly, we can use them to answer many more questions
• densities
• even order statistics (median, quantiles, etc.)
The Versatile CountMin Sketch

63. More Statistics
• Count-Range Queries
• Median
• Quantiles
• Histograms

64. 0001020304050607080910111213141516171819202122232425262728293031
Count(x=13)
CountMin: Point Queries

65. 0001020304050607080910111213141516171819202122232425262728293031
Count(x ∊ [14-15])
CountMin(⌊x/2⌋): Pair Queries

66. 0001020304050607080910111213141516171819202122232425262728293031
Count(x ∊ [16-19])
CountMin(⌊x/4⌋): Quartet Queries

67. 0001020304050607080910111213141516171819202122232425262728293031
Maintain all of these, and answer arbitrary range queries.
Count(x ∊ [13-24])
Dyadic CountMin: log2 CountMins
x
x/2
x/4
x/8
x/16

68. 0001020304050607080910111213141516171819202122232425262728293031
Maintain all of these, and answer arbitrary range queries.
Count(x ∊ [13-24])
Dyadic CountMin: log2 CountMins
x
x/2
x/4
x/8
x/16

69. More Statistics
• Count-Range Queries
• Median
• Quantiles
• Histograms

70. 0001020304050607080910111213141516171819202122232425262728293031
Median
Via binary search.
(Suppose we have N elements, and the real median is 14)

71. 0001020304050607080910111213141516171819202122232425262728293031
Median
Via binary search.
(Suppose we have N elements, and the real median is 14)

72. 0001020304050607080910111213141516171819202122232425262728293031
Median
Via binary search.
(Suppose we have N elements, and the real median is 14)

73. 0001020304050607080910111213141516171819202122232425262728293031
Median
Via binary search.
(Suppose we have N elements, and the real median is 14)

74. 0001020304050607080910111213141516171819202122232425262728293031
Median
Via binary search.
(Suppose we have N elements, and the real median is 14)

75. More Statistics
• Count-Range Queries
• Median
• Quantiles: generalization of Median
• Histograms
0001020304050607080910111213141516171819202122232425262728293031

76. More Statistics
• Count-Range Queries
• Median
• Quantiles
• Histograms:
• fixed-width bins: range queries
• fixed-height bins: quantiles
1-10 11-20 21-30 31-40

77. Putting It Together

78. Wrangling Revisited
Good EnoughAnomaliesBig PictureUnbox

79. Wrangling Revisited
Good EnoughAnomaliesBig PictureUnbox
Head-of-file

80. Wrangling Revisited
Good EnoughAnomaliesBig PictureUnbox
Head-of-file
Bernoulli
Block
Reservoir

81. Wrangling Revisited
Good EnoughAnomaliesBig PictureUnbox
Head-of-file
Bernoulli
Sketching
Stratified
Block
Reservoir

82. Summary
• ABP: Always Be Profiling
• Tradeoff latency and accuracy
• Approximation methods
• Heuristics and reasonable assumptions

83. Acknowledgments
Adam Silberstein, Joe Hellerstein