# Talk given at PyCon UK 2017 The first step in any data-intensive project is understanding the available data. To this end, data scientists spend a significant part of their time carrying out data quality assessments and data exploration. In spite of this being a crucial step, it usually requires repeating a series of menial tasks before the data scientist gains an understanding ofthe dataset and can progress to the next steps in the project. In this talk I will detail the inner workings of a Python package that we have built which automates this drudge work, enables efficient data exploration, and kickstarts data science projects. A summary is generated for each dataset, including: - General information about the dataset, including data quality of each of the columns; - Distribution of each of the columns through statistics and plots (histogram, CDF, KDE), optionally grouped by other categorical variables; - 2D distribution between pairs of columns; - Correlation coefficient matrix for all numerical columns. Building this tool has provided a unique view into the full Python data stack, from the parallelised analysis of a dataframe within a Dask custom execution graph, to the interactive visualisation with Jupyter widgets and Plotly. During the talk, I will also introduce how Dask works, and demonstrate how to migrate data pipelines to take advantage of its scalable capabilities.

1. Automated Data Exploration
Building efficient analysis
pipelines with Dask
Víctor Zabalza
victor.z@asidatascience.com
github.com/zblz

2. About me
• ASI Data Science on SherlockML.

3. About me
• ASI Data Science on SherlockML.
• Former astrophysicist.

4. About me
• ASI Data Science on SherlockML.
• Former astrophysicist.
• Main developer of naima, a
package to model non-thermal
astrophysical sources.
• matplotlib developer.

5. Data Exploration

6. First steps in a Data Science project
• Does the data fit in a single
computer?

7. First steps in a Data Science project
• Does the data fit in a single
computer?
• Data quality assessment
• Data exploration
• Data cleaning

8. 80% → >30 h/week

10. Can we automate
the drudge work?

11. Developing a tool
for data exploration
based on Dask

12. Lens
Open source library for
automated data exploration

13. Lens by example
Room occupancy dataset
• ML standard dataset
• Goal: predict whether room is
occupied from ambient
measurements.

14. Lens by example
Room occupancy dataset
• ML standard dataset
• Goal: predict whether room is
occupied from ambient
measurements.
• What can we learn about it with
Lens?

15. Python interface
>>> import lens

16. Python interface
>>> import lens
>>> df = pd.read_csv('room_occupancy.csv')
>>> ls = lens.summarise(df)
>>> type(ls)
<class 'lens.summarise.Summary'>
>>> ls.to_json('room_occupancy_lens.json')

17. Python interface
>>> import lens
>>> df = pd.read_csv('room_occupancy.csv')
>>> ls = lens.summarise(df)
>>> type(ls)
<class 'lens.summarise.Summary'>
>>> ls.to_json('room_occupancy_lens.json')
room_occupancy_lens.json now contains
all information needed for
exploration!

18. Python interface — Columns
>>> ls.columns
['date',
'Temperature',
'Humidity',
'Light',
'CO2',
'HumidityRatio',
'Occupancy']

19. Python interface — Categorical summary
>>> ls.summary('Occupancy')
{'name': 'Occupancy',
'desc': 'categorical',
'dtype': 'int64',
'nulls': 0, 'notnulls': 8143,
'unique': 2}
>>> ls.details('Occupancy')
{'desc': 'categorical',
'frequencies': {0: 6414, 1: 1729},
'name': 'Occupancy'}

20. Python interface — Numeric summary
>>> ls.details('Temperature')
{'name': 'Temperature',
'desc': 'numeric',
'iqr': 1.69,
'min': 19.0, 'max': 23.18,
'mean': 20.619, 'median': 20.39,
'std': 1.0169, 'sum': 167901.1980}

21. The lens.Summary
is a good building
block, but clunky for
exploration.
Can we do better?

22. Jupyter widgets

23. Jupyter widgets: Column distribution

24. Jupyter widgets: Correlation matrix

25. Jupyter widgets: Pair density

26. Jupyter widgets: Pair density

27. Building Lens

28. Our solution: Analysis
• A Python library computes
dataset metrics:
• Column-wise statistics
• Pairwise densities
• ...

29. Our solution: Analysis
• A Python library computes
dataset metrics:
• Column-wise statistics
• Pairwise densities
• ...
• Computation cost is paid up
front.
• The result is serialized to JSON.

30. Our Solution: Interactive exploration
• Using only the report, the user
can explore the dataset through
either:
• Jupyter widgets
• Web UI

31. The lens Python
library

32. Why Python?
• Data Scientists

33. Why Python?
• Data Scientists
• Portability
• Reusability

34. Why Python?
• Data Scientists
• Portability
• Reusability
• Scalability

35. Why Python?
• Data Scientists
• Portability
• Reusability
• Scalability
Can Python scale?

36. Out-of-core options in Python
Difficult
Flexible
Easy
Restrictive

37. Out-of-core options in Python
Difficult
Flexible
Easy
Restrictive
Threads, Processes, MPI, ZeroMQ
Concurrent.futures, joblib

38. Out-of-core options in Python
Difficult
Flexible
Easy
Restrictive
Threads, Processes, MPI, ZeroMQ
Concurrent.futures, joblib
Luigi
PySpark
Hadoop, SQL

39. Out-of-core options in Python
Difficult
Flexible
Easy
Restrictive
Threads, Processes, MPI, ZeroMQ
Concurrent.futures, joblib
Luigi
PySpark
Hadoop, SQL

40. Out-of-core options in Python
Difficult
Flexible
Easy
Restrictive
Threads, Processes, MPI, ZeroMQ
Concurrent.futures, joblib
Luigi
PySpark
Hadoop, SQL
Dask

41. Dask

42. Dask interface
• Dask objects are delayed.

43. Dask interface
• Dask objects are delayed.
• The user operates on them as
Python structures.

44. Dask interface
• Dask objects are delayed.
• The user operates on them as
Python structures.
• Dask builds a DAG of the
computation.

45. Dask interface
• Dask objects are delayed.
• The user operates on them as
Python structures.
• Dask builds a DAG of the
computation.
• DAG is executed when a result is
requested.

46. Dask data structures
• numpy.ndarray
• pandas.DataFrame
• list, set

47. Dask data structures
• numpy.ndarray → dask.array
• pandas.DataFrame →
dask.dataframe
• list, set → dask.bag

48. dask.delayed — Build you own DAG

49. dask.delayed — Build you own DAG
files = ['myfile.a.data',
'myfile.b.data',
'myfile.c.data']
loaded = [load(i) for i in files]
cleaned = [clean(i) for i in loaded]
analyzed = analyze(cleaned)
store(analyzed)

50. dask.delayed — Build you own DAG
@delayed
def load(filename):
...
@delayed
def clean(data):
...
@delayed
def analyze(sequence_of_data):
...
@delayed
def store(result):
with open(..., 'w') as f:
f.write(result)

51. dask.delayed — Build you own DAG
files = ['myfile.a.data', 'myfile.b.data',
'myfile.c.data']
loaded = [load(i) for i in files]
cleaned = [clean(i) for i in loaded]
analyzed = analyze(cleaned)
stored = store(analyzed)

52. dask.delayed — Build you own DAG
files = ['myfile.a.data', 'myfile.b.data',
'myfile.c.data']
loaded = [load(i) for i in files]
cleaned = [clean(i) for i in loaded]
analyzed = analyze(cleaned)
stored = store(analyzed)
clean-2
analyze
cleanload-2
analyze store
clean-3
clean-1
load
storecleanload-1
cleanload-3load
load

53. dask.delayed — Build you own DAG
files = ['myfile.a.data', 'myfile.b.data',
'myfile.c.data']
loaded = [load(i) for i in files]
cleaned = [clean(i) for i in loaded]
analyzed = analyze(cleaned)
stored = store(analyzed)
clean-2
analyze
cleanload-2
analyze store
clean-3
clean-1
load
storecleanload-1
cleanload-3load
load
stored.compute()

54. Dask DAG execution

55. Dask DAG execution
In memory Released from memory

56. Dask DAG execution
In memory Released from memory

57. How do we use
Dask in Lens?

58. Lens pipeline
DataFrame
colA
colB

59. Lens pipeline
DataFrame
colA
colB
PropA
PropB

60. Lens pipeline
DataFrame
colA
colB
PropA
PropB
SummA
SummB

61. Lens pipeline
DataFrame
colA
colB
PropA
PropB
SummA
SummB
OutA
OutB

62. Lens pipeline
DataFrame
colA
colB
PropA
PropB
SummA
SummB
OutA
OutB
Corr

63. Lens pipeline
DataFrame
colA
colB
PropA
PropB
SummA
SummB
OutA
OutB
Corr PairDensity

64. Lens pipeline
DataFrame
colA
colB
PropA
PropB
SummA
SummB
OutA
OutB
Corr PairDensity
Report

65. • Graph for
two-column
dataset
generated by
lens.

66. • Graph for
two-column
dataset
generated by
lens.
• The same
code can be
used for
much wider
datasets.

69. Integration with
SherlockML

70. SherlockML integration
• Every dataset entering the
platform is analysed by Lens.

71. SherlockML integration
• Every dataset entering the
platform is analysed by Lens.
• We can use the same Python
library!

72. SherlockML integration
• Every dataset entering the
platform is analysed by Lens.
• We can use the same Python
library!
• The web frontend is used to
interact with datasets.

73. SherlockML: Column information

74. SherlockML: Column distribution

75. SherlockML: Correlation matrix

76. SherlockML: Pair density

77. Data Exploration with Lens

78. Data Exploration with Lens
• Scalable compute with dask.

79. Data Exploration with Lens
• Scalable compute with dask.
• Snappy interactive exploration.

80. Data Exploration with Lens
• Scalable compute with dask.
• Snappy interactive exploration.
• Lens is open source:
• Github: ASIDataScience/lens
• Docs: https://lens.rtfd.io
• PyPi: pip install lens