1. Lens: Data exploration with
Dask and Jupyter widgets
Víctor Zabalza
vzabalza@gmail.com
@zblz

2. About me
• ASI Data Science on SherlockML.

3. About me
• ASI Data Science on SherlockML.
• Former astrophysicist.

4. About me
• ASI Data Science on SherlockML.
• Former astrophysicist.
• Main developer of naima, a package to
model non-thermal astrophysical sources.
• matplotlib developer.

5. Data Exploration

6. First steps in a Data Science project
• Does the data fit in a single computer?

7. First steps in a Data Science project
• Does the data fit in a single computer?
• Data quality assessment
• Data exploration
• Data cleaning

8. 80% → >30 h/week

10. Can we automate the
drudge work?

11. Developing a tool for
data exploration based
on Dask

12. Lens
Open source library for
automated data exploration

13. Lens by example
Room occupancy dataset
• ML standard dataset
• Goal: predict whether room is occupied
from ambient measurements.

14. Lens by example
Room occupancy dataset
• ML standard dataset
• Goal: predict whether room is occupied
from ambient measurements.
• What can we learn about it with Lens?

15. Python interface
>>> import lens

16. Python interface
>>> import lens
>>> df = pd.read_csv('room_occupancy.csv')
>>> ls = lens.summarise(df)
>>> type(ls)
<class 'lens.summarise.Summary'>
>>> ls.to_json('room_occupancy_lens.json')

17. Python interface
>>> import lens
>>> df = pd.read_csv('room_occupancy.csv')
>>> ls = lens.summarise(df)
>>> type(ls)
<class 'lens.summarise.Summary'>
>>> ls.to_json('room_occupancy_lens.json')
room_occupancy_lens.json now contains all
information needed for exploration!

18. Python interface — Columns
>>> ls.columns
['date',
'Temperature',
'Humidity',
'Light',
'CO2',
'HumidityRatio',
'Occupancy']

19. Python interface — Categorical summary
>>> ls.summary('Occupancy')
{'name': 'Occupancy',
'desc': 'categorical',
'dtype': 'int64',
'nulls': 0, 'notnulls': 8143,
'unique': 2}
>>> ls.details('Occupancy')
{'desc': 'categorical',
'frequencies': {0: 6414, 1: 1729},
'name': 'Occupancy'}

20. Python interface — Numeric summary
>>> ls.details('Temperature')
{'name': 'Temperature',
'desc': 'numeric',
'iqr': 1.69,
'min': 19.0, 'max': 23.18,
'mean': 20.619, 'median': 20.39,
'std': 1.0169, 'sum': 167901.1980}

21. The lens.Summary is a
good building block, but
clunky for exploration.
Can we do better?

22. Jupyter widgets

23. Jupyter widgets: Column distribution

24. Jupyter widgets: Correlation matrix

25. Jupyter widgets: Pair density

26. Jupyter widgets: Pair density

27. Building Lens

28. Our solution: Analysis
• A Python function computes dataset
metrics:
• Column-wise statistics
• Pairwise densities
• ...

29. Our solution: Analysis
• A Python function computes dataset
metrics:
• Column-wise statistics
• Pairwise densities
• ...
• Computation cost is paid up front.
• The result is serialized to JSON.

30. Our Solution: Interactive exploration
• Using only the report, the user can explore
the dataset through either:
• Jupyter widgets
• Web UI

31. The lens Python library

32. Why Python?
• Data Scientists

33. Why Python?
• Data Scientists
• Portability
• Reusability

34. Why Python?
• Data Scientists
• Portability
• Reusability
• Scalability

35. Why Python?
• Data Scientists
• Portability
• Reusability
• Scalability
Can Python scale?

36. Out-of-core options in Python
Difficult
Flexible
Easy
Restrictive

37. Out-of-core options in Python
Difficult
Flexible
Easy
Restrictive
Threads, Processes, MPI, ZeroMQ
Concurrent.futures, joblib

38. Out-of-core options in Python
Difficult
Flexible
Easy
Restrictive
Threads, Processes, MPI, ZeroMQ
Concurrent.futures, joblib
Luigi
PySpark
Hadoop, SQL

39. Out-of-core options in Python
Difficult
Flexible
Easy
Restrictive
Threads, Processes, MPI, ZeroMQ
Concurrent.futures, joblib
Luigi
PySpark
Hadoop, SQL

40. Out-of-core options in Python
Difficult
Flexible
Easy
Restrictive
Threads, Processes, MPI, ZeroMQ
Concurrent.futures, joblib
Luigi
PySpark
Hadoop, SQL
Dask

41. Dask

42. Dask interface
• Dask objects are lazily computed.

43. Dask interface
• Dask objects are lazily computed.
• The user operates on them as Python
structures.

44. Dask interface
• Dask objects are lazily computed.
• The user operates on them as Python
structures.
• Dask builds a DAG of the computation.

45. Dask interface
• Dask objects are lazily computed.
• The user operates on them as Python
structures.
• Dask builds a DAG of the computation.
• DAG is executed when a result is requested.

46. dask.delayed — Build your own DAG

47. dask.delayed — Build your own DAG
files = ['myfile.a.data',
'myfile.b.data',
'myfile.c.data']
loaded = [load(i) for i in files]
cleaned = [clean(i) for i in loaded]
analyzed = analyze(cleaned)
store(analyzed)

48. dask.delayed — Build your own DAG
@delayed
def load(filename):
...
@delayed
def clean(data):
...
@delayed
def analyze(sequence_of_data):
...
@delayed
def store(result):
with open(..., 'w') as f:
f.write(result)

49. dask.delayed — Build your own DAG
files = ['myfile.a.data', 'myfile.b.data',
'myfile.c.data']
loaded = [load(i) for i in files]
cleaned = [clean(i) for i in loaded]
analyzed = analyze(cleaned)
stored = store(analyzed)

50. dask.delayed — Build your own DAG
files = ['myfile.a.data', 'myfile.b.data',
'myfile.c.data']
loaded = [load(i) for i in files]
cleaned = [clean(i) for i in loaded]
analyzed = analyze(cleaned)
stored = store(analyzed)
clean-2
analyze
cleanload-2
analyze store
clean-3
clean-1
load
storecleanload-1
cleanload-3load
load

51. dask.delayed — Build your own DAG
files = ['myfile.a.data', 'myfile.b.data',
'myfile.c.data']
loaded = [load(i) for i in files]
cleaned = [clean(i) for i in loaded]
analyzed = analyze(cleaned)
stored = store(analyzed)
clean-2
analyze
cleanload-2
analyze store
clean-3
clean-1
load
storecleanload-1
cleanload-3load
load
stored.compute()

52. Dask high-level collections
These implement a good fraction of the APIs of
their Python counterparts:
• dask.array
• dask.dataframe
• dask.bag

53. Dask tools for Machine Learning
• dask-ml
• dask-searchcv
• dask-XGBoost
• dask-tensorflow
• ...

54. Dask schedulers — How is the graph executed?
• Synchronous — good for testing

55. Dask schedulers — How is the graph executed?
• Synchronous — good for testing
• Threaded — I/O and GIL-releasing code

56. Dask schedulers — How is the graph executed?
• Synchronous — good for testing
• Threaded — I/O and GIL-releasing code
• Multiprocessing — bypass GIL

57. Dask schedulers — How is the graph executed?
• Synchronous — good for testing
• Threaded — I/O and GIL-releasing code
• Multiprocessing — bypass GIL
• Distributed — run in multiple nodes

58. Dask DAG execution

59. Dask DAG execution
In memory Released from memory

60. Dask DAG execution
In memory Released from memory

61. How do we use
Dask in Lens?

62. Lens pipeline
DataFrame
colA
colB

63. Lens pipeline
DataFrame
colA
colB
PropA
PropB

64. Lens pipeline
DataFrame
colA
colB
PropA
PropB
SummA
SummB

65. Lens pipeline
DataFrame
colA
colB
PropA
PropB
SummA
SummB
OutA
OutB

66. Lens pipeline
DataFrame
colA
colB
PropA
PropB
SummA
SummB
OutA
OutB
Corr

67. Lens pipeline
DataFrame
colA
colB
PropA
PropB
SummA
SummB
OutA
OutB
Corr PairDensity

68. Lens pipeline
DataFrame
colA
colB
PropA
PropB
SummA
SummB
OutA
OutB
Corr PairDensity
Report

69. • Graph for
two-column
dataset generated
by lens.

70. • Graph for
two-column
dataset generated
by lens.
• The same code
can be used for
much wider
datasets.

73. Integration with
SherlockML

74. SherlockML integration
• Every dataset entering the platform is
analysed by Lens.

75. SherlockML integration
• Every dataset entering the platform is
analysed by Lens.
• We can use the same Python library!

76. SherlockML integration
• Every dataset entering the platform is
analysed by Lens.
• We can use the same Python library!
• The web frontend is used to interact with
datasets.

77. SherlockML: Column information

78. SherlockML: Column distribution

79. SherlockML: Correlation matrix

80. SherlockML: Pair density

81. Data Exploration with Lens

82. Data Exploration with Lens
• Scalable compute with dask.

83. Data Exploration with Lens
• Scalable compute with dask.
• Snappy interactive exploration.

84. Data Exploration with Lens
• Scalable compute with dask.
• Snappy interactive exploration.
• Lens is open source:
• GitHub: ASIDataScience/lens
• Docs: https://lens.readthedocs.io
• PyPI: pip install lens