This talk will show what is possible huge datasets that are becoming more prevalent in the era of big data. I will demonstrate this and the 3d visualization in the Jupyter notebook, the by now almost standard environment of (data) scientists. With large astronomical catalogues containing more than a billion stars becoming common, we are preparing for methods to visualize and explore these large datasets. Data volumes of this size requires different visualization techniques, since scatter plots become too slow and meaningless due to overplotting. We solve the performance and visualization issue using binned statistics, e.g. histograms, density maps, and volume rendering in 3d. The calculation of statistics on N-dimensional grids is handled by Python library called vaex, which I will introduce. It can process at least a billion samples per second, to produce for instance the mean of a quantity on a regular grid. This statistics can be calculated for any mathematical expression on the data (numpy style) and can be on the full dataset or subsets, specified by queries/selections. However, to visualize higher dimensional data in the notebook interactively, no proper solution existed. This led to the development of ipyvolume, which can render 3d volumes and up to a million glyphs (scatter plots and quiver) in the Jupyter notebook as a widget. With the browser as a platform, and the release of ipywidgets 6.0, these 3d plots can also be embedded in static html files and renders on nbviewer. This allows for sharing with colleagues, rendering on your tablet (paperless office), outreach, press release material, etc. Full screen stereo rendering allows for a virtual reality experience using your phone and Google Cardboard, a minor investment compared to other VR head mountables. Overlaying 3d quiver plots on a 3d volume rendering allows exploring a 6d (or higher) space. Vaex and ipyvolume can be used together to explore and visualize any large tabular data set, or separately to calculate statistics, and render 3d plots in the notebook and outside.

1. A Billion stars in the Jupyter notebook
Maarten Breddels
Groningen - 2017
@MaartenBreddels

2. Outline
• Motivation/Problem
• Solution (vaex)
• Demo
• vaex
• ipwidgets / ipyvolume

3. Copyright ESA/ATG medialab; background: ESO/S. Brunier
• Gaia satellite
• launched by ESA in december 2013
• determine for >109
stars/objects in our
Milky Way
• positions
• velocities
• astrophysical parameters
• First catalogue (DR1) is out
• sky positions, G magnitude
Motivation

4. Motivation• We have Gaia DR1, soon DR2
• > 10
9
objects/stars
• Can we visualise and explore this?
• We want to ‘see’ the data
• Data/Quality checks
• Science/Discovery
• Trends, relations, clustering
• You are the (biological) neutral network

5. Motivation• We have Gaia DR1, soon DR2
• > 10
9
objects/stars
• Can we visualise and explore this?
• We want to ‘see’ the data
• Data/Quality checks
• Science/Discovery
• Trends, relations, clustering
• You are the (biological) neutral network
• Problem
• Scatter plots do not work well for 109
rows/objects
• Work with densities/statistics in 0,1,2 and 3d
• Interactive?
• Zoom, pan etc
• Explore: selections/queries

6. Motivation• We have Gaia DR1, soon DR2
• > 10
9
objects/stars
• Can we visualise and explore this?
• We want to ‘see’ the data
• Data/Quality checks
• Science/Discovery
• Trends, relations, clustering
• You are the (biological) neutral network
• Problem
• Scatter plots do not work well for 109
rows/objects
• Work with densities/statistics in 0,1,2 and 3d
• Interactive?
• Zoom, pan etc
• Explore: selections/queries

7. Motivation• We have Gaia DR1, soon DR2
• > 10
9
objects/stars
• Can we visualise and explore this?
• We want to ‘see’ the data
• Data/Quality checks
• Science/Discovery
• Trends, relations, clustering
• You are the (biological) neutral network
• Problem
• Scatter plots do not work well for 109
rows/objects
• Work with densities/statistics in 0,1,2 and 3d
• Interactive?
• Zoom, pan etc
• Explore: selections/queries

8. How fast can it be processed?
• What can be done?
• 10
9
* 2 * 8 bytes = 15 GiB (double is 8 bytes)
• Memory bandwidth: 10-20 GiB/s: ~1 second
• CPU: 3 Ghz (but multicore, say 4-8): 12-24 cycles/second
• Few cycles per row/object, simple algorithm
• Histograms/Density grids
• Yes, but
• If it fits/cached in memory, otherwise ssd/hdd speeds (10-100 seconds)
• proper storage and reading of data (no .csv!)
• simple and fast algorithm for binning
• ~1 second

9. How fast can it be processed?
• What can be done?
• 10
9
* 2 * 8 bytes = 15 GiB (double is 8 bytes)
• Memory bandwidth: 10-20 GiB/s: ~1 second
• CPU: 3 Ghz (but multicore, say 4-8): 12-24 cycles/second
• Few cycles per row/object, simple algorithm
• Histograms/Density grids
• Yes, but
• If it fits/cached in memory, otherwise ssd/hdd speeds (10-100 seconds)
• proper storage and reading of data (no .csv!)
• simple and fast algorithm for binning
• ~1 second

10. How fast can it be processed?
• What can be done?
• 10
9
* 2 * 8 bytes = 15 GiB (double is 8 bytes)
• Memory bandwidth: 10-20 GiB/s: ~1 second
• CPU: 3 Ghz (but multicore, say 4-8): 12-24 cycles/second
• Few cycles per row/object, simple algorithm
• Histograms/Density grids
• Yes, but
• If it fits/cached in memory, otherwise ssd/hdd speeds (10-100 seconds)
• proper storage and reading of data (no .csv!)
• simple and fast algorithm for binning
• ~1 second

11. How to store and read the data
cache
• Storage: native, column based (hdf5)
• Normal (POSIX read) method:
• Allocate memory
• read from disk to memory
• Actually: from disk, to OS cache, to memory
• Wastes memory/cache
• 15 GB data , requires 30 GB is you want to use the file system cache

12. How to store and read the data
cache
• Storage: native, column based (hdf5)
• Normal (POSIX read) method:
• Allocate memory
• read from disk to memory
• Actually: from disk, to OS cache, to memory
• Wastes memory/cache
• 15 GB data , requires 30 GB is you want to use the file system cache

13. How to store and read the data
cache
• Storage: native, column based (hdf5)
• Normal (POSIX read) method:
• Allocate memory
• read from disk to memory
• Actually: from disk, to OS cache, to memory
• Wastes memory/cache
• 15 GB data , requires 30 GB is you want to use the file system cache
• Memory mapping:
• get direct access to OS memory cache, no copy, no
setup (apart from the kernel doing setting up the pages)
• avoid memory copies, more cache available
• In previous example:
• copying 15 GB will take about ~1.0 second, at 10-20
GB/s
• Can be 2-3x slower (cpu cache helps a bit)

15. 2d
1d

16. 2d
1d
3d

17. 2d
1d
3d
0d 330,000 rows

18. 2d
1d
3d
0d 330,000 rows

19. 2d
1d
3d
0d 330,000 rows

20. 2d
1d
3d
0d 330,000 rows mean: -0.083

21. 2d
1d
3d
0d 330,000 rows mean: -0.083

22. vaex
• Python library (conda/pip installable)
• Think pandas-like but for large datasets
• simple api: 1 class (Dataset), the rest are strings/dicts/tuples/lists/ndarray
• From/to pandas dataframes, plain numpy arrays, astropy tables
• Focusses mostly on statistics on N-d grids (count/mean/max/std/…)
• >1 billion rows / sec on a `decent` desktop (quad core 3Gz)
• >50x faster than scipy.stats.binned_statistic_2d
• 0.1-0.2 billion rows on this 2yr old Macbook Air
• CPU bound
• Does visualisation / matplotlib / bqplot / ipyvolume / ipyleaflet

23. For what?
• Astronomical catalogues
• Gaia > 1 billion
• SDSS > 150 million
• PANSTARRS > 1 billion
• LSST > 10 billion
• N-body simulations (0.1-100 billion)
• columns: x,y,z,vx,vy,vz
• SPH: density, temperature, metallicity…
• rows: particle
• Any tabular data
• rows and columns

24. For what?
• Astronomical catalogues
• Gaia > 1 billion
• SDSS > 150 million
• PANSTARRS > 1 billion
• LSST > 10 billion
• N-body simulations (0.1-100 billion)
• columns: x,y,z,vx,vy,vz
• SPH: density, temperature, metallicity…
• rows: particle
• Any tabular data
• rows and columns

25. For what?
• Astronomical catalogues
• Gaia > 1 billion
• SDSS > 150 million
• PANSTARRS > 1 billion
• LSST > 10 billion
• N-body simulations (0.1-100 billion)
• columns: x,y,z,vx,vy,vz
• SPH: density, temperature, metallicity…
• rows: particle
• Any tabular data
• rows and columns

26. For what?
• Astronomical catalogues
• Gaia > 1 billion
• SDSS > 150 million
• PANSTARRS > 1 billion
• LSST > 10 billion
• N-body simulations (0.1-100 billion)
• columns: x,y,z,vx,vy,vz
• SPH: density, temperature, metallicity…
• rows: particle
• Any tabular data
• rows and columns

27. Transformation of the data
• Not ‘z = x + y’
• for 1 billion rows: 8GB of RAM wasted
• Everything is an expression: dataset.mean(‘x + y’)
• Computed in chunks not to waste RAM
• Virtual columns:
• dataset.add_virtual_column(‘r’, ‘sqrt(x**2+y**2)’)
• dataset.mean(‘r’)

28. Selections/subsets
• You don’t use all data / filtering
• Efficient subsets (do NOT copy the data)
• vaex
• Selections result in a boolean mask
• dataset.select(‘x > 0’, name=‘xpos’)
• Use in any statistic
• dataset.mean(‘x + y’, selection=‘xpos’)

29. Demo library

30. Conclusions
• vaex
• N-d statistics on regular grids
• ~1 billion rows/sec
• efficient transformations/selections
• regular grids can be used for visualisation
• Interactive in the Notebook due to ipywidgets/bqplot/ipyleaflet
• ipyvolume
• adds interactive 3d plotting to the notebook, (as an ipywidget)
• can be used as backend for vaex
• works outside the notebook, in the browser
• tablet (paperless office)
• sharing, outreach, press release material

31. Questions

32. • maartenbreddels@gmail.com
• twitter @maartenbreddels
• vaex
• http://vaex.astro.rug.nl
• https://github.com/maartenbreddels/vaex
• pip install —pre vaex / conda install -c conda-forge vaex
• ipyvolume
• https://ipywidgets.readthedocs.io
• https://github.com/maartenbreddels/ipyvolume
• pip install ipyvolume / conda install -c conda-forge ipyvolume
• if pip: jupyter nbextension enable --py --user ipyvolume (but really, use anaconda)
• Notebook:
• https://github.com/maartenbreddels/pygrunn-2017