Bolt is a distributed ndarray built on PySpark that conforms to NumPy's API. It can handle large multidimensional datasets exceeding 1 TB in size with up to 1011 elements. By distributing arrays across a cluster, Bolt enables operations on large neuroscience, astronomy, geospatial and climate science datasets that would be impossible on a single machine due to limits of memory and processing power. Key features include indexing, slicing, transpose, and applying functions along axes.

1. Bolt:
Building a distributed ndarray
Jason Wittenbach
Janelia Research Campus (HHMI)
Freeman Lab

4. t, (x, y, z)
time ~ 104
space ~ 107
~ 1011 elements
~ 1 TB

5. (x, y, t)
(x, y, z, t)
(x, y, z, c, t)
(n, k)

8. time

9. time

10. time

11. x
y

13. neuroscience
astronomy
geospatial
climate science

14. • a distributed ndarray
• built on PySpark
• conforms to NumPy API

15. data.mean(axis=0)
data[2, 4:10]
data.T

16. (t, x, y, z) (x, y, z, t)

17. (v, w, x, y, z)
(v,w) y
x z

18. (v,w,x)
(v, w, x, y, z)
y
z

19. (v, w, x, y, z)
(v)

20. indexing slicing
apply-along-axis

21. indexing
transpose
slicing
apply-along-axis
reshape

22. indexing
transpose
map
slicing
apply-along-axis
reshape
reduce filter
chunking padding

23. indexing
filter
map
(u,v) x
y

24. apply-along-axis
reduceByKey
(u,v) x
y
map

25. transpose
(u,v) x
y
map

26. transpose
(u,v) x
y
map

27. transpose
(u,v) x
y
shuffle

28. (t x,y,z) (x,y,z t)

29. (t x, y, z)
(t)
x y
z

30. (t x, y, z)
(t)
x y
z

31. (t x, y, z)
(t,chunk) (t,chunk)
(t,chunk)

38. chunking
transpose
+ = shuffle
optimization

39. ???

40. thanks join us!
Jeremy Freeman
Nicholas Sofroniew
Andrew Osheroff
Freeman Lab
Ken Carlile
Robert Lines
Janelia Scientific
Computing
bolt-project
thunder-project
GitHub
@jsonWittenbach
Twitter