The document discusses the challenges of big data in astronomy. It describes several upcoming and current astronomical surveys that will produce enormous amounts of data, such as Euclid which will collect 15 terabytes per day starting in 2022. Dealing with such large datasets requires new techniques for storage, processing, analyzing, classifying, compressing and visualizing the data. Machine learning and citizen science are helping to process the data more efficiently. Future surveys like the Square Kilometer Array telescope will pose even greater challenges by producing data at terabytes per second.

1. The
Dark Matter
Mystery
The
big data universe.
Literally
D r M a g g i e L i e u
@space_mog
The
big data universe.
Literally
Bruno Merin & Beatriz Martinez

2. 1 black hole

3. M87’s
super massive
black hole
10,000,000 x smaller

4. 5x smaller
M87
0.5 deg
0.1 deg

5. A telescope the size of the Earth!
Event Horizon Telescope EHT
- Radio telescopes from around the world

6. 📸 : Flora Graham
5PB of data

7. 1 blackhole = 5000TB
1TB/day 8TB/day

8. Global average broadband
Download Upload
57.91Mbs
28.68Mbs

9. Global average broadband
57.91
Download
Mbs
28.68
Upload
Mbs
Upload time =
data volume
rate
= 5000000000[MB] x 8[Mb/MB]
28.68 Mbs
=
44 years!!!!

10. European space agency

11. 📸 : MIT Haystack Observatory

12. 📸 : Katie Bouman
2 years of processing

13. But space data is different….

17. ‣ coming soon in 2022
Euclid

19. Gravitational lensing

20. background galaxies
Weak gravitational lensing

21. foreground mass
More mass = more distorted galaxies!

22. real galaxy shear
atmosphere &
telescope
blur
pixelised by
detectors noise
photo cred: CFHT

24. Euclid Wide Survey
0°
30°
60°
-30°
-60°
315°0°45° 270° 225°90°135°
Moon for
comparison
‣ 15,000 deg2

25. Flagship simulations ‣ End-2-end simulations
‣ Largest simulated galaxy
catalogue ever built
‣ 2 trillion dark matter particles

26. ‣ Swiss National Computing Centre - 6th fastest computer in the world
‣ 80hrs
‣ 270,000 EUR
Piz Daint - over 5000 GPU nodes

27. ‣ 5000 deg2
‣ Raw simulation data: 0.4PB
‣ Compress to catalogs
‣ Rockstar (5.5TB)
‣ 2D dark matter count maps (1TB)

28. Sun
L1 L2
Moon
Earth
1,500,000km
Collecting the data

29. DSA-2
DSA-3
ESA-ESOC
‣ ESA deep space tracking stations
DSA-1

30. ‣ Ground station: Cebreros, Spain
‣ 4 hr communication window
‣ Steerable K-band (26 GHz)
‣ X-band (8.5 GHz)
‣ Data rate: 850 Gbit/day
‣ On board 4Tbit flash memory
Telemetry

31. ‣ Science centre: ESAC, Spain
‣ Ex-telemetry, tracking & commanding station
‣ Quick look analysis, archiving and distribution

32. Storage

33. Attractive data
No longer era where we fight for data, but era that we choose data!

34. Visualising the data
‣ ESA sky
‣ HiPS map, based on HEALpix
‣ Visualise TB’s of data
‣ Render kB of data
Fernique et al 2015

35. www.sky.esa.int

36. Code-to-data platform
‣ Science Exploration platform:
‣ Jupyter notebooks on SPARK clusters
‣ Cloud computing:
‣ Amazon web services,
‣ Google cloud, etc

37. See Lieu+18
‣ Raw data is nasty!
‣ with GB’s data per day, traditional methods are not efficient
Analysing the data

38. Machine learning

39. Classification and detection
‣ K-means
Characterising
spectra of galaxies,
Rahmani+18

40. Classification and detection
‣ DBSCAN
core
core border
noise
Stars in a star forming
region, Canovas+19

41. Classification and detection
‣ K-nearest neighbour
Defining types of
supernovae,
Lochner+16

42. Classification and detection
‣ Decision tree
T > 2 S > 5
F>0. 3
D
CB
A
T F
T T FF
Finding weird
galaxies, Baron+16

43. Classification and detection
‣ Convolutional neural networks (& object detection)
Strong gravitational
lensing, Schaefer+17
Asteroids, Lieu+19

44. Classification and detection
‣ Transfer learning

45. Freeze
Replace
Classification and detection
‣ Transfer learning

46. Citizen science
‣ Outsource tasks to the general public
‣ Zooniverse platform: easy to build projects
‣ 100’s of Projects
‣ 250M classifications
‣ 2M Volunteers

47. Citizen science
Classify galaxy morphologies:
Lintott+08

48. Citizen science
Find star forming bubbles:
Kendrew+12

49. Citizen science
Discover glitches in
gravitational wave signals:
Zevity+17

50. Data & model compression
‣ Neural networks & emulators
Emulate the halo mass
function with mixture
density networks,
Lieu+in. prep
Emulate cosmology with
neural density estimators,
Alsing+2019
Scaling relations with
principle component
analyses PCA,
Bothwell+2016

51. ‣ Edge computing:
‣ Some form in Gaia
‣ Mars rovers
Upcoming methods

52. ‣ Continuous learning with GANS
Upcoming methods

53. ‣ Federated learning
Upcoming methods
Super Model
ESAC
Data

54. 10PB of data is nothing…

56. ‣ First light: 2022
‣ Duration: 10 years
‣ Data: 15TB/day

57. ‣ Search for transients (supernova, asteroids, comets, gamma ray bursts)
‣ Gravitational lensing img: Pursiainen

58. Can it get any worse…?

59. ‣ SKA - square kilometre array
‣ First light: 2030
‣ Data: 2TB/sec

60. Karoo, South Africa
SKA-Mid
MRO, Australia
SKA-Low

61. Centaurus A
Optical
Hubble

62. Centaurus A
Radio
VLA

63. Centaurus A
Composite
Hubble/VLA/Chandra

64. EHT
5PB
10PB
60PB 200,000,000PB
EUCLID LSST SKA

65. Biggest challenges in Astronomy
‣ Collecting the data
‣ Retrieval
‣ Filtering good from bad
‣ Data Storage
‣ Distributing the data
‣ Upload/Download
‣ Combining data
‣ complementary observations and multi-wavelength observations
‣ Data analysis
‣ Compression
‣ Source detection
‣ Visualising the data
Let’s collaborate!

66. Thank you for your attentioN