1. Python and MongoDB
in Astronomy
Dan Foreman-Mackey
Center for Cosmology and Particle Physics
Department of Physics @ NYU
In collaboration with:
David W. Hogg (NYU), Larry Widrow (Queen’s),
Dustin Lang (Princeton), Jonathan Sick (Queen’s),
Micha Gorelick (NYU) and many others...
2. Astronomy 101
How to Study the Cosmos
Python, MongoDB, etc.
Case Studies
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
3. Astronomy 101
How to Study the Cosmos
Python, MongoDB, etc.
Andromeda Case Studies The Internet
The Milky Way
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
4. The Universe
Galaxies
Stars
Planets
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
5. What is the The Universe
Universe
Made of?
Galaxies
Stars
Planets
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
6. What is the The Universe
Universe
Made of?
Galaxies
Stars
Are there
other Earth-
Planets like planets?
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
7. What is the The Universe
Universe
Made of?
Galaxies
Stars
Are there
other Earth-
Planets like planets?
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
8. What is the Universe Made of?
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
9. What is the Universe Made of?
Source: Wikipedia (Adam Evans)
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
10. What is the Universe Made of?
Rotational Speed
Radius
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
11. What is the Universe Made of?
Observed
Rotational Speed
Radius
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
12. What is the Universe Made of?
Observed
Rotational Speed
WTF?
Radius
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
13. What is the Universe Made of?
Rotational Speed ? Observed
WTF?
Radius
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
14. What is the Universe Made of?
Rotational Speed ? Observed
WTF?
Radius
?
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
15. What is the Universe Made of?
Observed
Rotational Speed
WTF?
Radius
?
?
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
16. What is the Universe Made of?
Dark Matter
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
17. What is the Universe Made of?
Size of the Universe
PyGotham
b le
rva
obse
Time
Source: NASA / WMAP Science Team
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
18. What is the Universe Made of?
Dark Energy
73%
Heavy
Elements
0.03%
Dark Matter
23%
Atoms
4%
Source: NASA / WMAP Science Team
WMAP Year 7 (Larson et al. 2011)
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
19. What is the Universe Made of?
Dark Energy
73%
Source: DFM & Widrow (in prep)
Heavy
Elements
0.03%
Dark Matter
23%
Atoms
4%
Source: NASA / WMAP Science Team
WMAP Year 7 (Larson et al. 2011)
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
20. What is the Universe Made of? Source: http://apod.nasa.gov
Dark Energy
73%
Source: DFM & Widrow (in prep)
Heavy
Elements
0.03%
Dark Matter
23%
Atoms
4%
Source: NASA / WMAP Science Team
WMAP Year 7 (Larson et al. 2011)
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
21. What is the Universe Made of? Source: http://apod.nasa.gov
Dark Energy
73%
Source: DFM & Widrow (in prep)
Heavy
Elements
0.03%
Dark Matter
23%
Atoms
4%
Source: NASA / WMAP Science Team
WMAP Year 7 (Larson et al. 2011)
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
22. Dan Foreman-Mackey CCPP@NYU
Credit: The Millennium Simulation Project
dfm.github.com
23. Dan Foreman-Mackey CCPP@NYU
Credit: The Millennium Simulation Project
dfm.github.com
24. Dan Foreman-Mackey CCPP@NYU
Credit: The Millennium Simulation Project
dfm.github.com
26. Data in Astronomy
Credit: Jonathan Sick
jonathansick.ca
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
27. Data in Astronomy
Credit: Jonathan Sick
jonathansick.ca
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
28. Data in Astronomy
MegaCam: 340 MegaPixels Credit: Jonathan Sick
jonathansick.ca
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
29. Data in Astronomy
MegaCam: 340 MegaPixels Credit: NASA
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
30. Data in Astronomy
Imaging
Source: NASA / ESA
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
31. Data in Astronomy
Imaging
Spectroscopy
Source: NASA / ESA
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
32. Data in Astronomy
Source: Riaud & Schneider (2007)
Imaging
Spectroscopy
Spectroscopy
Source: NASA / ESA
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
33. Data in Astronomy
Source: Riaud & Schneider (2007)
Imaging
Spectroscopy
Spectroscopy
Source: NASA / ESA
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
34. a
Data in Astronomy
lot
is Open
of
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
35. a
Data in Astronomy
lot
is Open
of
!
and there’s a lot of it
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
36. Data in Astronomy is Open
1990– 2000– 1997–2001
SDSS
Hubble archive.stsci.edu/hst sdss.org
2MASS
www.ipac.caltech.edu/2mass
GAIA
Pan-STARRS LSST
Planned
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
37. Data in Astronomy is Open
1990– 2000– 1997–2001
SDSS
Hubble archive.stsci.edu/hst sdss.org
2MASS
www.ipac.caltech.edu/2mass
GAIA
Pan-STARRS LSST
Planned
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
38. Data in Astronomy is Open
1990– 2000– 1997–2001
SDSS
Hubble archive.stsci.edu/hst sdss.org
2MASS
www.ipac.caltech.edu/2mass
GAIA
Pan-STARRS LSST
Planned
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
49. Case Studies
Variable Stars in Stripe 82
SDSS
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
50. Case Studies
Variable Stars in Stripe 82
SDSS
728 SESAR ET AL.
Ses
S07 Labela Ntot
A 84
B 144
C 54
D 8
E 11
F 11
Source: Sesar et al. (2010) G 10
H 7
I 4
J 26
K 8
Dan Foreman-Mackey CCPP@NYU L 3
dfm.github.com
M 5
51. Case Studies
Variable Stars in Stripe 82
SDSS
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
52. Case Studies
Variable Stars in Stripe 82
SDSS
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
53. Case Studies
Variable Stars in Stripe 82
SDSS
800k “Fields” ~ 12TB Imaging data
> 1M “Target Stars”
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
54. Case Studies
Variable Stars in Stripe 82
SDSS
Photons/Brightness
Time
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
55. Case Studies
Variable Stars in Stripe 82
SDSS
Y N
p(X|⇥) = [(1 Pvar )pconst (X ↵ |⇥) + Pvar pvar (X ↵ |⇥)]
↵=1
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
56. Case Studies
Variable Stars in Stripe 82
SDSS
Y N
p(X|⇥) = [(1 Pvar )pconst (X ↵ |⇥) + Pvar pvar (X ↵ |⇥)]
↵=1
Stars
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
57. Case Studies
Variable Stars in Stripe 82
SDSS
YN
p(X|⇥) = [(1 Pvar )pconst (X ↵ |⇥) + Pvar pvar (X ↵ |⇥)]
↵=1
Stars
M
Y M
Y
pconst ⌘ [(1 Pbad )pgood + Pbad pbad ] pvar ⌘ [(1 Pbad )pvar,good + Pbad pbad ]
i=1 i=1
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
58. Case Studies
Variable Stars in Stripe 82
SDSS
YN
p(X|⇥) = [(1 Pvar )pconst (X ↵ |⇥) + Pvar pvar (X ↵ |⇥)]
↵=1
Stars
M
Y M
Y
pconst ⌘ [(1 Pbad )pgood + Pbad pbad ] pvar ⌘ [(1 Pbad )pvar,good + Pbad pbad ]
i=1 i=1
Runs Runs
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
59. Case Studies
Variable Stars in Stripe 82
SDSS
YN
p(X|⇥) = [(1 Pvar )pconst (X ↵ |⇥) + Pvar pvar (X ↵ |⇥)]
↵=1
Stars
M
Y M
Y
pconst ⌘ [(1 Pbad )pgood + Pbad pbad ] pvar ⌘ [(1 Pbad )pvar,good + Pbad pbad ]
i=1 i=1
Runs Runs
⇤ ⇤
pgood ⌘ N (Ci↵ |fi0 f↵ , 2
i↵ + 2
i↵ ) pvar,good ⌘ N (Ci↵ |fi0 f↵ , 2
i↵ + 2
i↵ + ⌃2 )
var
“Constant & Good” “Variable & Good”
⇤
pbad ⌘ N (Ci↵ |fi0 f↵ , 2
i↵ + 2
i↵ + ⌃2 )
bad
“Bad”
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
60. Case Studies
Variable Stars in Stripe 82
SDSS
YN
p(X|⇥) = [(1 Pvar )pconst (X ↵ |⇥) + Pvar pvar (X ↵ |⇥)]
↵=1
Stars
M
Y M
Y
pconst ⌘ [(1 Pbad )pgood + Pbad pbad ] pvar ⌘ [(1 Pbad )pvar,good + Pbad pbad ]
i=1
Npars = Nstars + Nruns + 6
i=1
Runs ~ 0 , ~ ⇤ , , ⌘, ⌃2 , PRuns , Pbad }
2
⇥ = {f f var var , ⌃bad
⇤ ⇤
pgood ⌘ N (Ci↵ |fi0 f↵ , 2
i↵ + 2
i↵ ) pvar,good ⌘ N (Ci↵ |fi0 f↵ , 2
i↵ + 2
i↵ + ⌃2 )
var
“Constant & Good” “Variable & Good”
⇤
pbad ⌘ N (Ci↵ |fi0 f↵ , 2
i↵ + 2
i↵ + ⌃2 )
bad
“Bad”
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
61. Case Studies
Variable Stars in Stripe 82
SDSS
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
62. Case Studies
Variable Stars in Stripe 82
SDSS
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
63. Case Studies
Variable Stars in Stripe 82
SDSS
N
Y
p(X|⇥) = [(1 Pvar )pconst (X ↵ |⇥) + Pvar pvar (X ↵ |⇥)]
↵=1
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
64. Case Studies
Variable Stars in Stripe 82
SDSS
N
X
f ⇤ (t) = A0 + [An sin(!t) + Bn cos(!t)]
n=1
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
65. Case Studies
CFHT
4.7 Gigapixel mosaic of M31
Source: Jonathan Sick
(Queen’s University)
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
66. Dan Foreman-Mackey CCPP@NYU
dfm.github.com Jonathan Sick
Source:
(Queen’s University)
67. Dan Foreman-Mackey CCPP@NYU
dfm.github.com Jonathan Sick
Source:
(Queen’s University)
68. Dan Foreman-Mackey CCPP@NYU
dfm.github.com Jonathan Sick
Source:
(Queen’s University)
69. Case Studies
CFHT
Cosmic ray removal Flat-fielding Sky subtraction ... Mosaic making
MongoDB
Persistent Metadata + GeoSpatial Indexing
img1.fits img2.fits ... img4000.fits
Source: Jonathan Sick
(Queen’s University)
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
70. Astrometry.net Case Studies
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
71. Astrometry.net Case Studies
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
83. Growing
Datasets
User
Interaction
Number Data And Much
Crunching Management More...
( )
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
84. Growing
Datasets
User
Interaction
Easy!
Number Data And Much
Crunching Management More...
( )
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
85. Growing Big Questions
Datasets
User
Interaction
Easy!
Number Data And Much
Crunching Management More...
( )
Dan Foreman-Mackey CCPP@NYU
dfm.github.com
86. Dan Foreman-Mackey
Center for Cosmology & Particle Physics (NYU)
dfm.github.com
@__dfm__
Dan Foreman-Mackey CCPP@NYU
dfm.github.com