Python and MongoDB in Astronomy

1,161 views

Published on

An intro to galactic Astronomy for programmers and how Python and MongoDB fit into our development cycle.

Published in: Technology, Education
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,161
On SlideShare
0
From Embeds
0
Number of Embeds
18
Actions
Shares
0
Downloads
0
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Python and MongoDB in Astronomy

  1. 1. Python and MongoDB in AstronomyDan Foreman-MackeyCenter for Cosmology and Particle PhysicsDepartment of Physics @ NYU In collaboration with: David W. Hogg (NYU), Larry Widrow (Queen’s), Dustin Lang (Princeton), Jonathan Sick (Queen’s), Micha Gorelick (NYU) and many others...
  2. 2. Astronomy 101 How to Study the Cosmos Python, MongoDB, etc. Case StudiesDan Foreman-Mackey CCPP@NYU dfm.github.com
  3. 3. Astronomy 101 How to Study the Cosmos Python, MongoDB, etc. Andromeda Case Studies The Internet The Milky WayDan Foreman-Mackey CCPP@NYU dfm.github.com
  4. 4. The Universe Galaxies Stars PlanetsDan Foreman-Mackey CCPP@NYU dfm.github.com
  5. 5. What is the The Universe UniverseMade of? Galaxies Stars Planets Dan Foreman-Mackey CCPP@NYU dfm.github.com
  6. 6. What is the The Universe UniverseMade of? Galaxies Stars Are there other Earth- Planets like planets? Dan Foreman-Mackey CCPP@NYU dfm.github.com
  7. 7. What is the The Universe UniverseMade of? Galaxies Stars Are there other Earth- Planets like planets? Dan Foreman-Mackey CCPP@NYU dfm.github.com
  8. 8. What is the Universe Made of?Dan Foreman-Mackey CCPP@NYU dfm.github.com
  9. 9. What is the Universe Made of? Source: Wikipedia (Adam Evans)Dan Foreman-Mackey CCPP@NYU dfm.github.com
  10. 10. What is the Universe Made of? Rotational Speed RadiusDan Foreman-Mackey CCPP@NYU dfm.github.com
  11. 11. What is the Universe Made of? Observed Rotational Speed RadiusDan Foreman-Mackey CCPP@NYU dfm.github.com
  12. 12. What is the Universe Made of? Observed Rotational Speed WTF? RadiusDan Foreman-Mackey CCPP@NYU dfm.github.com
  13. 13. What is the Universe Made of? Rotational Speed ? Observed WTF? RadiusDan Foreman-Mackey CCPP@NYU dfm.github.com
  14. 14. What is the Universe Made of? Rotational Speed ? Observed WTF? Radius ?Dan Foreman-Mackey CCPP@NYU dfm.github.com
  15. 15. What is the Universe Made of? Observed Rotational Speed WTF? Radius ? ?Dan Foreman-Mackey CCPP@NYU dfm.github.com
  16. 16. What is the Universe Made of? Dark MatterDan Foreman-Mackey CCPP@NYU dfm.github.com
  17. 17. What is the Universe Made of? Size of the Universe PyGotham b le rvaobse Time Source: NASA / WMAP Science Team Dan Foreman-Mackey CCPP@NYU dfm.github.com
  18. 18. What is the Universe Made of? Dark Energy 73% Heavy Elements 0.03% Dark Matter 23% Atoms 4% Source: NASA / WMAP Science Team WMAP Year 7 (Larson et al. 2011)Dan Foreman-Mackey CCPP@NYU dfm.github.com
  19. 19. What is the Universe Made of? Dark Energy 73%Source: DFM & Widrow (in prep) Heavy Elements 0.03% Dark Matter 23% Atoms 4% Source: NASA / WMAP Science Team WMAP Year 7 (Larson et al. 2011) Dan Foreman-Mackey CCPP@NYU dfm.github.com
  20. 20. What is the Universe Made of? Source: http://apod.nasa.gov Dark Energy 73%Source: DFM & Widrow (in prep) Heavy Elements 0.03% Dark Matter 23% Atoms 4% Source: NASA / WMAP Science Team WMAP Year 7 (Larson et al. 2011) Dan Foreman-Mackey CCPP@NYU dfm.github.com
  21. 21. What is the Universe Made of? Source: http://apod.nasa.gov Dark Energy 73%Source: DFM & Widrow (in prep) Heavy Elements 0.03% Dark Matter 23% Atoms 4% Source: NASA / WMAP Science Team WMAP Year 7 (Larson et al. 2011) Dan Foreman-Mackey CCPP@NYU dfm.github.com
  22. 22. Dan Foreman-Mackey CCPP@NYU Credit: The Millennium Simulation Project dfm.github.com
  23. 23. Dan Foreman-Mackey CCPP@NYU Credit: The Millennium Simulation Project dfm.github.com
  24. 24. Dan Foreman-Mackey CCPP@NYU Credit: The Millennium Simulation Project dfm.github.com
  25. 25. Data in AstronomyDan Foreman-Mackey CCPP@NYU dfm.github.com
  26. 26. Data in Astronomy Credit: Jonathan Sick jonathansick.caDan Foreman-Mackey CCPP@NYU dfm.github.com
  27. 27. Data in Astronomy Credit: Jonathan Sick jonathansick.caDan Foreman-Mackey CCPP@NYU dfm.github.com
  28. 28. Data in Astronomy MegaCam: 340 MegaPixels Credit: Jonathan Sick jonathansick.caDan Foreman-Mackey CCPP@NYU dfm.github.com
  29. 29. Data in Astronomy MegaCam: 340 MegaPixels Credit: NASADan Foreman-Mackey CCPP@NYU dfm.github.com
  30. 30. Data in Astronomy Imaging Source: NASA / ESADan Foreman-Mackey CCPP@NYU dfm.github.com
  31. 31. Data in Astronomy Imaging Spectroscopy Source: NASA / ESADan Foreman-Mackey CCPP@NYU dfm.github.com
  32. 32. Data in Astronomy Source: Riaud & Schneider (2007) Imaging Spectroscopy Spectroscopy Source: NASA / ESADan Foreman-Mackey CCPP@NYU dfm.github.com
  33. 33. Data in Astronomy Source: Riaud & Schneider (2007) Imaging Spectroscopy Spectroscopy Source: NASA / ESADan Foreman-Mackey CCPP@NYU dfm.github.com
  34. 34. a Data in Astronomy lot is Open ofDan Foreman-Mackey CCPP@NYU dfm.github.com
  35. 35. a Data in Astronomy lot is Open of ! and there’s a lot of itDan Foreman-Mackey CCPP@NYU dfm.github.com
  36. 36. Data in Astronomy is Open 1990– 2000– 1997–2001 SDSSHubble archive.stsci.edu/hst sdss.org 2MASS www.ipac.caltech.edu/2mass GAIAPan-STARRS LSST PlannedDan Foreman-Mackey CCPP@NYU dfm.github.com
  37. 37. Data in Astronomy is Open 1990– 2000– 1997–2001 SDSSHubble archive.stsci.edu/hst sdss.org 2MASS www.ipac.caltech.edu/2mass GAIAPan-STARRS LSST PlannedDan Foreman-Mackey CCPP@NYU dfm.github.com
  38. 38. Data in Astronomy is Open 1990– 2000– 1997–2001 SDSSHubble archive.stsci.edu/hst sdss.org 2MASS www.ipac.caltech.edu/2mass GAIAPan-STARRS LSST PlannedDan Foreman-Mackey CCPP@NYU dfm.github.com
  39. 39. Where does Python fit in?Dan Foreman-Mackey CCPP@NYU dfm.github.com
  40. 40. Where does Python fit in?Dan Foreman-Mackey CCPP@NYU dfm.github.com
  41. 41. Where does Python fit in?Dan Foreman-Mackey CCPP@NYU dfm.github.com
  42. 42. Where does Python fit in?Dan Foreman-Mackey CCPP@NYU dfm.github.com
  43. 43. Where does Python fit in?Dan Foreman-Mackey CCPP@NYU dfm.github.com
  44. 44. Where does Python fit in? + Scientific Python StackDan Foreman-Mackey CCPP@NYU dfm.github.com
  45. 45. Where does Python fit in? + Scientific Python StackDan Foreman-Mackey CCPP@NYU dfm.github.com
  46. 46. Where does Python fit in? + Scientific Python StackDan Foreman-Mackey CCPP@NYU dfm.github.com
  47. 47. What about ? Easy Flexible Pythonic ScalableDan Foreman-Mackey CCPP@NYU dfm.github.com
  48. 48. Case StudiesDan Foreman-Mackey CCPP@NYU dfm.github.com
  49. 49. Case Studies Variable Stars in Stripe 82 SDSSDan Foreman-Mackey CCPP@NYU dfm.github.com
  50. 50. Case Studies Variable Stars in Stripe 82 SDSS 728 SESAR ET AL. Ses S07 Labela Ntot A 84 B 144 C 54 D 8 E 11 F 11 Source: Sesar et al. (2010) G 10 H 7 I 4 J 26 K 8Dan Foreman-Mackey CCPP@NYU L 3 dfm.github.com M 5
  51. 51. Case Studies Variable Stars in Stripe 82 SDSSDan Foreman-Mackey CCPP@NYU dfm.github.com
  52. 52. Case Studies Variable Stars in Stripe 82 SDSSDan Foreman-Mackey CCPP@NYU dfm.github.com
  53. 53. Case Studies Variable Stars in Stripe 82 SDSS 800k “Fields” ~ 12TB Imaging data > 1M “Target Stars”Dan Foreman-Mackey CCPP@NYU dfm.github.com
  54. 54. Case Studies Variable Stars in Stripe 82 SDSS Photons/Brightness TimeDan Foreman-Mackey CCPP@NYU dfm.github.com
  55. 55. Case Studies Variable Stars in Stripe 82 SDSS Y Np(X|⇥) = [(1 Pvar )pconst (X ↵ |⇥) + Pvar pvar (X ↵ |⇥)] ↵=1 Dan Foreman-Mackey CCPP@NYU dfm.github.com
  56. 56. Case Studies Variable Stars in Stripe 82 SDSS Y Np(X|⇥) = [(1 Pvar )pconst (X ↵ |⇥) + Pvar pvar (X ↵ |⇥)] ↵=1 Stars Dan Foreman-Mackey CCPP@NYU dfm.github.com
  57. 57. Case Studies Variable Stars in Stripe 82 SDSS YN p(X|⇥) = [(1 Pvar )pconst (X ↵ |⇥) + Pvar pvar (X ↵ |⇥)] ↵=1 Stars M Y M Ypconst ⌘ [(1 Pbad )pgood + Pbad pbad ] pvar ⌘ [(1 Pbad )pvar,good + Pbad pbad ] i=1 i=1 Dan Foreman-Mackey CCPP@NYU dfm.github.com
  58. 58. Case Studies Variable Stars in Stripe 82 SDSS YN p(X|⇥) = [(1 Pvar )pconst (X ↵ |⇥) + Pvar pvar (X ↵ |⇥)] ↵=1 Stars M Y M Ypconst ⌘ [(1 Pbad )pgood + Pbad pbad ] pvar ⌘ [(1 Pbad )pvar,good + Pbad pbad ] i=1 i=1 Runs Runs Dan Foreman-Mackey CCPP@NYU dfm.github.com
  59. 59. Case Studies Variable Stars in Stripe 82 SDSS YN p(X|⇥) = [(1 Pvar )pconst (X ↵ |⇥) + Pvar pvar (X ↵ |⇥)] ↵=1 Stars M Y M Ypconst ⌘ [(1 Pbad )pgood + Pbad pbad ] pvar ⌘ [(1 Pbad )pvar,good + Pbad pbad ] i=1 i=1 Runs Runs ⇤ ⇤ pgood ⌘ N (Ci↵ |fi0 f↵ , 2 i↵ + 2 i↵ ) pvar,good ⌘ N (Ci↵ |fi0 f↵ , 2 i↵ + 2 i↵ + ⌃2 ) var “Constant & Good” “Variable & Good” ⇤ pbad ⌘ N (Ci↵ |fi0 f↵ , 2 i↵ + 2 i↵ + ⌃2 ) bad “Bad” Dan Foreman-Mackey CCPP@NYU dfm.github.com
  60. 60. Case Studies Variable Stars in Stripe 82 SDSS YN p(X|⇥) = [(1 Pvar )pconst (X ↵ |⇥) + Pvar pvar (X ↵ |⇥)] ↵=1 Stars M Y M Ypconst ⌘ [(1 Pbad )pgood + Pbad pbad ] pvar ⌘ [(1 Pbad )pvar,good + Pbad pbad ] i=1 Npars = Nstars + Nruns + 6 i=1 Runs ~ 0 , ~ ⇤ , , ⌘, ⌃2 , PRuns , Pbad } 2 ⇥ = {f f var var , ⌃bad ⇤ ⇤ pgood ⌘ N (Ci↵ |fi0 f↵ , 2 i↵ + 2 i↵ ) pvar,good ⌘ N (Ci↵ |fi0 f↵ , 2 i↵ + 2 i↵ + ⌃2 ) var “Constant & Good” “Variable & Good” ⇤ pbad ⌘ N (Ci↵ |fi0 f↵ , 2 i↵ + 2 i↵ + ⌃2 ) bad “Bad” Dan Foreman-Mackey CCPP@NYU dfm.github.com
  61. 61. Case Studies Variable Stars in Stripe 82 SDSSDan Foreman-Mackey CCPP@NYU dfm.github.com
  62. 62. Case Studies Variable Stars in Stripe 82 SDSSDan Foreman-Mackey CCPP@NYU dfm.github.com
  63. 63. Case Studies Variable Stars in Stripe 82 SDSS N Y p(X|⇥) = [(1 Pvar )pconst (X ↵ |⇥) + Pvar pvar (X ↵ |⇥)] ↵=1Dan Foreman-Mackey CCPP@NYU dfm.github.com
  64. 64. Case Studies Variable Stars in Stripe 82 SDSS N X f ⇤ (t) = A0 + [An sin(!t) + Bn cos(!t)] n=1Dan Foreman-Mackey CCPP@NYU dfm.github.com
  65. 65. Case Studies CFHT4.7 Gigapixel mosaic of M31 Source: Jonathan Sick (Queen’s University)Dan Foreman-Mackey CCPP@NYU dfm.github.com
  66. 66. Dan Foreman-Mackey CCPP@NYU dfm.github.com Jonathan Sick Source: (Queen’s University)
  67. 67. Dan Foreman-Mackey CCPP@NYU dfm.github.com Jonathan Sick Source: (Queen’s University)
  68. 68. Dan Foreman-Mackey CCPP@NYU dfm.github.com Jonathan Sick Source: (Queen’s University)
  69. 69. Case Studies CFHTCosmic ray removal Flat-fielding Sky subtraction ... Mosaic making MongoDB Persistent Metadata + GeoSpatial Indexing img1.fits img2.fits ... img4000.fits Source: Jonathan Sick (Queen’s University) Dan Foreman-Mackey CCPP@NYU dfm.github.com
  70. 70. Astrometry.net Case StudiesDan Foreman-Mackey CCPP@NYU dfm.github.com
  71. 71. Astrometry.net Case StudiesDan Foreman-Mackey CCPP@NYU dfm.github.com
  72. 72. Dan Foreman-Mackey CCPP@NYU dfm.github.com
  73. 73. Dan Foreman-Mackey CCPP@NYU dfm.github.com
  74. 74. Dan Foreman-Mackey CCPP@NYU dfm.github.com
  75. 75. Crowdsourcing Comet Holmes Case Studies Lang & Hogg (2011) ~2500 JPGsDan Foreman-Mackey CCPP@NYU dfm.github.com
  76. 76. Crowdsourcing Comet Holmes Case StudiesDan Foreman-Mackey CCPP@NYU dfm.github.com
  77. 77. Crowdsourcing Comet Holmes Case StudiesDan Foreman-Mackey CCPP@NYU dfm.github.com
  78. 78. Crowdsourcing Comet Holmes Case StudiesDan Foreman-Mackey CCPP@NYU dfm.github.com
  79. 79. Crowdsourcing Comet Holmes Case Studies Source: Lang & Hogg (2011)Dan Foreman-Mackey CCPP@NYU dfm.github.com
  80. 80. Crowdsourcing Comet Holmes Case Studies Source: Lang & Hogg (2011)Dan Foreman-Mackey CCPP@NYU dfm.github.com
  81. 81. Crowdsourcing Comet Holmes Case Studies Source: Lang & Hogg (2011)github.com/dfm/MarkovPyDan Foreman-Mackey CCPP@NYU dfm.github.com
  82. 82. Crowdsourcing Comet Holmes Case StudiesDan Foreman-Mackey CCPP@NYU dfm.github.com
  83. 83. Growing Datasets UserInteraction Number Data And MuchCrunching Management More... ( )Dan Foreman-Mackey CCPP@NYU dfm.github.com
  84. 84. Growing Datasets UserInteraction Easy! Number Data And MuchCrunching Management More... ( )Dan Foreman-Mackey CCPP@NYU dfm.github.com
  85. 85. Growing Big Questions Datasets UserInteraction Easy! Number Data And MuchCrunching Management More... ( )Dan Foreman-Mackey CCPP@NYU dfm.github.com
  86. 86. Dan Foreman-Mackey Center for Cosmology & Particle Physics (NYU) dfm.github.com @__dfm__Dan Foreman-Mackey CCPP@NYU dfm.github.com

×