Lofar python meetup jan9 2013

460 views
286 views

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
460
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Lofar python meetup jan9 2013

  1. 1. LOFAR & Python Gijs Molenaar gijs@pythonic.nl
  2. 2. Agenda● LOFAR● AARTFAAC / TKP● Python● MonetDB
  3. 3. Non Python stuff
  4. 4. What is LOFAR
  5. 5. What is LOFAR● Low-Frequency Array● Distributed sensor network● No dishes● Simple and cheap antennas● Omni-directional● Concurrent observations● Software telescope – Flexible – Requires supercomputer
  6. 6. What is LOFAR Biggest Radio telescope array in the world● 7000 antennas● 10 to 80 MHz (LBA)● 120 to 240 MHz (HBA)● Multiple small stations● NL, DE, UK, FR, SE● New (unique) design
  7. 7. New design● Many unsolved problems● Multi-terabit/s data scale● How to – Process – Store – Distribute – Make results useful● Small scale example for SKA
  8. 8. Usage● Epoch of re-ionization● Cosmic rays● Extragalactic surveys● Transients● Pulsars
  9. 9. Correlation
  10. 10. Correlation● IBM blue Gene/P● Groningen● Alotta computer power● Will be replaced with FPGAs (for our purpose)● No Python
  11. 11. Example output – Cygnus A
  12. 12. Actual data
  13. 13. Transient Key Project● Static sources are boring● Transient – something that changes● Detect sources● Store extracted information● Associate with existing observations● Did something change?● This is where the Python fun starts!
  14. 14. ● The glue● Distributing computation● Image processing● Source extraction● Source association● Database interactions
  15. 15. Why Python● A lot of existing code● Good (scientific) libraries● Good astronomical libraries● Easy to use● Get it working first, then fast● NumPy is quite fast when used correctly
  16. 16. What do we use?● Python● NumPy / SciPy● Django● MonetDB● Pyrap (casacore wrapper) – Does all astro shizzle you can imagine
  17. 17. Distributed Computation● Home made (ASTRON)● SSH based● Calling remote scripts● Shared code base (NFS)● Protocol for passing arguments and results● Uses logging module a lot● Difficult to debug● Difficult to profile● Hadoop probably better
  18. 18. The Lofar Cluster● 100 nodes● 24 cores each● 20 TB raid5 storage● 40 Gbps InfiniBand● But for AARTFAAC we build our own● Eventually
  19. 19. Main loop● Extract source properties● Store properties● Do association● Detect transients
  20. 20. The data● 1 datacube per second● 10 frequency bands● In the future 10 images per second● In the future 4 different polarization● Non stop
  21. 21. Source extraction
  22. 22. Source extraction & NumPy● Calculate average noise● Subtract● Find islands● Fit gauss
  23. 23. Store the data
  24. 24. Database● Move calculation to the data● Independent data● Naturally separable by sky coordinates● ~100 TB/year● 10.000 insert/second
  25. 25. MonetDB● Dont believe the hype● NoNoSQL● Relational● Column store DB● Fast● Auto tuning!● Developers next door (CWI)● Cross research funding
  26. 26. INSERT INTO tempbasesources(xtrsrc_id,datapoints,I_peak_sum,I_peak_sq_sum,weight_peak_sum,weight_I_peak_sum,weight_I_peak_sq_sum)SELECT b0.xtrsrc_id,b0.datapoints+ 1 AS datapoints,b0.I_peak_sum+ x0.I_peak AS i_peak_sum,b0.I_peak_sq_sum+ x0.I_peak * x0.I_peak AS i_peak_sq_sum,b0.weight_peak_sum+ 1 / (x0.I_peak_err * x0.I_peak_err) AS weight_peak_sum,b0.weight_I_peak_sum+ x0.I_peak / (x0.I_peak_err * x0.I_peak_err)AS weight_i_peak_sum,b0.weight_I_peak_sq_sum+ x0.I_peak * x0.I_peak / (x0.I_peak_err * x0.I_peak_err)AS weight_i_peak_sq_sumFROM basesources b0,extractedsources x0WHERE x0.image_id = @imageidAND b0.zone BETWEEN CAST(FLOOR((x0.decl - @theta) / x0.zoneheight) AS INTEGER)AND CAST(FLOOR((x0.decl + @theta) / x0.zoneheight) AS INTEGER)AND ASIN(SQRT((x0.x - b0.x)*(x0.x - b0.x)+(x0.y - b0.y)*(x0.y - b0.y)+(x0.z - b0.z)*(x0.z - b0.z))/2)/SQRT(x0.ra_err * x0.ra_err + b0.ra_err * b0.ra_err+x0.decl_err * x0.decl_err + b0.decl_err * b0.decl_err)< @assoc_r;
  27. 27. MonetDB vs MySQL
  28. 28. MonetDB and Python● We maintain the MonetDB Python API● http://pypi.python.org/pypi/python-monetdb/● Problems? Ask me :)
  29. 29. Challenges● Debugging queries● MonetDB still in active development
  30. 30. Djonet● MonetDB backend for Django● https://github.com/gijzelaerr/djonet● brew install monetdb● pip install python-monetdb djonet● Contributions are welcome!
  31. 31. VO events● Standardized language● Report observations of astronomical events● Hey world, check this supernova out over there● http://comet.transientskp.org
  32. 32. VOevents
  33. 33. Visualisation● Web interface● Django!● Not public (yet)
  34. 34. Source code?● No sorry.● Maybe some day● Legal issues, bureaucracy● UvA, Astron●
  35. 35. More● http://www.transientskp.org/● http://www.lofar.org/● http://www.aartfaac.org/
  36. 36. Questions?

×