LOFAR - finding transients in the radio spectrum

418
-1

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
418
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

LOFAR - finding transients in the radio spectrum

  1. 1. LOFAR Transient Detection Pipeline Gijs Molenaar gijs@pythonic.nl
  2. 2. Agenda● Transient detection● Pipeline layout● Software
  3. 3. Transients Detection● Static sources are boring● Transient – something that changes● Huge amount of data – automation
  4. 4. Image data
  5. 5. The data● 1 datacube per second● 10 frequency bands● In the future 10 images per second● In the future 4 different polarization● Non stop
  6. 6. Transient detection Pipeline Extract metadata Quality check Source extraction Source extraction Association Detection ClassificationDatabase
  7. 7. ● The glue● Distributing computation● Image processing● Statistics● Source extraction● Database interactions
  8. 8. Distributed Computation● Home made libary● SSH based● Difficult to debug● Difficult to profile● Doing research on Celery and Hadoop
  9. 9. Database● Move calculation to the data● Highly structured● Independent data● Naturally separable by sky coordinates● ~100 TB/year● 10.000 insert/second
  10. 10. MonetDB● Relational● Column store DB● Fast● Auto tuning!● Developers next door (CWI)
  11. 11. Challenges● Debugging queries● MonetDB still in active development
  12. 12. INSERT INTO tempbasesources(xtrsrc_id,datapoints,I_peak_sum,I_peak_sq_sum,weight_peak_sum,weight_I_peak_sum,weight_I_peak_sq_sum)SELECT b0.xtrsrc_id,b0.datapoints+ 1 AS datapoints,b0.I_peak_sum+ x0.I_peak AS i_peak_sum,b0.I_peak_sq_sum+ x0.I_peak * x0.I_peak AS i_peak_sq_sum,b0.weight_peak_sum+ 1 / (x0.I_peak_err * x0.I_peak_err) AS weight_peak_sum,b0.weight_I_peak_sum+ x0.I_peak / (x0.I_peak_err * x0.I_peak_err)AS weight_i_peak_sum,b0.weight_I_peak_sq_sum+ x0.I_peak * x0.I_peak / (x0.I_peak_err * x0.I_peak_err)AS weight_i_peak_sq_sumFROM basesources b0,extractedsources x0WHERE x0.image_id = @imageidAND b0.zone BETWEEN CAST(FLOOR((x0.decl - @theta) / x0.zoneheight) AS INTEGER)AND CAST(FLOOR((x0.decl + @theta) / x0.zoneheight) AS INTEGER)AND ASIN(SQRT((x0.x - b0.x)*(x0.x - b0.x)+(x0.y - b0.y)*(x0.y - b0.y)+(x0.z - b0.z)*(x0.z - b0.z))/2)/SQRT(x0.ra_err * x0.ra_err + b0.ra_err * b0.ra_err+x0.decl_err * x0.decl_err + b0.decl_err * b0.decl_err)< @assoc_r;
  13. 13. MonetDB and Python● We maintain the MonetDB Python API● http://pypi.python.org/pypi/python-monetdb/● Problems? Ask me :)
  14. 14. Djonet● MonetDB backend for Django● https://github.com/gijzelaerr/djonet● brew install monetdb● pip install python-monetdb djonet● Contributions are welcome!
  15. 15. VO events● Standardized language● Report observations of astronomical events● Hey world, check this supernova out over there● http://comet.transientskp.org
  16. 16. Visualisation● Web interface● Django!● Not public (yet)
  17. 17. More● http://www.transientskp.org/● http://www.lofar.org/● http://www.aartfaac.org/
  18. 18. Questions?
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×