0
LOFAR Transient Detection Pipeline           Gijs Molenaar          gijs@pythonic.nl
Agenda●   Transient detection●   Pipeline layout●   Software
Transients Detection●   Static sources are boring●   Transient – something that changes●   Huge amount of data – automation
Image data
The data●   1 datacube per second●   10 frequency bands●   In the future 10 images per second●   In the future 4 different...
Transient detection Pipeline    Extract metadata   Quality check    Source extraction                                     ...
●   The glue●   Distributing computation●   Image processing●   Statistics●   Source extraction●   Database interactions
Distributed Computation●   Home made libary●   SSH based●   Difficult to debug●   Difficult to profile●   Doing research o...
Database●   Move calculation to the data●   Highly structured●   Independent data●   Naturally separable by sky coordinate...
MonetDB●   Relational●   Column store DB●   Fast●   Auto tuning!●   Developers next door (CWI)
Challenges●   Debugging queries●   MonetDB still in active development
INSERT INTO tempbasesources(xtrsrc_id,datapoints,I_peak_sum,I_peak_sq_sum,weight_peak_sum,weight_I_peak_sum,weight_I_peak_...
MonetDB and Python●   We maintain the MonetDB Python API●   http://pypi.python.org/pypi/python-monetdb/●   Problems? Ask m...
Djonet●   MonetDB backend for Django●   https://github.com/gijzelaerr/djonet●   brew install monetdb●   pip install python...
VO events●   Standardized language●   Report observations of    astronomical events●   Hey world, check this supernova    ...
Visualisation●   Web interface●   Django!●   Not public (yet)
More●   http://www.transientskp.org/●   http://www.lofar.org/●   http://www.aartfaac.org/
Questions?
Upcoming SlideShare
Loading in...5
×

LOFAR - finding transients in the radio spectrum

357

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
357
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "LOFAR - finding transients in the radio spectrum"

  1. 1. LOFAR Transient Detection Pipeline Gijs Molenaar gijs@pythonic.nl
  2. 2. Agenda● Transient detection● Pipeline layout● Software
  3. 3. Transients Detection● Static sources are boring● Transient – something that changes● Huge amount of data – automation
  4. 4. Image data
  5. 5. The data● 1 datacube per second● 10 frequency bands● In the future 10 images per second● In the future 4 different polarization● Non stop
  6. 6. Transient detection Pipeline Extract metadata Quality check Source extraction Source extraction Association Detection ClassificationDatabase
  7. 7. ● The glue● Distributing computation● Image processing● Statistics● Source extraction● Database interactions
  8. 8. Distributed Computation● Home made libary● SSH based● Difficult to debug● Difficult to profile● Doing research on Celery and Hadoop
  9. 9. Database● Move calculation to the data● Highly structured● Independent data● Naturally separable by sky coordinates● ~100 TB/year● 10.000 insert/second
  10. 10. MonetDB● Relational● Column store DB● Fast● Auto tuning!● Developers next door (CWI)
  11. 11. Challenges● Debugging queries● MonetDB still in active development
  12. 12. INSERT INTO tempbasesources(xtrsrc_id,datapoints,I_peak_sum,I_peak_sq_sum,weight_peak_sum,weight_I_peak_sum,weight_I_peak_sq_sum)SELECT b0.xtrsrc_id,b0.datapoints+ 1 AS datapoints,b0.I_peak_sum+ x0.I_peak AS i_peak_sum,b0.I_peak_sq_sum+ x0.I_peak * x0.I_peak AS i_peak_sq_sum,b0.weight_peak_sum+ 1 / (x0.I_peak_err * x0.I_peak_err) AS weight_peak_sum,b0.weight_I_peak_sum+ x0.I_peak / (x0.I_peak_err * x0.I_peak_err)AS weight_i_peak_sum,b0.weight_I_peak_sq_sum+ x0.I_peak * x0.I_peak / (x0.I_peak_err * x0.I_peak_err)AS weight_i_peak_sq_sumFROM basesources b0,extractedsources x0WHERE x0.image_id = @imageidAND b0.zone BETWEEN CAST(FLOOR((x0.decl - @theta) / x0.zoneheight) AS INTEGER)AND CAST(FLOOR((x0.decl + @theta) / x0.zoneheight) AS INTEGER)AND ASIN(SQRT((x0.x - b0.x)*(x0.x - b0.x)+(x0.y - b0.y)*(x0.y - b0.y)+(x0.z - b0.z)*(x0.z - b0.z))/2)/SQRT(x0.ra_err * x0.ra_err + b0.ra_err * b0.ra_err+x0.decl_err * x0.decl_err + b0.decl_err * b0.decl_err)< @assoc_r;
  13. 13. MonetDB and Python● We maintain the MonetDB Python API● http://pypi.python.org/pypi/python-monetdb/● Problems? Ask me :)
  14. 14. Djonet● MonetDB backend for Django● https://github.com/gijzelaerr/djonet● brew install monetdb● pip install python-monetdb djonet● Contributions are welcome!
  15. 15. VO events● Standardized language● Report observations of astronomical events● Hey world, check this supernova out over there● http://comet.transientskp.org
  16. 16. Visualisation● Web interface● Django!● Not public (yet)
  17. 17. More● http://www.transientskp.org/● http://www.lofar.org/● http://www.aartfaac.org/
  18. 18. Questions?
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×