Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
LOFAR Transient Detection Pipeline           Gijs Molenaar          gijs@pythonic.nl
Agenda●   Transient detection●   Pipeline layout●   Software
Transients Detection●   Static sources are boring●   Transient – something that changes●   Huge amount of data – automation
Image data
The data●   1 datacube per second●   10 frequency bands●   In the future 10 images per second●   In the future 4 different...
Transient detection Pipeline    Extract metadata   Quality check    Source extraction                                     ...
●   The glue●   Distributing computation●   Image processing●   Statistics●   Source extraction●   Database interactions
Distributed Computation●   Home made libary●   SSH based●   Difficult to debug●   Difficult to profile●   Doing research o...
Database●   Move calculation to the data●   Highly structured●   Independent data●   Naturally separable by sky coordinate...
MonetDB●   Relational●   Column store DB●   Fast●   Auto tuning!●   Developers next door (CWI)
Challenges●   Debugging queries●   MonetDB still in active development
INSERT INTO tempbasesources(xtrsrc_id,datapoints,I_peak_sum,I_peak_sq_sum,weight_peak_sum,weight_I_peak_sum,weight_I_peak_...
MonetDB and Python●   We maintain the MonetDB Python API●   http://pypi.python.org/pypi/python-monetdb/●   Problems? Ask m...
Djonet●   MonetDB backend for Django●   https://github.com/gijzelaerr/djonet●   brew install monetdb●   pip install python...
VO events●   Standardized language●   Report observations of    astronomical events●   Hey world, check this supernova    ...
Visualisation●   Web interface●   Django!●   Not public (yet)
More●   http://www.transientskp.org/●   http://www.lofar.org/●   http://www.aartfaac.org/
Questions?
Upcoming SlideShare
Loading in …5
×

LOFAR - finding transients in the radio spectrum

707 views

Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

LOFAR - finding transients in the radio spectrum

  1. 1. LOFAR Transient Detection Pipeline Gijs Molenaar gijs@pythonic.nl
  2. 2. Agenda● Transient detection● Pipeline layout● Software
  3. 3. Transients Detection● Static sources are boring● Transient – something that changes● Huge amount of data – automation
  4. 4. Image data
  5. 5. The data● 1 datacube per second● 10 frequency bands● In the future 10 images per second● In the future 4 different polarization● Non stop
  6. 6. Transient detection Pipeline Extract metadata Quality check Source extraction Source extraction Association Detection ClassificationDatabase
  7. 7. ● The glue● Distributing computation● Image processing● Statistics● Source extraction● Database interactions
  8. 8. Distributed Computation● Home made libary● SSH based● Difficult to debug● Difficult to profile● Doing research on Celery and Hadoop
  9. 9. Database● Move calculation to the data● Highly structured● Independent data● Naturally separable by sky coordinates● ~100 TB/year● 10.000 insert/second
  10. 10. MonetDB● Relational● Column store DB● Fast● Auto tuning!● Developers next door (CWI)
  11. 11. Challenges● Debugging queries● MonetDB still in active development
  12. 12. INSERT INTO tempbasesources(xtrsrc_id,datapoints,I_peak_sum,I_peak_sq_sum,weight_peak_sum,weight_I_peak_sum,weight_I_peak_sq_sum)SELECT b0.xtrsrc_id,b0.datapoints+ 1 AS datapoints,b0.I_peak_sum+ x0.I_peak AS i_peak_sum,b0.I_peak_sq_sum+ x0.I_peak * x0.I_peak AS i_peak_sq_sum,b0.weight_peak_sum+ 1 / (x0.I_peak_err * x0.I_peak_err) AS weight_peak_sum,b0.weight_I_peak_sum+ x0.I_peak / (x0.I_peak_err * x0.I_peak_err)AS weight_i_peak_sum,b0.weight_I_peak_sq_sum+ x0.I_peak * x0.I_peak / (x0.I_peak_err * x0.I_peak_err)AS weight_i_peak_sq_sumFROM basesources b0,extractedsources x0WHERE x0.image_id = @imageidAND b0.zone BETWEEN CAST(FLOOR((x0.decl - @theta) / x0.zoneheight) AS INTEGER)AND CAST(FLOOR((x0.decl + @theta) / x0.zoneheight) AS INTEGER)AND ASIN(SQRT((x0.x - b0.x)*(x0.x - b0.x)+(x0.y - b0.y)*(x0.y - b0.y)+(x0.z - b0.z)*(x0.z - b0.z))/2)/SQRT(x0.ra_err * x0.ra_err + b0.ra_err * b0.ra_err+x0.decl_err * x0.decl_err + b0.decl_err * b0.decl_err)< @assoc_r;
  13. 13. MonetDB and Python● We maintain the MonetDB Python API● http://pypi.python.org/pypi/python-monetdb/● Problems? Ask me :)
  14. 14. Djonet● MonetDB backend for Django● https://github.com/gijzelaerr/djonet● brew install monetdb● pip install python-monetdb djonet● Contributions are welcome!
  15. 15. VO events● Standardized language● Report observations of astronomical events● Hey world, check this supernova out over there● http://comet.transientskp.org
  16. 16. Visualisation● Web interface● Django!● Not public (yet)
  17. 17. More● http://www.transientskp.org/● http://www.lofar.org/● http://www.aartfaac.org/
  18. 18. Questions?

×