This document discusses LOFAR (Low-Frequency Array), a radio telescope array in the Netherlands, and how Python is used for data processing tasks related to LOFAR such as transient detection. It describes how Python is used for distributed computation, image processing, source extraction from datacubes, database storage in MonetDB, and creating VOEvents to report astronomical observations. Python libraries like NumPy, SciPy, Django, and custom libraries are crucial to the data processing pipeline.

1. LOFAR & Python
Gijs Molenaar
gijs@pythonic.nl

2. Agenda
● LOFAR
● AARTFAAC / TKP
● Python
● MonetDB

3. Non Python stuff

4. What is LOFAR

5. What is LOFAR
● Low-Frequency Array
● Distributed sensor network
● No dishes
● Simple and cheap antenna's
● Omni-directional
● Concurrent observations
● Software telescope
– Flexible
– Requires supercomputer

6. What is LOFAR
Biggest Radio telescope
array in the world
● 7000 antennas
● 10 to 80 MHz (LBA)
● 120 to 240 MHz (HBA)
● Multiple small stations
● NL, DE, UK, FR, SE
● New (unique) design

7. New design
● Many unsolved problems
● Multi-terabit/s data scale
● How to
– Process
– Store
– Distribute
– Make results useful
● Small scale example for SKA

8. Usage
● Epoch of re-ionization
● Cosmic rays
● Extragalactic surveys
● Transients
● Pulsars

10. Correlation

11. Correlation
● IBM blue Gene/P
● Groningen
● Alotta computer power
● Will be replaced with
FPGA's (for our
purpose)
● No Python

12. Example output – Cygnus A

13. Actual data

14. Transient Key Project
● Static sources are boring
●
Transient – something that changes
● Detect sources
●
Store extracted information
●
Associate with existing observations
●
Did something change?
●
This is where the Python fun starts!

15. ● The glue
● Distributing computation
● Image processing
● Source extraction
● Source association
● Database interactions

16. Why Python
● A lot of existing code
● Good (scientific) libraries
● Good astronomical libraries
● Easy to use
● Get it working first, then fast
● NumPy is quite fast when used correctly

17. What do we use?
● Python
● NumPy / SciPy
● Django
● MonetDB
● Pyrap (casacore wrapper)
– Does all astro shizzle you can imagine

18. Distributed Computation
● Home made (ASTRON)
●
SSH based
●
Calling remote scripts
● Shared code base (NFS)
● Protocol for passing arguments and results
●
Uses logging module a lot
●
Difficult to debug
● Difficult to profile
●
Hadoop probably better

19. The Lofar Cluster
● 100 nodes
● 24 cores each
● 20 TB raid5 storage
● 40 Gbps InfiniBand
● But for AARTFAAC we build our own
● Eventually

20. Main loop
● Extract source properties
● Store properties
● Do association
● Detect transients

21. The data
● 1 datacube per second
● 10 frequency bands
● In the future 10 images per second
● In the future 4 different polarization
● Non stop

22. Source extraction

23. Source extraction & NumPy
● Calculate average noise
● Subtract
● Find 'islands'
● Fit gauss

24. Store the data

25. Database
● Move calculation to the data
● Independent data
● Naturally separable by sky coordinates
● ~100 TB/year
● 10.000 insert/second

26. MonetDB
● Don't believe the hype
● NoNoSQL
● Relational
● Column store DB
● Fast
● Auto tuning!
● Developers next door (CWI)
● Cross research funding

27. INSERT INTO tempbasesources
(xtrsrc_id
,datapoints
,I_peak_sum
,I_peak_sq_sum
,weight_peak_sum
,weight_I_peak_sum
,weight_I_peak_sq_sum
)
SELECT b0.xtrsrc_id
,b0.datapoints
+ 1 AS datapoints
,b0.I_peak_sum
+ x0.I_peak AS i_peak_sum
,b0.I_peak_sq_sum
+ x0.I_peak * x0.I_peak AS i_peak_sq_sum
,b0.weight_peak_sum
+ 1 / (x0.I_peak_err * x0.I_peak_err) AS weight_peak_sum
,b0.weight_I_peak_sum
+ x0.I_peak / (x0.I_peak_err * x0.I_peak_err)
AS weight_i_peak_sum
,b0.weight_I_peak_sq_sum
+ x0.I_peak * x0.I_peak / (x0.I_peak_err * x0.I_peak_err)
AS weight_i_peak_sq_sum
FROM basesources b0
,extractedsources x0
WHERE x0.image_id = @imageid
AND b0.zone BETWEEN CAST(FLOOR((x0.decl - @theta) / x0.zoneheight
) AS INTEGER)
AND CAST(FLOOR((x0.decl + @theta) / x0.zoneheight
) AS INTEGER)
AND ASIN(SQRT((x0.x - b0.x)*(x0.x - b0.x)
+(x0.y - b0.y)*(x0.y - b0.y)
+(x0.z - b0.z)*(x0.z - b0.z)
)/2
)
/
SQRT(x0.ra_err * x0.ra_err + b0.ra_err * b0.ra_err
+x0.decl_err * x0.decl_err + b0.decl_err * b0.decl_err)
< @assoc_r;

28. MonetDB vs MySQL

29. MonetDB and Python
● We maintain the MonetDB Python API
● http://pypi.python.org/pypi/python-monetdb/
● Problems? Ask me :)

30. Challenges
● Debugging queries
● MonetDB still in active development

31. Djonet
● MonetDB backend for Django
● https://github.com/gijzelaerr/djonet
● brew install monetdb
● pip install python-monetdb djonet
● Contributions are welcome!

32. VO events
● Standardized language
● Report observations of astronomical events
● Hey world, check this supernova out over
there
● http://comet.transientskp.org

33. VOevents

34. Visualisation
● Web interface
● Django!
● Not public (yet)

35. Source code?
● No sorry.
● Maybe some day
● Legal issues, bureaucracy
● UvA, Astron
●

36. More
● http://www.transientskp.org/
● http://www.lofar.org/
● http://www.aartfaac.org/

37. Questions?