Transient Detection
and SQL
Gijs Molenaar
!

Anton Pannekoek Institute (UvA)
!

@gijzelaerr
gijs@pythonic.nl
LOFAR
•

Radio telescope

•

~10–240 MHz

•

Software telescope

•

omni-directional
AARTFAAC
•

Lofar sub project

•

24/7 full sky transient
detection

•

‘Changes’

•

99% Python code
Transients
•

Super-nova

•

Gamma-ray burst

•

Pulsars

•

exo-planets

•

?
Images
•

10-240 MHz

•

Split in 10 bands

•

1 image cube per second

•

future 10 images per second
Data
•

Source

•

Position

•

Flux (intensity)

•

Shape (2d gaussian)
Data rate

•

500 - 5000 inserts per second

•

Higher in other modes
Calculations

•

Associate in frequency

•

Associate in time
Quite a lot of data

•

move calculations to data (base)
nonosql
•

Structured data

•

Natural sharding by
sky position

•

MonetDB
Why MonetDB
•

Dutch (CWI)

•

Fast with huge
datasets

•

Developers next door
calculations to data

•

No moving around data

•

one huge big monolithic query per ‘operation’
Example
HORRIBLE
•

Huge monolithic queries

•

hard to read

•

hard to reason about

•

hard to test

•

hard to maintain

•

ha...
Solution?

•

Functions?
NO!
•

DB revisions, versioning hell

•

For every code change new version

•

Schema migrations

•

Init functions on sta...
SOLUTION
Why?
•

Object Relational Mapper

•

Better Query Builder Language!

•

Support multiple database dialect

•

Functions pe...
example
Why not earlier?

•

Started by non programmers

•

No MonetDB dialect for SQLAlchemy
sqlalchemy-monetdb
•

pip install sqlalchemy-monetdb

•

https://github.com/gijzelaerr/sqlalchemy-monetdb

•

https://gith...
Questions?
Upcoming SlideShare
Loading in...5
×

Python User Group November 2013, SQL(alchemy) and transient detection

391

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
391
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Python User Group November 2013, SQL(alchemy) and transient detection"

  1. 1. Transient Detection and SQL Gijs Molenaar ! Anton Pannekoek Institute (UvA) ! @gijzelaerr gijs@pythonic.nl
  2. 2. LOFAR • Radio telescope • ~10–240 MHz • Software telescope • omni-directional
  3. 3. AARTFAAC • Lofar sub project • 24/7 full sky transient detection • ‘Changes’ • 99% Python code
  4. 4. Transients • Super-nova • Gamma-ray burst • Pulsars • exo-planets • ?
  5. 5. Images • 10-240 MHz • Split in 10 bands • 1 image cube per second • future 10 images per second
  6. 6. Data • Source • Position • Flux (intensity) • Shape (2d gaussian)
  7. 7. Data rate • 500 - 5000 inserts per second • Higher in other modes
  8. 8. Calculations • Associate in frequency • Associate in time
  9. 9. Quite a lot of data • move calculations to data (base)
  10. 10. nonosql • Structured data • Natural sharding by sky position • MonetDB
  11. 11. Why MonetDB • Dutch (CWI) • Fast with huge datasets • Developers next door
  12. 12. calculations to data • No moving around data • one huge big monolithic query per ‘operation’
  13. 13. Example
  14. 14. HORRIBLE • Huge monolithic queries • hard to read • hard to reason about • hard to test • hard to maintain • hard to port
  15. 15. Solution? • Functions?
  16. 16. NO! • DB revisions, versioning hell • For every code change new version • Schema migrations • Init functions on startup? • DB specific
  17. 17. SOLUTION
  18. 18. Why? • Object Relational Mapper • Better Query Builder Language! • Support multiple database dialect • Functions per dialect ! • SQLAlchemy -> SQL is like C -> assembler
  19. 19. example
  20. 20. Why not earlier? • Started by non programmers • No MonetDB dialect for SQLAlchemy
  21. 21. sqlalchemy-monetdb • pip install sqlalchemy-monetdb • https://github.com/gijzelaerr/sqlalchemy-monetdb • https://github.com/gijzelaerr/djonet • (banana)
  22. 22. Questions?
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×