Python User Group November 2013, SQL(alchemy) and transient detection

683 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
683
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Python User Group November 2013, SQL(alchemy) and transient detection

  1. 1. Transient Detection and SQL Gijs Molenaar ! Anton Pannekoek Institute (UvA) ! @gijzelaerr gijs@pythonic.nl
  2. 2. LOFAR • Radio telescope • ~10–240 MHz • Software telescope • omni-directional
  3. 3. AARTFAAC • Lofar sub project • 24/7 full sky transient detection • ‘Changes’ • 99% Python code
  4. 4. Transients • Super-nova • Gamma-ray burst • Pulsars • exo-planets • ?
  5. 5. Images • 10-240 MHz • Split in 10 bands • 1 image cube per second • future 10 images per second
  6. 6. Data • Source • Position • Flux (intensity) • Shape (2d gaussian)
  7. 7. Data rate • 500 - 5000 inserts per second • Higher in other modes
  8. 8. Calculations • Associate in frequency • Associate in time
  9. 9. Quite a lot of data • move calculations to data (base)
  10. 10. nonosql • Structured data • Natural sharding by sky position • MonetDB
  11. 11. Why MonetDB • Dutch (CWI) • Fast with huge datasets • Developers next door
  12. 12. calculations to data • No moving around data • one huge big monolithic query per ‘operation’
  13. 13. Example
  14. 14. HORRIBLE • Huge monolithic queries • hard to read • hard to reason about • hard to test • hard to maintain • hard to port
  15. 15. Solution? • Functions?
  16. 16. NO! • DB revisions, versioning hell • For every code change new version • Schema migrations • Init functions on startup? • DB specific
  17. 17. SOLUTION
  18. 18. Why? • Object Relational Mapper • Better Query Builder Language! • Support multiple database dialect • Functions per dialect ! • SQLAlchemy -> SQL is like C -> assembler
  19. 19. example
  20. 20. Why not earlier? • Started by non programmers • No MonetDB dialect for SQLAlchemy
  21. 21. sqlalchemy-monetdb • pip install sqlalchemy-monetdb • https://github.com/gijzelaerr/sqlalchemy-monetdb • https://github.com/gijzelaerr/djonet • (banana)
  22. 22. Questions?

×