Python User Group November 2013, SQL(alchemy) and transient detection
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Python User Group November 2013, SQL(alchemy) and transient detection

on

  • 473 views

 

Statistics

Views

Total Views
473
Views on SlideShare
468
Embed Views
5

Actions

Likes
0
Downloads
1
Comments
0

2 Embeds 5

http://www.linkedin.com 4
https://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Python User Group November 2013, SQL(alchemy) and transient detection Presentation Transcript

  • 1. Transient Detection and SQL Gijs Molenaar ! Anton Pannekoek Institute (UvA) ! @gijzelaerr gijs@pythonic.nl
  • 2. LOFAR • Radio telescope • ~10–240 MHz • Software telescope • omni-directional
  • 3. AARTFAAC • Lofar sub project • 24/7 full sky transient detection • ‘Changes’ • 99% Python code
  • 4. Transients • Super-nova • Gamma-ray burst • Pulsars • exo-planets • ?
  • 5. Images • 10-240 MHz • Split in 10 bands • 1 image cube per second • future 10 images per second
  • 6. Data • Source • Position • Flux (intensity) • Shape (2d gaussian)
  • 7. Data rate • 500 - 5000 inserts per second • Higher in other modes
  • 8. Calculations • Associate in frequency • Associate in time
  • 9. Quite a lot of data • move calculations to data (base)
  • 10. nonosql • Structured data • Natural sharding by sky position • MonetDB
  • 11. Why MonetDB • Dutch (CWI) • Fast with huge datasets • Developers next door
  • 12. calculations to data • No moving around data • one huge big monolithic query per ‘operation’
  • 13. Example
  • 14. HORRIBLE • Huge monolithic queries • hard to read • hard to reason about • hard to test • hard to maintain • hard to port
  • 15. Solution? • Functions?
  • 16. NO! • DB revisions, versioning hell • For every code change new version • Schema migrations • Init functions on startup? • DB specific
  • 17. SOLUTION
  • 18. Why? • Object Relational Mapper • Better Query Builder Language! • Support multiple database dialect • Functions per dialect ! • SQLAlchemy -> SQL is like C -> assembler
  • 19. example
  • 20. Why not earlier? • Started by non programmers • No MonetDB dialect for SQLAlchemy
  • 21. sqlalchemy-monetdb • pip install sqlalchemy-monetdb • https://github.com/gijzelaerr/sqlalchemy-monetdb • https://github.com/gijzelaerr/djonet • (banana)
  • 22. Questions?