The Weather of the Century Part 3: Visualization

593
-1

Published on

MongoDB natively supports geospatial indexing and querying, and it integrates easily with open source visualization tools. In this presentation, learn high-performance techniques for querying and retrieving geospatial data, and how to create a rich visual representation of global weather data using Python, Monary, and Matplotlib.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
593
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
16
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

The Weather of the Century Part 3: Visualization

  1. 1. A. Jesse Jiryu Davis #MongoDBWorld The Weather of the Century! Part III:! Visualization Senior Python Engineer, MongoDB
  2. 2. Serious MongoDB Talk
  3. 3. Serious MongoDB Talk Database
  4. 4. Serious MongoDB Talk
  5. 5. This Talk
  6. 6. Where’s the data from?
  7. 7. Where’s the data from?
  8. 8. How Much Is There?
  9. 9. Deployment
  10. 10. Visualization
  11. 11. Visualization Pipeline MongoDB PyMongo NumPy Matplotlib Python dicts SciPy
  12. 12. import numpy! import pymongo! ! data = []! db = pymongo.MongoClient().my_database! ! for doc in db.collection.find(query):! data.append((! doc['position']['coordinates'][0],! doc['position']['coordinates'][1],! doc['airTemperature']['value']))! ! arrays = numpy.array(data)!
  13. 13. # NumPy column access syntax.! lons = arrays[:, 0]! lats = arrays[:, 1]! temps = arrays[:, 2]!
  14. 14. from scipy import griddata! from matplotlib import pyplot! ! xs = numpy.linspace(-180, 180, 361)! ys = numpy.linspace(-90, 90, 181)! zs = griddata(lats, lons, temps,! (xs, ys),! method='linear')! ! pyplot.contour(xs, ys, zs)! Magic!! Also magic!!
  15. 15. from matplotlib import pyplot! ! xs = numpy.linspace(-180, 180, 361)! ys = numpy.linspace(-90, 90, 181)! zs = griddata(lats, lons, temps,! (xs, ys),! method='linear')! ! pyplot.contour(xs, ys, zs)!
  16. 16. Triangulation
  17. 17. Triangulation
  18. 18. What temperature? Triangulation
  19. 19. Barycentric Interpolation What temperature? 53 48 54 Weighted Average 51.1
  20. 20. Interpolation 51.1
  21. 21. Interpolation
  22. 22. Interpolation
  23. 23. Contours
  24. 24. Contours
  25. 25. import numpy! import pymongo! ! data = []! db = pymongo.MongoClient().my_database! ! for doc in db.collection.find(query):! data.append((! doc['position']['coordinates'][0],! doc['position']['coordinates'][1],! doc['airTemperature']['value']))! ! arrays = numpy.array(data)! Not terrifically fast
  26. 26. Analyzing large datasets • Querying: 109k documents per second • (On localhost) • Can we go faster? • Enter“Monary”
  27. 27. MongoDB PyMongo NumPy Matplotlib Python dicts MongoDB Monary NumPy Matplotlib Monary by David Beach
  28. 28. import monary! ! data = []! connection = monary.Monary()! ! arrays = monary_connection.query(! db='my_database',! coll='collection',! query=query,! fields=['lon', 'lat', 'temp'],! types=[! 'float32', 'float32', 'float32'])!
  29. 29. Monary • PyMongo: 109k documents per second • Monary: 817k documents per second
  30. 30. Visualization
  31. 31. {! ts: ISODate("1991-01-01T00:00:00Z"),! position: {! type: "Point",! coordinates: [! -94.6,! 39.117! ]! },! airTemperature: {! value: 45,! quality: "1"! }! }! Original Schema
  32. 32. {! ts: ISODate("1991-01-01T00:00:00Z"),! lon: 39.117,! lat: -94.6,! temp: 45! }! Target Schema
  33. 33. Future of Monary • Author:
 David Beach • Interns:
 Kyle Suarez
 Matt Cotter • Mentors:
 A.Jesse Jiryu Davis
 Jason Carey
  34. 34. Future of Monary • Subdocuments: "airTemperature.value" • Aggregation cursor • Packaging • Bugfixes • Python 3
  35. 35. Thanks • Monary • NumPy • SciPy • Matplotlib
  36. 36. Thanks
  37. 37. Thank you #MongoDBWorld A. Jesse Jiryu Davis Senior Python Engineer, MongoDB

×