• Like

The Weather of the Century Part 3: Visualization

  • 286 views
Uploaded on

MongoDB natively supports geospatial indexing and querying, and it integrates easily with open source visualization tools. In this presentation, learn high-performance techniques for querying and …

MongoDB natively supports geospatial indexing and querying, and it integrates easily with open source visualization tools. In this presentation, learn high-performance techniques for querying and retrieving geospatial data, and how to create a rich visual representation of global weather data using Python, Monary, and Matplotlib.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
286
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
15
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. A. Jesse Jiryu Davis #MongoDBWorld The Weather of the Century! Part III:! Visualization Senior Python Engineer, MongoDB
  • 2. Serious MongoDB Talk
  • 3. Serious MongoDB Talk Database
  • 4. Serious MongoDB Talk
  • 5. This Talk
  • 6. Where’s the data from?
  • 7. Where’s the data from?
  • 8. How Much Is There?
  • 9. Deployment
  • 10. Visualization
  • 11. Visualization Pipeline MongoDB PyMongo NumPy Matplotlib Python dicts SciPy
  • 12. import numpy! import pymongo! ! data = []! db = pymongo.MongoClient().my_database! ! for doc in db.collection.find(query):! data.append((! doc['position']['coordinates'][0],! doc['position']['coordinates'][1],! doc['airTemperature']['value']))! ! arrays = numpy.array(data)!
  • 13. # NumPy column access syntax.! lons = arrays[:, 0]! lats = arrays[:, 1]! temps = arrays[:, 2]!
  • 14. from scipy import griddata! from matplotlib import pyplot! ! xs = numpy.linspace(-180, 180, 361)! ys = numpy.linspace(-90, 90, 181)! zs = griddata(lats, lons, temps,! (xs, ys),! method='linear')! ! pyplot.contour(xs, ys, zs)! Magic!! Also magic!!
  • 15. from matplotlib import pyplot! ! xs = numpy.linspace(-180, 180, 361)! ys = numpy.linspace(-90, 90, 181)! zs = griddata(lats, lons, temps,! (xs, ys),! method='linear')! ! pyplot.contour(xs, ys, zs)!
  • 16. Triangulation
  • 17. Triangulation
  • 18. What temperature? Triangulation
  • 19. Barycentric Interpolation What temperature? 53 48 54 Weighted Average 51.1
  • 20. Interpolation 51.1
  • 21. Interpolation
  • 22. Interpolation
  • 23. Contours
  • 24. Contours
  • 25. import numpy! import pymongo! ! data = []! db = pymongo.MongoClient().my_database! ! for doc in db.collection.find(query):! data.append((! doc['position']['coordinates'][0],! doc['position']['coordinates'][1],! doc['airTemperature']['value']))! ! arrays = numpy.array(data)! Not terrifically fast
  • 26. Analyzing large datasets • Querying: 109k documents per second • (On localhost) • Can we go faster? • Enter“Monary”
  • 27. MongoDB PyMongo NumPy Matplotlib Python dicts MongoDB Monary NumPy Matplotlib Monary by David Beach
  • 28. import monary! ! data = []! connection = monary.Monary()! ! arrays = monary_connection.query(! db='my_database',! coll='collection',! query=query,! fields=['lon', 'lat', 'temp'],! types=[! 'float32', 'float32', 'float32'])!
  • 29. Monary • PyMongo: 109k documents per second • Monary: 817k documents per second
  • 30. Visualization
  • 31. {! ts: ISODate("1991-01-01T00:00:00Z"),! position: {! type: "Point",! coordinates: [! -94.6,! 39.117! ]! },! airTemperature: {! value: 45,! quality: "1"! }! }! Original Schema
  • 32. {! ts: ISODate("1991-01-01T00:00:00Z"),! lon: 39.117,! lat: -94.6,! temp: 45! }! Target Schema
  • 33. Future of Monary • Author:
 David Beach • Interns:
 Kyle Suarez
 Matt Cotter • Mentors:
 A.Jesse Jiryu Davis
 Jason Carey
  • 34. Future of Monary • Subdocuments: "airTemperature.value" • Aggregation cursor • Packaging • Bugfixes • Python 3
  • 35. Thanks • Monary • NumPy • SciPy • Matplotlib
  • 36. Thanks
  • 37. Thank you #MongoDBWorld A. Jesse Jiryu Davis Senior Python Engineer, MongoDB