The Weather of the Century Part 3: Visualization
 

The Weather of the Century Part 3: Visualization

on

  • 251 views

MongoDB natively supports geospatial indexing and querying, and it integrates easily with open source visualization tools. In this presentation, learn high-performance techniques for querying and ...

MongoDB natively supports geospatial indexing and querying, and it integrates easily with open source visualization tools. In this presentation, learn high-performance techniques for querying and retrieving geospatial data, and how to create a rich visual representation of global weather data using Python, Monary, and Matplotlib.

Statistics

Views

Total Views
251
Views on SlideShare
215
Embed Views
36

Actions

Likes
0
Downloads
13
Comments
0

3 Embeds 36

http://www.mongodb.com 18
https://www.mongodb.com 17
https://comwww-drupal.10gen.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

The Weather of the Century Part 3: Visualization The Weather of the Century Part 3: Visualization Presentation Transcript

  • A. Jesse Jiryu Davis #MongoDBWorld The Weather of the Century! Part III:! Visualization Senior Python Engineer, MongoDB
  • Serious MongoDB Talk
  • Serious MongoDB Talk Database
  • Serious MongoDB Talk
  • This Talk
  • Where’s the data from?
  • Where’s the data from?
  • How Much Is There?
  • Deployment
  • Visualization
  • Visualization Pipeline MongoDB PyMongo NumPy Matplotlib Python dicts SciPy
  • import numpy! import pymongo! ! data = []! db = pymongo.MongoClient().my_database! ! for doc in db.collection.find(query):! data.append((! doc['position']['coordinates'][0],! doc['position']['coordinates'][1],! doc['airTemperature']['value']))! ! arrays = numpy.array(data)!
  • # NumPy column access syntax.! lons = arrays[:, 0]! lats = arrays[:, 1]! temps = arrays[:, 2]!
  • from scipy import griddata! from matplotlib import pyplot! ! xs = numpy.linspace(-180, 180, 361)! ys = numpy.linspace(-90, 90, 181)! zs = griddata(lats, lons, temps,! (xs, ys),! method='linear')! ! pyplot.contour(xs, ys, zs)! Magic!! Also magic!!
  • from matplotlib import pyplot! ! xs = numpy.linspace(-180, 180, 361)! ys = numpy.linspace(-90, 90, 181)! zs = griddata(lats, lons, temps,! (xs, ys),! method='linear')! ! pyplot.contour(xs, ys, zs)!
  • Triangulation
  • Triangulation
  • What temperature? Triangulation
  • Barycentric Interpolation What temperature? 53 48 54 Weighted Average 51.1
  • Interpolation 51.1
  • Interpolation
  • Interpolation
  • Contours
  • Contours
  • import numpy! import pymongo! ! data = []! db = pymongo.MongoClient().my_database! ! for doc in db.collection.find(query):! data.append((! doc['position']['coordinates'][0],! doc['position']['coordinates'][1],! doc['airTemperature']['value']))! ! arrays = numpy.array(data)! Not terrifically fast
  • Analyzing large datasets • Querying: 109k documents per second • (On localhost) • Can we go faster? • Enter“Monary”
  • MongoDB PyMongo NumPy Matplotlib Python dicts MongoDB Monary NumPy Matplotlib Monary by David Beach
  • import monary! ! data = []! connection = monary.Monary()! ! arrays = monary_connection.query(! db='my_database',! coll='collection',! query=query,! fields=['lon', 'lat', 'temp'],! types=[! 'float32', 'float32', 'float32'])!
  • Monary • PyMongo: 109k documents per second • Monary: 817k documents per second
  • Visualization
  • {! ts: ISODate("1991-01-01T00:00:00Z"),! position: {! type: "Point",! coordinates: [! -94.6,! 39.117! ]! },! airTemperature: {! value: 45,! quality: "1"! }! }! Original Schema
  • {! ts: ISODate("1991-01-01T00:00:00Z"),! lon: 39.117,! lat: -94.6,! temp: 45! }! Target Schema
  • Future of Monary • Author:
 David Beach • Interns:
 Kyle Suarez
 Matt Cotter • Mentors:
 A.Jesse Jiryu Davis
 Jason Carey
  • Future of Monary • Subdocuments: "airTemperature.value" • Aggregation cursor • Packaging • Bugfixes • Python 3
  • Thanks • Monary • NumPy • SciPy • Matplotlib
  • Thanks
  • Thank you #MongoDBWorld A. Jesse Jiryu Davis Senior Python Engineer, MongoDB