A. Jesse Jiryu Davis
#MongoDBWorld
The Weather of the Century!
Part III:!
Visualization
Senior Python Engineer, MongoDB
Serious MongoDB Talk
Serious MongoDB Talk
Database
Serious MongoDB Talk
This Talk
Where’s the data from?
Where’s the data from?
How Much Is There?
Deployment
Visualization
Visualization Pipeline
MongoDB PyMongo NumPy Matplotlib
Python
dicts
SciPy
import numpy!
import pymongo!
!
data = []!
db = pymongo.MongoClient().my_database!
!
for doc in db.collection.find(query):...
# NumPy column access syntax.!
lons = arrays[:, 0]!
lats = arrays[:, 1]!
temps = arrays[:, 2]!
from scipy import griddata!
from matplotlib import pyplot!
!
xs = numpy.linspace(-180, 180, 361)!
ys = numpy.linspace(-90,...
from matplotlib import pyplot!
!
xs = numpy.linspace(-180, 180, 361)!
ys = numpy.linspace(-90, 90, 181)!
zs = griddata(lat...
Triangulation
Triangulation
What temperature?
Triangulation
Barycentric Interpolation
What temperature? 53
48
54
Weighted Average
51.1
Interpolation
51.1
Interpolation
Interpolation
Contours
Contours
import numpy!
import pymongo!
!
data = []!
db = pymongo.MongoClient().my_database!
!
for doc in db.collection.find(query):...
Analyzing large datasets
• Querying: 109k documents per second
• (On localhost)
• Can we go faster?
• Enter“Monary”
MongoDB PyMongo NumPy Matplotlib
Python
dicts
MongoDB Monary NumPy Matplotlib
Monary
by David Beach
import monary!
!
data = []!
connection = monary.Monary()!
!
arrays = monary_connection.query(!
db='my_database',!
coll='co...
Monary
• PyMongo: 109k documents per second
• Monary: 817k documents per second
Visualization
{!
ts: ISODate("1991-01-01T00:00:00Z"),!
position: {!
type: "Point",!
coordinates: [!
-94.6,!
39.117!
]!
},!
airTemperatur...
{!
ts: ISODate("1991-01-01T00:00:00Z"),!
lon: 39.117,!
lat: -94.6,!
temp: 45!
}!
Target Schema
Future of Monary
• Author:

David Beach
• Interns:

Kyle Suarez

Matt Cotter
• Mentors:

A.Jesse Jiryu Davis

Jason Carey
Future of Monary
• Subdocuments:
"airTemperature.value"
• Aggregation cursor
• Packaging
• Bugfixes
• Python 3
Thanks
• Monary
• NumPy
• SciPy
• Matplotlib
Thanks
Thank you
#MongoDBWorld
A. Jesse Jiryu Davis
Senior Python Engineer, MongoDB
The Weather of the Century Part 3: Visualization
The Weather of the Century Part 3: Visualization
The Weather of the Century Part 3: Visualization
The Weather of the Century Part 3: Visualization
The Weather of the Century Part 3: Visualization
The Weather of the Century Part 3: Visualization
The Weather of the Century Part 3: Visualization
The Weather of the Century Part 3: Visualization
Upcoming SlideShare
Loading in...5
×

The Weather of the Century Part 3: Visualization

435

Published on

MongoDB natively supports geospatial indexing and querying, and it integrates easily with open source visualization tools. In this presentation, learn high-performance techniques for querying and retrieving geospatial data, and how to create a rich visual representation of global weather data using Python, Monary, and Matplotlib.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
435
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
16
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "The Weather of the Century Part 3: Visualization"

  1. 1. A. Jesse Jiryu Davis #MongoDBWorld The Weather of the Century! Part III:! Visualization Senior Python Engineer, MongoDB
  2. 2. Serious MongoDB Talk
  3. 3. Serious MongoDB Talk Database
  4. 4. Serious MongoDB Talk
  5. 5. This Talk
  6. 6. Where’s the data from?
  7. 7. Where’s the data from?
  8. 8. How Much Is There?
  9. 9. Deployment
  10. 10. Visualization
  11. 11. Visualization Pipeline MongoDB PyMongo NumPy Matplotlib Python dicts SciPy
  12. 12. import numpy! import pymongo! ! data = []! db = pymongo.MongoClient().my_database! ! for doc in db.collection.find(query):! data.append((! doc['position']['coordinates'][0],! doc['position']['coordinates'][1],! doc['airTemperature']['value']))! ! arrays = numpy.array(data)!
  13. 13. # NumPy column access syntax.! lons = arrays[:, 0]! lats = arrays[:, 1]! temps = arrays[:, 2]!
  14. 14. from scipy import griddata! from matplotlib import pyplot! ! xs = numpy.linspace(-180, 180, 361)! ys = numpy.linspace(-90, 90, 181)! zs = griddata(lats, lons, temps,! (xs, ys),! method='linear')! ! pyplot.contour(xs, ys, zs)! Magic!! Also magic!!
  15. 15. from matplotlib import pyplot! ! xs = numpy.linspace(-180, 180, 361)! ys = numpy.linspace(-90, 90, 181)! zs = griddata(lats, lons, temps,! (xs, ys),! method='linear')! ! pyplot.contour(xs, ys, zs)!
  16. 16. Triangulation
  17. 17. Triangulation
  18. 18. What temperature? Triangulation
  19. 19. Barycentric Interpolation What temperature? 53 48 54 Weighted Average 51.1
  20. 20. Interpolation 51.1
  21. 21. Interpolation
  22. 22. Interpolation
  23. 23. Contours
  24. 24. Contours
  25. 25. import numpy! import pymongo! ! data = []! db = pymongo.MongoClient().my_database! ! for doc in db.collection.find(query):! data.append((! doc['position']['coordinates'][0],! doc['position']['coordinates'][1],! doc['airTemperature']['value']))! ! arrays = numpy.array(data)! Not terrifically fast
  26. 26. Analyzing large datasets • Querying: 109k documents per second • (On localhost) • Can we go faster? • Enter“Monary”
  27. 27. MongoDB PyMongo NumPy Matplotlib Python dicts MongoDB Monary NumPy Matplotlib Monary by David Beach
  28. 28. import monary! ! data = []! connection = monary.Monary()! ! arrays = monary_connection.query(! db='my_database',! coll='collection',! query=query,! fields=['lon', 'lat', 'temp'],! types=[! 'float32', 'float32', 'float32'])!
  29. 29. Monary • PyMongo: 109k documents per second • Monary: 817k documents per second
  30. 30. Visualization
  31. 31. {! ts: ISODate("1991-01-01T00:00:00Z"),! position: {! type: "Point",! coordinates: [! -94.6,! 39.117! ]! },! airTemperature: {! value: 45,! quality: "1"! }! }! Original Schema
  32. 32. {! ts: ISODate("1991-01-01T00:00:00Z"),! lon: 39.117,! lat: -94.6,! temp: 45! }! Target Schema
  33. 33. Future of Monary • Author:
 David Beach • Interns:
 Kyle Suarez
 Matt Cotter • Mentors:
 A.Jesse Jiryu Davis
 Jason Carey
  34. 34. Future of Monary • Subdocuments: "airTemperature.value" • Aggregation cursor • Packaging • Bugfixes • Python 3
  35. 35. Thanks • Monary • NumPy • SciPy • Matplotlib
  36. 36. Thanks
  37. 37. Thank you #MongoDBWorld A. Jesse Jiryu Davis Senior Python Engineer, MongoDB
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×