"Geohash: Integration of Disparate Geospatial Data," Abe Usher, Chief Innovation Officer, HumanGeo Group

1. Heatmaps & Geohash:
Integration of Multi-Source Geospatial Data
Abe Usher, CIO
abe@thehumangeo.com
703.955.1540
@abeusher
AKA Heatmaps are the Heat
INFORMATION INTO INSIGHT

2. Our Menu of Subtopics
A LITTLE HISTORY WHY HEATMAPS? GEOHASH? DATA GONE WILD
Big data requires new approaches. A new organizing construct for
Why geospatial data?
information analysis.
KITCHEN MODEL SPECIFIC EXAMPLES EASTER EGG
New ways to combine internal data Concrete take-aways. Treats for making it through another
with external new media for presentation.
maximum insight.
2

3. What’s in it for you?
3

4. What’s in it for you?
 Make custom heatmaps
 Three iron laws of geospatial data
integration and analysis
4

5. Whoami?
People Data Beverages
5

6. Geohash
Geohash is a coordinate
transformation
that facilitates combining two
variables (latitude and longitude)
into a single(text) variable
that represents a bounding-box
containing the point of interest.
6

7. Heatmap
“Discrete & continuous methods of kernel density
estimation”
7

8. A rose by any other name
“Discrete & continuous methods of kernel density
estimation”
 Gaussian
 Quartic
 Exponential
 Triangular
 Uniform
 Epanechnikov
8

9. A rose by any other name
“Discrete methods of kernel density estimation”
9

10. About
Big Data (Digital) Human Geography
 Predictive models of social drift &
human behavior
 Streaming media analytics
 Micro-demographics
We’re hiring! info@thehumangeo.com
10

11. Why Heatmaps and Geohash?
 Too much data
11

12. Why Heatmaps and Geohash?
 Too much data
 Trust in Internet data
12

13. Why Heatmaps and Geohash?
 Too much data
 Trust in Internet data
 Heatmaps look cool
13

14. Why Heatmaps and Geohash?
 Too much data
 Trust in Internet data
 Heatmaps look cool
 Geohash helps quantify
data
14

15. Why Heatmaps and Geohash?
Visual summary
 Too much data
 Trust in Internet data
 Heatmaps look cool
 Geohash helps quantify
data
Quantitative methods
15

16. Trust and Internet Information
Tracy Morrow aka “Ice T”
16

17. Trust and Internet Information
“Game knows
game, baby.”
Tracy Morrow aka “Ice T”
17

18. Trust and Internet Information
“If you have expert
knowledge, then
you are capable of
recognizing expert
knowledge.”
[paraphrased]
Tracy Morrow aka “Ice T”
18

19. Trust and Internet Information
Can we actually trust this Internet stuff?
19

20. Trust and Internet Information
20

21. Trust and Internet Information
21

22. Trust and Internet Information
22

23. Salami Slicing
Salami slicing: series of minor observations, resulting
in a larger observation that would be difficult to perform
http://en.wikipedia.org/wiki/Salami_slicing
23

24. Seven Layer GLT
1. OpenStreetMap data
2. Flickr
3. Panoramio
4. Geonames.org
5. Twitter
6. Wikimapia
7. 4Square
* Geospatial Lattice of Trusted Data
24

25. Seven Layer GLT
1. OpenStreetMap data
2. Flickr
3. Panoramio
4. Geonames.org
5. Twitter
6. Wikimapia
7. 4Square
Spatial Temporal User Finds From the Field (STUFFF)
25

26. Rule #1: Think in terms of aggregation
Twitter data
Panoramio Tourist photos
Classified data
Twitter geohash ez420 – coffee shop
Panoramio geohash ez420 – Starbucks
Classified geohash ez420 - Wifi
Trust through aggregation
26

27. Rule #1: Think in terms of aggregation
Twitter data
Twitter geohash ez420 – coffee shop
Panoramio geohash ez420 – Starbucks
Classified geohash ez420 - Wifi
Panoramio Tourist photos
Geohash creates simple string variables. Classified data
Matching strings = super easy
Matching similar coordinates = impossible
Trust through aggregation
27

28. Rule #1: Think in terms of aggregation
Twitter data
Panoramio Tourist photos
Use geohash to apply collaborative
filtering techniques to develop new Classified data
models of trust & data confidence.
Trust through aggregation
28

29. Rule #2: Selectively throw away precision
Entity #1
Latitude Longitude
40.998946 28.9232
41.005164 28.973668
41.018765 29.016412
41.062268 29.030145
Entity #2
Latitude Longitude
40.999100 28.92111
41.018112 28.973991
41.018880 29.016902
41.062110 29.030122
29

30. Rule #2: Selectively throw away precision
Entity #1
Latitude Longitude Geohash
40.998946 28.9232 SXK94
41.005164 28.973668 SXK97
41.018765 29.016412 SXK9K
41.062268 29.030145 SXK9S
30

31. Rule #2: Selectively throw away precision
Entity #1
Latitude Longitude Geohash
40.998946 28.9232 SXK94
41.005164 28.973668 SXK97
41.018765 29.016412 SXK9K
41.062268 29.030145 SXK9S
Entity #2
Latitude Longitude Geohash
40.999100 28.92111 SXK94
41.018112 28.973991 SXK97
41.018880 29.016902 SXK9K
41.062110 29.030122 SXK9S
31

32. Kitchen Model for
Spatial Analysis
32

33. Kitchen Model for
Spatial Analysis
Chef
33

34. Kitchen Model for
Spatial Analysis
Chef Ingredients
34

35. Kitchen Model for
Spatial Analysis
Chef Ingredients Utensils
35

36. Kitchen Model for
Spatial Analysis
Chef Ingredients Utensils Presentation
36

37. Kitchen Model for
Spatial Analysis
Chef Ingredients Utensils Presentation
37

38. Types of Heatmaps
Turnkey
 GeoCommons
 SpatialKey
 MapBox/TileMill
 ArcGIS Desktop
 QGIS
Custom
 Python
 R
 Javascript 38

39. Types of Heatmaps
Turnkey
 GeoCommons
 SpatialKey
 MapBox/TileMill
 ArcGIS Desktop
 QGIS
Custom
 Python
 R
 Javascript 39

40. Heatmap: Recipe One
“OSM Style”
Get Python http://python.org
Get the sethoscope library
http://www.sethoscope.net/heatmap/
Get data
http://bit.ly/geotweet_sc
https://dev.twitter.com/docs/streaming-api/methods#locations
Command line:
heatmap.py -g portland.gpx -o output.png --height 800 --osm
40

41. 41

42. Heatmap: Recipe One
Stitch it together in an MP4 movie!
Get the CLI app:
http://ffmpeg.org/
Command line:
heatmap.py -g portland.gpx -o output.png --height 800 –osm –a
ffmpeg -i frame-%05d.png OSM_is_awesome.mp4
42

43. Heatmap: Recipe One
Props to
Seth Golub from Google
http://www.sethoscope.net/
43

44. Rule #3:
Beware of population effects
http://xkcd.com/1138/
44

45. Rule #3:
Beware of population effects
Absolute value
Normalized value =
Population estimate
45

46. Rule #3:
Beware of population effects
34
72 52
22
46

47. Rule #3:
Beware of population effects
34
Absolute value
2,000
Normalized value =
Population estimate
72 52
16,000 25,000
22
2,000
47

48. Conclusion
1. Think in terms of aggregation
2. Selectively throw away precision
3. Beware of population effects
48

49. Contact Us
HumanGeo NY HumanGeo DC
1221 Avenue of the Americas 2500 Wilson Boulevard
Suite 4200 | New York, NY 10020 Suite 310 | Arlington, VA 22201
info@thehumangeo.com | 703.955.1540 | www.thehumangeo.com
49

50. Easter Egg
http://bit.ly/geotweet_sc 50