This document discusses processing large geospatial data at scale. It provides background on geospatial concepts like raster and vector data. It then discusses big data frameworks like Hadoop, Spark, and Accumulo that can be used to process geospatial data in parallel across large clusters. Finally, it presents several LocationTech projects like GeoTrellis, GeoJinni, and GeoWave that build geospatial capabilities on top of these frameworks to allow distributed processing and querying of large raster and vector maps.

1. PROCESSING GEOSPATIAL
DATA @ SCALE
Rob Emanuele

2. What we’ll be covering…
Background on geospatial concepts
What is LocationTech?
Background on big data frameworks
Overview of LocationTech projects for
processing big geo data.

3. PROCESSING GEOSPATIAL DATA
@ SCALE

4. PROCESSING GEOSPATIAL DATA
@ SCALE

5. Geospatial Data
Core of GIS (Geographic information system)
Raster (images, weather data)
Vector (points of interest, country boundries)

6. Raster Data

7. Raster Data

8. Raster Data

9. Raster Data

10. Vector Data (Points)

11. Vector Data (Lines)

12. Vector Data (Polygons)
Source: https://ryouready.wordpress.com/2009/11/16/infomaps-using-r-visualizing-german-unemployment-rates-by-color-on-a-map/

13. Vector Data

14. PROCESSING GEOSPATIAL DATA
@ SCALE

15. Contains

16. Contains

17. Heatmap (Kernel Density)

18. Zonal Statistics

19. Feature Extraction (Image Segmentation)
Source: http://www.professeurs.polymtl.ca/christopher.pal/

20. PROCESSING GEOSPATIAL DATA
@ SCALE

21. Large geospatial data
Landsat 8 on AWS: 311,405 scenes @ ~800 MB
each.That's 250 TB and counting.
OpenStreetMap: planet.osm is 617 GB.
3 years of geotagged tweets: 3 TB

22. PROCESSING GEOSPATIAL DATA
@ SCALE

23. WHAT IS ?

27. PROCESSING GEOSPATIAL DATA
@ SCALE

29. Project to build a better search engine, back in
the early 2000’s.
Worked for small datasets, but was not scalable.

30. The Google papers

31. After reading the papers, Nutch developers
added a distributed file system and MapReduce
model to Nutch.
In 2006, those portions were spun out of Nutch
to form…

33. Hadoop

34. Matei Zaharia
Worked with Hadoop at UC Berklee
Noticed Hadoop was not a good fit for
Machine Learning algorithms and other
iterative models.
So in 2009, he created…

41. 2006

43. Apache Accumulo
Created by the NSA in 2008
Donated to the Apache Foundation in 2011
Graduated to a top level project in 2012
Almost defunded by the US government the
same year.

44. (Sec. 929) Prohibits any DOD component from utilizing the
cloud computing database developed by the National Security
Agency (NSA) and known as "Accumulo" after the end of
FY2013, unless the DOD CIO certifies that: (1) there are no
viable commercial open source databases that have such security
features, or (2) Accumulo itself has become a successful open
source database project. Requires DOD and intelligence
community officials to coordinate the use by DOD components
of cloud computing infrastructure and services offered by the
intelligence community for purposes other than intelligence
analysis.

45. (Sec. 929) Prohibits any DOD component from utilizing the
cloud computing database developed by the National Security
Agency (NSA) and known as "Accumulo" after the end of
FY2013, unless the DOD CIO certifies that: (1) there are no
viable commercial open source databases that have such
security features, or (2) Accumulo itself has become a
successful open source database project. Requires DOD and
intelligence community officials to coordinate the use by DOD
components of cloud computing infrastructure and services
offered by the intelligence community for purposes other than
intelligence analysis.

46. Data
Node
Data
Node
Data
Node
Name
Node
Master
Tablet
Server
Tablet
Server
Tablet
Server
Accumulo
BigTable clone (columnar database)
Records stored on HDFS
Lexicographically sorted table index

49. GEOJINNI
(FORMERLY SPATIALHADOOP)

50. Spatial language Built-in spatial data types
Spatial Indexes Spatial Operations

52. R-TREE INDEX OF A 400 GB ROAD NETWORK

53. 72 Frames × 14 Billion points per frame
Total = 1 Trillion points
Generated in three hours on a 10-node cluster
HEAT MAP FROM 2009 TO 2014 MONTH-BY-MONTH

56. Geo +
accessed through

58. SELECT
tweet.text,
user.name
FROM
tweet, user
WHERE
bbox(tweet.location, -115, 45, -110, 50) AND
tweet.user_id = user.user_id
+

60. GeoTrellis
a Scala library for geospatial data types and
operations.
enables Spark with geospatial capabilities (raster
now, soon vector!).
storage and query raster from HDFS,
Accumulo, and S3

63. Zonal Summaries

64. Zonal Summaries

65. Benchmark Results
439.5 GB of monthly temperature model output data
USA temperature yearly average, 2006 to 2100

66. Benchmark Results
439.5 GB of monthly temperature model output data
USA temperature yearly average, 2006 to 2100
40 m3.xlarge instances
(estimated $2.00 USD per hour
on spot market)

68. GEOWAVE

70. Geo +
accessed through
GEOWAVE

71. GEOWAVE

74. Three dimensional Z-order curve

76. PROCESSING GEOSPATIAL
DATA @ SCALE

77. THANKYOU
@lossyrob
gitter.im/geotrellis/geotrellis
github.com/geotrellis/geotrellis
remanuele@azavea.com