During the rise and innovation of “big data,” the geospatial analytics landscape has grown and evolved. We are beyond just analyzing static maps. Geospatial data is streaming from devices, sensors, infrastructure systems, or social media, and our applications and use cases must dynamically scale to meet the increased demands. Cloud can provide cost-effective storage and that ephemeral resource-burst needed for fast processing and low latency, all to monetize the immediate value of fresh geospatial data. Geospatial analytics require optimized spatial data types and algorithms to distill data to knowledge. Such processing, especially with strict latency requirements, has always been a challenge. We propose an open source big data stack for geospatial analytics on Cloud based on Apache NiFi, Apache Spark and LocationTech GeoMesa. GeoMesa is a geospatial framework deployed in a modern big data platform that provides a scalable and low latency solution for indexing volumes of historical data and generating live views and streaming geospatial analytics. Presentation and live demo performed at DataWorks 2018 Conference - San Jose: https://bit.ly/2xthAGD

1. 1
High Performance and Scalable Geospatial
Analytics on Cloud with Open Source
James Hughes – CCRI
Constantin Stanca – Hortonworks

2. 2
Summary
⬢ Loading Geospatial data into the cloud and GeoTools datastores never seems as easy as
it should be. There's sensors network, GPS devices, Twitter streams, FTP servers and all
sorts of other data that you need to parse, convert to SimpleFeatures, and then ingest.
⬢ GeoMesa, NiFi and Spark provides a fully open source solution to ease the pain of
ingesting and analyzing data using ANY GeoTools data store.
⬢ DataPlane Services Cloud Manager (powered by Cloudbreak) helps you to deploy
ephemeral geospatial analytics clusters to support increased computation
requirements, all decoupled from storage.
⬢ We will show how real-time streaming data such as satellite AIS can be ingested and
managed in real-time with NiFi. Also, show how geospatial data stored in S3, HDFS, or
HBase, ORC or Parquet, can be queried at scale using GeoMesa, Spark and Zeppelin.

3. 3
Geospatial Analytics Challenges

4. 4
Data Movement & System Complexity with Added Pressure of Big Data
Acquire
Data
Store
Data
Acquire
Data
Store
Data
Store
Data
Store
Data
Store
Data
Process
and
Analyze
Data
Data
Flow
Acquire
Data
Acquire
Data

5. 5
If that was not enough …
Spatial Data Types
Points
Locations
Events
Instantaneous
Positions
Lines
Road networks
Voyages
Trips
Trajectories
Polygons
Administrative
Regions
Airspaces

6. 6
If that was not enough …
Spatial Data Relationships
equals
disjoint
intersects
touches
crosses
within
contains
overlaps

7. 7
If that was not enough …
Topology Operations
7
Algorithms
● Convex Hull
● Buffer
● Validation
● Dissolve
● Polygonization
● Simplification
● Triangulation
● Voronoi
● Linear Referencing
● and more...

8. 8
Requirements for a High Performance
Geospatial Analytics Platform

9. 9
Traditional Approach
⬢ GIS, data crunching and web serving were three very separate worlds.
⬢ If a web app wanted access to the analysis there was a long process of ETL, DB work,
imports and exports, and bribing various network and storage people for the
resources you needed.

10. 10
Requirements for a High Performance Geospatial Analytics Platform
⬢ IoT sensors present an opportunity to understand the world right now
– A map of the current state of the world enables faster reactions
⬢ The variety of sensors and data source present data management challenges
– Adding new, varied data sources must be easy
⬢ Big data requires distributed storage / computation and scalable infrastructure
– The data layer has to scale
– Analysis has to be easy

11. 11
Scalable Geospatial Analytics on the Cloud

12. 12
How Cloud Helps to Address Geospatial Big Data Challenges
Challenges:
⬢ Big data problem (derive insights from all data)
⬢ Compute resources when they are needed (easy scale, easy access to data)
Solution:
⬢ Cloud provides elastically the needed compute resources, all decoupled from the
storage, whether that is an object store, file system or NoSQL.

13. 13
Importance for Geospatial Analytics
⬢ Spatial streaming visualizations and analytics can present near real-time insights
– Decision makers can respond more rapidly when they see live data feeds on a map
⬢ Spatial batch analytics can fuse multiple data sources together to understand a region
– Patterns of life emerge
– Advertisers can plan their next campaigns
– Business can locate their new store sites

14. 14
Cloudbreak
⬢ Cloudbreak can be utilized to address
Geospatial computational capacity needs
⬢ Easily spin auto-scalable clusters for
different workloads and purposes, whether
is a Geospatial Ingest Cluster with NiFi and
GeoMesa, or Geospatial Analytics cluster
with Spark and GeoMesa.
⬢ Data can reside in your object store or even
in a persistent data store.
⬢ These ephemeral clusters can be scheduled
for a period of time or only until the job is
done so you pay only what you use.

15. 15
LocationTech GeoMesa

16. 16
How GeoMesa Helps with Geospatial Data Type Challenges
Challenges:
⬢ Vector & raster data
⬢ Geospatial data types
Solution:
⬢ GeoMesa tools for streaming, persisting, managing, and analyzing spatio-temporal data at scale

17. 17
What is GeoMesa?
A suite of tools for streaming, persisting, managing, and analyzing spatio-temporal
data at scale

18. 18
What is GeoMesa?
A suite of tools for streaming, persisting, managing, and analyzing spatio-temporal
data at scale

19. 19
What is GeoMesa?
A suite of tools for streaming, persisting, managing, and analyzing spatio-temporal
data at scale

20. 20
What is GeoMesa?
A suite of tools for streaming, persisting, managing, and analyzing spatio-temporal
data at scale

21. 21
What is GeoMesa?
A suite of tools for streaming, persisting, managing, and analyzing spatio-temporal
data at scale

22. 22
Proposed Reference Architecture

23. 23
How does HDP/HDF + GeoMesa stream data?
⬢ The GeoMesa Kafka DataStore allows data produces to write CRUD messages to a
Kafka topic.
⬢ Consumers off that topic build up an in-memory representation of the current state of
the world.
⬢ This allows for
– live maps,
– real time analytics, and
– complex event processing.

24. 24
How does HDP/HDF + GeoMesa persist data?
GeoMesa integrates with HBase and Accumulo:
⬢ Key structures use space filling curves
⬢ Complex geospatial filters and processing can be
‘pushed down’ using Filters, Coprocessors, and Iterators
Geomesa’s File System Datastore provides the ability to
store spatio-temporally indexed data on S3 cloud object
store or storage formats like ORC or Parquet.

25. 25
Geospatial Data Flow Transformation
with NiFi and GeoMesa

26. 26
Geo Data in
Motion
(Cloud)
Geo Data
in
Motion
(on-premises)
Geo Data
at Rest
(on-premises)
Edge
Geo Data
Geo Data
in Motion
Edge
Analytics
Geo Data
at Rest
(Cloud)
Edge
Geo Data
Geo Data
at Rest
(on-premises)
Closed
Loop
Analytics
Machine
Learning
Deep
Historical
Analysis
Geospatial Data Flow Transformation with NiFi and GeoMesa
On-Prem
Cloud
Satellite AIS
Spatial Data

27. 27
GeoMesa NiFi
⬢ GeoMesa-NiFi allows you to ingest data into GeoMesa straight from NiFi by
leveraging custom processors.
⬢ NiFi allows you to ingest data into GeoMesa from every source GeoMesa supports
and more.
Data
SimpleFeatureType
Schema
GeoMesa NiFi
Processors enabled datastores

28. 28
GeoMesa NiFi Processors
⬢ PutGeoMesaAccumulo: Ingest data into a GeoMesa Accumulo datastore with a
GeoMesa converter or from geoavro
⬢ PutGeoMesaHBase: Ingest data into a GeoMesa HBase datastore with a GeoMesa
converter or from geoavro
⬢ PutGeoMesaFileSystem: Ingest data into a GeoMesa File System datastore with a
GeoMesa converter or from geoavro
⬢ PutGeoMesaKafka: Ingest data into a GeoMesa Kafka datastore with a GeoMesa
converter or from geoavro
⬢ PutGeoTools: Ingest data into an arbitrary GeoTools datastore using a GeoMesa
converter or avro
⬢ ConvertToGeoAvro: Use a GeoMesa converter to create geoavro

29. 29
Analyze Geospatial Data with
Geomesa and Spark

30. 30
How does HDP + GeoMesa analyze geospatial data?
GeoMesa integrates deeply with Spark to:
⬢ create spatial User Defined Types and User Defined Functions
– (based on LocationTech JTS, a geometry library)
⬢ optimize spatial queries against GeoMesa DataSources
⬢ persist output data back to GeoMesa
⬢ leverage Zeppelin notebooks to allow for rapid innovation and creativity
⬢ Zeppelin allows analysts to visualize results easily

31. 31
DEMO
Data Ingest and Interactive Insights with
GeoMesa, NiFi, Spark and Zeppelin

32. 32
Demo
⬢ Introduce EE dataset
⬢ Data management / NiFi overview
⬢ Real-time view + historical recall
⬢ Spark Analysis

33. 33
NiFi-GeoMesa Data Flow

34. 34
Satellite AIS

35. 35
Setup
⬢ Import GeoMesa
dependency
⬢ Create dataframe
backed by
GeoMesa relation
⬢ Create SQL
temporary view so
we can query it

36. 36
Sub-select Data
⬢ Create rough sub-
selection of data
■ Bound by time
■ Bound by bounding
box roughly around
the Gulf of Mexico
⬢ Create a new
temporary view from
this sub-selection
⬢ Cache the data (pull
into memory)

37. 37
Data Exploration
⬢ Query for Tankers
in the Gulf
⬢ Get counts for
each type of
Tanker
⬢ Group the counts
by day
⬢ Graph counts to
see trends

38. 38
Data Exploration
⬢ Restrict our search to
just Trinity Bay

39. 39
Data Exploration
⬢ Create a new
temporary view of
the number of ships
in Trinity Bay

40. 40
Extra Data
⬢ Pull in Gas price data
⬢ Acquired from
EIA.gov
⬢ Two Gas Price
Indexes
⬢ NYH: New York
Harbor
⬢ GC: Gulf Coast
⬢ Create temporary
view so we can
analyze with SQL

41. 41
Data Exploration
⬢ Graph data over
time period of
Harvey
⬢ Notice we don’t
have daily values

42. 42
Data Exploration
⬢ Create temporary
view of gas price
data around our
time of interest

43. 43
Data Exploration
⬢ Backfill the price data
with the last value to
give us day-continuous
data
⬢ Min/Max Normalize gas
and ship counts
⬢ Graph gas prices and
ship counts together

44. 44
Resources

45. 45
Resources
⬢ GeoMesa Project: http://www.geomesa.org/
⬢ Geomesa-NiFi: http://www.geomesa.org/documentation/user/nifi.html
⬢ GeoMesa-Spark: http://www.geomesa.org/documentation/user/spark/index.html
⬢ Articles:
⬢ http://www.ccri.com/2017/03/20/new-geomesa-spark-sql-zeppelin-notebooks-support/
⬢ http://www.ccri.com/2018/02/26/interactive-insights-hurricane-harveys-impact-energy-
production-geomesa-jupyter-notebooks/

46. 46
Thank you