Magellan is a geospatial analytics library for Apache Spark that allows users to perform spatial queries and analysis on large geospatial datasets in a scalable manner. It provides custom data types and expressions to represent spatial objects like points, polygons and perform spatial operations. Magellan reads geospatial file formats like Shapefiles and GeoJSON, integrates with Spark SQL to allow spatial joins, and aims to simplify building geospatial applications at scale on Spark. The current version supports basic functionality and future versions will add more operators, optimizations, and support for additional formats and use cases.
2. Page2
What is geospatial context?
•Given a point = (-122.412651, 37.777748) which
city is it in?
•Does shape X intersect shape Y?
–Compute the intersection
•Given a sequence of points and a system of roads
–Compute best path representing points
3. Page3
Geospatial context is useful
What neighborhoods do people go to on weekends?
Predict the drop off neighborhood of a user?
Predict the location where next pick up can be expected?
How does usage pattern change with time?
Identify crime hotspot neighborhoods
How do these hotspots evolve with time?
Predict the likelihood of crime occurring at a given neighborhood
Predict climate at fairly granular level
Climate insurance: do I need to buy insurance for my crops?
Climate as a factor in crime: Join climate dataset with Crimes
7. Page7
Parsing!
•ESRI Shapefiles
–Spec for Shapes, no spec for metadata
–Worse, metadata = Dbase Format (really??)
•GeoJSON
–Verbose
–But atleast parseable
–Unfortunately not common
•ESRI Format
–JSON but not GeoJSON!
9. Page9
Scalability (or the lack thereof)
•ESRI Hive (runs on Hadoop but lacks spatial joins)
•JTS, Geos, Shapely (no support for scalability)
•Other proprietary engines = black boxes
11. Page11
Feature Extractors
Language integration simplifies exploratory analytics
Q-Q
Q-A
similarity
Parse +
Clean
Logs
Ad
category
mapping
Query
category
mapping
Poly
Exp
(Q-A)
Features
Model
Convex
Solver
Train/T
est
Split
train
Test/validation
Metrics
Ad Server
HDFS
Data Prep
Score Model - Real-time
Data
Flow
Stage
Data Flow Stage - Batch
Feedback
Spatial
Context
12. Page12
Not all is lost!
• local computations w/ ESRI Java API
• Scale out computation w/ Spark
• Python + R support without compromising
performance via Pyspark , SparkR
• Catalyst + Data Sources + Data Frames
= Flexibility + Simplicity + Performance
• Stitch it all together + Allow extension points
=> Success!
13. Page13
Magellan: a complete story for geospatial?
Create geospatial analytics applications
faster:
• Use your favorite language (Python/ Scala), even R
• Get best in class algorithms for common spatial analytics
• Write less code
• Read data efficiently
• Let the optimizer do the heavy lifting
14. Page14
How does it work?
Custom Data Types for Shapes:
• Point, Line, PolyLine, Polygon extend Shape
• Local Computations using ESRI Java API
• No need for Scala -> SQL serialization
Expressions for Operators:
• Literals e.g point(-122.4, 37.6)
• Boolean Expressions e.g Intersects, Contains
• Binary Expressions e.g Intersection
Custom Data Sources:
• Schema = [point, polyline, polygon, metadata]
• Metadata = Map[String, String]
• GeoJSON and Shapefile implementations
Custom Strategies for Spatial Join:
• Broadcast Cartesian Join
• Geohash Join (in progress)
• Plug into Catalyst as experimental strategies
15. Page15
Magellan in a nutshell
• Read Shapefiles/ GeoJSON as DataSources:
–sqlContext.read("magellan").load(“$path”)
–sqlContext.read(“magellan”).option(“type”, “geojson”).load(“$path”)
• Spatial Queries using Expressions
–point(-122.5, 37.6) = Shape Literal
–$”point” within $”polygon” = Boolean Expression
–$”polygon1” intersection $”polygon2” = Binary Expression
• Joins using Catalyst + Spatial Optimizations
–points.join(polygons).where($”point” within $”polygon”)
16. Page16
Where are we at?
Magellan 1.0.3 is out on Spark Packages, go give it a try!:
• Scala support, Python support will be functional in 1.0.4 (needs Spark 1.5)
• Github: https://github.com/harsha2010/magellan
• Spark Packages: http://spark-packages.org/package/harsha2010/magellan
• Data Formats: ESRI Shapefile + metadata, GeoJSON
• Operators: Intersects, Contains, Within, Intersection
• Joins: Broadcast
• Blog: http://hortonworks.com/blog/magellan-geospatial-analytics-in-spark/
• Zeppelin Notebook Example: http://bit.ly/1GwLyrV
17. Page17
What is next?
Magellan 1.0.4 expected release December:
• Python support
• MultiPolygon (Polygon Collection), MultiLineString (PolyLine Collection)
• Spark 1.5, 1.6
• Spatial Join Optimization
• Map Matching Algorithms
• More Operators based on requirements
• Support for other common geospatial data formats (WKT, others?)