Advertisement
HBaseCon 2012 | Scaling GIS In Three Acts
May. 31, 2012•0 likes•3,628 views
2 likes
Be the first to like this
Show More
views
Total views
0
On Slideshare
0
From embeds
0
Number of embeds
0
Download to read offline
Report
Technology
Economy & Finance
Business
Scaling geospatial data is hard. State of the art GIS technologies available to the general public are locked in the realm of relational databases with PostGIS as the prominent leader. Though a number of location-based startups have walked this path before, few have marked their trail along the way. Act one proveds a survey of the landscape, defining terms, and highlighting pitfalls. Act two explores the world of open source, horizontally scalable GIS and outlines the problems they solve. Act three explores implementations backed by HBase. No previous GIS knowledge is required.
Cloudera, Inc.Follow
Cloudera, Inc.Advertisement
Advertisement
Advertisement
Recommended
2012 remax vs the industryLisa Borders
Renaissance capital russian mining dayevraz_company
February Realtor ReportSouthwest Riverside County Association of Realtors
Tricumen Rbs Update 13 Jan 12 Publicsebwalker
Graphs, Edges & Nodes - Untangling the Social WebJoël Perras
SAIA 2012 PresentationRobynVeckranges
HBaseCon 2012 | Scaling GIS In Three Acts
- SCALING GIS IN 3 ACTS presented by: Nick Dimiduk May 22, 2012 1
- SCALING GIS IN 3 ACTS - Lightning Edition! - presented by: Nick Dimiduk May 22, 2012 2
- HBase in Action Manning Press, Fall 2012 MULTI-BAR CHART TITLE, LEFT ALIGNED Thousands 10 9 8 7 6 hbaseinaction.com 5 Discount code: 12hb10 4 3 2 1 0 Series 1 Series 2 Series 3 Series 4 © 2012 The Climate Corporation. All Rights Reserved. Policies are underwritten by State 3 National Insurance Company, Inc. and administered by The Climate Insurance Agency LLC.
- Act I: What is GIS?
- GIS: Data (on maps!) MULTI-BAR CHART TITLE, LEFT ALIGNED MULTI-BAR CHART TITLE, LEFT ALIGNED Thousands 10 Thousands 10 9 9 8 8 7 7 6 6 5 5 4 4 3 3 2 2 1 1 0 0 Series 1 Series 2 Series 3 Series 4 Series 1 Series 2 Series 3 Series 4 © 2012 The Climate Corporation. All Rights Reserved. Policies are underwritten by State 5 National Insurance Company, Inc. and administered by The Climate Insurance Agency LLC.
- HC SVNT DRAGONES MULTI-BAR CHART TITLE, LEFT ALIGNED Thousands 10 9 “Here are Dragons” 8 7 6 5 4 3 DRAGONES!!! 2 1 Image: Psalter World Map, 1265 0 http://en.wikipedia.org/wiki/Here_be_dragons Series 1 Series 2 Series 3 Series 4 © 2012 The Climate Corporation. All Rights Reserved. Policies are underwritten by State 6 National Insurance Company, Inc. and administered by The Climate Insurance Agency LLC.
- Act II: What to do with GIS?
- Geospatial Queries © 2012 The Climate Corporation. All Rights Reserved. Policies are underwritten by State 8 National Insurance Company, Inc. and administered by The Climate Insurance Agency LLC.
- Non-Euclidean Geometry “Know thy surface” Image: Trigonometry on a Spehere http://en.wikipedia.org/wiki/Non-Euclidean_geometry © 2012 The Climate Corporation. All Rights Reserved. Policies are underwritten by State 9 National Insurance Company, Inc. and administered by The Climate Insurance Agency LLC.
- Act III: GIS on HBase
- The devil is in the Indices Image: Six iterations of the Hilbert curve http://en.wikipedia.org/wiki/Space-filling_curve © 2012 The Climate Corporation. All Rights Reserved. Policies are underwritten by State 12 National Insurance Company, Inc. and administered by The Climate Insurance Agency LLC.
- Spatial Partitioning Image: USA night lights http://www.noaanews.noaa.gov/stories/s2015.htm © 2012 The Climate Corporation. All Rights Reserved. Policies are underwritten by State 14 National Insurance Company, Inc. and administered by The Climate Insurance Agency LLC.
- The devil is in the (Spatial) Indices Image: German zipcodes, R*Tree http://en.wikipedia.org/wiki/R*_tree © 2012 The Climate Corporation. All Rights Reserved. Policies are underwritten by State 16 National Insurance Company, Inc. and administered by The Climate Insurance Agency LLC.
- Thank you!
Editor's Notes
- IntrouctionsWho am INick Dimiduk, Data Platform teamwhat I do:Help growers manage risk. Sell insurance.3 bad years = lose your farm”what willthe weather be like this spring in Jasper County, IL?”“How many consecutive days above 74 degs?”“How similar is the weather Sioux Falls, SD vs. Dayton OH?”the data:*get data stats from zimmer*
- IntrouctionsWho am INick Dimiduk, Data Platform teamwhat I do:Help growers manage risk. Sell insurance.3 bad years = lose your farm”what willthe weather be like this spring in Jasper County, IL?”“How many consecutive days above 74 degs?”“How similar is the weather Sioux Falls, SD vs. Dayton OH?”the data:*get data stats from zimmer*
- On the sideCoauthor: Amandeep Khurana, ClouderaIn print this fall“Generous” discount code
- USGS (US Geological Survey) has a boring definition“Software system capable of capturing, storing, analyzing, displaying geographically referenced information”
- TN River Gorge, ChattanoogaMaps + Data actionable insight from dataThat happens to make really pretty picturesDifferent views for different peopleReally not all that different from this whole “Big Data” thingMap = dataBase layers:Terrain information: dataRiver and lake boundaries: dataCities, roads: dataReally just data + data in a picture (interactive?)Quick Poll: “how many of you make pretty pictures in your day-to-day data activities?”
- Not all fun and gamesMost GIS built by geographers… for bureaucratsNo software engineering experience or motivation“state of the art” is ArcGISthis is not a modern technical landscapeThe World isn’t flatMostly 3D informationWhich, btw, often changes over time (4D)Reduced to 2D for everyone’s convenienceStored in a 1D world (disk platters read linearly)“Here are Dragons”Hunt-Lenox Globe, 1503-07Unknown areas, stories of lions and dragons
- What does Climate do with Geo data?Historically:StoringAnalyzingNow and future:CapturingDisplaying
- Queries against spatial dataGeometry/Geography as first-class citizensIntersperse spatial queries with other attributesit’s all just data, remember?Geometric queries:Example of an “intersection” queryAlso: containment, overlap.Describe intersection between two geometries: “Dimensionally Extended 9-intersection model (DE-9IM)”Nearest neighborNo linear measurements (miles, kilometers) involved!GIS visualizing query results from early prototype.
- Measuring units becomes trickyAngular distance is not linear distance.Know that old joke about physicists? “Assume a spherical horse”Earth is an irregular sphereApproximated into idealUsing planar (2D) coordinatesCoordinate Reference Systems180 deg does not a triangle make. Validate your assumptions.
- Or “Horizontal scaling of geospatial systems.” In the cloud.What’s the data?Vector data: 1.5B features (geometry + metadata)Raster data: whole US on 10m resolutionReference into 30-100+ years’ worth of historical time-seriesAll on AWS.
- Preformant access to data requires indexingLinearization via Space-filling curveIndex 2D data in a single dimensionPreserve locality as much as possibleZ curve, Hilbert curve, etcGeohashingFar from perfect, edge-cases still hurtWorks okay for points but not for arbitrary geometries
- Horizontal scaling requires partitioningMany (2) ways to slice it (boo)Domain-partitioning: cut the world into chunksLon, lat are fixed domain. Simple to split evenly (hemispheres)Poor distribution of work.Range-partitioning: split according to the values you haveScale according to data densityEffective partitioning requires knowledge of your dataOr a specialized data-structure (foreshadowing)
- Many (2) ways to slice it (boo)Domain-partitioning: cut the world into chunksLon, lat are fixed domain. Simple to split evenly (hemispheres)Poor distribution of work.Range-partitioning: split according to the values you haveScale according to data density
- Preformant access to data requires indexingDimensionally aware indicesKD-Trees work great for point data (nearest neighbors)R-Tree variants for arbitrary geometries, but costly to constructUniform partitions => uniform trees => uniform access performanceTwo approaches to scaling:2-layer indices1st layer: coarse-grained partition2nd layer: specialized indexThis is MD-HBaseEasier to implement. Potentially miss geometries => incomplete results!!!Persisted spatial indicesImplement persisted R-TreeCustom regions via RegionSplitPolicy (0.94+)Should be more correct… “there are dragons”
- Questions?
Advertisement