Distributed georeferenced raster processing on Spark with GeoTrellis
Report
Share
•0 likes•434 views
Distributed georeferenced raster processing on Spark with GeoTrellis
•0 likes•434 views
Report
Share
Download to read offline
GeoTrellis is a geographic data processing engine for high performance applications. This presentation is focused on how Spark RDD partitioning scheme can influent the whole Spark application behaviour.
Recommended
More Related Content
What's hot
What's hot(20)
Similar to Distributed georeferenced raster processing on Spark with GeoTrellis
Similar to Distributed georeferenced raster processing on Spark with GeoTrellis(20)
Recently uploaded
Recently uploaded(20)
Distributed georeferenced raster processing on Spark with GeoTrellis
- 1. DISTRIBUTED GEOREFERENCED RASTER PROCESSING ON SPARK Grigory Pomadchin @daunnc / @pomadchin
- 2. GEO + VectorTiles + PointClouds + ——————————————————————————————— Raster +Vector + /w
- 3. • Raster Foundry (Spark SQL & ML) • Raster Frames (Spark SQL & ML, Datasets query API) • GeoPy Spark (Python bindings) • Vector Pipes (Vector tiles on Spark) • PDAL intergration (PointClouds on Spark) GEOTRELLIS ECOSYSTEM
- 4. WHATS UNDER THE COVERS?
- 7. • RDD (a basic core spark type from the past (no)) • Manual partitioning control • DATASET • Query planning optimizations, more related to already well partitioned and structured data. PARTITIONING SCHEME SPECIAL BROWN COLORED FUNCTIONS • Join • groupByKey • reduceByKey • combineByKey • Repartition • Each function that has no preservePartitioning flag or can accept partitioner as an argument, probably map?
- 8. MAP IS A FUNCTION OF A DIFFERENT KIND?
- 9. join reduceByKey join map reduceByKey join mapVlues reduceByKey MAP IS A FUNCTION OF A DIFFERENT KIND?
- 10. • inspired by Eugene Cheipesh slides
- 11. DATA PREPARATION • {hadoop | s3}GeoTiffRDD loads data from {HDFS / LocalFS | S3} into Spark • (I, V) - {ProjectedExtent(extent, crs) | TemporalProjectedExtent(extent, crs, time)}, {Multiband | Singleband}Tile • K - {Spatial(col, row) | SpaceTime(col, row, time)}
- 12. • inspired by Eugene Cheipesh slides
- 13. WAT?! • Load data into Spark memory according to some partitioning scheme • Ahead of shuffle: smaller chunks are better for Spark (as the max shuffle block size is only 2GBs) • Are we dependent on the input data type? (yes) • Window reading (what’s the desired / perfect window size?)
- 14. SPARK SHUFFLE BLOCK FEATURE • ~ 128mb per partition (rule of a thumb) • if(partitionsNumber ~ 2000) repartition(> 2000)
- 15. WINDOWED READS
- 16. WINDOWED READS • Here we have a sort of some crop function by grid bounds on each element: tiff.crop(gridBounds) (it is the meaning of rr.readWindows func)
- 17. WINDOWED READS • 13 GB loads not efficient into memory of three AWS m3.xlarge instances .
- 18. WINDOWED READS • Instead of 13Gb it fetches even 40Gb per partition…
- 19. WINDOWED READS • The solution is to pack segments into desired windows based on the input format requirements • After all the main idea is to leverage the gains by having a good partion scheme
- 20. READ / WRITE • SFC index and parallelism level control • Cassandra and range queries example (range queries and compare it to spark Cassandra connector, queries parallelism inside Spark partitions)
- 21. READ / WRITE
- 22. API & SPARK PROBLEMS • Spark has its limitations • It’s not required for a small data amount (In the real time case even milliseconds are important, otherwise we have to live somehow with the Spark slow responses) • The second API in addition to the RDD API is the answer? (Collections API; does it make any sense to abstract over RDDs and Collections?)
- 23. • https://github.com/locationtech/geotrellis • https://geotrellis.io • https://www.azavea.com • https://www.azavea.com/blog • https://yuns-stacy.github.io/geotrellis-angular-demo/dashboard • http://rasterframes.io/ml/statistics.html • https://github.com/pomadchin