Downloaded 33 times
Uploaded byRob Emanuele
Processing Geospatial Data At Scale @locationtech
This document discusses processing large geospatial data at scale. It provides background on big data frameworks like Apache Hadoop, Apache Spark, and geospatial projects like GeoTrellis, GeoWave, and SpatialHadoop that enable processing geospatial data using these frameworks. The document outlines how these tools allow geospatial data from sources like satellite imagery, OpenStreetMap, and geotagged social media to be analyzed using distributed computing platforms and algorithms.
More Related Content
Similar to Processing Geospatial Data At Scale @locationtech
Processing Geospatial Data At Scale @locationtech
- 1. PROCESSING GEOSPATIAL DATA @SCALE Rob Emanuele GEO(MESA/WAVE/TRELLIS/JINNI)
- 2. What we’ll becovering… What does “processing geospatial data at scale” mean? Background on big data frameworks What is LocationTech? Overview of LocationTech projects for processing big geo data.
- 3.
- 4.
- 5. Large geospatial data Landsat8 on AWS: 465,68 scenes @ ~800 MB each.That's 355 TB and counting. OpenStreetMap edit history: 75 GB compressed. 3 years of geotagged tweets: 3 TB
- 6. NED 1/3 arcsecond
- 7. NED 1/3 arcsecond
- 8. NED 1/3 arcsecond
- 9.
- 10.
- 11.
- 13. Project to builda better search engine, back in the early 2000’s. Worked for small datasets, but was not scalable.
- 14.
- 15. After reading thepapers, Nutch developers added a distributed file system and MapReduce model to Nutch. In 2006, those portions were spun out of Nutch to form…
- 17. Apache Hadoop Heavily supportedbyYahoo, which moved it’s large data processing to Hadoop. by 2007,Twitter, Facebook, LinkedIn and many others were doing serious work with Hadoop 2008 Hadoop graduated to a top level Apache project
- 18.
- 19. Matei Zaharia Worked withHadoop at UC Berklee Noticed Hadoop was not a good fit for Machine Learning algorithms and other iterative models. So in 2009, he created…
- 22. Open sourced in2010 under BSD license Maintained by UC Berkeley’s AMPLab Donated to the Apache Software Foundation in 2013 and relicensed as Apache 2.0 Graduated to a top level Apache project in 2014 Apache Spark
- 23. Apache Spark a distributedcomputation engine. An API that lets you work with distributed data as a collection. Written in Scala, with language bindings for use with Java, Python, and R.
- 30.
- 32. Apache Accumulo Created bythe NSA in 2008 Donated to the Apache Foundation in 2011 Graduated to a top level project in 2012 Almost defunded by the US government the same year.
- 33. (Sec. 929) Prohibitsany DOD component from utilizing the cloud computing database developed by the National Security Agency (NSA) and known as "Accumulo" after the end of FY2013, unless the DOD CIO certifies that: (1) there are no viable commercial open source databases that have such security features, or (2) Accumulo itself has become a successful open source database project. Requires DOD and intelligence community officials to coordinate the use by DOD components of cloud computing infrastructure and services offered by the intelligence community for purposes other than intelligence analysis.
- 34. (Sec. 929) Prohibitsany DOD component from utilizing the cloud computing database developed by the National Security Agency (NSA) and known as "Accumulo" after the end of FY2013, unless the DOD CIO certifies that: (1) there are no viable commercial open source databases that have such security features, or (2) Accumulo itself has become a successful open source database project. Requires DOD and intelligence community officials to coordinate the use by DOD components of cloud computing infrastructure and services offered by the intelligence community for purposes other than intelligence analysis.
- 35. Data Node Data Node Data Node Name Node Master Tablet Server Tablet Server Tablet Server Accumulo BigTable clone (columnardatabase) Records stored on HDFS Lexicographically sorted table index
- 38.
- 39.
- 43.
- 47.
- 48. Spatial language Built-inspatial data types Spatial Indexes Spatial Operations
- 50. 72 Frames ×14 Billion points per frame Total = 1 Trillion points Generated in three hours on a 10-node cluster HEAT MAP FROM 2009 TO 2014 MONTH-BY-MONTH
- 54.
- 62. SELECT tweet.text, user.name FROM tweet, user WHERE bbox(tweet.location, -115,45, -110, 50) AND tweet.user_id = user.user_id +
- 64. GeoTrellis a Scala libraryfor geospatial data types and operations. enables Spark with geospatial capabilities (mainly raster, currently working on vector) storage and query raster from HDFS, Accumulo, and S3 (Cassandra support in development) 0.10 is released!
- 69.
- 70.
- 71. 100 spot instancem3.xlarge workers @ $0.04 / hr = $4.00 / hr 400 CPUs / ≈1.5 TB memory 1 master m3.xlarge on-demand instance @ $0.26 / hr EMR cluster charge, $0.07 / hr $4.37 / hr Rendering elevation with hillshade + NLCD on AWS EMR
- 76.
- 77. Index type Z-order spatial& spatiotemporal binned by week Hilbert in N-dimensions with tiered indexing and binning Backends supported Accumulo (main), Cassandra, HBase, DynamoDB, Google cloud Bigtable Accumulo (main), HBase Servers supported GeoServer GeoServer, Mapnik Processing Frameworks supported Hadoop, Spark, Storm, Kafka Hadoop, Spark Language Scala Java
- 78. Index type Z-order spatial& spatiotemporal binned by week Hilbert in N-dimensions with tiered indexing and binning Backends supported Accumulo (main), Cassandra, HBase, DynamoDB, Google cloud Bigtable Accumulo (main), HBase Servers supported GeoServer GeoServer, Mapnik Processing Frameworks supported Hadoop, Spark, Storm, Kafka Hadoop, Spark Language Scala Java
- 79.
- 80.
- 81. Binning (time dimension) 19971998 1999
- 82.
- 84.
- 85.
- 87. • Working togetherto learn to collaborate • Making the connections necessary that allow collaboration to flourish
- 88. • Join thelocationtech-iwg mailing list • Share you big geospatial data challenges • Propose projects Get involved!
- 89.