Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Big Spatial(!) Data Processing mit GeoMesa. AGIT 2019, Salzburg, Austria.

16 views

Published on

This talk introduces GeoMesa and discusses how it can be used to store and analyze massive amounts of movement data.

Talk recording: https://av.tib.eu/media/42874

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Big Spatial(!) Data Processing mit GeoMesa. AGIT 2019, Salzburg, Austria.

  1. 1. BIG SPATIAL(!) DATA PROCESSING MIT Anita Graser Center for Mobility Systems, AIT Austrian Institute of Technology
  2. 2. Research results WHAT WE DO Movement data Spatial context data Evaluation & tuning Model application Algorithms Trained models Exploration & hypothesis formulation Model building & training
  3. 3. #1 Taxis in Vienna > 3.5 billion records since 2005 #2 Automatic Identification System (AIS) Data 500 million records per day #3 Mobile phone network data 3 billion records per day (one big Austrian provider) ANALYZING MASSIVE MOVEMENT DATA
  4. 4. WHY WE BOTHER? Too much waiting  Not enough time for data exploration & method development
  5. 5. OPEN & SPATIAL & SCALABLE 5https://projects.eclipse.org/wg/locationtech/projects
  6. 6. Short answer: * and other big data stores WHAT IS GEOMESA? GeoMesa is to Accumulo* what PostGIS is to PostgreSQL
  7. 7. WHAT IS GEOMESA? Source: Constantin Stanca “High Performance and Scalable Geospatial Analytics on Cloud with Open Source”
  8. 8. Features  Store gigabytes to petabytes of spatial data (tens of billions of points or more)  Serve up tens of millions of points in seconds  Ingest data faster than 10,000 records per second per node  Scale horizontally easily (add more servers to add more capacity)  Support Spark analytics  Drive a map through GeoServer or other OGC Clients GEOMESA 829/08/2019 http://www.geomesa.org/documentation/user/introduction.html#what-is-geomesa
  9. 9. GEOMESA https://www.geomesa.org/documentation/user/architecture.html
  10. 10. Spatial extension for Accumulo  Distributed  Spatially indexed GEOMESA Zoo keeper Hadoop
  11. 11. GEOMESA – SPATIAL INDEX
  12. 12. … make 2/3D data sortable SPACE-FILLING CURVES 12 Fox, A., Eichelberger, C., Hughes, J., & Lyon, S. (2013, October). Spatio-temporal indexing in non-relational distributed databases. In Big Data, 2013 IEEE International Conference on (pp. 291-299). IEEE.
  13. 13. geomesa export -c geomesa.gdelt -f gdelt -u root -p GisPwd -q "CONTAINS(POLYGON ((0 0, 0 90, 90 90, 90 0, 0 0)),geom)" -m 3 Using GEOMESA_ACCUMULO_HOME = /opt/geomesa id,globalEventId:String,...,dtg:Date,*geom:Point:srid=4326 139...,671713129,...,2017-07-10T00:00:00.000Z,POINT (5.43827 5.35886) 9e8...,671928676,...,2017-07-10T00:00:00.000Z,POINT (5.43827 5.35886) d6c...,671817380,...,2017-07-09T00:00:00.000Z,POINT (5.43827 5.35886) More complex queries & analyses  Spark(SQL)! SPATIAL QUERIES
  14. 14. GEOMESA Source: Constantin Stanca “High Performance and Scalable Geospatial Analytics on Cloud with Open Source”
  15. 15. http://www.geomesa.org/documentation/user/spark/sparksql_functions.html Geometry Constructors • st_geometryFromText • st_makeBBOX • st_makeLine • st_makePoint • st_makePolygon • … Geometry Accessors • st_geometryN • st_isValid • st_pointN • st_x • … Geometry Outputs • st_asGeoJSON • st_asText • … Spatial Relationships • st_area • st_centroid • st_closestPoint • st_contains • st_covers • st_crosses • st_disjoint • st_distance • st_distanceSphere • st_distanceSpheroid • st_equals • st_intersects • st_length • st_lengthSphere • st_lengthSpheroid • st_overlaps • st_relate • st_touches • st_within Geometry Processing • st_bufferPoint • st_convexHull • … GEOMESA-SPARK-SQL MODULE
  16. 16. 1629/08/2019
  17. 17. 1729/08/2019
  18. 18. 1829/08/2019
  19. 19. 1929/08/2019
  20. 20. 2029/08/2019
  21. 21. 2129/08/2019
  22. 22. Plugin for GeoServer  GeoMesa data store GEOMESA & GEOSERVER 22
  23. 23. WMS-T ttp://10.101.21.11:8080/geoserver/geomesa/wms?service=WMS&version=1.1.0&request=GetMap &layers=geomesa:aisdk&styles=point&bbox=-180.0,-90.0,180.0,90.0&width=1500&height=780 &srs=EPSG:4326&format=application/openlayers &TIME=2017-06-01T06:00:00.000Z/2017-06-01T06:30:00.000Z
  24. 24. WMS-T & QGIS TIME MANAGER
  25. 25. Large technology stack  Only specific versions work together  Challenging to set up & manage PRACTICAL ASPECTS
  26. 26. Setup on one machine for experimental purposes  GeoDocker https://github.com/geodocker/geodocker-geomesa  CCRI’s cloud-local https://github.com/ccri/cloud-local FIRST STEPS
  27. 27. CONTACT Anita Graser anita.graser@ait.ac.at @underdarkGIS anitagraser.com

×