Exploratory Analysis of Massive Movement Data (RGS-IBG GIScience Research Group Seminar 2021)

1. EXPLORATORY ANALYSIS OF MASSIVE MOVEMENT DATA RGS-IBG GIScRG webinar series Anita Graser Supported by the Austrian Federal Ministry of Climate Action, Environment, Energy, Mobility, Innovation and Technology (BMK) within the programme “IKT der Zukunft” under Grant 861258 (project MARNG).

2.  Gathering data, massaging it into a tractable form, making it tell its story, and presenting that story to others  Dealing with data that incorporates spatial and often temporal elements  Turning Big Spatiotemporal Data into insight and understanding GEOGRAPHIC DATA SCIENCE Big Data & Data Science Geography Geographic / Spatial Data Science Quant. Geogr. & Spatial Statistics & GIScience & Geomatics CS & Math & Statistics AI / ML Humanities STEM Critical Data Studies ICT & Engineering Andrienko et al. (2017) Geographic Data Science Singleton & Arribas-Bel (2021) Geographic Data Science

3.  Opportunistic reuse of data  Black box / undocumented data collection  Usually biased & messy data MOVEMENT DATA SCIENCE Geotagged social media posts Cell phone network data Mobile app usage data Vehicle tracking systems Check-in data WiFi tracking “All metadata records are incomplete as it is impossible to foresee future uses” Janowicz et al. (2020) GeoAI Payment data

4.  Accept messiness in data  Need to understand  Causes of bias & messiness  Consequences of using such data in analyses  Data visualization & exploratory approaches CHALLENGES Brunsdon & Comber (2020) Big issues for big data Graser & Dragaschnig (2020) Open Geospatial Tools for Movement Data Exploration

5.  Complex spatiotemporal phenomena  Context & scale dependent  Spatial, temporal & attribute uncertainty EXPLORATORY DATA ANALYSIS (EDA) OF MOVEMENT DATA Demšar & Virrantaus (2010) Space-time density of trajectories Andrienko & Andrienko (2011) Spatial generalization and aggregation of massive movement data Andrienko et al. (2017) Visual exploration of movement and event data with interactive time masks + Lack of established tools & practices

6. MARITIME MOVEMENT DATA  Unconstrained movement  Strongly varying trip properties: e.g. regarding duration & spatial extent  Observation gaps: data sources usually limited to certain regions  Unreliable information: e.g. movement speed and direction, trip destination, vessel identity, time stamps Image source: https://anitagraser.com/2017/10/28/movement-data-in-gis-10-open-tools-for-ais-tracks-from-marinecadastre-gov/

7. EDA OF MASSIVE MARITIME MOVEMENT DATA Input: AIS by Danish Maritime Authority for 2017  4 billion records by 89,926 distinct vessels Graser et al. (2020) The M³ massive movement model

8. AGGREGATING UNCONSTRAINED MOVEMENT DATA

10. © Eric Fischer © Ravi Shekhar © Marine Traffic © Strava

11. Why density grids are not good enough: M³ MASSIVE MOVEMENT MODEL PROTOTYPE AGGREGATION Graser et al. (2020) The M³ massive movement model

12. Multiple (up to nmax) prototypes per cell Gaussian mixture models (GMMs) including  Location  Speed  Direction  Count M³ MASSIVE MOVEMENT MODEL PROTOTYPE AGGREGATION Graser et al. (2020) The M³ massive movement model

13. Graser et al. (2020) The M³ massive movement model  Scalable  Grid cells independent  Independent of input order  Fast  Spark implementation (41min for 4 billion records on 8 node cluster)  Supports streaming data M³ MASSIVE MOVEMENT MODEL PROTOTYPE AGGREGATION

14. QUICK TECHNICAL BACKGROUND

15. Link to “GeoAI” or Predictive Analytics  Anomaly detection  Trajectory prediction M³ MASSIVE MOVEMENT MODEL PROTOTYPE AGGREGATION Graser & Widhalm (2018) Modelling Massive AIS Streams with Quad Trees and Gaussian Mixtures Graser et al. (2019) Data-driven Trajectory Prediction and Spatial Variability of Prediction Performance in Maritime Location Based Services

17. Cleaning & splitting at stops & observation gaps TRAJECTORY AGGREGATION Graser et al. (2020) Exploratory Trajectory Analysis for Massive Historical AIS Datasets

18. Challenges  No straight-forward spatial or temporal binning  Records of individual ships need to be processed in chronological order  Individual trips can be weeks long  collected records exceed available memory Implementation  Needs to iteratively process the sorted records WITHOUT materializing them in memory all at once  1 hour runtime for 6 months of AIS (on 6 node cluster) Graser et al. (2020) Exploratory Trajectory Analysis for Massive Historical AIS Datasets TRAJECTORY AGGREGATION

19. Challenges  No straight-forward spatial or temporal binning  Records of individual ships need to be processed in chronological order  Individual trips can be weeks long  collected records exceed available memory Implementation  Needs to iteratively process the sorted records WITHOUT materializing them in memory all at once  1 hour runtime for 6 months of AIS (on 6 node cluster) Graser et al. (2020) Exploratory Trajectory Analysis for Massive Historical AIS Datasets TRAJECTORY AGGREGATION

20. Link to “GeoAI” or Predictive Analytics  Travel time prediction  Destination prediction TRAJECTORY AGGREGATION Travel times between Gothenburg and Gibraltar Graser et al. (2020) Exploratory Trajectory Analysis for Massive Historical AIS Datasets

22. Movement between prototypes  Prototype pair  Speed  Count  Scalable ✘ Not suitable for data streams M³ MASSIVE MOVEMENT MODEL FLOW AGGREGATION Graser et al. (2020) Extracting Patterns from Large Movement Datasets

23. Passenger patterns Tanker patterns Graser et al. (2020) Extracting Patterns from Large Movement Datasets M³ MASSIVE MOVEMENT MODEL FLOW AGGREGATION

24. Passenger vessel speed & variability Graser et al. (2020) Extracting Patterns from Large Movement Datasets M³ MASSIVE MOVEMENT MODEL FLOW AGGREGATION

25. Link to “GeoAI” or Predictive Analytics  “Routing graph”  Improved trajectory prediction M³ MASSIVE MOVEMENT MODEL FLOW AGGREGATION

27. Continued need for domain-specific movement EDA tools 1. Best practices / checklist tools  Systematic evaluation of data quality * 2. Openness & reproducibility  Document all operations undertaken on the data 3. Privacy  Built-in privacy protection 4. Uncertainty  Efficient evaluation & visualization concepts for uncertainty SOME FUTURE CHALLENGES * Graser (in press) An exploratory data analysis protocol for identifying problems in continuous movement data.

28. anita.graser@ait.ac.at @underdarkGIS ANITA GRASER