This document presents GPSInsights, a framework for efficiently storing and mining massive real-time vehicle location data. The framework uses an architecture based on scalable open-source components like Apache Spark, Storm, MongoDB and GeoMesa to store GPS data in real-time and perform mining tasks like scalable map-matching. The document describes experiments on 12 million GPS records which showed the framework can complete map-matching within seconds and scale to support millions of vehicles.

1. GPSInsights: Towards an efficient
framework for storing and mining
massive real-time vehicle location data
Linh-Truong Hoang, Duy-Khanh Bui, Viet-
Trung Tran
Hanoi University of Science and Technology
1

2. Agenda
• Motivation
• System architecture
• Scalable map-matching
• Experimentation
• Conclusion
2

3. Global Navigation Satellite
System (GNSS)
• Autonomous geo-spatial positioning
– position
– velocity
– time
• "Great" points about GNSS
– Free
– Real-time
– No required local infrastructures
3

4. GNSS as part of Intelligent
transport system (ITS)
• "precious" data for real-time traffic
managements
– traffic dashboard
– speed control
– traffic jams monitoring
4
Need
for
collec-ng
and
mining
massive
GNSS
data
in
REAL-­‐TIME

5. GNSS data characteristics
• Real-time
– reported every
second
• Massive in volume
– from millions cars
• "bad" data
• Need to be
processed within
digital map topology
5

6. GNSS data is Bigdata's 5V
6

7. SYSTEM ARCHITECTURE
Store massive GNSS data
Real-time mining
7

8. 8
Elas(city
High-­‐throughput
Fault-­‐tolerance
Scalable
First-­‐class
spa(o-­‐temporal
API
High-­‐thoughput
Fault-­‐tolerance
Online
processing
Scalable
Fault-­‐tolerence
Leverage
opensource
components

9. 9

10. Apache spark processing
• Resilient Distributed dataset (RDD)
– In-memory, backed by persistent storage (HDFS)
– fault-tolerance by lineage
– Support interactive – iterative analysis
10

11. Spark streaming
11

12. Apache storm
12

13. MongoDb with geo-indexing
13

14. Geomesa: Accumulo + geo-indexing
14

15. SCALABLE MAP-MATCHING
ALGORITHM
15

16. Map-matching
• Online vs. Offline
• OSM map
16

17. Algorithm
• OSM map format
• Filling intermediate points
– Millions more points
– Massive data
– but simple calculations
• real-time, scalable
17

18. K-d tree for closest neighbours
• Run by apache spark/storm
18

19. EXPERIMENTATION
19

20. Experiment setup
• 12 millions GPS records collected by
vehicles equipped with the GPS receiver in
March 2014
• 4 nodes cluster
– 8-cores Intel Xeon 2.6GHz CPU, 32GB memory
20

21. Map-matching completion time
21

22. Latency
22

23. "Scalability"
23

24. Demonstration
24

25. Real-time traffic monitoring
25

26. Real-time shortest path
26

27. Conclusion
• GPSInsights: Scalable framework for storing
and mining massive location data
– built on open-source scalable components
– scalable storage + real-time mining
– Plug-able components
– Demonstration with scalable map-matching
algorithm
• Future work
– Advance map-matching algorithms
– Traffic jam prediction
27

28. Current state-of-the-arts
• PostGIS
– Spatial objects management
over Postgres
– Small size
– No mining supported
28