This document summarizes a fast offline reverse geocoding library in Python. It discusses the motivation for developing the library to geocode terabytes of data from OpenSignal at a faster speed than online services or other offline options. The library improves on an existing one by supporting Python 2 and 3, geocoding more information, and with high performance through a multi-process parallelized k-d tree implementation. Demo and performance results are shown, and applications for the library include analyzing mobile data regions and speed tests.

1. A FAST OFFLINE REVERSE
GEOCODER IN PYTHON
Ajay Thampi
Data Scientist, OpenSignal

2. OUTLINE
OpenSignal
Motivation
The Library
Demo
Performance Results
Applications
Contributions from the Community

3. OPENSIGNAL
http://opensignal.com | http://wifimapper.com
Cellular data points: 41 billion
WiFi data points: 50 billion
Speed tests: 93 million

4. MOTIVATION
Reverse geocode terabytes of data (~50M coordinates / day)
Options:
Online web services (Google Maps, OpenStreetMap)
Restrictive
Slow
Offline (PostGIS, Python libraries)
Complex
Slow

5. THE LIBRARY
Improves on an existing library by Richard Penman
Supports Python 2 and 3
Geocodes a lot more information
High Performance
Open Source (LGPL license)
Statistics: (since 27/03/2015)
Downloads: 2,649
Commits: 41
Committers: 5
Stars: 1,089
Forks: 40
#notsohumblebrag
• Place name
• Country Code (ISO-3166)
• Admin region 1
• Admin region 2
• Coordinates

6. IMPLEMENTATION
Two modes:
Mode 1: Single-process
Mode 2 (Default): Multi-process
Source of data: GeoNames
Places with a population > 1000 (Total = 144,859)
GPS coordinates of cities loaded into a K-D Tree
Nearest neighbour (NN) algorithm
Mode 1: cKDTree class in scipy
Mode 2: Parallelised cKDTree
Dependencies:
numpy
scipy

7. PARALLELISED K-DTREE
Uses the multiprocessing module
Pros over threading:
Exploits multiple CPUs and cores
No GIL limitation
Cons over threading:
Separate memory space => IPC or Shared Memory
Static Scheduling
K-D Tree Settings:
Euclidean distance (Minkowski p-norm where p = 2)
Distance upper bound: Inf
Refer multiprocessing tutorial by Sturla Molden, University of Oslo

8. DEMO

9. PERFORMANCE RESULTS

10. APPLICATIONS (1/2)
• Top 20 regions in the UK where OpenSignal users
run speed tests
Data from Sep-Dec 2014

11. APPLICATIONS (2/2)
• Speed test data points from the Greater London region
Data from Sep-Dec 2014Visualisation using Google FusionTables

12. HATTIP
Python 3 Support (Brandon Liu and David J. Felix)
C++ Wrapper (Mehdi Lauters)

13. @thampiman
ThankYou
Q & A