The presentation by Hannes Hapke at the Pydx Conference covers the essentials of geospatial data handling, including geocoding, serving geospatial data via APIs, and using tools like PostGIS, GDAL, and GeoDjango. It emphasizes the importance of location data and demonstrates various coding examples for geocoding addresses, generating KML files, and creating APIs with Django REST Framework. Additionally, it highlights privacy considerations and suggests useful resources for further learning in geospatial development.

1. Python, Geodata and
Maps
PyDX Conference, Oct 11th 2015
Hannes Hapke
@hanneshapke

2. Hello, my name is
Hannes.

3. Why could this presentation
be interesting to you?
Location data is key today!

4. What will you know in 30 min?
• Geospatial basics
• Geocode locations and other sources
• How to make geospatial queries
• How to serve geospatial data via APIs

5. Broad Field …
SRID, PostGIS, GDAL, tiles, Open Street
Map, Spatial queries, Mapnik, Shapely,
ogr2ogr, kml, GIS, Proj, TIGER,
SpatiaLite, Mapbox, leaflet, raster,
shapefiles, geometries, EPSG, WGS 84,
layers, ArcGIS, bbox, distance loopup,
datum, GEOS, geocode, WKT, WKB,
GeoDjango, GPS, projections, tiles

6. Broad Field …
SRID, PostGIS, GDAL, tiles, Open Street
Map, Spatial queries, Mapnik, Shapely,
ogr2ogr, kml, GIS, Proj, TIGER,
SpatiaLite, Mapbox, leaflet, raster,
shapefiles, geometries, EPSG, WGS 84,
layers, ArcGIS, bbox, distance loopup,
datum, GEOS, geocode, WKT, WKB,
GeoDjango, GPS, projections, tiles

7. Geospatial Basics

8. Just a very few …

9. Coordinate Systems
Source: https://developer.apple.com/library/ios/documentation/UserExperience/Conceptual/LocationAwarenessPG/Art/sphere_to_cylinder.jpg
projected
vs.
unprojected
coordinate systems

10. SRID
Spatial Reference System Identifier
spatialreference.org

11. Remember
4326

12. Projections
Source: http://gdb.voanews.com/9E85F1AA-B4C0-49FA-ADC0-27BA1FD1C426_mw1024_s_n.jpg

13. Projections
WGS-84 GCJ-02
Same, same, but different …

14. GIS Data
raster format
vector format
other formats

15. Storing Geospatial Data
Sources
SQLite: https://upload.wikimedia.org/wikipedia/commons/thumb/3/38/SQLite370.svg/640px-SQLite370.svg.png
Postgres: http://www.tivix.com/uploads/blog_pics/postgresql_1.png

16. Postgres is a defacto
standard
Source
PostGIS: http://www.moderncourse.com/wp-content/uploads/2015/03/postgis.png
Thanks to PostGIS

17. PostGIS, what?
Adds geospatial functionality
to Postgres
Source
PostGIS: https://en.wikipedia.org/wiki/PostGIS

18. Useful tools …

19. QGIS
Source
QGIS: https://underdark.files.wordpress.com/2012/07/stamen_watercolor1.png?w=700

20. CartoDB
Source
CartoDB: http://dmdphilly.org/wp-content/uploads/2015/07/cartodb-editor.2876823e.png

21. Enough basics!
Let’s get started …

22. Wait, one last
comment …

23. Consider the
user’s privacy …

24. Sources for Geodata

25. Geocode Addresses

27. Geocoder
• pip install geocoder
• Package by Denis Carriere
• Supports 20 geocoding services
• Multiple export data structures
• Command line interface

28. Code Example
#!/usr/bin/python
import geocoder
"""Sample code to demonstrate geocoding"""
address = "70 NW Couch St, Portland, OR 97209"
result = geocoder.google(address)
print ("Your address: %s" % address)
print ("%s, %s" % (result.lng, result.lat))
print ("Location type: %s" % result.accuracy)

29. Geolocating IP
Addresses

30. Code Example
#!/usr/bin/python
import geocoder
"""Sample code to demonstrate the ip geocoder
IP address obtained via http://fakeip.org/"""
ip_address = "192.169.226.73"
result = geocoder.ip(ip_address)
print ("Your ip address: %s" % ip_address)
print ("Location address: %s" % result.address)
print ("%s, %s" % (result.lng, result.lat))

31. Images as
geodata sources

32. Code Example
#!/usr/bin/python
from pyexif import pyexif
"""Sample code to demonstrate pyexif"""
file_name = “your_image.JPG"
result = pyexif.Exif(file_name)
print ("Your location: %s %s" % result.lat, result.lon)
GPS Latitude 44.100339 degrees
GPS Longitude121.772294 degrees
GPS Altitude 3109.044444 m
GPS Time Stamp 21:49:20.09
GPS Speed 0.26
GPS Img Direction Ref True North
GPS Img Direction 233.1646707
GPS Dest Bearing Ref True North
GPS Dest Bearing 233.1646707
GPS Date Stamp 2015:08:15

33. Shapefiles

34. What are Shapefiles?
Source: http://store.mapsofworld.com/image/cache/data/GIS/Oregon%20County-900x700.jpg

35. Obtain Shapefiles
# Install the gis stack (geos, proj4, postgres, postgis, gdal)
$ brew install postgis gdal
# Clone the Portland Atlas
$ git clone https://github.com/Caged/portland-atlas.git
$ cd portland-atlas/
$ make shp/neighborhoods.shp
# Check structure
$ ogrinfo -al -geom=OFF shp/neighborhoods.shp
# Check projection
$ ogrinfo -so shp/neighborhoods.shp -sql "SELECT * FROM neighborhoods"

36. Generate Model from Shapefile
$ python manage.py ogrinspect
sample_geodjango/data/neighborhoods.shp
PortlandNeighborhood --srid=4326 --mapping
# This is an auto-generated Django model module created by ogrinspect.
from django.contrib.gis.db import models
class PortlandNeighborhood(models.Model):
objectid = models.FloatField()
name = models.CharField(max_length=50)
shape_leng = models.FloatField()
…
geom = models.PolygonField(srid=4326)
objects = models.GeoManager()
# Auto-generated `LayerMapping` dictionary for PortlandNeighborhood model
portlandneighborhood_mapping = {
'objectid' : 'OBJECTID',
'name' : 'NAME',
'shape_leng' : 'SHAPE_LENG',
…
'geom' : 'POLYGON',
}

37. Load Shapefile into your DB
import os
from django.contrib.gis.utils import LayerMapping
from .models import Neighborhood
neighboorhood_mapping = {
'name': 'NAME',
'poly': 'POLYGON',
}
neighboorhood_shp = os.path.abspath(
os.path.join(
os.path.dirname(__file__),
'../../data', 'neighborhoods.shp'))
def run(verbose=True):
lm = LayerMapping(
Neighborhood,
neighboorhood_shp, neighboorhood_mapping,
transform=False, encoding='iso-8859-1')
lm.save(strict=True, verbose=verbose)

38. Now, what to do with
the data?

39. Generate KML files
with Python

40. How to create kml data
structures?
• Keyhole Markup Languange (KML)
• XML based
• Developed by Google for Google Earth
• Package SimpleKML
https://pypi.python.org/pypi/simplekml/1.2.8

41. Code Example
""" Example code which will generate a kml file of
Portland's neighborhoods
Execute this script with the runscript command
of the Django extensions
$ python manage.py runscript simplekml_example -v3
"""
import simplekml
from project.models import Neighborhood
def run():
kml = simplekml.Kml()
neighborhoods = Neighborhood.objects.all()
for neighborhood in neighborhoods:
# this is just bare example
# normally use neighborhood.poly.kml to the kml data
kml.newpolygon(
name=neighborhood.name,
outerboundaryis=list(
neighborhood.poly.boundary.coords[0]),
innerboundaryis=list(
neighborhood.poly.boundary.coords[0])
)
kml.save("portland_neighborhoods.kml")

42. Google Earth

43. Generate Static Maps

44. Motionless
• pip install motionless
• Package by Ryan Cox
• Supports street, satellite, and terrain maps
• Based on Google Static Map API
• Watch out for the rate limit

45. Code Example
#!/usr/bin/python
"""Sample code to demonstrate motionless"""
import requests
from StringIO import StringIO
from PIL import Image
from slugify import slugify
from motionless import DecoratedMap, AddressMarker
address = "70 NW Couch St, Portland, OR 97209"
# generate static map object
dmap = DecoratedMap(maptype='satellite', zoom=18)
dmap.add_marker(AddressMarker(address))
# download the static image
response = requests.get(dmap.generate_url())
if response.status_code == 200:
image = Image.open(StringIO(response.content))
image.save('.'.join([slugify(address), 'png']))
else:
print ("Download error with status code %s",
response.status_code)

46. Code Example

47. But what about
database queries?

48. GeoDjango

49. GeoDjango
• Created by Justin Bronn
• Part of the Django core
• Supports various spatial databases and libraries
• Let’s you define geospatial attributes in your
models

50. GeoDjango
• PointField
• LineStringField
• PolygonField
• MultiPointField
• MultiLineStringField
• MultiPolygonField

51. GeoDjango Queries

52. GeoDjango
Spatial Lookups
bbcontains, bboverlaps, contained, contains,
contains_properly, coveredby, covers,
crosses, disjoint, equals, exact, same_as,
intersects, overlaps, relate, touches, within,
left, right, overlaps_left, overlaps_right,
o v e r l a p s _ a b o v e , o v e r l a p s _ b e l o w,
strictly_above, strictly_below

53. GeoDjango
Distance Lookups
• distance_lt
• distance_lte
• distance_gt
• distance_gte
• dwithin

54. GeoDjango Example

55. GeoDjango - Example
• Simple GeoModel
• Demonstrate the GeoDjango Admin
• How to save Points
• How to search for Points within a Polygon
• Leaflet Maps

56. GeoDjango - Example

57. GeoDjango - Model
class CallLocation(TimeStampedModel):
address = models.CharField(
max_length=255)
…
location = models.PointField()
objects = models.GeoManager()
def save(self, *args, **kwargs):
try:
# get the geocoding results
result = geocoder.google(self.address)
# correct the address spelling
self.address = result.address.encode('utf-8')
self.location = fromstr(
'POINT(%s %s)' % (result.lng, result.lat),
srid=4326)
except (AttributeError, Exception):
print "Oops! Couldn't geocode address because of %s"
% sys.exc_info()[0]
super(CallLocation, self).save(*args, **kwargs)

58. GeoDjango - View
class CallLocationCreateView(CreateView):
model = CallLocation
form = CallLocationForm
success_url = reverse_lazy('home')
fields = ['address', 'call_type', 'comment']
def get_context_data(self, *args, **kwargs):
context = super(
CallLocationCreateView, self
).get_context_data(**kwargs)
context['neighborhoods'] = Neighborhood.objects.all()
context['pk'] = int(self.kwargs.get('pk', -1))
if context['pk'] > 0:
neighborhood = get_object_or_404(Neighborhood, pk=context['pk'])
context['object_list'] = CallLocation.objects.filter(
location__within=neighborhood.poly)
else:
context['object_list'] = CallLocation.objects.all()
return context

59. GeoDjango - Admin
from django.contrib.gis import admin
from .models import CallLocation, CallType, Neighborhood
class GeoLocationAdmin(admin.OSMGeoAdmin):
default_zoom = 11
list_display = (
'call_type', 'address',
'modified',)
class NeighboorhoodAdmin(admin.OSMGeoAdmin):
default_zoom = 11
list_display = ('name',)
admin.site.register(CallLocation, GeoLocationAdmin)
admin.site.register(Neighborhood, NeighboorhoodAdmin)
admin.site.register(CallType)

60. GeoDjango - Admin

61. GeoDjango - Template
<script type="text/javascript">
function initialize() {
var map = new L.Map(
"terrain", {
center: new L.LatLng(45.524221, -122.670749),
zoom: 16
}
);
…
var markers = [];
var bbox = [];
{% for call in object_list %}
markers.push(
L.marker(
[ {{ call.location.coords.1 }}, {{ call.location.coords.0 }} ]
).addTo(map));
bbox.push(
[ {{ call.location.coords.1 }}, {{ call.location.coords.0 }} ]
);
{% endfor %}
map.fitBounds(bbox);
};
…
</script>

62. GeoDjango - Example

63. Django REST
Framework

64. REST+GIS - Example
• Create an API endpoint
• Set up a GeoSerializer
• AngularJS ng-map
• something very practical …

65. REST+GIS - Example

66. What needs to change
in your API code?

67. REST+GIS - Model
Nothing needs to change

68. REST+GIS - Serializer
from .models import CallLocation
from rest_framework import serializers
from rest_framework_gis import serializers as gis_serializer
class CallSerializer(gis_serializer.GeoModelSerializer):
call_type = serializers.StringRelatedField()
class Meta:
model = CallLocation
fields = ('status_id', 'address', 'location',
'call_type', 'timestamp')
geo_field = 'point'
pip install djangorestframework-gis

69. REST+GIS - Serializer
{
"status_id": "650065994463838208",
"address": "2200 NE 36th Ave, Portland, OR 97212, USA",
"location": "SRID=4326;POINT (-122.6259952999999996 45.5384747000000019)",
"call_type": "PROPERTY LOST, FOUND, RECOVERED ",
"timestamp": "2015-10-02T21:53:05Z"
}
{
"status_id": "650864005439840258",
"address": "8900 SW 30th Ave, Portland, OR 97219, USA",
"location": {
"type": "Point",
"coordinates": [
-122.7077201,
45.4608926
]
},
"call_type": "THEFT - COLD ",
"timestamp": "2015-10-05T02:44:06Z"
}
Without the djangorestframework-gis
With the djangorestframework-gis

70. REST+GIS - View
# Django Packages
# 3rd party
from rest_framework import viewsets
# project packages
from .models import CallLocation
from .serializers import CallSerializer
class CallLocationViewSet(viewsets.ModelViewSet):
"""
API endpoint that allows users to be viewed or edited.
"""
serializer_class = CallSerializer
No magic here.

71. REST+GIS - Front end
$scope.getMarkers = function () {
$scope.state.markers = [];
Call.query().$promise.then(function(data) {
data.forEach(function(call){
$scope.state.markers.push((
[call.location.coordinates[1],
call.location.coordinates[0]]))
});
}, function(errResponse) {
console.log('REST error');
});
};
...
<div class="container-fluid">
<map
zoom="10" center="[45.5200, -122.6819]"
on-center-changed="getMarkers()">
<marker ng-repeat="p in state.markers"
position="{{ p }}"></marker>
</map>
</div>
AngularJS with ngmap

72. How to limit the
API requests?

73. Limit requests by the
using bounding box
# views.py
from rest_framework import viewsets
from rest_framework_gis.filters import InBBOXFilter
class CallLocationViewSet(viewsets.ModelViewSet):
"""
API endpoint that allows users to be viewed or edited.
"""
serializer_class = CallSerializer
# bbox defines which column is used to select the objects
returned
# to the frontend
bbox_filter_field = 'location'
# filter settings
filter_backends = (InBBOXFilter, )
# the overlapping allows polygons to be partially be part of the
bbox
bbox_filter_include_overlapping = True # not needed for
PointFields
pip install django-filters

74. Limit requests by the
using bounding box
// Change AngularJS call
_getPolygonStr = function (map){
var bounds = [];
bounds[0] = map.getBounds().getSouthWest().lng(); // min lon
bounds[1] = map.getBounds().getSouthWest().lat(); // min lat
bounds[2] = map.getBounds().getNorthEast().lng(); // max lon
bounds[3] = map.getBounds().getNorthEast().lat(); // max lat
return bounds.join(",");
};
...
if ($scope.map) {
bounds = _getPolygonStr($scope.map);
};
...
Call.query({in_bbox: bounds}).$promise.then(function(data) {
data.forEach(function(call){
…
});
}
/api/calls/?in_bbox=-123.093887304,45.327228800,-122.269912695,45.71211299

75. Recap
• SRID, Projections
• How to get geodata (geocoding)
• How to store it
• How to query it
• How to create an API endpoint

76. Useful Resources

77. Useful Resources
• Python Geospatial
Development
Erik Westra, O’Reilly

78. Useful Resources
• GeoDjango Tutorial
https://docs.djangoproject.com/en/1.8/ref/contrib/gis/
• Python & GeoData PyCon 2014 (thanks to @pdxmele):
https://github.com/pdxmele/python-geodata-bffs
• GeoDjango DjangoCon 2014 (thanks to @aleck_landgraf)
https://www.youtube.com/watch?v=mUhinowr3RY
• Creating maps with D3.js (thanks to @oceankidbilly)
http://wrobstory.github.io/2013/06/creating-with-d3-
dymo.html

79. Useful Resources
• Fake Addresses (thanks to @alphydan):
https://fakena.me/random-real-address/
• Portland Atlas (thanks to @Caged)
https://github.com/caged/portland-atlas
• Boston GIS Group
http://www.bostongis.com/

80. Big thanks to
@alphydan
@aleck_landgraf
@srikanth_chikoo
@patrickbeeson

81. Thank you.

82. Questions?
Hannes
@hanneshapke
code
bit.ly/pydx-python-geodata