This document discusses using AWS Lambda for geospatial analysis. It provides examples of serverless configuration files that define Lambda functions with Python 2.7 and 3.6 runtimes. It also describes a Dockerfile and build process for packaging Python dependencies within Lambda deployment packages to reduce their size and optimize startup performance.

1. Fast and cost effective geospatial
analysis pipeline with AWS lambda
© Civic Vision UG

2. Mila Frerichs
© Civic Vision UG

3. I help civic advocacy groups
© Civic Vision UG

4. © Civic Vision UG

5. © Civic Vision UG

6. © Civic Vision UG

7. © Civic Vision UG

8. © Civic Vision UG

9. © Civic Vision UG

10. © Civic Vision UG

11. © Civic Vision UG

12. © Civic Vision UG

13. © Civic Vision UG

14. © Civic Vision UG

15. © Civic Vision UG

16. © Civic Vision UG

17. © Civic Vision UG

18. Available packages
boto3
botocore
docutils
jmespath
pip
python-dateutil
s3transfer
setuptools
six
© Civic Vision UG

19. © Civic Vision UG

20. service: foss4g_oceania
provider:
name: aws
runtime: nodejs6.10
functions:
hello:
handler: handler.hello
© Civic Vision UG

21. © Civic Vision UG

22. service: foss4g_oceania
provider:
name: aws
runtime: python2.7
package:
include:
- venv/lib/python2.7/site-packages
- analysis.py
exclude:
- venv
functions:
hello:
handler: handler.hello
© Civic Vision UG

23. import sys, os, json
here = os.path.dirname(os.path.realpath(__file__))
env_path = os.path.join(here, "./venv/lib/python2.7/site-packages/")
sys.path.append(env_path)
© Civic Vision UG

24. © Civic Vision UG

25. plugins:
- serverless-python-requirements
© Civic Vision UG

26. service: foss4g_oceania
provider:
name: aws
runtime: python3.6
package:
artifact: package.zip
functions:
hello:
handler: handler.hello
© Civic Vision UG

27. FROM amazonlinux:latest
# Install apt dependencies
RUN yum install -y
gcc gcc-c++ freetype-devel yum-utils findutils openssl-devel
RUN yum -y groupinstall development
RUN curl https://www.python.org/ftp/python/3.6.1/Python-3.6.1.tar.xz | tar -xJ
&& cd Python-3.6.1
&& ./configure --prefix=/usr/local --enable-shared
&& make
&& make install
&& cd ..
&& rm -rf Python-3.6.1
ENV LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
# Install Python dependencies
RUN pip3 install xxx--no-binary -t /tmp/vendored -U
# Reduce Lambda package size to fit the 250Mb limit
# Mostly based on https://github.com/jamesandersen/aws-machine-learning-demo
RUN du -sh /tmp/vendored
# This is the list of available modules on AWS lambda Python 3
# ['boto3', 'botocore', 'docutils', 'jmespath', 'pip', 'python-dateutil', 's3transfer', 'setuptools', 'six']
RUN find /tmp/vendored -name "*-info" -type d -exec rm -rdf {} +
RUN rm -rdf /tmp/vendored/boto3/
RUN rm -rdf /tmp/vendored/botocore/
RUN rm -rdf /tmp/vendored/docutils/
RUN rm -rdf /tmp/vendored/dateutil/
RUN rm -rdf /tmp/vendored/jmespath/
RUN rm -rdf /tmp/vendored/s3transfer/
RUN rm -rdf /tmp/vendored/numpy/doc/
# Leave module precompiles for faster Lambda startup
RUN find /tmp/vendored -type f -name '*.pyc' | while read f; do n=$(echo $f | sed 's/__pycache__///' | sed 's/.cpython-36//'); cp $f $n; done;
RUN find /tmp/vendored -type d -a -name '__pycache__' -print0 | xargs -0 rm -rf
RUN find /tmp/vendored -type f -a -name '*.py' -print0 | xargs -0 rm -f
RUN du -sh /tmp/vendored
COPY app /tmp/vendored/app
# Create archive
RUN cd /tmp/vendored && zip -r9q /tmp/package.zip *
# Cleanup
RUN rm -rf /tmp/vendored/
© Civic Vision UG

28. # Install Python dependencies
RUN pip3 install xxx--no-binary -t /tmp/vendored -U
# Reduce Lambda package size to fit the 250Mb limit
# Mostly based on https://github.com/jamesandersen/aws-machine-learning-demo
RUN du -sh /tmp/vendored
© Civic Vision UG

29. Adapted from mapbox/landsat-tiler
© Civic Vision UG

30. all: build package
build:
docker build --tag fis-server:latest .
package:
docker run
-w /var/task/
--name fis
-itd
fis-server:latest
docker cp fis:/tmp/package.zip package.zip
docker stop fis
docker rm fis
deploy:
sls deploy
© Civic Vision UG

31. © Civic Vision UG

32. © Civic Vision UG

33. Divide & Conquer
© Civic Vision UG

34. © Civic Vision UG

35. © Civic Vision UG

36. © Civic Vision UG

37. © Civic Vision UG

38. lambci/docker-lambda
© Civic Vision UG

39. © Civic Vision UG

40. © Civic Vision UG

41. © Civic Vision UG

42. © Civic Vision UG

43. © Civic Vision UG

44. questions
Give me your two questions!
© Civic Vision UG

45. Resources available at
civicvision.de/f4goaws
© Civic Vision UG