Serving TF models from AWS Lambda is not easy, but possible.

1. Serving models
from AWS Lambda
Alexey Grigorev
01.08.2019

2. AWS Lambda

3. Messages from SQS

4. How about python&ML stuff?
pip install -r requirements.txt -t build
cp index.py build
cd build && zip -r ../build.zip & cd ..
specify output folder
this is sent to aws

5. How about python&ML stuff?
aws lambda update-function-code
--function-name ${FUNCTION_NAME}
--s3-bucket ${S3_BUCKET}
--s3-key ${ZIP_FILE}

6. Problems: Mac
Use Docker!

7. Problem: AWS Lambda Limits
● Limits:
○ 128MB - 3GB of RAM
○ 512MB storage in /tmp
○ Package size 50MB zipped / 250MB unzipped
https://docs.aws.amazon.com/lambda/latest/dg/limits.html

8. AWS Lambda Limits
$ du -sh * | sort -k 2
28K bin
56M build.zip
392K imagehash
32K ImageHash-4.0.dist-info
80M numpy
88K numpy-1.17.0.dist-info
6.8M PIL
48K Pillow-5.4.1.dist-info
32K __pycache__
28K PyWavelets-1.0.3.dist-info
8.4M pywt
88M scipy
148K scipy-1.3.0.dist-info
28K six-1.12.0.dist-info
32K six.py
More than 50MB

9. Solution: rm things
32M May 16 10:50 index-index-hasher.zip

10. It scalez!

11. Image index
https://tech.olx.com/detecting-image-duplicates-at-olx-scale-7f59e4b6aef4
s3
ObjectCreated:Put
ES
hashes
Image index
ingestor

12. What if we could use it for serving image models?

13. pip install numpy pillow tensorflow==1.7.0 -b build
size: 354M
I’ll just
deploy TF
to lambda

14. How to cut down the size of TF?
!!!

15. Yay!
original size 354M
striping SOs in tensorflow
striping SOs in numpy
stripped size 177M
compressed size 43M

16. Keras models in Lambda
● Convert Keras to TF
● Save the model to s3
● During lambda startup
○ Download the model from s3
○ Put to /tmp
○ Load the model
○ Don’t delete from /tmp - new lambdas startup will be faster

17. Keras → TF
Load Keras model
Convert to TF and save

18. Loading on startup

19. Loading on startup

20. Serve!

21. https://github.com/keras-team/keras-applications/blob/master/keras_applications/inception_resnet_v2.py#L38
https://github.com/keras-team/keras-applications/blob/master/keras_applications/imagenet_utils.py#L18
https://github.com/keras-team/keras-preprocessing/blob/master/keras_preprocessing/image/utils.py#L78

22. Demo

23. Demo: First request

24. Demo: Consequent requests

25. More RAM
● 1152MB: 50 sec load, 4 sec inference
● 1344MB: 43 sec load, 3.2 sec inference
● 1536MB: 30 sec load, 3 sec inference

26. Price
● 1024MB:
○ (4sec) 40 * 0.000001667
○ = 0.00006668 USD
○ 1 mln images: 66 USD
● 1344MB:
○ 1 mln images: 66 USD
● 1536MB:
○ 1 mln images: 75 USD
https://aws.amazon.com/lambda/pricing/