Slides from my talk at DevOpsCon Online

1. Serverless
Deep Learning
Alexey Grigorev
30.06.2020

2. Classify 1M images for $16.66

4. 4

5. 5
Car
Bike

6. Data scientist
(me)
DevOps engineer
(you)

8. I have a
model

9. Can we
deploy it?

10. Ehm
sure... Do
you know
how?

11. Yea I
read a
blogpost
about it

12. Metadata
Service
DB
S3
(images)
MMS
TF-Serving
Image
Quality
Image
Category
https://www.slideshare.net/AlexeyGrigorev/deep-learning-in-production-serving-image-models-at-scale
Client

14. 😐

15. 😐
6 months

17. Plan
● Deep learning in AWS Lambda
● AWS Lambda limitations
● Edge computing to the rescue

19. I’ll just
deploy TF to
lambda

22. S3
(model)

23. S3
(model)
{
"pug": 0.9993,
"elkhound": 0.0005,
"chow": 0.00004
}

24. S3
(model)
{
"pug": 0.9993,
"elkhound": 0.0005,
"chow": 0.00004
}
🥳🥳

25. import tensorflow as tf
import boto3
# ...
# load the model
# ...
def handler(event, context):
img = download_image(event['url'])
preds = predict(img)
result = decode_predictions(preds)[0]
return result
index.py:

26. # install dependencies to “build” folder
pip install -r requirements.txt -t build
# put our code there as well
cp index.py build
# pack everything into a zip file
cd build && zip -r ../build.zip * & cd ..
* do this in Docker (amazonlinux)
*

27. AWS Lambda Limits
● 512MB storage in /tmp
● Package size 50MB zipped / 250MB unzipped
https://docs.aws.amazon.com/lambda/latest/dg/limits.html

28. TF1.8 size
354M

29. I’ll just reduce
the size
using this
awesome
tutorial
https://www.slideshare.net/AlexeyGrigorev/serving-models-from-aws-lambda

30. find . -type d -name "tests" -exec rm -rf {} +
find . -name *.pyc -delete
find . -name "__pycache__" | xargs -n1 rm -rf
find . -name "*.dist-info" | xargs -n1 rm -rf
find . -name "*.so" | xargs strip
find . -name "*.so.*" | xargs strip

31. TF2.2
size 1.7G

32. find . -type d -name "tests" -exec rm -rf {} +
find . -name *.pyc -delete
find . -name "__pycache__" | xargs -n1 rm -rf
find . -name "*.dist-info" | xargs -n1 rm -rf
find . -name "*.so" | xargs strip
find . -name "*.so.*" | xargs strip

34. What to do?
● TensorFlow for JS or Java
● TF-Serving (C++)
● TF-Lite (mobile)

35. TensorFlow JS
Pros
● Fits within limits
Cons
● JavaScript
● No NumPy, re-writing pre- & post-processing
https://github.com/lucleray/tensorflow-lambda

36. TensorFlow JS
Pros
● Fits within limits
Cons
● JavaScript
● No NumPy, re-writing pre- & post-processing
https://github.com/lucleray/tensorflow-lambda

37. TensorFlow Serving
Pros
● Fits within limits
● We already know how to use it for serving TF models
Cons
● Tricky to use in Lambda
● Only C++, No Python bindings

38. TensorFlow Serving
Pros
● Fits within limits
● We already know how to use it for serving TF models
Cons
● Tricky to use in Lambda
● Only C++, No Python bindings

39. TensorFlow Lite

40. TensorFlow Lite

41. TensorFlow Lite
🤔

42. TensorFlow Lite runs on mobile phones,
embedded Linux devices, and microcontrollers.

43. TensorFlow Lite runs on mobile phones,
embedded Linux devices, and microcontrollers.
On Linux, you can use C++ and Python
https://www.tensorflow.org/lite/guide/inference

44. https://www.tensorflow.org/lite/guide/python

45. https://www.tensorflow.org/lite/guide/python
ImportError: /lib64/libm.so.6: version `GLIBC_2.27' not found
(required by /app/tflite_runtime/_interpreter_wrapper.so)

46. Building TF-Lite
● Docker: FROM amazonlinux
● Compile TF-Lite ourselves
● …
● Profit!
https://github.com/tpaul1611/python_tflite_for_amazonlinux

47. Using TF-Lite
60M numpy
6,0M PIL
5,7M tflite_runtime

48. Using TF-Lite
60M numpy
6,0M PIL
5,7M tflite_runtime
20M build.zip

49. Using TF-Lite
{
'pug': 0.999370,
'Norwegian_elkhound': 0.00053755,
'chow': 3.780921e-05
}

50. Using TF-Lite
{
'pug': 0.999370,
'Norwegian_elkhound': 0.00053755,
'chow': 3.780921e-05
}
🎉

52. First request

53. Other requests

54. RAM (MB)
First request
(ms)
2nd+ requests
(ms)
512 3000 1700
1024 1600 900
1536 1200 700

55. $16.66
per 1 mln images

56. Cost
● 512 MB
○ (2 sec) 20 * 0.0000008333 = 0.00001666
○ 1 mln: $16.666
● 1024 MB
○ (1 sec) 10 * 0.0000016667 = 0.00001666
○ 1 mln: $16.666
● 1532 MB
○ (0.7 sec) 7 * 0.0000025 = 0.0000175
○ 1 mln: $17.5
https://aws.amazon.com/lambda/pricing/

57. $166.6
per 10 mln images

58. ~$5000
per month (10 mln * 30)

59. Metadata
Service
DB
S3
(images)
MMS
TF-Serving
Image
Quality
Image
Category
https://www.slideshare.net/AlexeyGrigorev/deep-learning-in-production-serving-image-models-at-scale
Client

60. Metadata
Service
DB
S3
(images)
MMS
TF-Serving
Image
Quality
Image
Category
https://www.slideshare.net/AlexeyGrigorev/deep-learning-in-production-serving-image-models-at-scale
Client
~$500
per month

61. Deep Learning with AWS Lambda
Pros:
● Easy to deploy
● Serverless!
● Scales down to 0
● Great for small load (<1 mln
images)
Cons:
● Initially not straightforward
● Monitoring (CloudWatch)
● Undocumented behaviour
● Expensive for high load (>10
mln images daily)

62. Main takeaways
● Serving deep learning without infrastructure is possible
● TensorFlow 2.2 is heavy, but TensorFlow Lite is lite
● AWS Lambda is good for small load, but expensive under
heavy load

63. https://github.com/alexeygrigorev/serverless-deep-learning

64. olxgroup.com/careers

65. mlbookcamp.com
● Learn Machine Learning by doing
projects
● http://bit.ly/mlbookcamp
● Get 40% off with code “grigorevpc”
Machine Learning
Bookcamp

66. @Al_Grigoragrigorev
alexeygrigorev.com
mlbookcamp.com
https://forms.gle/gfsAeWLg72FhcHPZ8

67. Backup

68. Results stream
S3
(images)
S3
(results)
Athena tables
ObjectCreated:Put