https://www.linkedin.com/in/jpizarrom/

1. Practical Examples of Serverless
Architecture using AWS Lambda and
PyWren as SQS Python Workers
by Juan Pizarro
Data Science and Engineering Club Meetup 2018.06.27
https://www.meetup.com/Data-Science-and-Engineering-Club/events/251202735/

2. Juan Pizarro jpizarrom@gmail.com https://www.linkedin.com/in/jpizarrom/
● Half-time student (2018-2019)
● previous
○ Lead Software Engineer (2015-2018) at https://www.admetricks.com
https://www.youtube.com/watch?v=IvbiDE-lY_k.
○ Developer (2013-2015) at http://a-dedo.cl (now https://www.allrideapp.com).
○ Co-Founder (2009-2012)
○ Computer Science Engineer from University of Talca (2011)

3. Agenda
● Serverless/AWS Lambda
● PyWren
● PyWren as SQS worker
● PyWren use cases (maybe demo)
○ Hyperparameter optimization, e.g. scikit-learn distributed GridSearchCV
○ Web Scraping
○ ...

4. Serverless
Data centre: Hardware as the unit of scale. Abstracts the physical hosting environment.
IaaS: Operating system as the unit of scale. Abstracts the hardware.
PaaS: Application as the unit of scale. Abstracts the Operating System.
Serverless/FaaS: Functions as the unit of scale. Abstracts the language runtime.
Ref: https://read.acloud.guru/iaas-paas-serverless-the-next-big-deal-in-cloud-computing-34b8198c98a2

5. FaaS Providers

6. AWS Lambda
AWS Lambda Run code fully managed without thinking about servers
Pay only for the compute time you consume
Scales automatically to the incoming event rate
Runs your code in parallel
Ref: https://www.slideshare.net/AmazonWebServices/massively-parallel-data-processing-with-pywren-and-aws-lambda-srv424-reinvent-2017/5
Ref: https://gfycat.com/gifs/detail/FlawlessFlusteredCony

8. Async
Ref: https://es.slideshare.net/AmazonWebServices/serverless-architectural-patterns-86885685/8

9. Ref: https://es.slideshare.net/AmazonWebServices/serverless-architectural-patterns-86885685/8
Sync

10. Stream
Ref: https://es.slideshare.net/AmazonWebServices/serverless-architectural-patterns-86885685/8

11. [...] there is little or no dependency or need for communication between those
parallel tasks, or for results between them.
Ref: https://en.wikipedia.org/wiki/Embarrassingly_parallel
Ref: https://www.youtube.com/watch?v=p0aLQLkzP24&t=32s
Ref: https://www.slideshare.net/AmazonWebServices/massively-parallel-data-processing-with-pywren-and-aws-lambda-srv424-reinvent-2017/7
Embarrassingly parallel

12. Ref: https://www.youtube.com/watch?v=2tu5f3jGwV4&t=9m
Ref: https://dl.acm.org/citation.cfm?id=3128601
Word count program
The Amazon reviews dataset

13. “Microservices and Teraflops” October 2016 http://ericjonas.com/pywren.html
Pywren lets you run your existing python code at massive scale via AWS Lambda
Serializes local Python code and sends it to AWS Lambda for massive parallel
execution with Amazon S3 as the intermediary
Receive results back to your local machine or other data stores for further analysis
PyWren © 2017 University of California, Berkeley, Eric Jonas, Qifan Pu, Shivaram Venkataraman, Vaishaal Shankar, Allan Peng.
Ref: http://pywren.io
Ref: https://www.slideshare.net/AmazonWebServices/massively-parallel-data-processing-with-pywren-and-aws-lambda-srv424-reinvent-2017/9
https://github.com/pywren/pywren
https://github.com/pywren/pywren-ibm-cloud

14. Ref: https://www.slideshare.net/continuumio/microservices-teraflops-effortlessly-scaling-data-science-with-pywren-anacondacon-2017/27
Ref: http://pywren.io/pywren.html

15. Ref: https://www.slideshare.net/databricks/microservices-and-teraflops-effortlessly-scaling-data-science-with-pywren-with-eric-jonas-and-shivaram-venkataraman/58

16. Ref: https://www.slideshare.net/databricks/microservices-and-teraflops-effortlessly-scaling-data-science-with-pywren-with-eric-jonas-and-shivaram-venkataraman/59

17. Ref: https://www.slideshare.net/databricks/microservices-and-teraflops-effortlessly-scaling-data-science-with-pywren-with-eric-jonas-and-shivaram-venkataraman/60

18. Ref: https://www.slideshare.net/databricks/microservices-and-teraflops-effortlessly-scaling-data-science-with-pywren-with-eric-jonas-and-shivaram-venkataraman/61

19. Ref: https://www.slideshare.net/databricks/microservices-and-teraflops-effortlessly-scaling-data-science-with-pywren-with-eric-jonas-and-shivaram-venkataraman/62

20. Ref: https://www.slideshare.net/databricks/microservices-and-teraflops-effortlessly-scaling-data-science-with-pywren-with-eric-jonas-and-shivaram-venkataraman/63

21. Ref: https://www.slideshare.net/databricks/microservices-and-teraflops-effortlessly-scaling-data-science-with-pywren-with-eric-jonas-and-shivaram-venkataraman/64

22. Ref: https://www.slideshare.net/databricks/microservices-and-teraflops-effortlessly-scaling-data-science-with-pywren-with-eric-jonas-and-shivaram-venkataraman/65

23. Ref: https://www.slideshare.net/databricks/microservices-and-teraflops-effortlessly-scaling-data-science-with-pywren-with-eric-jonas-and-shivaram-venkataraman/66

24. Ref: https://www.slideshare.net/databricks/microservices-and-teraflops-effortlessly-scaling-data-science-with-pywren-with-eric-jonas-and-shivaram-venkataraman/67

25. Ref: https://www.slideshare.net/databricks/microservices-and-teraflops-effortlessly-scaling-data-science-with-pywren-with-eric-jonas-and-shivaram-venkataraman/68

26. Ref: https://www.slideshare.net/databricks/microservices-and-teraflops-effortlessly-scaling-data-science-with-pywren-with-eric-jonas-and-shivaram-venkataraman/69

27. Ref: https://www.slideshare.net/databricks/microservices-and-teraflops-effortlessly-scaling-data-science-with-pywren-with-eric-jonas-and-shivaram-venkataraman/70

28. Ref: https://www.slideshare.net/databricks/microservices-and-teraflops-effortlessly-scaling-data-science-with-pywren-with-eric-jonas-and-shivaram-venkataraman/71

29. Ref: https://www.slideshare.net/AmazonWebServices/massively-parallel-data-processing-with-pywren-and-aws-lambda-srv424-reinvent-2017/16

30. PyWren as
Serverless
Amazon SQS
Worker with
AWS Lambda

31. SQS to Lambda
● estimated invocation rate
○ AWS Lambda + Amazon CloudWatch Events
● fixed invocation rate
○ AWS Lambda + Amazon Kinesis
○ non AWS Lambda Invoker (AWS Fargate?)

32. PyWren + SQS support
Proof of concept
Ref: https://github.com/admetricks/pywren/pull/1/files pywren v0.2.0
Ref: https://github.com/jpizarrom/pywren/compare/cf1d70a7362ded1ebdab052af2dc37af44f49761...jpizarrom:6b7d4f967fce0ca7ab8c17453d153ae28e02f1d5 pywren v0.3.0

35. Map use cases
● ETL
● Hyperparameter tuning
● Web scraping
● ...

36. Ref: https://github.com/ucbrise/risecamp/blob/master/pywren/hyperparameter-optimization.ipynb
Ref: https://github.com/ucbrise/risecamp/blob/master/pywren/solution/hyperparameter-optimization-solution.ipynb

37. Ref: https://www.slideshare.net/AmazonWebServices/massively-parallel-data-processing-with-pywren-and-aws-lambda-srv424-reinvent-2017/19
Ref: https://github.com/aws-samples/pywren-workshops/tree/master/Lab-2-Common-Crawl

38. Ref: https://www.slideshare.net/AmazonWebServices/massively-parallel-data-processing-with-pywren-and-aws-lambda-srv424-reinvent-2017/20
Ref: https://github.com/aws-samples/pywren-workshops/tree/master/Lab-3-Scrape-Sentiment

39. Demo

40. Q&A
@jpizarrom
jpizarrom@gmail.com
https://www.linkedin.com/in/jpizarrom/

41. Ref
http://pywren.io
https://github.com/pywren/pywren
https://github.com/pywren/runtimes
https://github.com/pywren/pywren-ibm-cloud

42. Ref: Workshops
https://github.com/aws-samples/pywren-workshops
https://github.com/ucbrise/risecamp/tree/master/pywren

43. Ref: MapReduce on AWS Lambda
https://github.com/bcongdon/corral
https://github.com/d2si-oss/ooso
https://github.com/awslabs/lambda-refarch-mapreduce

44. PyWren Design Goals
1. Very little overhead for setup
2. As close to zero overhead for users as possible
3. Target jobs that run in the minutes-or-more regime
4. I don't want to run a service
5. It has to be from a cloud player that's likely to give out an academic grant
Ref: http://pywren.io/pywren.html

45. Build Custom PyWren Runtime
● AWS EC2 machine Ref: https://github.com/pywren/runtimes
● Docker Ref: https://github.com/admetricks/runtimes/tree/c0df0c7ff996dc95c8eb21eef5a3ff6c40c9715e

46. PyWren invokation message

47. Ref: https://github.com/ucbrise/risecamp/blob/master/pywren/analyze-wikipedia.ipynb
Ref: https://github.com/ucbrise/risecamp/blob/master/pywren/solution/analyze-wikipedia-solution.ipynb

48. Ref: https://www.slideshare.net/AmazonWebServices/massively-parallel-data-processing-with-pywren-and-aws-lambda-srv424-reinvent-2017/18
Ref: https://github.com/aws-samples/pywren-workshops/tree/master/Lab-1-Hello-World

49. Ref: https://www.slideshare.net/AmazonWebServices/massively-parallel-data-processing-with-pywren-and-aws-lambda-srv424-reinvent-2017/21
Ref: https://github.com/aws-samples/pywren-workshops/tree/master/Lab-4-Landsat-NDVI