Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AWS Lambda x Python Error Handling, Test and CI


Published on

- Our use case of AWS Lambda
- AWS Lambda and & Handling
- AWS Lambda and Unit Test

Published in: Technology
  • D0WNL0AD FULL ▶ ▶ ▶ ▶ ◀ ◀ ◀ ◀
    Are you sure you want to  Yes  No
    Your message goes here
  • Jeevan's revision guide has helped understand questions which I did not understand before in class. It has really helped me learn things the easy way. It's straightforward and shows you how to get a top grade in GCSE maths, in a step-by-step format... ●●●
    Are you sure you want to  Yes  No
    Your message goes here

AWS Lambda x Python Error Handling, Test and CI

  1. 1. AWS Labmda x Python - Error Handling, Test and CI 2017-02-17 Takahiro Ikeuchi eurie Inc.
  2. 2. Agenda Our use case of AWS Lambda AWS Lambda and & Handling AWS Lambda and Unit Test 2
  3. 3. Today's Goal Learn about error handlings in AWS Lambda Learn about a local environment, UT and CI to develop AWS Lamba 3
  4. 4. Takahiro Ikeuchi @iktakahiro Company / Community eurie Inc. Founder & CEO SQUEEZE Inc. Tech Adviser PyData.Tokyo Organizer Specialties (or just a dabbler :-D Go lang, Python, React.js, TypeScript Cloud Infrastructure, UI Design etc... 4
  5. 5. My Favorite AWS Service Amazon Aurora 5
  6. 6. eurie Inc. B2B SaaS ( Launched : Customer Support Solution for innovative team. "eurie Desk" Golang, React and Python (4:4:2) AWS heavy user 6
  7. 7. Our use cases of AWS Lambda 7
  8. 8. Architecture 8
  9. 9. 9
  10. 10. 10
  11. 11. Use Cases It is used as th hub of processing that allows async POST documents to Elasticsearch SMTP Service (e.g. Amazon SES, SendGrid) Notifications to external system (e.g. Slack) WebApp => SNS => AWS Lambda When WebApp calls Amazon SNS, the response is returned Immediately. 11
  12. 12. AWS Lambda and Error Handling 12
  13. 13. We don't know whether processing after SNS was successful or not. We must pray for safety :-D When WebApp calls Amazon SNS, the response is returned Immediately “ “ 13
  14. 14. Don't pray, Think about error handlings 14
  15. 15. Exeptions on AWS Lambda - 1 (in async action) If a process is failed, AWS Lambda retry automatically up to 2 times. When an exception is raised in Python, AWS determines processing to be unsuccessful def always_failed_handler(event, context): raise Exception('I failed!') exceptions.html 15
  16. 16. Exeptions on AWS Lambda - 2 The retry approach is not executed within a transaction. Idempotency (冪統性) is needed. 1. Register a user data => Success 2. Send an email => Failure When the 2nd process fails, the 1st process will also be retried. Consider separating them into different AWS Lambda function. 16
  17. 17. Don't pray, Implement a retry logic 17
  18. 18. Exponential backoff 18
  19. 19. What is Exponential backoff Exponential backoff - Wikipedia Exponential backoff is an algorithm that uses feedback to multiplicatively decrease the rate of some process, in order to gradually find an acceptable rate. “ “ 19
  20. 20. Python and Exponential backoff You Just write @decorator. my_function() is retried with the algorithm automatically when the exception is raised. @retry(wait_exponential_multiplier=1000, stop_max_delay=12000, wait_exponential_max=10 def my_functoin(): try: res =, data=payload, timeout=5) except Exception: raise 20
  21. 21. Notes on using Exponential backoff It is suitable for transient faults and system busy (500). Retrying to a Bad Request response is a waste of time. 5 minute execution time limit, we can not retry unlimitedly. 21
  22. 22. Note Dead Letter Queue 22
  23. 23. AWS Lambda and UT, CI 23
  24. 24. AWS Lambda and UnitTest - 1 Debugging is slightly more difficult than general Linux environment. However, AWS Lambda calls the entry point which is just a Python function. def lambda_handler(event, context): # main process 24
  25. 25. Write Test Code 25
  26. 26. AWS Lambda and UnitTest - 2 dummy event and context object for Test test_event = {"Records": [ { "EventVersion": "1.0", "EventSubscriptionArn": "arn:aws:sns:EXAMPLE", "EventSource": "aws:sns", "Sns": { "TopicArn": "arn:aws:sns:EXAMPLE", "Subject": "TestInvoke" } } ] } 26
  27. 27. AWS Lambda and UnitTest - 3 Write a test code to the entry point. class TestLambdaFunction(object): resp = lambda_function.lambda_handler(event, context) assert resp is True Use a mock object as necessary. 27
  28. 28. Local Enviroment You might adopt an emulator. difficult to implement the execution limit time of AWS Lambda. The emulator has it. 28
  29. 29. Continuous Integration - 1 Archive the package for deployment include 3rd party packages. pip install -U -r requirements.txt -t ./vendor Vendoring fits this case. ├── ├── requirements.txt └── vendor ├── requests └── slackpy Embedding a Python library in my own package - Stack Overflow 29
  30. 30. Continuous Integration - 2 We can adapt the knowledge we already got. 1. PUSH Github 2. Start CI Process (Codeship) 1. Build 2. Test 3. Deploy (aws cli) aws lambda update-function-code --function-name my_func --zip-file 30
  31. 31. Conclusioin We love Serverless Architecture!! Knowledges about UnitTest and CI are adapt to AWS Lambda. Write test code, not easy for human to notice the error occurred in the cloud. Prepare a well-developed deployment process using CI tool/service 31