AWS Lambda: Best Practices and Common Mistakes - Dev Ops West 2019

AWS Lambda: Best Practices
and Common Mistakes
Given by Derek C. Ashmore
DevOps West
June 5, 2019
©2018 Derek C. Ashmore, All Rights Reserved 1

Who am I?
• Professional Geek
since 1987
• Java/J2EE/Java EE
since 1999
• AWS since 2010
• Specialties
• Cloud
Workshops
• Cloud-native
Applications
• Yes – I still code!

Discussion Resources
• This slide deck
– https://www.slideshare.net/derekashmore/presentations
• Sample code on my Github
– https://github.com/Derek-Ashmore/
• Slide deck has hyper-links!
– Don’t bother writing down URLs

Agenda
The
“What”
and “Why”
of AWS
Lambda
Code-Level
Tips
Operation
and Design
Habits
When to
use
Lambdas
Summary /
Q&A

What are AWS Lambdas?
• You provide custom code -> AWS runs it
– Java, Go, PowerShell, Node.js, C#, Python, and Ruby
• Computing power with less management
– AWS manages that hardware
– AWS autoscales that hardware
– AWS maintains that hardware
• Lambdas are event driven
– API Gateway (e.g. RESTful Web Service call)
– Many more
• Lambdas are stateless
• Not to be confused with “Lambda Expressions” in Java 8

Lambda Event Sources
• API Gateway
• SNS Messaging
Subscriptions
• Schedule
• Storage writes
– S3, DynamoDB, Kenesis
• SES Email receipt
• Cloudwatch
– Schedule, Events, log entries
• Cognito (Security)
• CloudFormation
– Creation script

What’s the Business Benefit
• Less Maintenance Hassle
• Unlimited* Parallelism
• Current cost advantage
– Don’t pay for idle
– CPU cost currently lower
• Free tier
– 1 M executions and 400K compute seconds per month
– Memory allocated determines allowed free-tier runtime
• 20 cents per 1 M executions + memory/runtime cost
– Administration cost
• No O/S upgrades, server backups, etc.

There’s no free lunch
• Less control over environment
– Harder to tune
– Memory and time limits on execution
• Few Environment amenities
– No connection pooling, session support, caching
• Proprietary Interface
– Potential Technical Lock-in
• No Guarantee that AWS cost will be constant
– Potential Business Risk
• Modern version of CGI

Agenda
The
“What”
and “Why”
of AWS
Lambda
Code-Level
Tips
Operation
and Design
Habits
When to
use
Lambdas
Summary /
Q&A

What Makes a “Best Practice”?
• Makes Support Easier
• Increases Reuse
• Increases Performance
• Minimizes Resource Consumption
– Labor
– Runtime

Let’s start with Low-Hanging Fruit

Report Inputs/Env on Exception
• Place a Try / Catch in your handler
– Python Example
– Java Example
• Also check your arguments with a clear error message
def crossAccountHandler(event, context):
try:
………………
except Exception as e:
e.args += (event,vars(context))
raise

Check Arguments Up Front
• Check your arguments with a clear error message
– Python Example
– Java Example
def crossAccountHandler(event, context):
try:
if 'Assumed_Role' in event:
…………………
else:
raise Exception('Assumed_Role not provided as argument')
except Exception as e:

Specify Lambda Source Repo
• Explicitly put the source repository name in the Lambda comments
– In most organizations, the repository name isn’t obvious
– Others changing your code need it
– You don’t want source control to be out of date
"""
secretLambda.py
……………
Source Control: https://github.com/Derek-Ashmore/AWSDevOpsUtilities
"""

Separate Lambda from Business Logic
• Make business logic reusable
– Callable by other applications
– Usable on premises
• Easier to locally develop and debug
– Lambda-specific logic is thin!
def startStopHandler(event, context):
try:
executeStopStart(datetime.datetime.now()
, os.getenv('Scheduled_StartTime', ‘’)
, os.getenv('Scheduled_StopTime', ‘’)
, os.getenv('Scheduled_StartStop_Days', 'M,T,W,R,F’))
……………
return 0;

This is low-hanging fruit that will be appreciated by
your fellow developers!
• Log All Inputs and Environment on
Exception
• Check all arguments up front
• Document the source repo at the top.
• Repo readme can have other
developer specifics
• Separate Lambda code from business
logic
• Now let’s talk design and operations….

Agenda
The
“What”
and “Why”
of AWS
Lambda
Code-Level
Tips
Operation
and Design
Habits
When to
use
Lambdas
Summary /
Q&A

Automate builds and deployments!

Lambda Copies Everywhere!
• Changes / Bug Fixes need to be deployed everywhere
• Solving with automation solves the wrong problem!

One Copy for All!
• Scalable – only need to add accounts over time
• Bugfixes in one place
• Configuration usually in common DynamoDB table(s)
• Sample in Python here

Cross-Account Execution
• Algorithm is
– Assume a remote-account role using STS
• The response has temporary credentials
– Create a session using the remote account creds
– Do work in the remote account
• Example here: Derek-Ashmore/AWSDevOpsUtilities (Github)

For workloads over 5 min
• Executor that invokes lambda asynchronously for
each account
• Sample in Python here

Limit Custom Nesting to One Level
• Debugging with nested executions is
– Time consuming and difficult
– Can’t do locally
– Absolutely requires unique correlation id for the entire transaction
• Allows you to tell invocation history for one logical transaction
– Instead of deep custom nesting, use AWS Step Functions
• Use Step Functions if you need more

Nested Calls using AWS Step Functions
• AWS Step Functions
– Uses a State Machine model
• Think turn-style to get access to train
– States are “Locked” and “Unlocked”
– Locked → Payment input allowed, then “Unlocked”
– Unlocked → One person allowed through, then “Locked”
– Automatically provides correlation between invocations
• Unified logs for the entire transaction

Operations and Design Habits
• Automate Builds and Deployments
• Only install Lambda’s Once
• Limit Lambda nesting to One Level
• Step functions if you need more
• Now let’s talk dependencies and secrets

Use Configuration Injection
• No environment specifics hardcoded in the Lambda deployment
• Use Environment Variables on the Lambda Definition
– No un-encrypted secrets (e.g. database password)
• Use Arguments in the triggering event
– No un-encrypted secrets
• Anti-Example
– Splunk forwarding Lambda with hard-coded Splunk channels

Providing Secrets to Lambdas
• Secrets are needed items like credentials of any type.
• Use IAM Roles to grant permission to read secrets
• Options are:
– Use KMS
• Encrypt credential and base64 encode it
– Place encrypted version in environment variable
• Sample Lambda and Encryption Script (here)
– Use a Digital Vault (e.g. AWS Secrets Manager)
• Sample Lambda here

AWS Secrets Manager
• Use IAM Roles to grant
permission to read secrets
• You don’t need a “secret” to
get a “secret”!

Avoid Heavy-Footprint Dependencies
• Minimizes load time
– Mitigates cold-start problem
• Java
– Use Guice over Spring
• Python
– Use AWS provided deps first (list is
here)

Don’t Hog Resources
• 512 Mb temp space per invocation
• 1,024 file descriptors
• 1,024 Threads and processes
(combined)
• 6 MB payload size (synchronous)
• 128 KB payload size (asynchronous)
• 1000 concurrent lambda executions
per account per region

Idempotence
• Possible for your Lambda to be invoked multiple times for the same event
– Prevent repeat actions from having a different effect.
• Options
– Record the event id –> Skip repeated events
• Most event sources provide a unique request id
– Lambda invoking lambda does not!
• Negatively affects performance
– 1 extra read
– 1 extra write
• Need to roll-off old events
– Insure that the effect is the same each time
• Not perfect → You don’t control invocation order

Agenda
The
“What”
and “Why”
of AWS
Lambda
Code-Level
Tips
Operation
and Design
Habits
When to
use
Lambdas
Summary /
Q&A

Suitable workloads for Lambda’s
• Workloads that
– Take less than 15 min
– Are stateless
– Idempotent
• Evaluate cost with calculator: https://dashbird.io/lambda-cost-calculator/
• Typical examples
– Streaming data processors
– Dynamo DB Change Processors
– AWS-specific DevOps Tasks
• Security Enforcement
• Uptime Scheduling
• AWS Change Event processing
– CloudFormation Macros

Lambda can herd the cats!
• Using Lambda to enforce security
– Automatic Remediation
• Unlike AWS Config, Lambdas can take action!
• Unwanted port exposures
– Unauthorized exposure of 0.0.0.0/0 to the world
• Decentralized Management
– Empowers the organization
– Improves speed to market
• Less bottleneck by admin groups
– Still keeps the enterprise secure

Lambdas and Microservices

Using Lambdas as Microservices
• Lambda / API Gateway is a deployment option for microservices
– No differences in design principles
• Single purpose
• Self-contained
– Still design for failure
• Don’t “assume” that Lambda outages can’t happen
– A Lambda might need external resources that aren’t available
• Off Limits: Coding patterns that use state
– Lambdas must be stateless
– Fail fast patterns
• Service Call Mediator
• Circuit Breaker
– Performance Patterns
• Expiring Cache (API Gateway allows request caching)

Lambda Alternatives
• Use Kubernetes
– Operates as a Lambda type service in Kubernetes
– Functions get shut down when not used – like Lambda behind the
scenes
– Serverless workloads are Containerized
• Run on premises or on any Cloud
– Examples
• Fission
• Knative
• Kubeless

Further Reading
• This slide deck
– https://www.slideshare.net/derekashmore/presentations
• AWS Lambda Reading List
– http://www.derekashmore.com/2016/04/aws-lambda-reading-list.html
• Amazon’s Published Best Practice List
– https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html

Questions?
• Derek Ashmore:
– Blog: www.derekashmore.com
– LinkedIn: www.linkedin.com/in/derekashmore
• Connect Invites from attendees welcome
– Twitter: https://twitter.com/Derek_Ashmore
– GitHub: https://github.com/Derek-Ashmore
– Book: http://dvtpress.com/

AWS Lambda: Best Practices and Common Mistakes - Dev Ops West 2019

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to AWS Lambda: Best Practices and Common Mistakes - Dev Ops West 2019

Similar to AWS Lambda: Best Practices and Common Mistakes - Dev Ops West 2019 (20)

Recently uploaded

Recently uploaded (20)

AWS Lambda: Best Practices and Common Mistakes - Dev Ops West 2019