Your system is composed of highly decoupled, independent, fast, and modular microservices. But how can they share common configurations, dynamic endpoints, database references, and properly rotate secrets? Based on the size and complexity of your serverless system, you may simply use environment variables or eventually opt for some sort of centralized store. And then how do integrate all of this with monitoring and automation tooling? During this session, I will present the ideal solutions and some of the alternatives available on AWS (such as AWS Systems Manager Parameter Store and AWS Secrets Manager). I will also discuss the best use cases for each solution and the corresponding best practices to achieve the highest standards for security and performance.

1. Alex Casalboni
Technical Evangelist, AWS
@alex_casalboni
@ 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Configuration management and service
discovery in a serverless world

2. About me
• Software Engineer & Web Developer
• Worked in a startup for 4.5 years
• ServerlessDays Organizer
• AWS Customer since 2013

3. Agenda
1. Serverless security & configuration management
2. Live demo
3. Cost optimization techniques

4. Serverless security &
configuration management
@alex_casalboni

5. Lambda permission model
Fine-grained security controls for both execution and invocation
Execution policies
Define what AWS resources/API calls can this function access via AWS IAM
Used in streaming invocations
For example, “Lambda function A can read from DynamoDB table users”
Function policies
Used for sync and async invocations
Resource policies allow for cross account access
For example, “Actions on bucket X can invoke Lambda function Z"

6. Action: “s3:*”
… make puppies cry!Action: “dynamodb:*"
Action: “sns:*“
Photo by Matthew Henry on Unsplash

7. MyFunction:
Type: AWS::Serverless::Function
Properties:
Handler: index.handler
Runtime: python2.7
Policies:
- AWSLambdaExecute # Managed Policy
- Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- dynamodb:GetItem
Resource: !GetAtt MyDynamoDBTable.Arn
Fine-grained IAM policy with AWS SAM
MyFunction:
Type: AWS::Serverless::Function
Properties:
Handler: index.handler
Runtime: python2.7
Policies:
- AWSLambdaExecute # Managed Policy
- Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- dynamodb:GetItem
Resource: !GetAtt MyDynamoDBTable.Arn

8. Hardcoded secrets make fish cry!
Photo by Julieann Ragojo on Unsplash

9. AWS Lambda environment variables
Key-value pairs that you can dynamically pass to your function
Available via standard environment variable APIs (based on runtime)
Can optionally be encrypted via AWS KMS
Allows you to specify in IAM what roles have access to the keys to decrypt the information
Useful for creating environments per stage (such as dev, test, prod)

10. AWS Systems Manager―Parameter Store
Centralized store to manage your configuration data
Supports hierarchies
Plaintext or encrypted with AWS KMS
Can send notifications of changes to Amazon SNS or Lambda
Can be secured with IAM
Calls recorded in AWS CloudTrail
Can be tagged
Available via API/SDK
Useful for centralized environment variables, secrets control, feature flags

11. Parameter Store access via SDK
import json, boto3
ssm = boto3.client('ssm')
def get_parameter():
response = ssm.get_parameter(
Name=‘my_param’,
WithDecryption=True
)
return response['Parameter']['Value']
def lambda_handler(event, context):
value = get_parameter()
print(”value = %s" % value)

12. Parameter Store access via SDK with ssm_cache
import json, boto3
ssm = boto3.client('ssm')
def get_parameter():
response = ssm.get_parameter(
Name=‘my_param’,
WithDecryption=True
)
return response['Parameter']['Value']
def lambda_handler(event, context):
value = get_parameter()
print(”value = %s" % value)
from ssm_cache import SSMParameter
param = SSMParameter(‘my_param’)
def lambda_handler(event, context):
value = param.value
print(”value = %s" % value)
github.com/alexcasalboni/ssm-cache-python

13. AWS Secrets Manager
Allows you to manage, retrieve, and rotate credentials
Helps you rotate secrets regularly without breaking stuff
Keeps track of different password versions
Implements security controls associated with credential management
Built-in support for Amazon RDS

14. AWS Secrets Manager + Parameter Store
Uniform and consistent access to both services
You can reference Secrets Manager secrets with Parameter Store APIs
Rotation & Refresh delegated to the client
As simple as using a prefix: /aws/reference/secretsmanager/
+

15. Secrets access via Parameter Store
import json, boto3
ssm = boto3.client('ssm’)
prefix = ‘/aws/reference/secretsmanager’
def get_secret():
response = ssm.get_parameter(
Names=[‘%s/my_secret’ % prefix],
WithDecryption=True
)
return response['Parameter']['Value']
def lambda_handler(event, context):
value = get_secret()
print(”value = %s" % value)

16. Secrets access via Parameter Store with ssm_cache
import json, boto3
ssm = boto3.client('ssm’)
prefix = ‘/aws/reference/secretsmanager’
def get_secret():
response = ssm.get_parameter(
Names=[‘%s/my_secret’ % prefix],
WithDecryption=True
)
return response['Parameter']['Value']
def lambda_handler(event, context):
value = get_secret()
print(”value = %s" % value)
from ssm_cache import SecretsManagerParameter
secret = SecretsManagerParameter(‘my_secret’)
def lambda_handler(event, context):
value = secret.value
print(”value = %s" % value)
github.com/alexcasalboni/ssm-cache-python

17. Demo time!
@alex_casalboni
amzn.to/serverless-security

18. Serverless cost
optimization techniques
@alex_casalboni

19. Anatomy of a function
Your
function
Language
runtime
Function
container
Compute
substrate

20. The request lifecycle
Bootstrap
the runtime
Start your
code
Cold
start
Warm
start
Download
your code
Start new
container
AWS optimization Your optimization

21. Same view in AWS X-Ray

22. Efficient function code
Avoid monolithic functions (or “fat”)
Control the dependencies in your function's deployment package
Optimize for your language
Node.js – Browserfy, Minify, Webpack

23. Ephemeral function environment
Lambda processes a single event per-container
No need for non-blocking execution on the frontend
REMEMBER – containers are reused
Lazily load variables in global scope
Don’t load it if you don’t need it

24. Lazy initialization example (Python & boto3)
import boto3
S3_client = None
ddb_client = None
def get_objects(event, context):
if not s3_client:
s3_client = boto3.client("s3")
# business logic
def get_items(event, context):
if not ddb_client:
ddb_client = boto3.client(”dynamodb")
# business logic

25. Optimized dependency usage (Node.js SDK & X-Ray)
// const AWS = require('aws-sdk’)
const DynamoDB = require('aws-sdk/clients/dynamodb’) // 125ms faster
// const AWSXRay = require('aws-xray-sdk’)
const AWSXRay = require('aws-xray-sdk-core’) // 5ms faster
// const AWS = AWSXRay.captureAWS(require('aws-sdk’))
const dynamodb = new DynamoDB.DocumentClient()
AWSXRay.captureAWSClient(dynamodb.service) // 140ms faster
@theburningmonktheburningmonk.com/2019/03/just-how-expensive-is-the-full-aws-sdk/

26. Concise function logic
Separate Lambda handler from core logic
Use functions to TRANSFORM, not TRANSPORT
Read only what you need
Query filters in Amazon Aurora
Use Amazon S3 select

27. Concise function logic (example)
from mylib import MyLibClass
def lambda_handler(event, context):
operation = event['Operation’]
myobj = MyLibClass()
if operation == ‘do_this’:
my_obj.do_this()
elif operation == ‘do_that’:
myobj.do_that()
else:
raise ValueError(‘Invalid op’)

28. Concise function logic (example)
import boto3
ddb = boto3.client(‘dynamodb’)
class MyLibClass(object):
MY_CONSTANT = ‘blabla’
def __init__(…):
# constructor
def do_this(self):
# use ddb to do this
def do_that(self):
# use ddb to do that
from mylib import MyLibClass
def lambda_handler(event, context):
operation = event['Operation’]
myobj = MyLibClass()
if operation == ‘do_this’:
my_obj.do_this()
elif operation == ‘do_that’:
myobj.do_that()
else:
raise ValueError(‘Invalid op’)

29. Small changes, big difference
# Download and process all keys
for key in src_keys:
response = s3_client.get_object(…)
contents = response['Body'].read()
for line in contents.split('n')[:-1]:
line_count +=1
try:
data = line.split(',')
srcIp = data[0][:8]
…
# Select IP Address and Keys
for key in src_keys:
response = s3_client.select_object_content(
expression=“SELECT SUBSTR(obj._1, 1, 8),
obj._2 FROM s3object as obj”)
contents = response['Body'].read()
for line in contents:
line_count +=1
try:
…
After (95s, $0.028)Before (200s, $0.112)
https://github.com/awslabs/lambda-refarch-mapreduce

30. Smart resource allocation
Match resource allocation (up to 3 GB!) to logic
Stats for Lambda function that calculates 1000 times all prime
numbers <= 1000000
128 MB 11.722s $0.024628
256 MB 6.6789s $0.028035
512 MB 3.1949s $0.026830
1024 MB 1.4659s $0.024638

31. “AWS Lambda Power Tuning”
Data-driven cost & performance
optimization for AWS Lambda
github.com/alexcasalboni/aws-lambda-power-tuning
Don’t guesstimate!

32. No orchestration in codeSTARTJOB
JOB#XSTARTED
HTTPPOST
HTTPPOST
AREWETHEREYET?
NOPE!
WE’REDONE!
ZzZz
OR
time.sleep(10)

33. No orchestration in code

34. Gateways & routers
Choose suitable entry point for client applications
Single, custom client?
Use the AWS SDK
Not end user facing?
Use regional endpoints on API Gateway
Discard uninteresting events ASAP
S3 – Event prefix
SNS – Message filtering

35. Resilient: retry policies
Understand retry policies
Sync never retried
Async retried 2 times
Streams retried all the time
Leverage Dead Letter Queues (DLQ)
SQS or SNS for replays
REMEMBER: Retries count as invokes

36. Concurrency Controls
Concurrency a shared pool by default
Separate using per function concurrency settings
Acts as reservation
Also acts as max concurrency per function
Especially critical for data sources like RDS
“Kill switch” – set per function concurrency to zero

37. Should my
Lambda
function be
in a VPC?
Does my function
need to access
any specific
resources in a
VPC?
Does it also need to
access resources or
services in the
public internet?
Don’t put the
function in a
VPC
Put the function
in a private
subnet
Put the function
in a subnet with
a NAT’d route
to the internet
Yes Yes
No No
Do I need a VPC?

38. Alex Casalboni
Technical Evangelist, AWS
@alex_casalboni
@ 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Thank you!