Building Lambda Packages
In Python
Getting Past Hello World
Sponsors
Find me on LinkedIn
AWS Certifications
Presented by Adam Book
Lambda Uses
Using a serverless technology like AWS
Lambda takes a bit of shift in thinking
If you think about the quote from Werner Vogles:
“Everything fails all the time”
Then you might currently have an AutoScaling group of min 1 max 1 Desired 1
just for an instance that is processing a piece of data or doing a task on a daily or
scheduled basis.
Lambda may be the cost effective way to take over that event.
Lambda Uses
Using a serverless technology like AWS
Lambda takes a bit of shift in thinking
Event Based Processing
For more info
http://docs.aws.amazon.com/lambda/latest/dg/intro-core-components.html
AWS Services as Event Sources
S3 – Push Model
Dynamo DB – Pull Model
Kinesis – Pull Model
Simple Notification Service – Push Model
Amazon Cognito – Push Model
CloudWatchLogs - Push model
CloudWatch Events – Push Model
Scheduled Events (cloudwatch) – Push Model
AWS Config – Push model
Cascading Lambda Functions
One Lambda Events can be the triggering Event for another
Dynamo DB
redshift
S3
Ways to Invoke Lambda
Currently there are 2 ways to invoke Lambda
Event
This invocation type causes AWS Lambda to execute the Lambda function
asynchronously.
The event sources Amazon S3, Amazon SNS or Amazon DynamoDB use this
invocation type.
Request Response
This invocation type causes AWS Lambda to execute the function synchronously
and returns the response immediately to the calling application. This invocation
type is available for custom applications.
Cases where Lambda
isn’t a good fit
The AWS Certified Solutions Architect – Associate Level exam is
intended for individuals who perform a Solutions Architect role.
Scenario
The need for the latest OS patches and Security updates minutes or
hours after they’re released
The need to access the Security logs the host
The need to change the OS settings such as Memory Allocation or max
file handlers
An auditing and compliance requirement that requires you to take a
snapshot of a host after a security issue
Cases where Lambda
is a good Fit
The AWS Certified Solutions Architect – Associate Level exam is
intended for individuals who perform a Solutions Architect role.
Scenario
Automatically Tagging newly Created Instances
Automatically Shutting off
Performing Database functions such as bulk inserts and vacuums
Converting Bash Scripts which run automation tasks
Sending notifications to your Slack Channel
Lambda Pricing
AWS Lambda is one of the most Economical Services on AWS
Requests
• First 1 million requests per month are free
• $0.20 per 1 million requests thereafter ($0.0000002 per request)
If all of this sounds confusing then you might try this calculator found on
Matthew Fuller’s blog
For more info
https://aws.amazon.com/lambda/pricing/
Programming Model for
Lambda Functions in Python
In building your Python Lambda packages there are 4 main topics to
know :
• The Lambda Function Handler
• The Context Object
• Logging
• Exceptions
For more info
http://docs.aws.amazon.com/lambda/latest/dg/python-programming-model.html
NOTE: At the time of presentation you must
write your Python lambda code in 2.7
The Handler
The Handler is the function AWS Lambda calls to start execution of your Lambda
Function.
When a Lambda function is invoked then it starts executing your code by calling the
handler function.
It also passes any event data as the first parameter.
The Handler is the function AWS Lambda calls to start execution of
your Lambda Function.
For more info
http://docs.aws.amazon.com/lambda/latest/dg/programming-model-v2.html
The Handler
def lambda_handler(event, context):
myfile = retrieveFile(input_file, bucket)
newfile = converter(myfile,index,mytype)
return success
The Handler is the function AWS Lambda calls to start execution of
your Lambda Function.
For more info
http://docs.aws.amazon.com/lambda/latest/dg/programming-model-v2.html
Logging (in Python)
The following Python statements generate log entries:
• print statements.
• Logger functions in the logging module ( ie logging.Logger.info /
logging.Logger.errror)
Both print and logging.* function write logs to CloudWatch logs but the logging.*
functions write additional information to each log entry such as timestamp and log level.
The Handler is the function AWS Lambda calls to start execution of
your Lambda Function.
For more info
http://docs.aws.amazon.com/lambda/latest/dg/python-logging.html
Time to Test
Testing your Lambda
Image by http://www.splitshire.com/
Lambda Testing Checklist
Is the function timeout to low?
Does the function have the correct IAM Role and permissions for services that it’s
using and to send logs to Cloudwatch Logs?
Has the function been allocated enough memory?
Building the Package
Step 1 Start with a small / micro EC2 instance
running AWS linux
Once you’ve written your code (And Tested it) now you can build
your package with dependencies, zip it up and then upload.
NOTE: There’s other ways to build the package…this is the fastest
most effective way I’ve found for python
Building the Package
cont.
Step 2 SSH to your instance and create a directory for your Lambda code
NOTE: The name of the file doesn’t matter as much as the lambda
handler calling your other functions does.
Step 3 Move your code to that directory either via scp or by cutting and
pasting to a file with a .py extension
$mkdir /home/aws-user/cleaner_lambda
Building the Package
Adding Dependencies
Step 4 Install any libaries or dependencies that you need via pip directly to the
directory which you created.
NOTE: The name of the file doesn’t matter as much as the lambda
handler calling your other functions does.
$pip install requests –t /home/aws-user/cleaner_lambda
Building the Package
Creating the zip file
Step 5 Create the zip file and add NOT ONLY your python file but also the
directories of the packages which pip has downloaded for you
NOTE: using the –r flag will recursively add all the files in the
subsequent folders (in our case requests)
$zip –r lambda_cleanup.zip lambda_cleanup.py requests/
Python Lambda Packages
What about writing to files?
Image by http://www.gratisography.com/
Writing to files
def retrieveFile(filename, bucket):
s3conn = boto.connect_s3()
bucket = s3conn.get_bucket(bucket)
fn=filename
key = bucket.get_key(fn)
dl = key.get_contents_to_filename('/tmp/'+fn)
myfile = ("/tmp/"+fn)
return myfile
You can write to files (while processing) but you must do so in /tmp
After you have manipulated your file then you will need to save
somewhere else, most likely an S3 bucket
Which Packages don’t
need to be added
There are a handful of libraries that when imported do not need to
be built into a zip file.
• Boto / Boto3
• Time
• OS
For more info
http://docs.aws.amazon.com/lambda/latest/dg/python-programming-model.html
Python Lambda Packages
Real World Examples
Image by http://www.gratisography.com/
Python Lambda Packages
Real World Examples
A Cloudwatch to Slack Lambda Function
A Lambda Job to backup RDS instances
A Lambda Job that cleans S3 buckets
Word Analysis with Lambda
Nightly Shutdown (and startup) with Lambda
Lambda in other Flavors
Python 2.7 Java 8node.js 4.3
node.js 0.10
Questions?
Image by http://www.gratisography.com/
Interested in Sponsoring
AWS Atlanta?
Image by http://www.gratisography.com/
Contact us at
http://www.meetup.com/AWS-Atlanta/

Aws meetup building_lambda

  • 1.
    Building Lambda Packages InPython Getting Past Hello World
  • 2.
  • 3.
    Find me onLinkedIn AWS Certifications Presented by Adam Book
  • 4.
    Lambda Uses Using aserverless technology like AWS Lambda takes a bit of shift in thinking If you think about the quote from Werner Vogles: “Everything fails all the time” Then you might currently have an AutoScaling group of min 1 max 1 Desired 1 just for an instance that is processing a piece of data or doing a task on a daily or scheduled basis. Lambda may be the cost effective way to take over that event.
  • 5.
    Lambda Uses Using aserverless technology like AWS Lambda takes a bit of shift in thinking Event Based Processing For more info http://docs.aws.amazon.com/lambda/latest/dg/intro-core-components.html AWS Services as Event Sources S3 – Push Model Dynamo DB – Pull Model Kinesis – Pull Model Simple Notification Service – Push Model Amazon Cognito – Push Model CloudWatchLogs - Push model CloudWatch Events – Push Model Scheduled Events (cloudwatch) – Push Model AWS Config – Push model
  • 6.
    Cascading Lambda Functions OneLambda Events can be the triggering Event for another Dynamo DB redshift S3
  • 7.
    Ways to InvokeLambda Currently there are 2 ways to invoke Lambda Event This invocation type causes AWS Lambda to execute the Lambda function asynchronously. The event sources Amazon S3, Amazon SNS or Amazon DynamoDB use this invocation type. Request Response This invocation type causes AWS Lambda to execute the function synchronously and returns the response immediately to the calling application. This invocation type is available for custom applications.
  • 8.
    Cases where Lambda isn’ta good fit The AWS Certified Solutions Architect – Associate Level exam is intended for individuals who perform a Solutions Architect role. Scenario The need for the latest OS patches and Security updates minutes or hours after they’re released The need to access the Security logs the host The need to change the OS settings such as Memory Allocation or max file handlers An auditing and compliance requirement that requires you to take a snapshot of a host after a security issue
  • 9.
    Cases where Lambda isa good Fit The AWS Certified Solutions Architect – Associate Level exam is intended for individuals who perform a Solutions Architect role. Scenario Automatically Tagging newly Created Instances Automatically Shutting off Performing Database functions such as bulk inserts and vacuums Converting Bash Scripts which run automation tasks Sending notifications to your Slack Channel
  • 10.
    Lambda Pricing AWS Lambdais one of the most Economical Services on AWS Requests • First 1 million requests per month are free • $0.20 per 1 million requests thereafter ($0.0000002 per request) If all of this sounds confusing then you might try this calculator found on Matthew Fuller’s blog For more info https://aws.amazon.com/lambda/pricing/
  • 11.
    Programming Model for LambdaFunctions in Python In building your Python Lambda packages there are 4 main topics to know : • The Lambda Function Handler • The Context Object • Logging • Exceptions For more info http://docs.aws.amazon.com/lambda/latest/dg/python-programming-model.html NOTE: At the time of presentation you must write your Python lambda code in 2.7
  • 12.
    The Handler The Handleris the function AWS Lambda calls to start execution of your Lambda Function. When a Lambda function is invoked then it starts executing your code by calling the handler function. It also passes any event data as the first parameter. The Handler is the function AWS Lambda calls to start execution of your Lambda Function. For more info http://docs.aws.amazon.com/lambda/latest/dg/programming-model-v2.html
  • 13.
    The Handler def lambda_handler(event,context): myfile = retrieveFile(input_file, bucket) newfile = converter(myfile,index,mytype) return success The Handler is the function AWS Lambda calls to start execution of your Lambda Function. For more info http://docs.aws.amazon.com/lambda/latest/dg/programming-model-v2.html
  • 14.
    Logging (in Python) Thefollowing Python statements generate log entries: • print statements. • Logger functions in the logging module ( ie logging.Logger.info / logging.Logger.errror) Both print and logging.* function write logs to CloudWatch logs but the logging.* functions write additional information to each log entry such as timestamp and log level. The Handler is the function AWS Lambda calls to start execution of your Lambda Function. For more info http://docs.aws.amazon.com/lambda/latest/dg/python-logging.html
  • 15.
    Time to Test Testingyour Lambda Image by http://www.splitshire.com/
  • 16.
    Lambda Testing Checklist Isthe function timeout to low? Does the function have the correct IAM Role and permissions for services that it’s using and to send logs to Cloudwatch Logs? Has the function been allocated enough memory?
  • 17.
    Building the Package Step1 Start with a small / micro EC2 instance running AWS linux Once you’ve written your code (And Tested it) now you can build your package with dependencies, zip it up and then upload. NOTE: There’s other ways to build the package…this is the fastest most effective way I’ve found for python
  • 18.
    Building the Package cont. Step2 SSH to your instance and create a directory for your Lambda code NOTE: The name of the file doesn’t matter as much as the lambda handler calling your other functions does. Step 3 Move your code to that directory either via scp or by cutting and pasting to a file with a .py extension $mkdir /home/aws-user/cleaner_lambda
  • 19.
    Building the Package AddingDependencies Step 4 Install any libaries or dependencies that you need via pip directly to the directory which you created. NOTE: The name of the file doesn’t matter as much as the lambda handler calling your other functions does. $pip install requests –t /home/aws-user/cleaner_lambda
  • 20.
    Building the Package Creatingthe zip file Step 5 Create the zip file and add NOT ONLY your python file but also the directories of the packages which pip has downloaded for you NOTE: using the –r flag will recursively add all the files in the subsequent folders (in our case requests) $zip –r lambda_cleanup.zip lambda_cleanup.py requests/
  • 21.
    Python Lambda Packages Whatabout writing to files? Image by http://www.gratisography.com/
  • 22.
    Writing to files defretrieveFile(filename, bucket): s3conn = boto.connect_s3() bucket = s3conn.get_bucket(bucket) fn=filename key = bucket.get_key(fn) dl = key.get_contents_to_filename('/tmp/'+fn) myfile = ("/tmp/"+fn) return myfile You can write to files (while processing) but you must do so in /tmp After you have manipulated your file then you will need to save somewhere else, most likely an S3 bucket
  • 23.
    Which Packages don’t needto be added There are a handful of libraries that when imported do not need to be built into a zip file. • Boto / Boto3 • Time • OS For more info http://docs.aws.amazon.com/lambda/latest/dg/python-programming-model.html
  • 24.
    Python Lambda Packages RealWorld Examples Image by http://www.gratisography.com/
  • 25.
    Python Lambda Packages RealWorld Examples A Cloudwatch to Slack Lambda Function A Lambda Job to backup RDS instances A Lambda Job that cleans S3 buckets Word Analysis with Lambda Nightly Shutdown (and startup) with Lambda
  • 26.
    Lambda in otherFlavors Python 2.7 Java 8node.js 4.3 node.js 0.10
  • 27.
  • 28.
    Interested in Sponsoring AWSAtlanta? Image by http://www.gratisography.com/ Contact us at http://www.meetup.com/AWS-Atlanta/

Editor's Notes

  • #5 If you think about the quote from Werner Vogles that everything fails all the time, In the case of Dynamo DB Lambda polls the stream and invokes the Lambda function when it detects new records. This is the same case of polling the stream with Kinesis. In the cases of the Push Models, the services actually ”Push” an event which then triggers the Lambda function.
  • #6 If you think about the quote from Werner Vogles that everything fails all the time, In the case of Dynamo DB Lambda polls the stream and invokes the Lambda function when it detects new records. This is the same case of polling the stream with Kinesis. In the cases of the Push Models, the services actually ”Push” an event which then triggers the Lambda function.
  • #8 If you think about the quote from Werner Vogles that everything fails all the time, In the case of Dynamo DB Lambda polls the stream and invokes the Lambda function when it detects new records. This is the same case of polling the stream with Kinesis. In the cases of the Push Models, the services actually ”Push” an event which then triggers the Lambda function.
  • #11 Where could Lambda not be cost effective? Well take a use case of having a single Lambda function with 1GB of memory that uses the entire 5 min period to execute and then keeps getting invoked so that it is constantly running for a whole month. This could cost as much as (or over) 3 times a comparable EC2 instance
  • #13  One of the best uses of the Join is in the output section and to produce the output endpoint for your users.
  • #14  The two subsequent lines underneath the handler in this example are functions defined later in the code
  • #15  One of the best uses of the Join is in the output section and to produce the output endpoint for your users.
  • #16 Before you create your zip package and add all included dependencies to the zip file then you want to test, Here is a technique or two that can give you a head start.
  • #17 If you expect a function to execute in 1000 ms, cold boot times and potential external resource connection issues may mean the function occasionally takes 2000 ms. Increasing the timeout exponentially can prevent unexpected timeouts.
  • #18  The two subsequent lines underneath the handler in this example are functions defined later in the code
  • #19  The two subsequent lines underneath the handler in this example are functions defined later in the code
  • #20  The two subsequent lines underneath the handler in this example are functions defined later in the code
  • #21  The two subsequent lines underneath the handler in this example are functions defined later in the code
  • #23  The two subsequent lines underneath the handler in this example are functions defined later in the code