In today's competitive environment, startups are increasingly focused on eliminating any undifferentiated heavy lifting. Come learn about various architectural patterns for building a scalable, function-rich data processing systems, using AWS Lambda and other AWS managed services. Come see how PlayOn! Sports went from a multi-layered architecture for video streaming to a streamlined and serverless system using Lambda and Amazon S3.

1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Vyom Nagrani, Sr. Product Manager, AWS Lambda
Jay Sandhaus, CTO, PlayOn! Sports
October 2015
ARC308
The Serverless Company
Using AWS Lambda
Streamlining Architecture with AWS

2. What to Expect from the Session
• Evolution of Cloud Compute toward Serverless
• What is AWS Lambda?
• Building a serverless data processing architecture
• Example of a live video processing architecture
• Real-time data processing architecture patterns
• Tips and tricks for using AWS Lambda

3. Evolution of Cloud Compute
Toward Serverless

4. Evolution of Compute – Public Cloud
InfrastructureInstances
Application code

5. Evolution of Compute – Containers
InfrastructureInstances
Application codeContainers

6. Evolution of Compute – Serverless
Application code

7. What Is AWS Lambda?

8. AWS Lambda
Serverless, event-driven compute service
Lambda = microservice without servers

9. AWS Lambda – Benefits
EVENT-DRIVEN SCALESERVERLESS SUBSECOND BILLING

10. AWS Lambda – Capabilities
BRING YOUR OWN CODE SIMPLE RESOURCE MODEL
FLEXIBLE INVOCATION PATHS GRANULAR PERMISSIONS CONTROL

11. AWS Lambda – How it Works
AUTHORING STATELESS
DEPLOYMENT MONITORING & LOGGING

12. AWS Lambda – Partner Ecosystem

13. Building a Serverless Data
Processing Architecture

14. SOURCE of
data
Fleet of servers poll for
changes
Listening to source Pushes to queue Pull off queue
Fleet of servers act as
workers to process the data
Auto-scale
worker nodes
to adjust with
load
S3 objects
Amazon Kinesis records
DynamoDB tables
SNS messages
IoT devices
…
Cross-AZ
replication for
high
availability
Load
balancer to
distribute
traffic
Data Processing Architecture with Servers
Application code

15. SOURCE of
data
Attach a Lambda function
And that’s it!
Lambda does
Listening/polling
Queuing
Auto scaling
Redundancy
Load balancing
Data Processing Architecture Without Servers
Application code

16. Example of a Live Video
Processing Architecture

17. Company Overview & Background
 PlayOn! Sports streams live high school sports
 Tools to produce & stream a professional broadcast
 Inverse use case from typical media company
 Many simultaneous events
 Smaller audience for each (long tail)
 Challenge: transcode many live video streams at once
 Peak hour @ 300 live inbound streams and 20k viewers

18. Legacy Architecture
Limelight
Edge
Laptop
Encoders
RTMP
Wowza Transmux
Servers
Wowza Transcode
Servers
Wowza Record
Servers
RTMP
RTMP
RTMP
Auto
Uploader
MP4 HLS
S3 VOD
bucket
CloudFront mobile client
Live Streaming
VOD
CloudFront mobile client
HLS
HLS

19. Reasons for Change
 Move from RTMP to HLS
 Minimize operational overhead
 Adaptive bitrate streaming
 Mid-roll advertising
 Minimize expense
 Cleaner encoding workflow
 Dev focus video quality

20. Current Architecture
Laptop
Encoders
HLS
S3
Playback
VOD Stream
mobile client
CloudFront
Streaming
Live stream
mobile client
CloudFront S3 Ingest
480p
Transcode
HQ Copy
360p
Transcode
Audio-only
Transcode
Thumbnail
QOS
Analytics
Cascading Lambda Functions

21. What We Learned
 So far, Lambda works as expected
 Functions can be built and deployed with minimal effort
 Successfully able to focus on the domain-specific part of
our problem (video) with minimal ops work
 Need to ensure we handle Lambda jobs not completing
 Use SNS to work around S3 event notification method
 Cost savings appear to be one order of magnitude

22. Real-Time Data Processing
Architecture Patterns

23. Sample Real-Time File Processing Workflow
Using Amazon S3 & AWS Lambda
Amazon S3 AWS Lambda
Amazon S3
New file
uploaded
Amazon
DynamoDB

24. Demo – Video Clip Transcode
// Transcode video files uploaded to an S3 bucket
var async = require('async');
var fs = require('fs');
var exec = require('child_process').exec;
var aws = require('aws-sdk');
var s3 = new aws.S3();
// Lambda entry point
exports.handler = function(event, context) {
var inputVideo = '/tmp/myVideo.avi';
var outputVideo = '/tmp/myVideo.mp4';
var inputBucket = event.Records[0].s3.bucket.name;
var inputKey = event.Records[0].s3.object.key;
var outputBucket = inputBucket + '-out';
var outputKey = inputKey.substr(0,inputKey.length-4) + '.mp4';
// Runs the array of functions in series, each passing their results to the next
async.waterfall([
Set up variables
Serialize steps

25. Demo – Video Clip Transcode
// Download the object from S3 into a buffer
function download(next) {
s3.getObject({Bucket: inputBucket, Key: inputKey}, next); },
// Write the buffer to a file
function write(response, next) {
fs.writeFile(inputVideo, response.Body, next); },
// Spawn a process to transcode
function transcode(next) {
exec("./ffmpeg -i " + inputVideo + " -vcodec mpeg4 -b:v 1200k “
+ outputVideo, next); },
// Read the file to a buffer
function read(next) {
fs.readFile(outputVideo, next); },
// Upload the buffer to an object in S3
function upload(objectBuffer, next) {
s3.putObject({Bucket: outputBucket, Key: outputKey, Body: objectBuffer},
context.done); },
], context.fail);
};
Get file from S3
Write to disk
ffmpeg transcode
Read from disk
Upload to S3

26. Sample Real-Time Stream Processing Workflow
Using Amazon Kinesis & AWS Lambda
Amazon Kinesis AWS Lambda
Amazon Redshift
Amazon SNS
New records
available

27. Sample Real-Time DB Triggers Workflow Using
Amazon DynamoDB & AWS Lambda
New data
available
Amazon DynamoDB AWS Lambda
Amazon
DynamoDB
Amazon
Redshift

28. Sample Real-Time Message Handling Workflow
Using Amazon SNS & AWS Lambda
New message
published
Amazon SNS AWS Lambda
Amazon SNS
Amazon Kinesis

29. Sample CRUD Backend Workflow Using
Amazon API Gateway & AWS Lambda
New API
called
Amazon API Gateway AWS Lambda
Amazon S3
Amazon
DynamoDB

30. More Triggers for AWS Lambda
AWS
CloudFormation
Custom Resources
Amazon SES
Actions
Amazon Cognito
Sync Triggers
… and the list will
continue to grow!
AWS IoT ActionsAWS Lambda
Scheduled Events
Amazon Echo
Skills
Amazon SWF
Tasks

31. Tips and Tricks for Using AWS
Lambda

32. Using Lambda to Audit CloudTrail Activity
AWS
Lambda
Amazon S3Amazon CloudTrail
Amazon SNS
AWS IAM

33. Using Lambda for Automated Infrastructure
Management
AWS
Lambda
Amazon
SNS
Amazon
CloudWatch Alarm
ec2
runInstance
ecs
startTask
beanstalk
updateApp
kinesis
splitShard
Any API call
https://aws.amazon.com/blogs/compute/scaling-amazon-ecs-services-automatically-using-amazon-cloudwatch-and-aws-lambda/

34. Using Lambda to Forward AWS Events to
External Endpoints
http://danilop.net/aws/2015/07/26/sns2ifttt/ | https://github.com/danilop/SNS2IFTTT
AWS
Lambda
Amazon
SNS
IFTTT via
the Maker
channel
Amazon
CloudWatch Events
Auto Scaling

35. Using Lambda to Deploy Lambda Functions
https://aws.amazon.com/blogs/compute/dynamic-github-actions-with-aws-lambda/
AWS
Lambda
Amazon
SNS
GitHub Repo
lambda
createFn ()

36. A Few Other Tips and Tricks for Extending
AWS Lambda Functionality
 Use /tmp space as cache
 Run arbitrary binaries
 Use Grunt and Jenkins plugins for deploying
 Hook up to Slack
 Build event-driven Amazon ECS
 Try out JAWS: The Server-less Application Framework

37. Thank you!
Visit http://aws.amazon.com/lambda, the
AWS Compute blog, or the Lambda forum to
learn more and to get started using Lambda.

38. Remember to complete
your evaluations!

39. Related Sessions