This document discusses building a serverless data pipeline using AWS Lambda and other AWS managed services like DynamoDB, Kinesis Firehose and S3. It provides steps to create a DynamoDB table with streams enabled, a Lambda function to read from the DynamoDB streams and write to Kinesis Firehose, and a Kinesis Firehose delivery stream to deliver data to S3. With these serverless components, data can be ingested and processed without having to provision or manage any servers.

1. ©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved
A serverless data pipeline
Julien Simon, Principal Technical Evangelist, AWS
julsimon@amazon.fr
@julsimon

2. Werner Vogels, CTO, Amazon.com
AWS re:Invent 2015

3. Managed services
+
AWS Lambda
=
Serverless architecture

4. AWS Lambda
• Deploy pure funcJons in Java, Python and Node.js
• Works nicely with AWS managed services:
Amazon S3, Amazon DynamoDB, etc.
• Build event-driven applicaJons
• Build RESTful APIs in conjuncJon with Amazon API Gateway
• Pay as you go: number of requests + execuJon Jme (100ms slots)

5. Another way to put it…
Tim Wagner,
General Manager,
AWS Lambda
Serverless conference, NYC, May 2016

6. AdRoll
h^ps://www.youtube.com/watch?v=JFfvD2cw2IE
h^ps://aws.amazon.com/fr/dynamodb/adtech/
60 billion Real-Time Bidding events daily

7. Building a serverless data pipeline
Lambda
DynamoDB
Kinesis
Firehose
API Gateway
HTTP POST
/prod/logger writeTo
Kinesis
DynamoDB
ToFirehose
S3
eventTable
DynamoDB
streams
bucket
EMR,
Redshift,
…
firehoseToS3
Kinesis StreamsLambda Lambda
KinesisTo
DynamoDB
Web apps

8. CLI: create DynamoDB table
aws dynamodb create-table
--table-name eventTable
--attribute-definitions
AttributeName=userId,AttributeType=N
AttributeName=timestamp,AttributeType=N
--key-schema
AttributeName=userId,KeyType=HASH
AttributeName=timestamp,KeyType=RANGE
--provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5
--stream-specification StreamEnabled=true,StreamViewType=NEW_IMAGE
eventTable
DynamoDB

9. CLI: create Lambda funcIon
aws lambda create-function
--function-name DynamoDBToFirehose
--role DYNAMODBTOFIREHOSE_ROLE_ARN
--zip-file fileb://DynamoDBToFirehose.zip
--handler DynamoDBToFirehose.lambda_handler
--runtime python2.7
--memory-size 128
--description "Write DynamoDB stream to Kinesis Firehose"
aws lambda create-event-source-mapping
--function-name DynamoDBToFirehose
--event-source DYNAMODB_STREAM_ARN
--batch-size 10
--starting-position TRIM_HORIZON
Lambda
DynamoDB
ToFirehose
eventTable
DynamoDB
streams
DynamoDB

10. CLI: create Kinesis Firehose stream
aws firehose create-delivery-stream
--delivery-stream-name firehoseToS3
--s3-destination-configuration
RoleARN=FIREHOSETOS3_ROLE_ARN,
BucketARN="arn:aws:s3:::jsimon-public",
Prefix="firehose",
BufferingHints={SizeInMBs=1,IntervalInSeconds=60},
CompressionFormat="GZIP",
EncryptionConfiguration={NoEncryptionConfig="NoEncryption"}
Kinesis
Firehose
firehoseToS3
Lambda
DynamoDB
ToFirehose
eventTable
DynamoDB
streams
DynamoDB

11. Building a serverless data pipeline
Lambda
DynamoDB
Kinesis
Firehose
API Gateway
HTTP POST
/prod/logger writeTo
Kinesis
DynamoDB
ToFirehose
S3
eventTable
DynamoDB
streams
bucket
EMR,
Redshift,
…
firehoseToS3
Kinesis StreamsLambda Lambda
KinesisTo
DynamoDB
Web apps
Lines of code: 16
Number of servers: zero
Performance & scalability: maximum

12. Ready for some tesIng?
hMp://api.julien.org

13. Upcoming book on AWS Lambda
Wri^en by AWS Technical
Evangelist Danilo Poccia
Early release available at:
h^ps://www.manning.com/
books/aws-lambda-in-acJon

14. To go further
h^p://aws.amazon.com/lambda/
h^ps://aws.amazon.com/blogs/compute/
AWS User Group Turkey
h^p://www.meetup.com/AWS-User-Group-Turkey/
Qwiklabs: h^ps://qwiklabs.com
Serverless framework: h^ps://github.com/serverless/

15. Teşekkür ederim J
Julien Simon, Principal Technical Evangelist, AWS
julsimon@amazon.fr
@julsimon