Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A Serverless Data Pipeline

Managed services + AWS Lambda = Serverless architecture. Presented by Julien Simon, AWS Evangelist.

  • Login to see the comments

A Serverless Data Pipeline

  1. 1. ©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved A serverless data pipeline Julien Simon, Principal Technical Evangelist, AWS julsimon@amazon.fr @julsimon
  2. 2. Werner Vogels, CTO, Amazon.com AWS re:Invent 2015
  3. 3. Managed services + AWS Lambda = Serverless architecture
  4. 4. AWS Lambda •  Deploy pure funcJons in Java, Python and Node.js •  Works nicely with AWS managed services: Amazon S3, Amazon DynamoDB, etc. •  Build event-driven applicaJons •  Build RESTful APIs in conjuncJon with Amazon API Gateway •  Pay as you go: number of requests + execuJon Jme (100ms slots)
  5. 5. Another way to put it… Tim Wagner, General Manager, AWS Lambda Serverless conference, NYC, May 2016
  6. 6. AdRoll h^ps://www.youtube.com/watch?v=JFfvD2cw2IE h^ps://aws.amazon.com/fr/dynamodb/adtech/ 60 billion Real-Time Bidding events daily
  7. 7. Building a serverless data pipeline Lambda DynamoDB Kinesis Firehose API Gateway HTTP POST 
 /prod/logger writeTo Kinesis DynamoDB ToFirehose S3 eventTable DynamoDB streams bucket EMR, Redshift, … firehoseToS3 Kinesis StreamsLambda Lambda KinesisTo DynamoDB Web apps
  8. 8. CLI: create DynamoDB table aws dynamodb create-table --table-name eventTable --attribute-definitions AttributeName=userId,AttributeType=N AttributeName=timestamp,AttributeType=N --key-schema AttributeName=userId,KeyType=HASH AttributeName=timestamp,KeyType=RANGE --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5 --stream-specification StreamEnabled=true,StreamViewType=NEW_IMAGE eventTable DynamoDB
  9. 9. CLI: create Lambda funcIon aws lambda create-function --function-name DynamoDBToFirehose --role DYNAMODBTOFIREHOSE_ROLE_ARN --zip-file fileb://DynamoDBToFirehose.zip --handler DynamoDBToFirehose.lambda_handler --runtime python2.7 --memory-size 128 --description "Write DynamoDB stream to Kinesis Firehose" aws lambda create-event-source-mapping --function-name DynamoDBToFirehose --event-source DYNAMODB_STREAM_ARN --batch-size 10 --starting-position TRIM_HORIZON Lambda DynamoDB ToFirehose eventTable DynamoDB streams DynamoDB
  10. 10. CLI: create Kinesis Firehose stream aws firehose create-delivery-stream --delivery-stream-name firehoseToS3 --s3-destination-configuration RoleARN=FIREHOSETOS3_ROLE_ARN, BucketARN="arn:aws:s3:::jsimon-public", Prefix="firehose", BufferingHints={SizeInMBs=1,IntervalInSeconds=60}, CompressionFormat="GZIP", EncryptionConfiguration={NoEncryptionConfig="NoEncryption"} Kinesis Firehose firehoseToS3 Lambda DynamoDB ToFirehose eventTable DynamoDB streams DynamoDB
  11. 11. Building a serverless data pipeline Lambda DynamoDB Kinesis Firehose API Gateway HTTP POST 
 /prod/logger writeTo Kinesis DynamoDB ToFirehose S3 eventTable DynamoDB streams bucket EMR, Redshift, … firehoseToS3 Kinesis StreamsLambda Lambda KinesisTo DynamoDB Web apps Lines of code: 16 Number of servers: zero Performance & scalability: maximum
  12. 12. Ready for some tesIng? hMp://api.julien.org
  13. 13. Upcoming book on AWS Lambda Wri^en by AWS Technical Evangelist Danilo Poccia Early release available at: h^ps://www.manning.com/ books/aws-lambda-in-acJon
  14. 14. To go further h^p://aws.amazon.com/lambda/ h^ps://aws.amazon.com/blogs/compute/ AWS User Group Turkey h^p://www.meetup.com/AWS-User-Group-Turkey/ Qwiklabs: h^ps://qwiklabs.com Serverless framework: h^ps://github.com/serverless/
  15. 15. Teşekkür ederim J Julien Simon, Principal Technical Evangelist, AWS julsimon@amazon.fr @julsimon

×