SlideShare a Scribd company logo
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cecilia Deng
Software Developer
AWS Lambda
12/01/2016
SVR301
Real-Time Processing Using AWS
Lambda
Anders Fritz
Senior Manager
ThomsonReuters
Marco Pierleoni
Lead Software Developer
Thomson Reuters
What to Expect from the Session
• What kinds of real time events can trigger lambda?
• How does Lambda pull and process streams?
• What are some stream processing behaviors?
• Hear how Thomson Reuters went real time with AWS
Lambda
Flavors of real time event sources
Asynchronous Invoke
Push Event Source
Synchronous Invoke
Push Event Source
Stream
Pull Event Source
S3
async invoke
Alexa skill
sync invoke
Pull then sync invoke
DynamoDB
Update Stream
Real-time push
Real-time push
Who?
• Any integrator that uses AWS Lambda invoke API
• E.g., Amazon S3, Amazon SNS, Amazon Alexa, AWS IoT
What?
• Event sources sending events to Lambda for processing
How?
• Real-time triggered events owned by event source
• Real-time processing owned by Lambda invoke methods
Real-time push
Synchronous Invoke
Push Event Source
Asynchronous Invoke
Push Event Source
Real-time pull
Real-time pull
Who?
• Amazon Kinesis and DynamoDB update streams
What?
• Lambda grabbing events from a stream for processing
How?
• Mapping maintained by Lambda
• Real-time triggered events owned by DDB or Kinesis
producer
• Real-time processing owned by Lambda stream polling
component and invoke methods
Real-time pull
Stream
Pull Event Source
Processing streams
Processing streams: Kinesis setup
Streams
▪ Made up of shards
▪ Each shard supports writes up to 1 MB/s
▪ Each shard supports reads up to 2 MB/s
▪ Each shard supports 5 reads/s
Data
▪ All data is stored and replayable for 24 hours by default
▪ Make sure partition key distribution is even to optimize parallel throughput
▪ Pick a key with more groups than shards
Processing streams: Lambda setup
Memory
▪ CPU is proportional to the memory
configured
▪ More memory means faster execution,
if CPU bound
▪ More memory means larger sized
record batches can be processed
Timeout
• Increasing timeout allows for longer functions, but more wait in case of
errors
Permission model
• The execution role defined for Lambda must have permission to access
the stream
Processing streams: event source setup
Batch size
▪ Max number of records that Lambda will send in one invocation
▪ Not equivalent to how many records Lambda gets from Kinesis
▪ Effective batch size is
MIN(records available, batch size, 6 MB)
▪ Increasing batch size allows fewer Lambda function invocations with
more data processed per function
Processing streams: event source setup
Starting Position:
▪ The position in the stream where Lambda starts reading
▪ Set to “Trim Horizon” for reading from start of stream (all data)
▪ Set to “Latest” for reading most recent data (LIFO) (latest data)
Processing streams: event source setup
Amazon
Kinesis 1
AWS
Lambda 1
Amazon
CloudWatch
Amazon
DynamoDB
AWS
Lambda 2 Amazon
S3
• Multiple functions can be mapped to
one stream
• Multiple streams can be mapped to
one Lambda function
• Each mapping is a unique key pair
Kinesis stream to Lambda function
• Each mapping has unique shard
iterators
Amazon
Kinesis 2
Processing streams: under the hood
Event received by Lambda function is a collection of records from the stream
{ "Records": [ {
"kinesis": {
"partitionKey": "partitionKey-3",
"kinesisSchemaVersion": "1.0",
"data": "SGVsbG8sIHRoaXMgaXMgYSB0ZXN0IDEyMy4=",
"sequenceNumber": "49545115243490985018280067714973144582180062593244200961" },
"eventSource": "aws:kinesis",
"eventID": "shardId-
000000000000:49545115243490985018280067714973144582180062593244200961",
"invokeIdentityArn": "arn:aws:iam::account-id:role/testLEBRole",
"eventVersion": "1.0",
"eventName": "aws:kinesis:record",
"eventSourceARN": "arn:aws:kinesis:us-west-2:35667example:stream/examplestream",
"awsRegion": "us-west-2" } ] }
Processing streams: under the hood
Polling
▪ Concurrent polling and processing per shard
▪ Polls every 250 ms if no records found
▪ Grab as much as possible in one GetRecords call
Batching
▪ Sub batch in memory for invocation payload
Synchronous invocation
▪ Batches invoked as synchronous RequestResponse type
▪ Lambda honors Kinesis at least once semantics
▪ Each shard blocks on in order synchronous invocation
Processing streams: under the hood
Per Shard:
▪ Lambda calls GetRecords with max limit from Kinesis (10 k or 10 MB)
▪ If no record, wait 250 ms
▪ From in memory, sub batches and formats records into Lambda payload
▪ Invoke Lambda with synchronous invoke
… …
Source
Kinesis Lambda Polling Logic
Shards
Lambda will scale automaticallyScale Kinesis by adding shards
Batch sync invokesPolls
Processing streams: how it works
▪ Lambda blocks on ordered processing for each individual shard
▪ Increasing # of shards with even distribution allows increased concurrency
▪ Batch size may impact duration if the Lambda function takes longer to process
more records
… …
Source
Kinesis Lambda Polling Logic
Shards
Lambda will scale automaticallyScale Kinesis by adding shards
Batch sync invokesPolls
Processing streams: under the hood
▪ Retry execution failures until the record is expired
▪ Retry with exponential backoff up to 60 s
▪ Throttles and errors impacts duration and directly impacts throughput
Kinesis
…
Source
Scale Kinesis by adding shards
Lambda Polling Logic
Lambda will scale automatically
Polls
invoke fail
invoke fail
invoke success
Batch sync invokes
Processing streams: under the hood
▪ Maximum theoretical throughput:
# shards * 2 MB / (s)
▪ Effective theoretical throughput:
( # shards * batch size (MB) ) / ( function duration (s) * retries until expiry)
▪ If put / ingestion rate is greater than the theoretical throughput, consider
increasing number of shards of optimizing function duration to increase
throughput
Processing streams: how it looks
•GetRecords (effective throughput): bytes, latency, records, etc.
•PutRecord: bytes, latency, records, etc.
•GetRecords.IteratorAgeMilliseconds: how old your last processed records were.
If high, processing is falling behind. If close to 24 hours, records are close to
being dropped.
Processing streams: how it looks
Amazon CloudWatch Metrics
• Invocation count
• Duration
• Error count
• Throttle count
Amazon CloudWatch Logs
• All Metrics
• Custom logs
• RAM consumed
Processing streams: how it looks
Common observations:
▪ Effective batch size may be less than configured during low throughput
▪ Effective batch size will increase during higher throughput
▪ Increased Lambda duration -> decreased # of invokes and GetRecord calls
▪ Too many consumers of your stream may compete with Kinesis read limits and
induce ReadProvisionedThroughputExceeded errors and metrics
ANALYSING USAGE OF THOMSON
REUTERS PRODUCTS WITH AWS
Anders Fritz & Marco Pierleoni
CHALLENGE
To identify and define a solution for usage analytics tracking that enables product teams to
take ownership of the usage data collected. In addition to tracking and visualizing usage
data it had to;
1. Cross reference Usage
with Business data
4. Require Limited
Maintenance.
3. Auto Scale as data
flow fluctuates.
2. Follow TR Security &
Compliance rules.
5. Launch in 5
months.
SOLUTION
SOLUTION
SOLUTION
SOLUTION
SOLUTION
SOLUTION
SOLUTION
• Product Insight is live – adoption rate high.
• Tested 4,000 requests per second while targeting 5bn requests / month.
• Since March – very little maintenance required
• No Outages
• No Downtime
• Cloudwatch monitor everything.
• Latency – Data visible on chart within 10 seconds
• BrExit and US elections tested autoscaling.
• US elections ~16m events – normally ~ 6-8m events / day.
• UK EU referendum (BrExit) ~ 10m events – normally ~ 5m events / day
OUTCOME
EVENTS CAPTURED
UK EU Referendum June 23rd (BrExit)
time
#events
EVENTS CAPTURED
US Elections November 8th
time
#events
Thank you!
Remember to complete
your evaluations!

More Related Content

What's hot

AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most fro...
AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most fro...AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most fro...
AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most fro...
Amazon Web Services
 
AWS re:Invent 2016: Using AWS Lambda to Build Control Systems for Your AWS In...
AWS re:Invent 2016: Using AWS Lambda to Build Control Systems for Your AWS In...AWS re:Invent 2016: Using AWS Lambda to Build Control Systems for Your AWS In...
AWS re:Invent 2016: Using AWS Lambda to Build Control Systems for Your AWS In...
Amazon Web Services
 
Automate Migration to AWS with Datapipe
Automate Migration to AWS with DatapipeAutomate Migration to AWS with Datapipe
Automate Migration to AWS with Datapipe
Amazon Web Services
 
AWS re:Invent 2016: Wild Rydes Takes Off – The Dawn of a New Unicorn (SVR309)
AWS re:Invent 2016: Wild Rydes Takes Off – The Dawn of a New Unicorn (SVR309)AWS re:Invent 2016: Wild Rydes Takes Off – The Dawn of a New Unicorn (SVR309)
AWS re:Invent 2016: Wild Rydes Takes Off – The Dawn of a New Unicorn (SVR309)
Amazon Web Services
 
NEW LAUNCH! Developing Serverless C# Applications
NEW LAUNCH! Developing Serverless C# ApplicationsNEW LAUNCH! Developing Serverless C# Applications
NEW LAUNCH! Developing Serverless C# Applications
Amazon Web Services
 
Real-time Data Processing Using AWS Lambda
Real-time Data Processing Using AWS LambdaReal-time Data Processing Using AWS Lambda
Real-time Data Processing Using AWS Lambda
Amazon Web Services
 
AWS re:Invent 2016: Automating and Scaling Infrastructure Administration with...
AWS re:Invent 2016: Automating and Scaling Infrastructure Administration with...AWS re:Invent 2016: Automating and Scaling Infrastructure Administration with...
AWS re:Invent 2016: Automating and Scaling Infrastructure Administration with...
Amazon Web Services
 
Getting Started with AWS Lambda and the Serverless Cloud
Getting Started with AWS Lambda and the Serverless CloudGetting Started with AWS Lambda and the Serverless Cloud
Getting Started with AWS Lambda and the Serverless Cloud
Amazon Web Services
 
AWS re:Invent 2016: Running Lean Architectures: How to Optimize for Cost Effi...
AWS re:Invent 2016: Running Lean Architectures: How to Optimize for Cost Effi...AWS re:Invent 2016: Running Lean Architectures: How to Optimize for Cost Effi...
AWS re:Invent 2016: Running Lean Architectures: How to Optimize for Cost Effi...
Amazon Web Services
 
Real-time Data Processing Using AWS Lambda
Real-time Data Processing Using AWS LambdaReal-time Data Processing Using AWS Lambda
Real-time Data Processing Using AWS Lambda
Amazon Web Services
 
Real-Time Streaming Data on AWS
Real-Time Streaming Data on AWSReal-Time Streaming Data on AWS
Real-Time Streaming Data on AWS
Amazon Web Services
 
BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch Service
BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch ServiceBDA402 Deep Dive: Log Analytics with Amazon Elasticsearch Service
BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch Service
Amazon Web Services
 
AWS re:Invent 2016: 6 Million New Registrations in 30 Days: How the Chick-fil...
AWS re:Invent 2016: 6 Million New Registrations in 30 Days: How the Chick-fil...AWS re:Invent 2016: 6 Million New Registrations in 30 Days: How the Chick-fil...
AWS re:Invent 2016: 6 Million New Registrations in 30 Days: How the Chick-fil...
Amazon Web Services
 
WKS407 Wild Rydes Takes Off – The Dawn of a New Unicorn
WKS407 Wild Rydes Takes Off – The Dawn of a New Unicorn WKS407 Wild Rydes Takes Off – The Dawn of a New Unicorn
WKS407 Wild Rydes Takes Off – The Dawn of a New Unicorn
Amazon Web Services
 
AWS re:Invent 2016: Getting Started with Serverless Architectures (CMP211)
AWS re:Invent 2016: Getting Started with Serverless Architectures (CMP211)AWS re:Invent 2016: Getting Started with Serverless Architectures (CMP211)
AWS re:Invent 2016: Getting Started with Serverless Architectures (CMP211)
Amazon Web Services
 
NEW LAUNCH! Delivering Powerful Graphics-Intensive Applications from the AWS ...
NEW LAUNCH! Delivering Powerful Graphics-Intensive Applications from the AWS ...NEW LAUNCH! Delivering Powerful Graphics-Intensive Applications from the AWS ...
NEW LAUNCH! Delivering Powerful Graphics-Intensive Applications from the AWS ...
Amazon Web Services
 
Building a Real Time Dashboard with Amazon Kinesis, Amazon Lambda and Amazon ...
Building a Real Time Dashboard with Amazon Kinesis, Amazon Lambda and Amazon ...Building a Real Time Dashboard with Amazon Kinesis, Amazon Lambda and Amazon ...
Building a Real Time Dashboard with Amazon Kinesis, Amazon Lambda and Amazon ...
Amazon Web Services
 
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
Amazon Web Services
 
AWS re:Invent 2016: Configuration Management in the Cloud (DEV305)
AWS re:Invent 2016: Configuration Management in the Cloud (DEV305)AWS re:Invent 2016: Configuration Management in the Cloud (DEV305)
AWS re:Invent 2016: Configuration Management in the Cloud (DEV305)
Amazon Web Services
 
(ARC305) How J&J Manages AWS At Scale For Enterprise Workloads
(ARC305) How J&J Manages AWS At Scale For Enterprise Workloads(ARC305) How J&J Manages AWS At Scale For Enterprise Workloads
(ARC305) How J&J Manages AWS At Scale For Enterprise Workloads
Amazon Web Services
 

What's hot (20)

AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most fro...
AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most fro...AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most fro...
AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most fro...
 
AWS re:Invent 2016: Using AWS Lambda to Build Control Systems for Your AWS In...
AWS re:Invent 2016: Using AWS Lambda to Build Control Systems for Your AWS In...AWS re:Invent 2016: Using AWS Lambda to Build Control Systems for Your AWS In...
AWS re:Invent 2016: Using AWS Lambda to Build Control Systems for Your AWS In...
 
Automate Migration to AWS with Datapipe
Automate Migration to AWS with DatapipeAutomate Migration to AWS with Datapipe
Automate Migration to AWS with Datapipe
 
AWS re:Invent 2016: Wild Rydes Takes Off – The Dawn of a New Unicorn (SVR309)
AWS re:Invent 2016: Wild Rydes Takes Off – The Dawn of a New Unicorn (SVR309)AWS re:Invent 2016: Wild Rydes Takes Off – The Dawn of a New Unicorn (SVR309)
AWS re:Invent 2016: Wild Rydes Takes Off – The Dawn of a New Unicorn (SVR309)
 
NEW LAUNCH! Developing Serverless C# Applications
NEW LAUNCH! Developing Serverless C# ApplicationsNEW LAUNCH! Developing Serverless C# Applications
NEW LAUNCH! Developing Serverless C# Applications
 
Real-time Data Processing Using AWS Lambda
Real-time Data Processing Using AWS LambdaReal-time Data Processing Using AWS Lambda
Real-time Data Processing Using AWS Lambda
 
AWS re:Invent 2016: Automating and Scaling Infrastructure Administration with...
AWS re:Invent 2016: Automating and Scaling Infrastructure Administration with...AWS re:Invent 2016: Automating and Scaling Infrastructure Administration with...
AWS re:Invent 2016: Automating and Scaling Infrastructure Administration with...
 
Getting Started with AWS Lambda and the Serverless Cloud
Getting Started with AWS Lambda and the Serverless CloudGetting Started with AWS Lambda and the Serverless Cloud
Getting Started with AWS Lambda and the Serverless Cloud
 
AWS re:Invent 2016: Running Lean Architectures: How to Optimize for Cost Effi...
AWS re:Invent 2016: Running Lean Architectures: How to Optimize for Cost Effi...AWS re:Invent 2016: Running Lean Architectures: How to Optimize for Cost Effi...
AWS re:Invent 2016: Running Lean Architectures: How to Optimize for Cost Effi...
 
Real-time Data Processing Using AWS Lambda
Real-time Data Processing Using AWS LambdaReal-time Data Processing Using AWS Lambda
Real-time Data Processing Using AWS Lambda
 
Real-Time Streaming Data on AWS
Real-Time Streaming Data on AWSReal-Time Streaming Data on AWS
Real-Time Streaming Data on AWS
 
BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch Service
BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch ServiceBDA402 Deep Dive: Log Analytics with Amazon Elasticsearch Service
BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch Service
 
AWS re:Invent 2016: 6 Million New Registrations in 30 Days: How the Chick-fil...
AWS re:Invent 2016: 6 Million New Registrations in 30 Days: How the Chick-fil...AWS re:Invent 2016: 6 Million New Registrations in 30 Days: How the Chick-fil...
AWS re:Invent 2016: 6 Million New Registrations in 30 Days: How the Chick-fil...
 
WKS407 Wild Rydes Takes Off – The Dawn of a New Unicorn
WKS407 Wild Rydes Takes Off – The Dawn of a New Unicorn WKS407 Wild Rydes Takes Off – The Dawn of a New Unicorn
WKS407 Wild Rydes Takes Off – The Dawn of a New Unicorn
 
AWS re:Invent 2016: Getting Started with Serverless Architectures (CMP211)
AWS re:Invent 2016: Getting Started with Serverless Architectures (CMP211)AWS re:Invent 2016: Getting Started with Serverless Architectures (CMP211)
AWS re:Invent 2016: Getting Started with Serverless Architectures (CMP211)
 
NEW LAUNCH! Delivering Powerful Graphics-Intensive Applications from the AWS ...
NEW LAUNCH! Delivering Powerful Graphics-Intensive Applications from the AWS ...NEW LAUNCH! Delivering Powerful Graphics-Intensive Applications from the AWS ...
NEW LAUNCH! Delivering Powerful Graphics-Intensive Applications from the AWS ...
 
Building a Real Time Dashboard with Amazon Kinesis, Amazon Lambda and Amazon ...
Building a Real Time Dashboard with Amazon Kinesis, Amazon Lambda and Amazon ...Building a Real Time Dashboard with Amazon Kinesis, Amazon Lambda and Amazon ...
Building a Real Time Dashboard with Amazon Kinesis, Amazon Lambda and Amazon ...
 
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
 
AWS re:Invent 2016: Configuration Management in the Cloud (DEV305)
AWS re:Invent 2016: Configuration Management in the Cloud (DEV305)AWS re:Invent 2016: Configuration Management in the Cloud (DEV305)
AWS re:Invent 2016: Configuration Management in the Cloud (DEV305)
 
(ARC305) How J&J Manages AWS At Scale For Enterprise Workloads
(ARC305) How J&J Manages AWS At Scale For Enterprise Workloads(ARC305) How J&J Manages AWS At Scale For Enterprise Workloads
(ARC305) How J&J Manages AWS At Scale For Enterprise Workloads
 

Similar to AWS re:Invent 2016: Real-time Data Processing Using AWS Lambda (SVR301)

Real-time Data Processing Using AWS Lambda
Real-time Data Processing Using AWS LambdaReal-time Data Processing Using AWS Lambda
Real-time Data Processing Using AWS Lambda
Amazon Web Services
 
Real-time Data Processing Using AWS Lambda
Real-time Data Processing Using AWS LambdaReal-time Data Processing Using AWS Lambda
Real-time Data Processing Using AWS Lambda
Amazon Web Services
 
Real-time Data Processing Using AWS Lambda
Real-time Data Processing Using AWS LambdaReal-time Data Processing Using AWS Lambda
Real-time Data Processing Using AWS Lambda
Amazon Web Services
 
Real Time Data Processing Using AWS Lambda - DevDay Los Angeles 2017
Real Time Data Processing Using AWS Lambda - DevDay Los Angeles 2017Real Time Data Processing Using AWS Lambda - DevDay Los Angeles 2017
Real Time Data Processing Using AWS Lambda - DevDay Los Angeles 2017
Amazon Web Services
 
Real Time Data Processing Using AWS Lambda - DevDay Austin 2017
Real Time Data Processing Using AWS Lambda - DevDay Austin 2017Real Time Data Processing Using AWS Lambda - DevDay Austin 2017
Real Time Data Processing Using AWS Lambda - DevDay Austin 2017
Amazon Web Services
 
Building Big Data Applications with Serverless Architectures - June 2017 AWS...
Building Big Data Applications with Serverless Architectures -  June 2017 AWS...Building Big Data Applications with Serverless Architectures -  June 2017 AWS...
Building Big Data Applications with Serverless Architectures - June 2017 AWS...
Amazon Web Services
 
Raleigh DevDay 2017: Real time data processing using AWS Lambda
Raleigh DevDay 2017: Real time data processing using AWS LambdaRaleigh DevDay 2017: Real time data processing using AWS Lambda
Raleigh DevDay 2017: Real time data processing using AWS Lambda
Amazon Web Services
 
AWS May Webinar Series - Streaming Data Processing with Amazon Kinesis and AW...
AWS May Webinar Series - Streaming Data Processing with Amazon Kinesis and AW...AWS May Webinar Series - Streaming Data Processing with Amazon Kinesis and AW...
AWS May Webinar Series - Streaming Data Processing with Amazon Kinesis and AW...
Amazon Web Services
 
SMC303 Real-time Data Processing Using AWS Lambda
SMC303 Real-time Data Processing Using AWS LambdaSMC303 Real-time Data Processing Using AWS Lambda
SMC303 Real-time Data Processing Using AWS Lambda
Amazon Web Services
 
Real Time Data Processing Using AWS Lambda
Real Time Data Processing Using AWS LambdaReal Time Data Processing Using AWS Lambda
Real Time Data Processing Using AWS Lambda
Amazon Web Services
 
AWS Lambda Supports Parallelization Factor for Kinesis and DynamoDB Event Sou...
AWS Lambda Supports Parallelization Factor for Kinesis and DynamoDB Event Sou...AWS Lambda Supports Parallelization Factor for Kinesis and DynamoDB Event Sou...
AWS Lambda Supports Parallelization Factor for Kinesis and DynamoDB Event Sou...
Swapnil Pawar
 
Real-Time Event Processing
Real-Time Event ProcessingReal-Time Event Processing
Real-Time Event Processing
Amazon Web Services
 
Deep Dive and Best Practices for Real Time Streaming Applications
Deep Dive and Best Practices for Real Time Streaming ApplicationsDeep Dive and Best Practices for Real Time Streaming Applications
Deep Dive and Best Practices for Real Time Streaming Applications
Amazon Web Services
 
Real-time Data Processing with Amazon DynamoDB Streams and AWS Lambda
Real-time Data Processing with Amazon DynamoDB Streams and AWS LambdaReal-time Data Processing with Amazon DynamoDB Streams and AWS Lambda
Real-time Data Processing with Amazon DynamoDB Streams and AWS Lambda
Amazon Web Services
 
Deep dive and best practices on real time streaming applications nyc-loft_oct...
Deep dive and best practices on real time streaming applications nyc-loft_oct...Deep dive and best practices on real time streaming applications nyc-loft_oct...
Deep dive and best practices on real time streaming applications nyc-loft_oct...
Amazon Web Services
 
Serverless Architecture Patterns
Serverless Architecture PatternsServerless Architecture Patterns
Serverless Architecture Patterns
Amazon Web Services
 
AWS April 2016 Webinar Series - Getting Started with Real-Time Data Analytics...
AWS April 2016 Webinar Series - Getting Started with Real-Time Data Analytics...AWS April 2016 Webinar Series - Getting Started with Real-Time Data Analytics...
AWS April 2016 Webinar Series - Getting Started with Real-Time Data Analytics...
Amazon Web Services
 
Serverless Architectural Patterns and Best Practices | AWS
Serverless Architectural Patterns and Best Practices | AWSServerless Architectural Patterns and Best Practices | AWS
Serverless Architectural Patterns and Best Practices | AWS
AWS Germany
 
Serverless Architectural Patterns and Best Practices
Serverless Architectural Patterns and Best PracticesServerless Architectural Patterns and Best Practices
Serverless Architectural Patterns and Best Practices
Amazon Web Services
 
AWS re:Invent 2016: [JK REPEAT] Serverless Architectural Patterns and Best Pr...
AWS re:Invent 2016: [JK REPEAT] Serverless Architectural Patterns and Best Pr...AWS re:Invent 2016: [JK REPEAT] Serverless Architectural Patterns and Best Pr...
AWS re:Invent 2016: [JK REPEAT] Serverless Architectural Patterns and Best Pr...
Amazon Web Services
 

Similar to AWS re:Invent 2016: Real-time Data Processing Using AWS Lambda (SVR301) (20)

Real-time Data Processing Using AWS Lambda
Real-time Data Processing Using AWS LambdaReal-time Data Processing Using AWS Lambda
Real-time Data Processing Using AWS Lambda
 
Real-time Data Processing Using AWS Lambda
Real-time Data Processing Using AWS LambdaReal-time Data Processing Using AWS Lambda
Real-time Data Processing Using AWS Lambda
 
Real-time Data Processing Using AWS Lambda
Real-time Data Processing Using AWS LambdaReal-time Data Processing Using AWS Lambda
Real-time Data Processing Using AWS Lambda
 
Real Time Data Processing Using AWS Lambda - DevDay Los Angeles 2017
Real Time Data Processing Using AWS Lambda - DevDay Los Angeles 2017Real Time Data Processing Using AWS Lambda - DevDay Los Angeles 2017
Real Time Data Processing Using AWS Lambda - DevDay Los Angeles 2017
 
Real Time Data Processing Using AWS Lambda - DevDay Austin 2017
Real Time Data Processing Using AWS Lambda - DevDay Austin 2017Real Time Data Processing Using AWS Lambda - DevDay Austin 2017
Real Time Data Processing Using AWS Lambda - DevDay Austin 2017
 
Building Big Data Applications with Serverless Architectures - June 2017 AWS...
Building Big Data Applications with Serverless Architectures -  June 2017 AWS...Building Big Data Applications with Serverless Architectures -  June 2017 AWS...
Building Big Data Applications with Serverless Architectures - June 2017 AWS...
 
Raleigh DevDay 2017: Real time data processing using AWS Lambda
Raleigh DevDay 2017: Real time data processing using AWS LambdaRaleigh DevDay 2017: Real time data processing using AWS Lambda
Raleigh DevDay 2017: Real time data processing using AWS Lambda
 
AWS May Webinar Series - Streaming Data Processing with Amazon Kinesis and AW...
AWS May Webinar Series - Streaming Data Processing with Amazon Kinesis and AW...AWS May Webinar Series - Streaming Data Processing with Amazon Kinesis and AW...
AWS May Webinar Series - Streaming Data Processing with Amazon Kinesis and AW...
 
SMC303 Real-time Data Processing Using AWS Lambda
SMC303 Real-time Data Processing Using AWS LambdaSMC303 Real-time Data Processing Using AWS Lambda
SMC303 Real-time Data Processing Using AWS Lambda
 
Real Time Data Processing Using AWS Lambda
Real Time Data Processing Using AWS LambdaReal Time Data Processing Using AWS Lambda
Real Time Data Processing Using AWS Lambda
 
AWS Lambda Supports Parallelization Factor for Kinesis and DynamoDB Event Sou...
AWS Lambda Supports Parallelization Factor for Kinesis and DynamoDB Event Sou...AWS Lambda Supports Parallelization Factor for Kinesis and DynamoDB Event Sou...
AWS Lambda Supports Parallelization Factor for Kinesis and DynamoDB Event Sou...
 
Real-Time Event Processing
Real-Time Event ProcessingReal-Time Event Processing
Real-Time Event Processing
 
Deep Dive and Best Practices for Real Time Streaming Applications
Deep Dive and Best Practices for Real Time Streaming ApplicationsDeep Dive and Best Practices for Real Time Streaming Applications
Deep Dive and Best Practices for Real Time Streaming Applications
 
Real-time Data Processing with Amazon DynamoDB Streams and AWS Lambda
Real-time Data Processing with Amazon DynamoDB Streams and AWS LambdaReal-time Data Processing with Amazon DynamoDB Streams and AWS Lambda
Real-time Data Processing with Amazon DynamoDB Streams and AWS Lambda
 
Deep dive and best practices on real time streaming applications nyc-loft_oct...
Deep dive and best practices on real time streaming applications nyc-loft_oct...Deep dive and best practices on real time streaming applications nyc-loft_oct...
Deep dive and best practices on real time streaming applications nyc-loft_oct...
 
Serverless Architecture Patterns
Serverless Architecture PatternsServerless Architecture Patterns
Serverless Architecture Patterns
 
AWS April 2016 Webinar Series - Getting Started with Real-Time Data Analytics...
AWS April 2016 Webinar Series - Getting Started with Real-Time Data Analytics...AWS April 2016 Webinar Series - Getting Started with Real-Time Data Analytics...
AWS April 2016 Webinar Series - Getting Started with Real-Time Data Analytics...
 
Serverless Architectural Patterns and Best Practices | AWS
Serverless Architectural Patterns and Best Practices | AWSServerless Architectural Patterns and Best Practices | AWS
Serverless Architectural Patterns and Best Practices | AWS
 
Serverless Architectural Patterns and Best Practices
Serverless Architectural Patterns and Best PracticesServerless Architectural Patterns and Best Practices
Serverless Architectural Patterns and Best Practices
 
AWS re:Invent 2016: [JK REPEAT] Serverless Architectural Patterns and Best Pr...
AWS re:Invent 2016: [JK REPEAT] Serverless Architectural Patterns and Best Pr...AWS re:Invent 2016: [JK REPEAT] Serverless Architectural Patterns and Best Pr...
AWS re:Invent 2016: [JK REPEAT] Serverless Architectural Patterns and Best Pr...
 

More from Amazon Web Services

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Amazon Web Services
 
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Amazon Web Services
 
Esegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateEsegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS Fargate
Amazon Web Services
 
Costruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSCostruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWS
Amazon Web Services
 
Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot
Amazon Web Services
 
Open banking as a service
Open banking as a serviceOpen banking as a service
Open banking as a service
Amazon Web Services
 
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Amazon Web Services
 
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
Amazon Web Services
 
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsMicrosoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Amazon Web Services
 
Computer Vision con AWS
Computer Vision con AWSComputer Vision con AWS
Computer Vision con AWS
Amazon Web Services
 
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareDatabase Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatare
Amazon Web Services
 
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSCrea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Amazon Web Services
 
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAPI moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e web
Amazon Web Services
 
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareDatabase Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Amazon Web Services
 
Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWS
Amazon Web Services
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch Deck
Amazon Web Services
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without servers
Amazon Web Services
 
Fundraising Essentials
Fundraising EssentialsFundraising Essentials
Fundraising Essentials
Amazon Web Services
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
Amazon Web Services
 
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceIntroduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container Service
Amazon Web Services
 

More from Amazon Web Services (20)

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
 
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
 
Esegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateEsegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS Fargate
 
Costruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSCostruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWS
 
Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot
 
Open banking as a service
Open banking as a serviceOpen banking as a service
Open banking as a service
 
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
 
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
 
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsMicrosoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
 
Computer Vision con AWS
Computer Vision con AWSComputer Vision con AWS
Computer Vision con AWS
 
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareDatabase Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatare
 
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSCrea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
 
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAPI moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e web
 
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareDatabase Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
 
Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWS
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch Deck
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without servers
 
Fundraising Essentials
Fundraising EssentialsFundraising Essentials
Fundraising Essentials
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
 
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceIntroduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container Service
 

Recently uploaded

Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Hiroshi SHIBATA
 
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptxThe Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
operationspcvita
 
Apps Break Data
Apps Break DataApps Break Data
Apps Break Data
Ivo Velitchkov
 
Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)
Jakub Marek
 
From Natural Language to Structured Solr Queries using LLMs
From Natural Language to Structured Solr Queries using LLMsFrom Natural Language to Structured Solr Queries using LLMs
From Natural Language to Structured Solr Queries using LLMs
Sease
 
Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
c5vrf27qcz
 
Demystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through StorytellingDemystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through Storytelling
Enterprise Knowledge
 
Christine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptxChristine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptx
christinelarrosa
 
Must Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during MigrationMust Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during Migration
Mydbops
 
Christine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptxChristine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptx
christinelarrosa
 
Mutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented ChatbotsMutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented Chatbots
Pablo Gómez Abajo
 
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-EfficiencyFreshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
ScyllaDB
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
Fwdays
 
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and StandardsLeveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
Neo4j
 
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and BioinformaticiansBiomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Neo4j
 
What is an RPA CoE? Session 1 – CoE Vision
What is an RPA CoE?  Session 1 – CoE VisionWhat is an RPA CoE?  Session 1 – CoE Vision
What is an RPA CoE? Session 1 – CoE Vision
DianaGray10
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
Miro Wengner
 
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectorsConnector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
DianaGray10
 
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin..."$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
Fwdays
 
Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation ParametersEssentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
Safe Software
 

Recently uploaded (20)

Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
 
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptxThe Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
 
Apps Break Data
Apps Break DataApps Break Data
Apps Break Data
 
Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)
 
From Natural Language to Structured Solr Queries using LLMs
From Natural Language to Structured Solr Queries using LLMsFrom Natural Language to Structured Solr Queries using LLMs
From Natural Language to Structured Solr Queries using LLMs
 
Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
 
Demystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through StorytellingDemystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through Storytelling
 
Christine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptxChristine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptx
 
Must Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during MigrationMust Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during Migration
 
Christine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptxChristine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptx
 
Mutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented ChatbotsMutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented Chatbots
 
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-EfficiencyFreshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
 
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and StandardsLeveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
 
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and BioinformaticiansBiomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
 
What is an RPA CoE? Session 1 – CoE Vision
What is an RPA CoE?  Session 1 – CoE VisionWhat is an RPA CoE?  Session 1 – CoE Vision
What is an RPA CoE? Session 1 – CoE Vision
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
 
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectorsConnector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
 
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin..."$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
 
Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation ParametersEssentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
 

AWS re:Invent 2016: Real-time Data Processing Using AWS Lambda (SVR301)

  • 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cecilia Deng Software Developer AWS Lambda 12/01/2016 SVR301 Real-Time Processing Using AWS Lambda Anders Fritz Senior Manager ThomsonReuters Marco Pierleoni Lead Software Developer Thomson Reuters
  • 2. What to Expect from the Session • What kinds of real time events can trigger lambda? • How does Lambda pull and process streams? • What are some stream processing behaviors? • Hear how Thomson Reuters went real time with AWS Lambda
  • 3. Flavors of real time event sources Asynchronous Invoke Push Event Source Synchronous Invoke Push Event Source Stream Pull Event Source S3 async invoke Alexa skill sync invoke Pull then sync invoke DynamoDB Update Stream
  • 5. Real-time push Who? • Any integrator that uses AWS Lambda invoke API • E.g., Amazon S3, Amazon SNS, Amazon Alexa, AWS IoT What? • Event sources sending events to Lambda for processing How? • Real-time triggered events owned by event source • Real-time processing owned by Lambda invoke methods
  • 6. Real-time push Synchronous Invoke Push Event Source Asynchronous Invoke Push Event Source
  • 8. Real-time pull Who? • Amazon Kinesis and DynamoDB update streams What? • Lambda grabbing events from a stream for processing How? • Mapping maintained by Lambda • Real-time triggered events owned by DDB or Kinesis producer • Real-time processing owned by Lambda stream polling component and invoke methods
  • 11. Processing streams: Kinesis setup Streams ▪ Made up of shards ▪ Each shard supports writes up to 1 MB/s ▪ Each shard supports reads up to 2 MB/s ▪ Each shard supports 5 reads/s Data ▪ All data is stored and replayable for 24 hours by default ▪ Make sure partition key distribution is even to optimize parallel throughput ▪ Pick a key with more groups than shards
  • 12. Processing streams: Lambda setup Memory ▪ CPU is proportional to the memory configured ▪ More memory means faster execution, if CPU bound ▪ More memory means larger sized record batches can be processed Timeout • Increasing timeout allows for longer functions, but more wait in case of errors Permission model • The execution role defined for Lambda must have permission to access the stream
  • 13. Processing streams: event source setup Batch size ▪ Max number of records that Lambda will send in one invocation ▪ Not equivalent to how many records Lambda gets from Kinesis ▪ Effective batch size is MIN(records available, batch size, 6 MB) ▪ Increasing batch size allows fewer Lambda function invocations with more data processed per function
  • 14. Processing streams: event source setup Starting Position: ▪ The position in the stream where Lambda starts reading ▪ Set to “Trim Horizon” for reading from start of stream (all data) ▪ Set to “Latest” for reading most recent data (LIFO) (latest data)
  • 15. Processing streams: event source setup Amazon Kinesis 1 AWS Lambda 1 Amazon CloudWatch Amazon DynamoDB AWS Lambda 2 Amazon S3 • Multiple functions can be mapped to one stream • Multiple streams can be mapped to one Lambda function • Each mapping is a unique key pair Kinesis stream to Lambda function • Each mapping has unique shard iterators Amazon Kinesis 2
  • 16. Processing streams: under the hood Event received by Lambda function is a collection of records from the stream { "Records": [ { "kinesis": { "partitionKey": "partitionKey-3", "kinesisSchemaVersion": "1.0", "data": "SGVsbG8sIHRoaXMgaXMgYSB0ZXN0IDEyMy4=", "sequenceNumber": "49545115243490985018280067714973144582180062593244200961" }, "eventSource": "aws:kinesis", "eventID": "shardId- 000000000000:49545115243490985018280067714973144582180062593244200961", "invokeIdentityArn": "arn:aws:iam::account-id:role/testLEBRole", "eventVersion": "1.0", "eventName": "aws:kinesis:record", "eventSourceARN": "arn:aws:kinesis:us-west-2:35667example:stream/examplestream", "awsRegion": "us-west-2" } ] }
  • 17. Processing streams: under the hood Polling ▪ Concurrent polling and processing per shard ▪ Polls every 250 ms if no records found ▪ Grab as much as possible in one GetRecords call Batching ▪ Sub batch in memory for invocation payload Synchronous invocation ▪ Batches invoked as synchronous RequestResponse type ▪ Lambda honors Kinesis at least once semantics ▪ Each shard blocks on in order synchronous invocation
  • 18. Processing streams: under the hood Per Shard: ▪ Lambda calls GetRecords with max limit from Kinesis (10 k or 10 MB) ▪ If no record, wait 250 ms ▪ From in memory, sub batches and formats records into Lambda payload ▪ Invoke Lambda with synchronous invoke … … Source Kinesis Lambda Polling Logic Shards Lambda will scale automaticallyScale Kinesis by adding shards Batch sync invokesPolls
  • 19. Processing streams: how it works ▪ Lambda blocks on ordered processing for each individual shard ▪ Increasing # of shards with even distribution allows increased concurrency ▪ Batch size may impact duration if the Lambda function takes longer to process more records … … Source Kinesis Lambda Polling Logic Shards Lambda will scale automaticallyScale Kinesis by adding shards Batch sync invokesPolls
  • 20. Processing streams: under the hood ▪ Retry execution failures until the record is expired ▪ Retry with exponential backoff up to 60 s ▪ Throttles and errors impacts duration and directly impacts throughput Kinesis … Source Scale Kinesis by adding shards Lambda Polling Logic Lambda will scale automatically Polls invoke fail invoke fail invoke success Batch sync invokes
  • 21. Processing streams: under the hood ▪ Maximum theoretical throughput: # shards * 2 MB / (s) ▪ Effective theoretical throughput: ( # shards * batch size (MB) ) / ( function duration (s) * retries until expiry) ▪ If put / ingestion rate is greater than the theoretical throughput, consider increasing number of shards of optimizing function duration to increase throughput
  • 22. Processing streams: how it looks •GetRecords (effective throughput): bytes, latency, records, etc. •PutRecord: bytes, latency, records, etc. •GetRecords.IteratorAgeMilliseconds: how old your last processed records were. If high, processing is falling behind. If close to 24 hours, records are close to being dropped.
  • 23. Processing streams: how it looks Amazon CloudWatch Metrics • Invocation count • Duration • Error count • Throttle count Amazon CloudWatch Logs • All Metrics • Custom logs • RAM consumed
  • 24. Processing streams: how it looks Common observations: ▪ Effective batch size may be less than configured during low throughput ▪ Effective batch size will increase during higher throughput ▪ Increased Lambda duration -> decreased # of invokes and GetRecord calls ▪ Too many consumers of your stream may compete with Kinesis read limits and induce ReadProvisionedThroughputExceeded errors and metrics
  • 25. ANALYSING USAGE OF THOMSON REUTERS PRODUCTS WITH AWS Anders Fritz & Marco Pierleoni
  • 26. CHALLENGE To identify and define a solution for usage analytics tracking that enables product teams to take ownership of the usage data collected. In addition to tracking and visualizing usage data it had to; 1. Cross reference Usage with Business data 4. Require Limited Maintenance. 3. Auto Scale as data flow fluctuates. 2. Follow TR Security & Compliance rules. 5. Launch in 5 months.
  • 34. • Product Insight is live – adoption rate high. • Tested 4,000 requests per second while targeting 5bn requests / month. • Since March – very little maintenance required • No Outages • No Downtime • Cloudwatch monitor everything. • Latency – Data visible on chart within 10 seconds • BrExit and US elections tested autoscaling. • US elections ~16m events – normally ~ 6-8m events / day. • UK EU referendum (BrExit) ~ 10m events – normally ~ 5m events / day OUTCOME
  • 35. EVENTS CAPTURED UK EU Referendum June 23rd (BrExit) time #events
  • 36. EVENTS CAPTURED US Elections November 8th time #events