This document provides an overview of real-time data processing using AWS Lambda and Amazon Kinesis. It discusses serverless real-time data processing, processing real-time streaming data with Lambda and Kinesis, setting up a Kinesis stream and Lambda functions, tuning throughput, and an example Kinesis/Lambda pipeline. It also discusses distributed computing with Lambda using a map-reduce model and a case study of Fannie Mae's use of serverless high performance computing with Lambda.
The document discusses how AWS services can help organizations increase speed and agility. It provides an overview of AWS services for compute, storage, databases, analytics and more. It also discusses how AWS enables continuous delivery and automation through services like CodeDeploy, CodePipeline, CloudFormation and Elastic Beanstalk. The document argues that AWS allows organizations to provision resources on demand, pay as they go, and build infrastructure as code.
AWS October Webinar Series - AWS Lambda Best Practices: Python, Scheduled Job...Amazon Web Services
AWS Lambda lets you run code without provisioning or managing servers. We have introduced a few new features this year at re:Invent and would like to share with you some of the best practices.
This webinar will introduce you to scheduled AWS Lambda functions and how to use long running functions to handle large volume data ingestion and processing jobs. We will demonstrate how to use versioning to control which Lambda function version is being executed in your development, testing, and production environments. We will also show you how to run your Python code in AWS Lambda.
AWS Lambda from the trenches (Serverless London)Yan Cui
AWS Lambda has changed the way we deploy and run software, but this new serverless paradigm has created new challenges to old problems - how do you test a cloud-hosted function locally? How do you monitor them? What about logging and config management? And how do we start migrating from existing architectures?
In this talk Yan will discuss solutions to these challenges by drawing from real-world experience running Lambda in production and migrating from an existing monolithic architecture.
AWS Lambda is a compute service that runs your code in response to events and automatically manages the compute resources for you, making it easy to build applications that respond quickly to new information. AWS
Lambda starts running your code within milliseconds of an event such as an image upload, in-app activity, website click, or output from a connected device. In this Live-Demo we will show you what Lambda is and which capabilities it provides, how to actually build a Lambda function that thumbnails an image and how to build the entire backend-logic of a funny little web-based game - with Lambda only. "
AWS Lambda is a new compute service that runs your code in response to events and automatically manages compute resources for you. In this session, you learn what you need to get started quickly, including a review of key features, a live demonstration, how to use AWS Lambda with Amazon S3 event notifications and Amazon DynamoDB streams, and tips on getting the most out of Lambda functions.
This document provides an overview of AWS Lambda and how it enables serverless application development. It discusses how AWS Lambda allows developers to run code without provisioning or managing servers, and how it can automatically scale in response to triggers from various AWS services. It also summarizes some key capabilities like bringing custom code, parallel execution, integration with other AWS services, and event-driven or pull-based invocation models.
The document discusses various ways to use the ZFS file system on Amazon EC2 instances. It describes ZFS as a file system and logical volume manager with features like data protection, high storage capacities, and snapshots. It provides instructions for installing and using ZFS on Linux, OmniOS, and FreeBSD AMIs, including attaching EBS volumes to an instance and creating ZFS pools and filesystems on them similar to using physical disks.
The document discusses how AWS services can help organizations increase speed and agility. It provides an overview of AWS services for compute, storage, databases, analytics and more. It also discusses how AWS enables continuous delivery and automation through services like CodeDeploy, CodePipeline, CloudFormation and Elastic Beanstalk. The document argues that AWS allows organizations to provision resources on demand, pay as they go, and build infrastructure as code.
AWS October Webinar Series - AWS Lambda Best Practices: Python, Scheduled Job...Amazon Web Services
AWS Lambda lets you run code without provisioning or managing servers. We have introduced a few new features this year at re:Invent and would like to share with you some of the best practices.
This webinar will introduce you to scheduled AWS Lambda functions and how to use long running functions to handle large volume data ingestion and processing jobs. We will demonstrate how to use versioning to control which Lambda function version is being executed in your development, testing, and production environments. We will also show you how to run your Python code in AWS Lambda.
AWS Lambda from the trenches (Serverless London)Yan Cui
AWS Lambda has changed the way we deploy and run software, but this new serverless paradigm has created new challenges to old problems - how do you test a cloud-hosted function locally? How do you monitor them? What about logging and config management? And how do we start migrating from existing architectures?
In this talk Yan will discuss solutions to these challenges by drawing from real-world experience running Lambda in production and migrating from an existing monolithic architecture.
AWS Lambda is a compute service that runs your code in response to events and automatically manages the compute resources for you, making it easy to build applications that respond quickly to new information. AWS
Lambda starts running your code within milliseconds of an event such as an image upload, in-app activity, website click, or output from a connected device. In this Live-Demo we will show you what Lambda is and which capabilities it provides, how to actually build a Lambda function that thumbnails an image and how to build the entire backend-logic of a funny little web-based game - with Lambda only. "
AWS Lambda is a new compute service that runs your code in response to events and automatically manages compute resources for you. In this session, you learn what you need to get started quickly, including a review of key features, a live demonstration, how to use AWS Lambda with Amazon S3 event notifications and Amazon DynamoDB streams, and tips on getting the most out of Lambda functions.
This document provides an overview of AWS Lambda and how it enables serverless application development. It discusses how AWS Lambda allows developers to run code without provisioning or managing servers, and how it can automatically scale in response to triggers from various AWS services. It also summarizes some key capabilities like bringing custom code, parallel execution, integration with other AWS services, and event-driven or pull-based invocation models.
The document discusses various ways to use the ZFS file system on Amazon EC2 instances. It describes ZFS as a file system and logical volume manager with features like data protection, high storage capacities, and snapshots. It provides instructions for installing and using ZFS on Linux, OmniOS, and FreeBSD AMIs, including attaching EBS volumes to an instance and creating ZFS pools and filesystems on them similar to using physical disks.
AWS Lambda is a new compute service that runs your code in response to events and automatically manages compute resources for you. In this session, you will learn what you need to get started quickly, including a review of key
features, a live demonstration, how to use AWS Lambda with Amazon S3 event notifications and Amazon DynamoDB streams, and tips on getting the most out of AWS Lambda functions.
Speakers:
Dean Bryen, AWS Solutions Architect and
Andrew Wheat, Senior Developer Media Services BBC
This presentation is from the AWS Lambda session of Container Days Conference in NYC. AWS Lambda is a new compute service that runs your code in response to events and automatically and dynamically manages infra resources for you. Tara will talk about AWS's event-driven compute strategy and explain how Lambda works to respond to events from various Amazon services.
Tara will describe what you need to easily build scalable microservices for mobile, web, and IoT applications that use AWS Lambda as a serverless back-end, how you can expose these services using Amazon API Gateway, and how to extend both AWS and third party services by triggering Lambda functions. She'll also cover the updated Lambda features announced at reInvent 2015, its programming model, and tips on getting the most out of Lambda.
AWS Lambda is a serverless compute platform that allows users to upload code and create functions that can be triggered by events from other AWS services like S3, DynamoDB, SNS, and Kinesis. Lambda handles provisioning and managing servers so users do not have to worry about infrastructure management. It provides a pay-per-use model where users are charged only for the compute time used to process events. The presentation provided examples of using Lambda for image thumbnailing from S3 uploads, sending notifications from DynamoDB updates, and processing streaming data from Kinesis.
Aws lambda and accesing AWS RDS - ClouddictiveClouddictive
Implement a Lambda function which integrates with RDS. How to implement this new function in Java using Spring Framework.
1) Setup RDS instance
2) Implement RequestHandler in java
4) Create lambda function
The document discusses serverless architectures using AWS Lambda and Amazon API Gateway. It provides background on moving from monolithic to microservices architectures. It then covers AWS Lambda functions, event sources, and networking environments. Amazon API Gateway is presented as a way to build multi-tier serverless applications. Common serverless architecture patterns and best practices for AWS Lambda, API Gateway, and general serverless development are outlined. The document concludes with a demonstration of a simple CRUD backend using Lambda and DynamoDB with API Gateway.
AWS Lambda and Serverless framework: lessons learned while building a serverl...Luciano Mammino
The document discusses lessons learned from building a serverless company. It introduces Planet 9 Energy and their use of AWS Lambda and the Serverless framework. Key topics covered include security, quality assurance, developer experience, costs, and lessons learned. Some challenges discussed are debugging, API Gateway custom domains, and Lambda limitations. The document emphasizes that serverless architectures provide infinite scalability at low cost but also have some limitations that require management.
AWS Lambda allows any Node.js app to be run at scale in a massively parallel environment with no up-front costs or planning. This session shows how to use Lambda to build dynamic analytic data flows that can be tuned as they execute, based on initial results, to provide real-time output streamed to web clients. This process enables a cost-effective and responsive user experience for ad hoc big data jobs and lets developers focus on how data is consumed and presented, instead of how it is obtained.
Aws Lambda Cart Microservice Server LessDhanu Gupta
This document describes an AWS serverless architecture for a cart microservice using AWS Lambda, API Gateway, and DynamoDB. It includes components like API Gateway for the REST API frontend, Lambda functions to run the application code, and DynamoDB for the database. It provides instructions on setting up the resources, mapping the API to Lambda, and deploying the API for testing. The goal is to build a serverless REST API for basic cart operations like read, create, delete that avoids managing servers and scales automatically.
Building Serverless Backends with AWS Lambda and Amazon API GatewayAmazon Web Services
AWS Lambda is a compute service that runs your code without provisioning or managing servers. Amazon API Gateway is a fully managed service that makes it easy for developers to publish, maintain, monitor, and secure APIs at any scale.
This session will familiarize you with the basics of AWS Lambda and Amazon API Gateway and demonstrate how to build web, mobile, and IoT backends using these services. You will learn how to setup API endpoints that trigger AWS Lambda functions to handle mobile, web, IoT, and 3rd party API requests. You will also learn how to use Lambda to read and write to Amazon DynamoDB. We will run through a demo of setting up a simple serverless blogging web application that allows user authentication and the ability to create posts and comments.
(CMP407) Lambda as Cron: Scheduling Invocations in AWS LambdaAmazon Web Services
Do you need to run an AWS Lambda function on a schedule, without an event to trigger the invocation? This session shows how to use an Amazon CloudWatch metric and CloudWatch alarms, Amazon SNS, and Lambda so that Lambda triggers itself every minute—no external services required! From here, other Lambda jobs can be scheduled in crontab-like format, giving minute-level resolution to your Lambda scheduled tasks. During the session, we build this functionality up from scratch with a Lambda function, CloudWatch metric and alarms, sample triggers, and tasks.
The AWS Lambda is now available in Singapore and we are excited to invite you to participate in a webinar to learn more about the service and ask questions live throughout the webinar and receive responses during the Q&A session. In this one hour session, you will get to understand key AWS Lambda features, learn the AWS Lambda programming model and get tips on getting the most out of AWS Lambda.
AWS Lambda is a new compute service that runs your code in response to events and automatically manages compute resources for you. In this webinar you’ll learn what you need to quickly begin building mobile, tablet, or IoT applications that use AWS Lambda as a serverless back-end. You’ll also hear about Amazon Web Service’s Event-Driven Compute strategy and see demonstrations that use Lambda to respond to events from Amazon S3 notifications and Amazon DynamoDB streams.
Slides for a short presentation I gave on AWS Lambda, which "lets you run code without provisioning or managing servers". Lambda is to running code as Amazon S3 is to storing objects.
Do you want to run your code without the cost and effort of provisioning and managing servers? Find out how in this deep dive session on AWS Lambda, which allows you to run code for virtually any type of application or back end service – all with zero administration. During the session, we’ll look at a number of key AWS Lambda features and benefits, including automated application scaling with high availability; pay-as-you-consume billing; and the ability to automatically trigger your code from other AWS services or from any web or mobile app.
This document provides tips and patterns for building serverless applications. It discusses how serverless architectures can simplify operations by removing the need to manage servers. It then demonstrates how to design a media sharing application using serverless technologies like AWS Lambda, Amazon S3, DynamoDB, and API Gateway, driven by events. The document shows how the application architecture can be refined from an initial feature-based view to use specific serverless services and functions.
1. The document discusses using AWS Lambda and Amazon Kinesis for real-time data processing in a serverless architecture. It describes how Lambda functions can be triggered by data ingestion in Kinesis streams to process streaming data without needing to manage servers.
2. Key benefits highlighted include automatic scaling of compute capacity, paying only for resources used, and focusing on business logic rather than infrastructure management. Best practices discussed include monitoring for errors/throttling and distributing load evenly across shards.
3. The demo portion shows how to set up a Kinesis stream, Lambda function, and configure the integration between the two for processing streaming data in real-time at scale in a serverless manner.
This document discusses serverless architectures and provides examples of building serverless applications. It introduces serverless computing and explains why developers want to adopt serverless and DevOps approaches. Examples are given for static and dynamic websites built with serverless technologies like AWS Lambda, API Gateway, and S3 storage. The document also provides a case study of building a serverless application with services like Cognito, DynamoDB, and Lambda. Alternatives to AWS serverless options are mentioned along with takeaways about paying only for resources used.
AWS Lambda is a compute service that runs your code in response to events and automatically manages the compute resources for you, making it easy to build applications that respond quickly to new information. AWS Lambda starts running your code within milliseconds of an event such as an image upload, in-app activity, website click, or output from a connected device.
Lambda and serverless - DevOps North East Jan 2017Mike Shutlar
Introduction to AWS Lambda, serverless architectures, & the new AWS Serverless Application Model.
Source code for demo serverless application available here:
https://github.com/infectedsoundsystem/lambda-refarch-webapp
10 Tips For Serverless Backends With NodeJS and AWS LambdaJim Lynch
The document provides 10 tips for building serverless backends with Node.js and AWS Lambda. It discusses how serverless architectures are cheaper and easier to manage than traditional servers. It then outlines each tip which includes how to create "Hello World" functions, pass data to Lambda functions, set up REST APIs, secure functions, send emails/texts, schedule functions, view logs, add additional Node.js libraries, and use Lambda with IoT devices.
Real Time Data Processing Using AWS Lambda - DevDay Austin 2017Amazon Web Services
1) The document discusses serverless real-time data processing using AWS Lambda and Amazon Kinesis.
2) It provides an example of how streaming data can be captured in an Amazon Kinesis stream and processed by AWS Lambda functions to output the results to databases or cloud services.
3) The document also discusses how Fannie Mae used a distributed computing approach with AWS Lambda to perform mortgage simulations, achieving a 3x performance increase over their existing process.
This document provides an overview and agenda for a presentation on real-time data processing using AWS Lambda. The presentation covers serverless real-time data processing concepts, processing streaming data with Lambda and Kinesis, a streaming data processing demo, a data processing pipeline with Lambda and MapReduce, and a big data processing solution demo. It also discusses a customer story of Fannie Mae using distributed computing with Lambda for financial modeling. Key topics include serverless processing of real-time streaming data, a map-reduce model for serverless distributed computing, benchmarks of serverless distributed computing, and Fannie Mae's journey migrating their high performance computing workloads to AWS Lambda.
AWS Lambda is a new compute service that runs your code in response to events and automatically manages compute resources for you. In this session, you will learn what you need to get started quickly, including a review of key
features, a live demonstration, how to use AWS Lambda with Amazon S3 event notifications and Amazon DynamoDB streams, and tips on getting the most out of AWS Lambda functions.
Speakers:
Dean Bryen, AWS Solutions Architect and
Andrew Wheat, Senior Developer Media Services BBC
This presentation is from the AWS Lambda session of Container Days Conference in NYC. AWS Lambda is a new compute service that runs your code in response to events and automatically and dynamically manages infra resources for you. Tara will talk about AWS's event-driven compute strategy and explain how Lambda works to respond to events from various Amazon services.
Tara will describe what you need to easily build scalable microservices for mobile, web, and IoT applications that use AWS Lambda as a serverless back-end, how you can expose these services using Amazon API Gateway, and how to extend both AWS and third party services by triggering Lambda functions. She'll also cover the updated Lambda features announced at reInvent 2015, its programming model, and tips on getting the most out of Lambda.
AWS Lambda is a serverless compute platform that allows users to upload code and create functions that can be triggered by events from other AWS services like S3, DynamoDB, SNS, and Kinesis. Lambda handles provisioning and managing servers so users do not have to worry about infrastructure management. It provides a pay-per-use model where users are charged only for the compute time used to process events. The presentation provided examples of using Lambda for image thumbnailing from S3 uploads, sending notifications from DynamoDB updates, and processing streaming data from Kinesis.
Aws lambda and accesing AWS RDS - ClouddictiveClouddictive
Implement a Lambda function which integrates with RDS. How to implement this new function in Java using Spring Framework.
1) Setup RDS instance
2) Implement RequestHandler in java
4) Create lambda function
The document discusses serverless architectures using AWS Lambda and Amazon API Gateway. It provides background on moving from monolithic to microservices architectures. It then covers AWS Lambda functions, event sources, and networking environments. Amazon API Gateway is presented as a way to build multi-tier serverless applications. Common serverless architecture patterns and best practices for AWS Lambda, API Gateway, and general serverless development are outlined. The document concludes with a demonstration of a simple CRUD backend using Lambda and DynamoDB with API Gateway.
AWS Lambda and Serverless framework: lessons learned while building a serverl...Luciano Mammino
The document discusses lessons learned from building a serverless company. It introduces Planet 9 Energy and their use of AWS Lambda and the Serverless framework. Key topics covered include security, quality assurance, developer experience, costs, and lessons learned. Some challenges discussed are debugging, API Gateway custom domains, and Lambda limitations. The document emphasizes that serverless architectures provide infinite scalability at low cost but also have some limitations that require management.
AWS Lambda allows any Node.js app to be run at scale in a massively parallel environment with no up-front costs or planning. This session shows how to use Lambda to build dynamic analytic data flows that can be tuned as they execute, based on initial results, to provide real-time output streamed to web clients. This process enables a cost-effective and responsive user experience for ad hoc big data jobs and lets developers focus on how data is consumed and presented, instead of how it is obtained.
Aws Lambda Cart Microservice Server LessDhanu Gupta
This document describes an AWS serverless architecture for a cart microservice using AWS Lambda, API Gateway, and DynamoDB. It includes components like API Gateway for the REST API frontend, Lambda functions to run the application code, and DynamoDB for the database. It provides instructions on setting up the resources, mapping the API to Lambda, and deploying the API for testing. The goal is to build a serverless REST API for basic cart operations like read, create, delete that avoids managing servers and scales automatically.
Building Serverless Backends with AWS Lambda and Amazon API GatewayAmazon Web Services
AWS Lambda is a compute service that runs your code without provisioning or managing servers. Amazon API Gateway is a fully managed service that makes it easy for developers to publish, maintain, monitor, and secure APIs at any scale.
This session will familiarize you with the basics of AWS Lambda and Amazon API Gateway and demonstrate how to build web, mobile, and IoT backends using these services. You will learn how to setup API endpoints that trigger AWS Lambda functions to handle mobile, web, IoT, and 3rd party API requests. You will also learn how to use Lambda to read and write to Amazon DynamoDB. We will run through a demo of setting up a simple serverless blogging web application that allows user authentication and the ability to create posts and comments.
(CMP407) Lambda as Cron: Scheduling Invocations in AWS LambdaAmazon Web Services
Do you need to run an AWS Lambda function on a schedule, without an event to trigger the invocation? This session shows how to use an Amazon CloudWatch metric and CloudWatch alarms, Amazon SNS, and Lambda so that Lambda triggers itself every minute—no external services required! From here, other Lambda jobs can be scheduled in crontab-like format, giving minute-level resolution to your Lambda scheduled tasks. During the session, we build this functionality up from scratch with a Lambda function, CloudWatch metric and alarms, sample triggers, and tasks.
The AWS Lambda is now available in Singapore and we are excited to invite you to participate in a webinar to learn more about the service and ask questions live throughout the webinar and receive responses during the Q&A session. In this one hour session, you will get to understand key AWS Lambda features, learn the AWS Lambda programming model and get tips on getting the most out of AWS Lambda.
AWS Lambda is a new compute service that runs your code in response to events and automatically manages compute resources for you. In this webinar you’ll learn what you need to quickly begin building mobile, tablet, or IoT applications that use AWS Lambda as a serverless back-end. You’ll also hear about Amazon Web Service’s Event-Driven Compute strategy and see demonstrations that use Lambda to respond to events from Amazon S3 notifications and Amazon DynamoDB streams.
Slides for a short presentation I gave on AWS Lambda, which "lets you run code without provisioning or managing servers". Lambda is to running code as Amazon S3 is to storing objects.
Do you want to run your code without the cost and effort of provisioning and managing servers? Find out how in this deep dive session on AWS Lambda, which allows you to run code for virtually any type of application or back end service – all with zero administration. During the session, we’ll look at a number of key AWS Lambda features and benefits, including automated application scaling with high availability; pay-as-you-consume billing; and the ability to automatically trigger your code from other AWS services or from any web or mobile app.
This document provides tips and patterns for building serverless applications. It discusses how serverless architectures can simplify operations by removing the need to manage servers. It then demonstrates how to design a media sharing application using serverless technologies like AWS Lambda, Amazon S3, DynamoDB, and API Gateway, driven by events. The document shows how the application architecture can be refined from an initial feature-based view to use specific serverless services and functions.
1. The document discusses using AWS Lambda and Amazon Kinesis for real-time data processing in a serverless architecture. It describes how Lambda functions can be triggered by data ingestion in Kinesis streams to process streaming data without needing to manage servers.
2. Key benefits highlighted include automatic scaling of compute capacity, paying only for resources used, and focusing on business logic rather than infrastructure management. Best practices discussed include monitoring for errors/throttling and distributing load evenly across shards.
3. The demo portion shows how to set up a Kinesis stream, Lambda function, and configure the integration between the two for processing streaming data in real-time at scale in a serverless manner.
This document discusses serverless architectures and provides examples of building serverless applications. It introduces serverless computing and explains why developers want to adopt serverless and DevOps approaches. Examples are given for static and dynamic websites built with serverless technologies like AWS Lambda, API Gateway, and S3 storage. The document also provides a case study of building a serverless application with services like Cognito, DynamoDB, and Lambda. Alternatives to AWS serverless options are mentioned along with takeaways about paying only for resources used.
AWS Lambda is a compute service that runs your code in response to events and automatically manages the compute resources for you, making it easy to build applications that respond quickly to new information. AWS Lambda starts running your code within milliseconds of an event such as an image upload, in-app activity, website click, or output from a connected device.
Lambda and serverless - DevOps North East Jan 2017Mike Shutlar
Introduction to AWS Lambda, serverless architectures, & the new AWS Serverless Application Model.
Source code for demo serverless application available here:
https://github.com/infectedsoundsystem/lambda-refarch-webapp
10 Tips For Serverless Backends With NodeJS and AWS LambdaJim Lynch
The document provides 10 tips for building serverless backends with Node.js and AWS Lambda. It discusses how serverless architectures are cheaper and easier to manage than traditional servers. It then outlines each tip which includes how to create "Hello World" functions, pass data to Lambda functions, set up REST APIs, secure functions, send emails/texts, schedule functions, view logs, add additional Node.js libraries, and use Lambda with IoT devices.
Real Time Data Processing Using AWS Lambda - DevDay Austin 2017Amazon Web Services
1) The document discusses serverless real-time data processing using AWS Lambda and Amazon Kinesis.
2) It provides an example of how streaming data can be captured in an Amazon Kinesis stream and processed by AWS Lambda functions to output the results to databases or cloud services.
3) The document also discusses how Fannie Mae used a distributed computing approach with AWS Lambda to perform mortgage simulations, achieving a 3x performance increase over their existing process.
This document provides an overview and agenda for a presentation on real-time data processing using AWS Lambda. The presentation covers serverless real-time data processing concepts, processing streaming data with Lambda and Kinesis, a streaming data processing demo, a data processing pipeline with Lambda and MapReduce, and a big data processing solution demo. It also discusses a customer story of Fannie Mae using distributed computing with Lambda for financial modeling. Key topics include serverless processing of real-time streaming data, a map-reduce model for serverless distributed computing, benchmarks of serverless distributed computing, and Fannie Mae's journey migrating their high performance computing workloads to AWS Lambda.
This document discusses serverless real-time data processing using AWS Lambda. It provides an overview of serverless real-time data processing and processing streaming data with Lambda and Amazon Kinesis. It also demonstrates streaming data processing and a data processing pipeline with Lambda and MapReduce. Finally, it discusses Fannie Mae's use of distributed computing with Lambda for financial modeling.
Building Big Data Applications with Serverless Architectures - June 2017 AWS...Amazon Web Services
Learning Objectives:
- Use cases and best practices for serverless big data applications
- Leverage AWS technologies such as AWS Lambda and Amazon Kinesis
- Learn to perform ETL, event processing, ad-hoc analysis, real-time processing, and MapReduce with serverless
Building data processing applications is challenging and time-consuming, and often requires specialized expertise to deploy and operate. With serverless computing, you can perform real-time stream processing of multiple data types without needing to spin up servers or install software, allowing you to deploy big data applications quickly and more easily. Come learn how you can use AWS Lambda with Amazon Kinesis to analyze streaming data in real-time and then store the results in a managed NoSQL database such as Amazon DynamoDB. You’ll learn tips and tricks for doing in-line processing, data manipulation, and even distributed MapReduce on large data sets.
Raleigh DevDay 2017: Real time data processing using AWS LambdaAmazon Web Services
This document provides an overview of serverless real-time data processing using AWS Lambda and Amazon Kinesis. It discusses how Lambda functions can be used to process streaming data from Kinesis in real-time. An example pipeline is shown where data is ingested into Kinesis and Lambda functions are triggered to process the data and output results. Distributed computing with Lambda is also briefly discussed.
Serverless architecture can eliminate the need to provision and manage servers required to process files or streaming data in real time.
In this session, we will cover the fundamentals of using AWS Lambda to process data from sources such as Amazon DynamoDB Streams, Amazon Kinesis, and Amazon S3. We will walk through sample use cases for real-time data processing and discuss best practices on using these services together. We will then demonstrate how to set up a real-time stream processing solution using just Amazon Kinesis and AWS Lambda, all without the need to run or manage servers.
Serverless architectures can eliminate the need to provision and manage servers required to process files or streaming data in real time. In this session, we will cover the fundamentals of using AWS Lambda to process data from sources such as Amazon DynamoDB Streams, Amazon Kinesis, and Amazon S3. We will walk through sample use cases for real-time data processing and discuss best practices on using these services together. We will then demonstrate run a live demonstration on how to set up a real-time stream processing solution using just Amazon Kinesis and AWS Lambda, all without the need to run or manage servers.
Learning Objectives:
• Learn the fundamentals of using AWS Lambda with various AWS data sources
• Understand best practices of using AWS Lambda with Amazon Kinesis
Who Should Attend:
• Developers
The document describes a serverless real-time data processing pipeline using AWS Lambda and Amazon Kinesis. Streaming data is captured in an Amazon Kinesis stream. Lambda functions are used to process the streaming data in real-time and aggregate metrics. The aggregated metrics are output to another Kinesis stream and stored in DynamoDB for persistence. SQL queries against streaming data in Kinesis Analytics are used to calculate distinct user counts and metrics grouped by time period.
AWS re:Invent 2016: Real-time Data Processing Using AWS Lambda (SVR301)Amazon Web Services
Serverless architecture can eliminate the need to provision and manage servers required to process files or streaming data in real time.
In this session, we will cover the fundamentals of using AWS Lambda to process data in real-time from push sources such as AWS Iot and pull sources such as Amazon DynamoDB Streams or Amazon Kinesis. We will walk through sample use cases and demonstrate how to set up some of these real-time data processing solutions. We'll also discuss best practices and do a deep dive into AWS Lambda real-time stream processing.
You also hear from speakers from Thomson Reuters, who discuss how the company leverages AWS for its Product Insight service. The service provides insights to collect usage analytics for Thomson Reuters products. The speakers walk through its architecture and demonstrate how they leverage Amazon Kinesis Streams, Amazon Kinesis Firehose, AWS Lambda, Amazon S3, Amazon Route 53, and AWS KMS for near real-time access to data being collected around the globe. They also outline how applying AWS methodologies benefited its business, such as time-to-market and cross-region ingestion, auto-scaling capabilities, low-latency, security features, and extensibility.
AWS May Webinar Series - Streaming Data Processing with Amazon Kinesis and AW...Amazon Web Services
If you are interested to know more about AWS Chicago Summit, please use the following to register: http://amzn.to/1RooPPL
Amazon Kinesis is a fully managed, cloud-based service for real-time data processing over large, distributed data streams. AWS Lambda is a compute service that runs your code in response to events and automatically manages the compute resources for you. AWS Lambda can run code in response to data in Amazon Kinesis streams, making it easy to build big data applications that respond quickly to new information. In this webinar, we will cover key Kinesis and Lambda features, walk through sample use cases for stream processing, and discuss best practices on using the services together. We'll then demonstrate setting up an Amazon Kinesis stream and an associated Lambda function to capture and perform custom computations on click-stream data, all without setting up any infrastructure.
Learning Objectives: • Understand key Amazon Kinesis and AWS Lambda features • Learn how to setup streaming data capture and processing framework using AWS Lambda • Learn sample use cases, best practices and tips on using AWS Lambda with Amazon Kinesis
Who Should Attend: • Developers, Devops Engineers, IT Operations Professionals
Lambda allows real-time event processing from various event sources like S3, DynamoDB streams, and Kinesis streams. Events can either be pushed to Lambda through asynchronous or synchronous invokes, or pulled from streams using Lambda's polling logic. Lambda processes streams efficiently by sub-batching records into invocations and processing shards concurrently with retries. Thomson Reuters used this to build a scalable usage analytics solution on AWS that processes millions of events daily from their products, automatically scales, and requires little maintenance.
Real-time data processing serverless architecture can eliminate the need to provision and manage servers required to process files or streaming data in real time. In this session, we will cover the fundamentals of using AWS Lambda to process data in real-time from push sources such as AWS Iot and pull sources such as Amazon DynamoDB Streams or Amazon Kinesis. We'll also discuss best practices and do a deep dive into AWS Lambda real-time stream processing.
This document discusses real-time data processing using Amazon Web Services. It describes how to use Amazon Kinesis for real-time data ingestion and processing and Amazon Elastic MapReduce (EMR) for batch processing. It provides examples of using EMR for batch processing large amounts of log data and for interactive querying of data stored in Amazon S3. It also discusses using Kinesis as a data broker to distribute streaming data to multiple applications and using Kinesis with EMR, Spark, and Storm for real-time analytics.
Deep Dive and Best Practices for Real Time Streaming ApplicationsAmazon Web Services
This document summarizes a presentation on real-time streaming data on AWS. It discusses Amazon Kinesis, Spark Streaming, AWS Lambda, and Amazon EMR. The presentation covers an overview of streaming vs batch processing, common streaming data use cases and design patterns, a deep dive on Amazon Kinesis, examples of ingesting and processing streaming data, and a case study of how Sizmek uses these services for their real-time analytics needs.
AWS April 2016 Webinar Series - Getting Started with Real-Time Data Analytics...Amazon Web Services
It is becoming increasingly important to analyze real time streaming data. It allows organizations to remain competitive by uncovering relevant, actionable insights. AWS makes it easy to capture, store, and analyze real-time streaming data.
In this webinar, we will guide you through some of the proven architectures for processing streaming data, using a combination of tools including Amazon Kinesis Streams, AWS Lambda, and Spark Streaming on Amazon Elastic MapReduce (EMR). We will then talk about common use cases and best practices for real-time data analysis on AWS.
Learning Objectives:
Understand how you can analyze real-time data streams using Amazon Kinesis, AWS Lambda, and Spark running on Amazon EMR
Learn use cases and best practices for streaming data applications on AWS
Deep dive and best practices on real time streaming applications nyc-loft_oct...Amazon Web Services
This document provides an overview of real-time streaming data on AWS and best practices for using Amazon Kinesis, Spark Streaming, AWS Lambda, and Amazon EMR. It discusses ingesting streaming data using Kinesis Streams and Firehose, processing data with Kinesis Client Library, Spark Streaming, and AWS Lambda, and integrating with data stores like S3, Redshift and Elasticsearch. Example use cases are also presented from companies like Sonos, publishers and gaming companies.
NASA LandSat data can be stored, transformed, navigated, and visualized. In this session we will explore how the LandSat dataset is stored in Amazon Simple Storage Service (S3), one of the recommended cloud storage services in AWS for storage of petabytes of data, and how data stored in S3 can be processed on the server with the Lambda service, visualized for users, and made available to search engines.
Create by: Ben Snively, Senior Solutions Architect
The world is producing an ever increasing volume, velocity, and variety of big data. Consumers and businesses are demanding up-to-the-second (or even millisecond) analytics on their fast-moving data, in addition to classic batch processing. AWS delivers many technologies for solving big data problems. But what services should you use, why, when, and how? In this session, we simplify big data processing as a data bus comprising various stages: ingest, store, process, and visualize. Next, we discuss how to choose the right technology in each stage based on criteria such as data structure, query latency, cost, request rate, item size, data volume, durability, and so on. Finally, we provide reference architecture, design patterns, and best practices for assembling these technologies to solve your big data problems at the right cost.
Presented by: Arie Leeuwesteijn, Principal Solutions Architect, Amazon Web Services
Customer Guest: Sander Kieft, Sanoma
- Amazon Kinesis Data Streams and Amazon Managed Streaming for Kafka (MSK) are services for stream storage and processing. Kinesis Data Streams uses shards that can scale out, while MSK uses Kafka brokers that require more manual scaling.
- Key metrics to monitor for stream processing include request/response queues, produce/consume rates, network traffic, and disk usage. Monitoring helps identify bottlenecks or imbalances.
- Common streaming architectures include using Kinesis/MSK as an event bus, log aggregation from IoT devices, event sourcing with CQRS, and real-time analytics with Kinesis Analytics. These patterns are useful for building real-time applications and analytics.
Similar to Real Time Data Processing Using AWS Lambda - DevDay Los Angeles 2017 (20)
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Amazon Web Services
Il Forecasting è un processo importante per tantissime aziende e viene utilizzato in vari ambiti per cercare di prevedere in modo accurato la crescita e distribuzione di un prodotto, l’utilizzo delle risorse necessarie nelle linee produttive, presentazioni finanziarie e tanto altro. Amazon utilizza delle tecniche avanzate di forecasting, in parte questi servizi sono stati messi a disposizione di tutti i clienti AWS.
In questa sessione illustreremo come pre-processare i dati che contengono una componente temporale e successivamente utilizzare un algoritmo che a partire dal tipo di dato analizzato produce un forecasting accurato.
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Amazon Web Services
La varietà e la quantità di dati che si crea ogni giorno accelera sempre più velocemente e rappresenta una opportunità irripetibile per innovare e creare nuove startup.
Tuttavia gestire grandi quantità di dati può apparire complesso: creare cluster Big Data su larga scala sembra essere un investimento accessibile solo ad aziende consolidate. Ma l’elasticità del Cloud e, in particolare, i servizi Serverless ci permettono di rompere questi limiti.
Vediamo quindi come è possibile sviluppare applicazioni Big Data rapidamente, senza preoccuparci dell’infrastruttura, ma dedicando tutte le risorse allo sviluppo delle nostre le nostre idee per creare prodotti innovativi.
Ora puoi utilizzare Amazon Elastic Kubernetes Service (EKS) per eseguire pod Kubernetes su AWS Fargate, il motore di elaborazione serverless creato per container su AWS. Questo rende più semplice che mai costruire ed eseguire le tue applicazioni Kubernetes nel cloud AWS.In questa sessione presenteremo le caratteristiche principali del servizio e come distribuire la tua applicazione in pochi passaggi
Vent'anni fa Amazon ha attraversato una trasformazione radicale con l'obiettivo di aumentare il ritmo dell'innovazione. In questo periodo abbiamo imparato come cambiare il nostro approccio allo sviluppo delle applicazioni ci ha permesso di aumentare notevolmente l'agilità, la velocità di rilascio e, in definitiva, ci ha consentito di creare applicazioni più affidabili e scalabili. In questa sessione illustreremo come definiamo le applicazioni moderne e come la creazione di app moderne influisce non solo sull'architettura dell'applicazione, ma sulla struttura organizzativa, sulle pipeline di rilascio dello sviluppo e persino sul modello operativo. Descriveremo anche approcci comuni alla modernizzazione, compreso l'approccio utilizzato dalla stessa Amazon.com.
Come spendere fino al 90% in meno con i container e le istanze spot Amazon Web Services
L’utilizzo dei container è in continua crescita.
Se correttamente disegnate, le applicazioni basate su Container sono molto spesso stateless e flessibili.
I servizi AWS ECS, EKS e Kubernetes su EC2 possono sfruttare le istanze Spot, portando ad un risparmio medio del 70% rispetto alle istanze On Demand. In questa sessione scopriremo insieme quali sono le caratteristiche delle istanze Spot e come possono essere utilizzate facilmente su AWS. Impareremo inoltre come Spreaker sfrutta le istanze spot per eseguire applicazioni di diverso tipo, in produzione, ad una frazione del costo on-demand!
In recent months, many customers have been asking us the question – how to monetise Open APIs, simplify Fintech integrations and accelerate adoption of various Open Banking business models. Therefore, AWS and FinConecta would like to invite you to Open Finance marketplace presentation on October 20th.
Event Agenda :
Open banking so far (short recap)
• PSD2, OB UK, OB Australia, OB LATAM, OB Israel
Intro to Open Finance marketplace
• Scope
• Features
• Tech overview and Demo
The role of the Cloud
The Future of APIs
• Complying with regulation
• Monetizing data / APIs
• Business models
• Time to market
One platform for all: a Strategic approach
Q&A
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Amazon Web Services
Per creare valore e costruire una propria offerta differenziante e riconoscibile, le startup di successo sanno come combinare tecnologie consolidate con componenti innovativi creati ad hoc.
AWS fornisce servizi pronti all'utilizzo e, allo stesso tempo, permette di personalizzare e creare gli elementi differenzianti della propria offerta.
Concentrandoci sulle tecnologie di Machine Learning, vedremo come selezionare i servizi di intelligenza artificiale offerti da AWS e, anche attraverso una demo, come costruire modelli di Machine Learning personalizzati utilizzando SageMaker Studio.
OpsWorks Configuration Management: automatizza la gestione e i deployment del...Amazon Web Services
Con l'approccio tradizionale al mondo IT per molti anni è stato difficile implementare tecniche di DevOps, che finora spesso hanno previsto attività manuali portando di tanto in tanto a dei downtime degli applicativi interrompendo l'operatività dell'utente. Con l'avvento del cloud, le tecniche di DevOps sono ormai a portata di tutti a basso costo per qualsiasi genere di workload, garantendo maggiore affidabilità del sistema e risultando in dei significativi miglioramenti della business continuity.
AWS mette a disposizione AWS OpsWork come strumento di Configuration Management che mira ad automatizzare e semplificare la gestione e i deployment delle istanze EC2 per mezzo di workload Chef e Puppet.
Scopri come sfruttare AWS OpsWork a garanzia e affidabilità del tuo applicativo installato su Instanze EC2.
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsAmazon Web Services
Vuoi conoscere le opzioni per eseguire Microsoft Active Directory su AWS? Quando si spostano carichi di lavoro Microsoft in AWS, è importante considerare come distribuire Microsoft Active Directory per supportare la gestione, l'autenticazione e l'autorizzazione dei criteri di gruppo. In questa sessione, discuteremo le opzioni per la distribuzione di Microsoft Active Directory su AWS, incluso AWS Directory Service per Microsoft Active Directory e la distribuzione di Active Directory su Windows su Amazon Elastic Compute Cloud (Amazon EC2). Trattiamo argomenti quali l'integrazione del tuo ambiente Microsoft Active Directory locale nel cloud e l'utilizzo di applicazioni SaaS, come Office 365, con AWS Single Sign-On.
Dal riconoscimento facciale al riconoscimento di frodi o difetti di fabbricazione, l'analisi di immagini e video che sfruttano tecniche di intelligenza artificiale, si stanno evolvendo e raffinando a ritmi elevati. In questo webinar esploreremo le possibilità messe a disposizione dai servizi AWS per applicare lo stato dell'arte delle tecniche di computer vision a scenari reali.
Amazon Web Services e VMware organizzano un evento virtuale gratuito il prossimo mercoledì 14 Ottobre dalle 12:00 alle 13:00 dedicato a VMware Cloud ™ on AWS, il servizio on demand che consente di eseguire applicazioni in ambienti cloud basati su VMware vSphere® e di accedere ad una vasta gamma di servizi AWS, sfruttando a pieno le potenzialità del cloud AWS e tutelando gli investimenti VMware esistenti.
Molte organizzazioni sfruttano i vantaggi del cloud migrando i propri carichi di lavoro Oracle e assicurandosi notevoli vantaggi in termini di agilità ed efficienza dei costi.
La migrazione di questi carichi di lavoro, può creare complessità durante la modernizzazione e il refactoring delle applicazioni e a questo si possono aggiungere rischi di prestazione che possono essere introdotti quando si spostano le applicazioni dai data center locali.
Crea la tua prima serverless ledger-based app con QLDB e NodeJSAmazon Web Services
Molte aziende oggi, costruiscono applicazioni con funzionalità di tipo ledger ad esempio per verificare lo storico di accrediti o addebiti nelle transazioni bancarie o ancora per tenere traccia del flusso supply chain dei propri prodotti.
Alla base di queste soluzioni ci sono i database ledger che permettono di avere un log delle transazioni trasparente, immutabile e crittograficamente verificabile, ma sono strumenti complessi e onerosi da gestire.
Amazon QLDB elimina la necessità di costruire sistemi personalizzati e complessi fornendo un database ledger serverless completamente gestito.
In questa sessione scopriremo come realizzare un'applicazione serverless completa che utilizzi le funzionalità di QLDB.
Con l’ascesa delle architetture di microservizi e delle ricche applicazioni mobili e Web, le API sono più importanti che mai per offrire agli utenti finali una user experience eccezionale. In questa sessione impareremo come affrontare le moderne sfide di progettazione delle API con GraphQL, un linguaggio di query API open source utilizzato da Facebook, Amazon e altro e come utilizzare AWS AppSync, un servizio GraphQL serverless gestito su AWS. Approfondiremo diversi scenari, comprendendo come AppSync può aiutare a risolvere questi casi d’uso creando API moderne con funzionalità di aggiornamento dati in tempo reale e offline.
Inoltre, impareremo come Sky Italia utilizza AWS AppSync per fornire aggiornamenti sportivi in tempo reale agli utenti del proprio portale web.
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareAmazon Web Services
Molte organizzazioni sfruttano i vantaggi del cloud migrando i propri carichi di lavoro Oracle e assicurandosi notevoli vantaggi in termini di agilità ed efficienza dei costi.
La migrazione di questi carichi di lavoro, può creare complessità durante la modernizzazione e il refactoring delle applicazioni e a questo si possono aggiungere rischi di prestazione che possono essere introdotti quando si spostano le applicazioni dai data center locali.
In queste slide, gli esperti AWS e VMware presentano semplici e pratici accorgimenti per facilitare e semplificare la migrazione dei carichi di lavoro Oracle accelerando la trasformazione verso il cloud, approfondiranno l’architettura e dimostreranno come sfruttare a pieno le potenzialità di VMware Cloud ™ on AWS.
1) The document discusses building a minimum viable product (MVP) using Amazon Web Services (AWS).
2) It provides an example of an MVP for an omni-channel messenger platform that was built from 2017 to connect ecommerce stores to customers via web chat, Facebook Messenger, WhatsApp, and other channels.
3) The founder discusses how they started with an MVP in 2017 with 200 ecommerce stores in Hong Kong and Taiwan, and have since expanded to over 5000 clients across Southeast Asia using AWS for scaling.
This document discusses pitch decks and fundraising materials. It explains that venture capitalists will typically spend only 3 minutes and 44 seconds reviewing a pitch deck. Therefore, the deck needs to tell a compelling story to grab their attention. It also provides tips on tailoring different types of decks for different purposes, such as creating a concise 1-2 page teaser, a presentation deck for pitching in-person, and a more detailed read-only or fundraising deck. The document stresses the importance of including key information like the problem, solution, product, traction, market size, plans, team, and ask.
This document discusses building serverless web applications using AWS services like API Gateway, Lambda, DynamoDB, S3 and Amplify. It provides an overview of each service and how they can work together to create a scalable, secure and cost-effective serverless application stack without having to manage servers or infrastructure. Key services covered include API Gateway for hosting APIs, Lambda for backend logic, DynamoDB for database needs, S3 for static content, and Amplify for frontend hosting and continuous deployment.
This document provides tips for fundraising from startup founders Roland Yau and Sze Lok Chan. It discusses generating competition to create urgency for investors, fundraising in parallel rather than sequentially, having a clear fundraising narrative focused on what you do and why it's compelling, and prioritizing relationships with people over firms. It also notes how the pandemic has changed fundraising, with examples of deals done virtually during this time. The tips emphasize being fully prepared before fundraising and cultivating connections with investors in advance.
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...Amazon Web Services
This document discusses Amazon's machine learning services for building conversational interfaces and extracting insights from unstructured text and audio. It describes Amazon Lex for creating chatbots, Amazon Comprehend for natural language processing tasks like entity extraction and sentiment analysis, and how they can be used together for applications like intelligent call centers and content analysis. Pre-trained APIs simplify adding machine learning to apps without requiring ML expertise.
Amazon Elastic Container Service (Amazon ECS) è un servizio di gestione dei container altamente scalabile, che semplifica la gestione dei contenitori Docker attraverso un layer di orchestrazione per il controllo del deployment e del relativo lifecycle. In questa sessione presenteremo le principali caratteristiche del servizio, le architetture di riferimento per i differenti carichi di lavoro e i semplici passi necessari per poter velocemente migrare uno o più dei tuo container.
5. Agenda
What’s Serverless Real-Time Data Processing?
Serverless Processing of Real-Time Streaming Data
Example Kinesis/Lambda Pipeline
Streaming Demo
Customer Story: Fannie Mae-Distributed Computing with Lambda
7. AWS Lambda
Bring your own code
• Node.js, Java, Python,
C#
• Bring your own libraries
(even native ones)
Simple resource model
• Select power rating from
128 MB to 1.5 GB
• CPU and network
allocated proportionately
Flexible use
• Synchronous or
asynchronous
• Integrated with other
AWS services
Flexible authorization
• Securely grant access to
resources and VPCs
• Fine-grained control for
invoking your functions
9. Amazon
DynamoDB
Amazon
Kinesis
Amazon
S3
Amazon
SNS
ASYNCHRONOUS PUSH MODEL
STREAM PULL MODEL
Lambda Real-Time Event Sources
Amazon
Alexa
AWS
IoT
SYNCHRONOUS PUSH MODEL
Mapping owned by Event Source
Mapping owned by Lambda
Invokes Lambda via Event Source API
Lambda function invokes when new
records found on stream
Resource-based policy permissions
Lambda Execution role policy permissions
Concurrent executions
Sync invocation
Async Invocation
Sync invocation
Lambda polls the streams
HOW IT WORKS
10. Serverless Real-Time Data Processing Is..
Capture Data
Streams
IoT Data
Financial
Data
Log Data
No servers to
provision or
manage
EVENT SOURCE
Node.js
Python
Java
C#
Process Data
Streams
FUNCTION
Clickstream
Data
Output
Data
DATABASE
CLOUD
SERVICES
12. Amazon Kinesis
Amazon Kinesis Offering: Managed services for streaming data
ingestion and processing.
• Amazon Kinesis Streams: Build applications that process or
analyze streaming data.
• Amazon Kinesis Firehose: Load massive volumes of
streaming data into Amazon S3, Amazon Redshift, and
Elasticsearch.
• Amazon Kinesis Analytics: Analyze data streams using SQL
queries.
Easy to use: Focus on quickly launching data streaming
applications instead of managing infrastructure.
Real-Time: Collect real-time data streams and promptly
respond to key business events and operational triggers.
Real-time latencies.
13. Processing Real-Time Streams: Lambda + Amazon Kinesis
Streaming data sent to Amazon
Kinesis and stored in shards
Multiple Lambda functions can be
triggered to process same Amazon
Kinesis stream for “fan out”
Lambda can process data and store
results ex. to DynamoDB, S3
Lambda can aggregate data to
services like Amazon Elasticsearch
Service for analytics
Lambda sends event data and
function info to Amazon CloudWatch
for capturing metrics and monitoring
Amazon
Kinesis
AWS
Lambda
Amazon
CloudWatch
Amazon
DynamoDB
AWS
Lambda
Amazon
Elasticsearch Service
Amazon
S3
14. Processing Streams: Set Up Amazon Kinesis Stream
Streams
Made up of Shards
Each Shard ingests/reads data up to 1 MB/sec
Each Shard emits/writes data up to 2 MB/sec
Each Shard supports 5 read transactions/sec
Data
All data is stored and is replayable for 24 hours (default)
Retention window can be configured up to 7 days
Partition key used to distribute PUTs across shards
Even partition key distribution optimizes throughput
Best Practice
Determine an initial size/shards to plan for expected maximum demand
ü Leverage “Help me decide how many shards I need” option in Console
ü Use formula for Number Of Shards:
max(incoming_write_bandwidth_in_KB/1000, outgoing_read_bandwidth_in_KB / 2000)
15. Processing Streams: Create Lambda functions
Memory
CPU allocation proportional to the memory configured
Increasing memory makes your code execute faster (if CPU bound)
Increasing memory allows for larger record sizes processed
Timeout
Increasing timeout allows for longer functions, but longer wait in case of errors
Retries
With Amazon Kinesis, Lambda retries until the data expires
(i.e. 24 hours)
Permission model
Execution role defined for Lambda must have permission to access the stream
Best Practice
Write Lambda function code to be stateless
Instantiate AWS clients & database clients outside the scope of the function handler to take
advantage of connection re-use.
16. Processing Streams: Configure Event Source
Amazon Kinesis mapped as event source in Lambda
Batch size
Max number of records that Lambda will send to one invocation
Not equivalent to effective batch size
Effective batch size is every 1 second – Calculated as:
MIN(records available, batch size, 6MB)
Increasing batch size allows fewer Lambda function invocations with more data
processed per function
Best Practices
Set to “Trim Horizon” for reading from start of
stream (all data)
Set to “Latest” for reading most recent data (LIFO) (latest data)
Set to “At timestamp” to pick up at a specific time
17. Processing streams: How It Works
Polling
Concurrent polling and processing per shard
Lambda polls every 1s if no records found
Will grab as much data as possible in one GetRecords call (Batch)
Batching
Batches are passed for invocation to Lambda through
function parameters
Batch size may impact duration if the Lambda function
takes longer to process more records
Sub batch in memory for invocation payload
Synchronous invocation
Batches invoked as synchronous RequestResponse type
Lambda honors Amazon Kinesis at least once semantics
Each shard blocks in order of synchronous invocation
18. Processing streams: Tuning throughput
If put / ingestion rate is greater than the theoretical throughput, your
processing is at risk of falling behind
Maximum theoretical throughput
# shards * 2MB / Lambda function duration (s)
Effective theoretical throughput
# shards * batch size (MB) / Lambda function duration (s)
… …
Source
Amazon Kinesis
Destination
1
Lambda
Destination
2
FunctionsShards
Lambda will scale automaticallyScale Amazon Kinesis by splitting or merging shards
Waits for responsePolls a batch
19. Processing streams: Tuning Throughput w/ Retries
Retries
Will retry on execution failures until the record is expired
Throttles and errors impacts duration and directly impacts throughput
Best Practice
Retry with exponential back-off of up to 60s
Effective theoretical throughput with retries
( # shards * batch size (MB) ) / ( function duration (s) * retries until expiry)
… …
Source
Amazon Kinesis
Destination
1
Lambda
Destination
2
FunctionsShards
Lambda will scale automatically
Receives errorPolls a batch
Receives error
Receives success
20. Processing streams: Common observations
Effective batch size may be less than configured during low throughput
Effective batch size will increase during higher throughput
Increased Lambda duration -> decreased # of invokes and GetRecords calls
Too many consumers of your stream may compete with Amazon Kinesis read
limits and induce ReadProvisionedThroughputExceeded errors and metrics
Amazon
Kinesis
AWS
Lambda
21. Processing streams: Monitoring with Cloudwatch
• GetRecords: (effective throughput)
• PutRecord: bytes, latency, records, etc
• GetRecords.IteratorAgeMilliseconds: how old your
last processed records were
Monitoring Amazon Kinesis Streams
Monitoring Lambda functions
• Invocation count: Time function invoked
• Duration: Execution/processing time
• Error count: Number of Errors
• Throttle count: Number of time function throttled
• Iterator Age: Time elapsed from batch received &
final record written to stream
• Review All Metrics
• Make Custom logs
• View RAM consumed
• Search for log events
Debugging
24. Kinesis/Lambda Demo
amzn.to/bigdata
Quiz: If I set my batch size to 100, each Lambda call…
A) Will get exactly 100 records
B) Will get 100 records or less
C) Will get an average of 100 records
D) Will get 95 ReadProvisionedThroughputExceeded
errors
25. Kinesis/Lambda Demo
amzn.to/bigdata
Quiz: If I set my batch size to 100, each Lambda call…
A) Will get exactly 100 records
B) Will get 100 records or less
C) Will get an average of 100 records
D) Will get 95 ReadProvisionedThroughputExceeded
errors
26. Kinesis/Lambda Demo
amzn.to/bigdata
I think this session…
A) Was really useful
B) Was a too technical
C) Was not deep enough
D) When is lunch?
E) This guy is totally confusing
30. The demo application
CREATE OR REPLACE STREAM DESTINATION_SQL_STREAM (UNIQUE_USER_COUNT INT, ANDROID_COUNT INT, IOS_COUNT INT, WINDOWS_PHONE_COUNT INT,
OTHER_OS_COUNT INT, QUADRANT_A_COUNT INT, QUADRANT_B_COUNT INT, QUADRANT_C_COUNT INT, QUADRANT_D_COUNT INT, WINDOW_TIME TIMESTAMP);
CREATE OR REPLACE STREAM DISTINCT_USER_STREAM (COGNITO_ID VARCHAR(64), DEVICE VARCHAR(32), OS VARCHAR(32), QUADRANT char(1), DT
TIMESTAMP);
CREATE OR REPLACE PUMP "DISTINCT_USER_PUMP" AS
INSERT INTO "DISTINCT_USER_STREAM"
SELECT STREAM DISTINCT
"cognitoId",
"device",
"os",
"quadrant",
FLOOR("SOURCE_SQL_STREAM_001".ROWTIME TO SECOND)
FROM "SOURCE_SQL_STREAM_001";
CREATE OR REPLACE PUMP "OUTPUT_PUMP" AS
INSERT INTO "DESTINATION_SQL_STREAM"
SELECT STREAM
COUNT("DISTINCT_USER_STREAM".COGNITO_ID) AS UNIQUE_USER_COUNT,
COUNT((CASE WHEN "DISTINCT_USER_STREAM".OS = 'Android' THEN COGNITO_ID ELSE null END)) AS ANDROID_COUNT,
COUNT((CASE WHEN "DISTINCT_USER_STREAM".OS = 'iOS' THEN COGNITO_ID ELSE null END)) AS IOS_COUNT,
COUNT((CASE WHEN "DISTINCT_USER_STREAM".OS = 'Windows Phone' THEN COGNITO_ID ELSE null END)) AS WINDOWS_PHONE_COUNT,
COUNT((CASE WHEN "DISTINCT_USER_STREAM".OS = 'other' THEN COGNITO_ID ELSE null END)) AS OTHER_OS_COUNT,
COUNT((CASE WHEN "DISTINCT_USER_STREAM".QUADRANT = 'A' THEN COGNITO_ID ELSE null END)) AS QUADRANT_A_COUNT,
COUNT((CASE WHEN "DISTINCT_USER_STREAM".QUADRANT = 'B' THEN COGNITO_ID ELSE null END)) AS QUADRANT_B_COUNT,
COUNT((CASE WHEN "DISTINCT_USER_STREAM".QUADRANT = 'C' THEN COGNITO_ID ELSE null END)) AS QUADRANT_C_COUNT,
COUNT((CASE WHEN "DISTINCT_USER_STREAM".QUADRANT = 'D' THEN COGNITO_ID ELSE null END)) AS QUADRANT_D_COUNT,
ROWTIME
FROM "DISTINCT_USER_STREAM"
GROUP BY
FLOOR("DISTINCT_USER_STREAM".ROWTIME TO SECOND);
32. Serverless Distributed Computing: Map-Reduce Model
Why Serverless Data Processing with Distributed
Computing?
Remove Difficult infrastructure management
ü Cluster administration
ü Complex configuration tools
Enable simple, elastic, user-friendly distributed data
processing
ü Eliminate complexity of state management
ü Bring Distributed Computing power to the masses
35. Fannie Mae’s Serverless HPC Performance
Lambda service configuration:
• Initial burst rate = 2,000, incremental rate = 100 per
minute, throttle limit = 15,000.
• Lambda ramps up automatically from 2,000 to 15,000
concurrent executions.
Application Result:
• One simulation run of ~ 20 million mortgages takes 2
hours, >3 times faster than the existing process.
• The performance does not degrade during the ramp up
period.
• Lambdas’ CPU efficiency is close to 100%. Actual
elapsed time is consistent with the estimated elapsed
time based on Lambda billing time.
Number of New
Lambda Invocations
every 5 Mins
Maximum
Concurrent
Lambdas =
15,000
36. Complex Serverless HPC Reference
Architecture
Breakdown complex workload into multiple simple ones:
…
Reducer
Final Reducer Result
Input Bucket Mapper Functions
Reducers
Reducersmapper
output
mapper
output
reducer
output
38. Data Processing with AWS: Next steps
ü Learn more about AWS Serverless at
https://aws.amazon.com/serverless
ü Explore Real-time Clickstream Anomaly Detection with
Amazon Kinesis Analytics on the AWS Big Data Blog at
https://aws.amazon.com/blogs/big-data/real-time-
clickstream-anomaly-detection-with-amazon-kinesis-
analytics/
ü Explore the AWS Lambda Reference Architecture on GitHub:
§ Real-Time Streaming: https://github.com/awslabs/lambda-refarch-
streamprocessing
§ Distributed Computing Reference Architecture (serverless MapReduce)
https://github.com/awslabs/lambda-refarch-mapreduce
39. Data Processing with AWS: Next steps
ü Create an Amazon Kinesis stream. Visit the Amazon Kinesis
Console and configure a stream to receive data Ex. data from
Social media feeds.
ü Create & test a Lambda function to process streams from Amazon
Kinesis by visiting Lambda console. First 1M requests each month
are on us!
ü Read the Developer Guide and try the Lambda and Amazon
Kinesis Tutorial:
§ http://docs.aws.amazon.com/lambda/latest/dg/with-
kinesis.html
ü Send questions, comments, feedback to the AWS Lambda Forums