The document discusses the AWS Community Day in Chicago, focusing on a project called 'Prism' aimed at efficiently uploading millions of images in a serverless environment using AWS Lambda. It details the project's objectives, the architecture involving various AWS resources, and key learnings from implementing the solution. The results from the fall 2017 season highlight a significant volume of images and metadata successfully processed using this serverless approach.

1. AWS Community Day Chicago
Uploading Millions of Images using a
Serverless Environment
Brett Sutter, Steve Miers, Kirtesh Garg
Minnesota User Group
June 2018

2. Lifetouch
● School Photography Company
● 30 + Million Subjects
Photographed Each Fall

3. Problem Statement
● Objectives
○ New project called “Prism” made up of 2 components. Prelab and PPS.
○ Make images available for new PPS fulfillment systems.
○ Capture all image metadata in image data management service. (AIR)
○ Minimal changes to existing applications/infrastructure as they would be in
peak. (Fox Systems)
○ Extra Credit: Can we do this without additional servers?????

4. Job 1 - Name The Project
Amazon Image Repository
Legacy Image File Transfer

5. AIRLIFT

6. AirLift Usecase

7. AWS Lambda
● Serverless Compute Resource.
● Invokes code in response to events.
● Manages all underlying resources.
● Pay for the actual usage.

8. Why Lambda ?
● Suits well for our seasonal compute demands.
○ No Auto Scaling to worry about.
● No servers to maintain.
● Tiny functions and faster cold starts.
● Well under the Request/Response size limits.

9. Lambda Invocation Types
● On Demand (Using AWS SDK)
○ Synchronous
○ Asynchronous
● Event Sources
○ S3 (Asynchronous)
○ Kinesis Streams & DynamoDB (Polls & Invokes Synchronously)
○ Many other services (Cognito, Iot, Alexa, etc.)

10. Lambda - 1 (Triggers Image Upload)

11. Lambda - 2 (MetaData Upload)

12. AWS Resources
● 7 SQS Queues
● 2 Lambda functions
● 2 S3 buckets
● 3 IAM roles
● 3 IAM policies

13. CloudFormation in CI/CD pipeline

14. Learnings
● Declare all initialization code outside of handler.
● Reduce cold start latency by allocating more memory.
● Lambda function invoked asynchronously is retried twice. Configuring DLQ is
helpful.
● If hosting Lambda inside VPC make sure your subnet has sufficient ENI capacity.
● AWS Lambda (Async invoked) auto retries in case of throttled event.
● Function level concurrent execution limit (AWS:ReInvent 2017).

15. Results
Fall 2017 Season - School Photography
◆ Subjects uploaded
● August: 6,712,193
● September: 13,031,102
● October: 13,973,025
● November: 5,641,892
● Season Total: 39,358,212 - Approx 39M
◆ Objects Moved
● August: 54,033,153
● September: 104,900,371
● October: 112,482,851
● November: 45,417,230
● Season Total: 316,833,605 - Approx 316M Images were uploaded using AirLift
platform. Data - 110TB

16. Thank You !