Talk: "AWS Lambda @ Scale: Designing for High Load" Event: Nerdearla 2023 (https://nerdear.la) Author: Alexis Cuadrado In the realm of serverless computing, AWS Lambda stands as a powerhouse, providing the backend for many of today's most scalable and robust applications. A fully managed compute service capable of running your code at any scale, its surface-level simplicity can be misleading: when operating under a high influx of requests, an adecquate management of the intricate dynamics of the Lambda concurrency model is key to success. In this talk, we will demystify Lambda concurrency and throughput to help you design your applications for optimal performance under high load. We will explore strategies for optimizing individual Lambda functions for latency and illustrate how to strategically employ concurrency controls. Tailored for AWS developers, architects, and system administrators, this session seeks to transform AWS Lambda from a 'black box' to a transparent and fine-tuned engine for your serverless applications.

1. BY ALEXIS CUADRADO
AWSLAMBDA@SCALE
DESIGNING FOR HIGH LOAD

2. LAMBDA @ SCALE #NERDEARLA
LOOK, IT SCALES
AUTOMATICALLY!
MANAGED SERVICE
ACCOUNT
CONCURRENCY
COLD STARTS
THROTTLING
BURST
CONCURRENCY
EXECUTION
COSTS

3. GETTINGOURLAMBDAFUNCTIONSTO
PERFORMASREQUIRED
THE CHALLENGE OF SCALE

4. THREEPERFORMANCEINDICATORS
$
LATENCY THROUGHPUT
COSTPER
TRANSACTION
seconds or milliseconds transactions per second (TPS) USD
MINIMIZE MAXIMIZE MINIMIZE
TIP: Define proper Service Level Objectives (SLOs)
LAMBDA @ SCALE #NERDEARLA
(meet demand)

5. MINIMIZING LATENCY
BY ALEXIS CUADRADO
AWSLAMBDA@SCALE
MAKE EVERY MILLISECOND COUNT

6. ANATOMYOFALAMBDAINVOCATION
1
2
RUNTIME
3
COMPUTE SUBSTRATE
EXECUTION ENVIRONMENT
HANDLER CODE
STATIC CODE
4 5
Download Code Start Environment Bootstrap Runtime Run Static Code Run Handler
1 2 3 4 5
LAMBDA @ SCALE #NERDEARLA

7. initializing
executing
available
invocation
LONGLIVETHEEXECUTIONENVIRONMENT!
1 2 3
TIME
EXECUTION ENVIRONMENT
LAMBDA @ SCALE #NERDEARLA

8. STARTEDONTHECOLDFOOT
initializing
executing
available
invocation
1
2
3
E1
E2
4
5
cold start
Download Code Start Environment Set up Runtime Run Static Code Run Handler
1 2 3 4 5
LAMBDA @ SCALE #NERDEARLA

9. DISSECTING LATENCY
Download Code Start Environment Bootstrap Runtime Run Static Code Run Handler
1 2 3 4 5
COLD START ( INITIALIZATION ) EXECUTION
LAMBDA @ SCALE #NERDEARLA

10. OPTIMIZEHANDLERLOGIC
EMPLOY EFFICIENT ALGORITHMS
A B
C
Put those hard-won
whiteboarding skills to use
Parallelize I/O operations
(e.g. S3 downloads)
Use Step Functions
LAMBDA @ SCALE #NERDEARLA
AVOID ORCHESTRATION
MOVE WORK OUTSIDE HANDLER
D
MULTI-THREADING
Download Code Start Environment Set up Runtime Run Static Code Run Handler
1 2 3 4 5
Reusable objects should be
statically initialized

11. TURNUPTHERAM!
Example Python Function: Return all prime numbers between 0 and 10K
MEMORY (MB) EXECUTION DURATION (MS)
128 170
256 80
512 40
1024 20
1536 17
3008 17
MEMORY = vCPU NETWORK THROUGHPUT
lowest latency
negative returns beyond this point
Beware of
LAMBDA @ SCALE #NERDEARLA
Download Code Start Environment Set up Runtime Run Static Code Run Handler
1 2 3 4 5
TIP: Enable Lambda Insights for profiling

12. STATICINITIALIZATIONDONERIGHT
+ 120 ms
LAMBDA @ SCALE #NERDEARLA
Download Code Start Environment Set up Runtime Run Static Code Run Handler
1 2 3 4 5

13. PACKAGESIZEMATTERS
Download Deployment Package Start Environment Bootstrap Runtime
Download Deployment
Package
Start Environment Bootstrap Runtime Run Static Code
Run
Handler
50 MB
20 MB
TIPS:
• Audit and remove unused dependencies
• Use minifiers (e.g. node-minify)
LAMBDA @ SCALE #NERDEARLA
1
1
3 4 5
2
3
2

14. ONCOLDSTARTFREQUENCY
LAMBDA @ SCALE #NERDEARLA
?
?
?
? COLD
WARM
Isn’t that what we’re all asking in our own lives?
HOW CAN WE GET MORE OF THIS?
... AND LESS OF THIS?

15. CONCURRENCY
t₀ t₁
E3
concurrency = 1 t₂
E1
E2
concurrency = 3
1 2
3
4
TIME
LAMBDA @ SCALE #NERDEARLA

16. PROVISIONALLTHECONCURRENCY!
t₀
E3
E1
E2
provisioned concurrency = 3
1
2
3
4
5
t₁
concurrency is provisioned
requests start arriving
TIME
NO COLD STARTS!
LAMBDA @ SCALE #NERDEARLA

17. AUTOSCALING
TIME
DEMAND /
CONCURRENCY
LAMBDA @ SCALE #NERDEARLA

18. MAXIMIZING
BY ALEXIS CUADRADO
AWSLAMBDA@SCALE
THROUGHPUT
LEAVE NO REQUEST BEHIND

19. AFUNCTION’STHROUGHPUT
1 2
1 2
1 2 3 4
= 2 TPS
THROUGHPUT =
CONCURRENCY
LATENCY
= 4 TPS
= 4 TPS
= 500 ms
= 1
= 500 ms
= 500 ms = 500 ms
= 2
= 250 ms = 250 ms = 250 ms = 250 ms
= 1 3 4
LAMBDA @ SCALE #NERDEARLA

20. REDUCINGLATENCYTOGAINTHROUGHPUT
250 ms
1 2 3 4 5 6 7 8
1 2 3 4 5 6 7 8 9 10
1 2 4
3
= 4 TPS
1 sec
100 ms
50 ms
125 ms
1
2
3
4
= 8 TPS
= 10 TPS
= 1
LAMBDA @ SCALE #NERDEARLA
?

21. OH,THERATELIMITS
1 2 3 4 5 6 7 8 9 10
1 2 3 4 5 6 7 8 9 10
100 ms
50 ms
50 ms
4A
3
4B
= 20 TPS
= 10 TPS
= STILL 10 TPS
= 1
Throughput capped due to rate limiting
Additional invocations will increase
concurrency or be throttled
LAMBDA @ SCALE #NERDEARLA
1 3 5 7 9 11 13 15 17 19
2 4 6 8 10 12 14 16 18 20
ASYNCHRONOUS
INVOCATIONS
SYNCHRONOUS
INVOCATIONS
No limit!

22. ACCOUNTCONCURRENCY
TIME
E1
E100
E1
E800
THROTTLED
X
concurrency = 200
concurrency = 800
account concurrency = 1000
ANOTHER FUNCTION
OUR FUNCTION
LAMBDA @ SCALE #NERDEARLA
Everyone gets their share

23. RESERVEDCONCURRENCY
TIME
E1
E300
E1
E100
THROTTLED
X
concurrency = 300
concurrency = 100
account concurrency = 1000
OUR FUNCTION
ANOTHER FUNCTION
reserved concurrency = 900
-1900
LAMBDA @ SCALE #NERDEARLA

24. BURSTCONCURRENCY
TIME (MINUTES)
CONCURRENCY
(UNITS)
5K
3000
2500
1000 1500
1000
0 0
500 1000
2500
1
0 3
2 4 6
5 7 9
8
2K
1K
3K
4K
+500 +500 +500 +500
+0 +0 +0 +500 +500
BURST QUOTA = 3000
AVAILABLE BURST
ACCOUNT CONCURRENCY = 5000
BURST THROTTLE ZONE
ACCOUNT THROTTLE ZONE
BURST THROTTLE ZONE
FUNCTION CONCURRENCY
Super
(OR HOW FAST CONCURRENCY CAN RISE)
LAMBDA @ SCALE #NERDEARLA

25. MINIMIZING
BY ALEXIS CUADRADO
AWSLAMBDA@SCALE
$ COSTS
GET THE BIGGEST BANG FOR YOUR BUCK

26. HOWLAMBDAPRICINGWORKS
$
COSTPER
TRANSACTION
also COST PER EXECUTION
$
= COMPUTE
CHARGES
REQUEST
CHARGES
Rates vary based on Region and CPU Architecture
Free Tier available
Elegible for Savings Plans
fixed fee per request
EXECUTION
DURATION
ALLOCATED
MEMORY
LATENCY
determines
influences
LAMBDA @ SCALE #NERDEARLA

27. POWERTUNING
alexcasalboni/aws-lambda-power-tuning
OPTIMAL
MEMORY
CONFIGURATION
LAMBDA @ SCALE #NERDEARLA

28. ARMYOURSELF(WITHGRAVITON2)
COST
LATENCY
LAMBDA @ SCALE #NERDEARLA

29. PUTTINGITALLTOGETHER
BY ALEXIS CUADRADO
AWSLAMBDA@SCALE
BEFORE OUR BRAINS BURST

30. WHATWE(HOPEFULLY)LEARNEDTODAY
KEY PERFORMANCE METRICS
Getting our Lambda Functions to perform as required
THE CHALLENGE OF SCALE
COLD STARTS
HOW LAMBDA PROCESSES INVOCATIONS
Execution Environments
CONCURRENCY
HOW LAMBDA SCALES
HOW PRICING WORKS
QUOTAS
CONTROLS
ACCOUNT CONCURRENCY
BURST CONCURRENCY
PROVISIONED CONCURRENCY
RESERVED CONCURRENCY
(how fast)
(how much)
(pre-warm)
(set aside)
POWER TUNING
$
LATENCY - THROUGHPUT - COST PER TRANSACTION
WHAT FACTORS AFFECT LATENCY
THROUGHPUT CONSIDERATIONS
Interplay between memory, execution time and costs
COLD START EXECUTION
Rate Limits, Concurrency Quotas and Controls
Relation to Latency and Concurrency
OPTIMIZATION TECHNIQUES Price-performance gains with arm64 (Graviton2)
LAMBDA @ SCALE #NERDEARLA

31. LAMBDA @ SCALE #NERDEARLA
THANKYOU
@alexscuadrado
/in/alexis-cuadrado
FOLLOW ME
𝕏
alexis.hashnode.dev