Serverless is a great pattern for modern cloud application. But this completly change our approach at monitoring. This is our story...

1. Carlo Pecchia
@carlopecchia
Logging, monitoring and tracing of a serverless app
Mariano Calandra
@MarianoCalandra

2. A real life experience
TRACKING
A tracking app for bikers
REST API
More than 70 API
TRAFFIC
1000+ RPD in beta
SOCIAL
Friends, group itineraries...
SERVERLESS
What else? :-)

3. Why serverless?
NO SERVERS TO MANAGE
Just write your code
FLEXIBLE SCALING
Scale just a given function
AUTOMATED HA
Provided by the service
NO UPFRONT COST
Just pay for what you use

4. Building blocks
≪Amazon API Gateway is a fully managed service that
makes it easy for developers to create, publish, maintain,
monitor, and secure APIs at any scale. [...] You can create an
API that acts as a “front door” for [...] your back-end services
[...] running on AWS Lambda.≫
≪AWS Lambda lets you run code without provisioning
or managing servers. You pay only for the compute time
you consume [...]. With Lambda, you can run code [...]
with zero administration. Just upload your code and
Lambda takes care of everything required to run and
scale your code with high availability.≫

5. Architecture (simplified)

6. Serverless application model – SAM
D0002GetItineraries:
Type: 'AWS::Serverless::Function'
Properties:
Handler: it.xxx.xxx.ItineraryHandler::getItineraries
Runtime: java8
MemorySize: 3008
Timeout: 15
FunctionName: D0002GetItineraries
Events:
D0002GetItineraries1:
Type: Api
Properties:
Path: /itinerary
Method: get

7. CI/CD Pipeline

8. Until the day...
≪The app is too slow...≫

9. Many questions
WHEN
Always? When error happens?
WHERE
Any critical regions in source code?
WHY
Low Memory? A bug? A leak?
WHAT
All Lambdas? Just one? Two?

10. Lambda console

11. CloudWatch custom dashboard

12. Lambda Log

13. Lambda REPORT Log
REPORT RequestId: 5d1...3f7
⤷Duration: 4529.12 ms
⤷Billed Duration: 4600 ms
⤷Memory Size: 3008 MB
⤷Max Memory Used: 157 MB

14. First issue
SCATTERED INFORMATIONS
Many log groups and log streams
IDEA
Collect everything in the same place

15. Amazon Elasticsearch Service
≪Amazon Elasticsearch Service is a
managed service that makes it easy to
deploy, operate, and scale Elasticsearch
clusters in the AWS Cloud.≫

16. Stream logs to ElasticSearch

17. Kibana

18. Second issue
IT’S JUST TEXT
No possibility to graph, sort, ...
IDEA
Index the REPORT line

19. Lambda REPORT indexing
// ...
var regex = /^.*REPORTs+RequestId:s+([0-9A-Fa-f-]{36})s+Duration:
s+([0-9]*[.][0-9]*)s+mss+Billed Duration:s+([0-9]*) mss+Memory Size:
([0-9]*) MBs+Max Memory Used: ([0-9]*) MB/g;
var parts = regex.exec(log.message);
// ...
body.lambda_request_id = parts[1];
body.lambda_duration = parseFloat(parts[2]);
body.lambda_billed_duration = Number(parts[3]);
body.lambda_memory_size = Number(parts[4]);
body.lambda_memory_used = Number(parts[5]);
body.lambda_function_name = loggroup_name.split("/")[3];
// ...

20. Lambda REPORT indexing

21. Lambda REPORT indexing

22. Third issue
IT’S JUST A PIECE
What about API Gateway latencies?
IDEA
Stream and index API Gateway logs

23. Integration latency
(5d1...bb3) Received response.
⤷Integration latency: 7126 ms
(*) The time between when API Gateway relays a request to the backend and when it receives a response from it.

24. Integration latency indexing
var regex = /^(([0-9A-Fa-f-]{36})) Received response. Integration latency: ([0-9]*) ms$/g;
var parts = regex.exec(log.message);
// ...
body["@id"] = parts[1];
body.api_gw_integration_latency = Number(parts[2]);
// ...

25. Lambda RequestId
(5d1...bb3)Endpoint response headers:
{...
x-amzn-RequestId=5d1...3f7
...}

26. Lambda RequestId indexing
// ...
var regex = /^(([0-9A-Fa-f-]{36})) Endpoint response headers:
.*x-amzn-RequestId=([0-9A-Fa-f-]{36}),.*$/g;
var parts = regex.exec(log.message);
// ...
body["@id"] = parts[1];
body.api_gw_request_id = parts[1];
body.api_gw_lambda_invoked_request_id = parts[2];
// ...

27. API Gateway log indexing

28. Fourth issue
HIGH LATENCY
Why Integration latency is much greater than Lambda duration?
7126 ms >> 4529.12 ms
IDEA
Digging deeper into Lambda executions

29. COLD START

30. Enable tracing

31. Request TraceId
(5d1...bb3)Method response headers:
{...
X-Amzn-Trace-Id=Root=1-5b2...340
...}

32. TraceId indexing
var regex = /^(([0-9A-Fa-f-]{36})) Method response headers:
{.*X-Amzn-Trace-Id=Root=([0-9A-Fa-f-]*),.*$/g;
var parts = regex.exec(log.message);
// ...
body["@id"] = parts[1];
body.api_gw_trace_id = parts[2];
// ...

33. TraceId indexing

34. Trace visualization
ColdStarttime
Function initialization time

35. Fifth issue
INSPECTING
How time is spent during Lambda execution?
IDEA
Instruments AWS X-Ray

36. Deep tracing

37. Further developments
MORE SERVICES
Push new log types into ElasticSearch (e.g. CodeBuild?, CloudTrail?)
GRAPHIC CORRELATION
Did this deploy negatively affect Lambda execution times?
CACHED REQUEST
Cache hits vs Cache misses

38. www.xpeppers.com
@xpeppers xpeppers.com/radar