Visibility into any system is a key component of creating a supportable platform. Without proper logging, support can be costly and inefficient. With the emergence of APIs, microservices, and distributed, decoupled architectures, logging becomes even more important because there are more components that make up a system than ever before. This is beneficial from the standpoint of creating reliable systems, but logging frameworks need to adapt to this architecture because the premise of logging remains the same as it always has: log clear messages that are easy to read with the goal of enhancing visibility into a system. In this Meetup hosted by Big Compass, we will explore techniques of logging from the typical iPaas or always-on managed system like a custom application on an EC2, and we will balance that with a discussion on logging from serverless microservices such as AWS Lambda also. We’ll walk through a real system we have created and discuss how a logging framework can be created using AWS serverless services to enhance the visibility and supportability of the system. You will learn: • Common best practices and blind spots of logging • Differences of logging from always-on systems versus serverless services (AWS Lambda) • Successful use cases where logging has been implemented to improve supportability of a system Who should attend: • IT leaders who want to decrease support cost and have a system visibility pain point • Developers struggling with implementing a robust, highly visible logging solution • Anyone considering using serverless technology for an upcoming implementation Reasons to attend: • Create a logging framework that garners deep visibility and a great experience for users, no matter the underlying architecture

1. Deep Visibility: Logging
From Distributed
Microservices
December 2020

2. 1. The importance of logging
2. Logging common pitfalls
3. Logging best practices
4. Comparison of logging from a traditional
application vs. a serverless implementation
5. Serverless logging
Topics:
Logging and
Beyond

3. ● Consulting - specializing in
integrations and related technologies
● AWS partners
● We build connections between
systems, applications, people, and
ideas
Big Compass

4. The Importance of
Logging

5. Quiz Question
What is the difference between traceability
and visibility?
01

6. ● Visibility
● Traceability
● Supportability
● Debugging
Why Is Logging So
Important?

7. Logging Pitfalls and
Best Practices

8. ● No design for logging
● Lack of standards
● Hard coding
● Lack of request identifiers
● Logging payloads or sensitive data
● Logging too much
● Logging too little
Logging Common
Pitfalls
Organize
Standardize
Stabilize
Optimize

9. Logging Common Pitfalls Example
No identifier or metadata logged and sensitive information in plaintext logged

10. ● Standards, standards, standards
● Informative messages
● Request identifier
● Dynamic error messages
● Persisting logs to an easily accessible
system
● Logging at appropriate levels
● Logging an appropriate amount of
data
Logging Best Practices
Organize
Standardize
Stabilize
Optimize

11. Logging Best Practices Example
All messages for a single transaction readily available

12. Quiz Question
Can you name 5 commonly used logging
levels?
01

13. Traditional Logging Vs.
Serverless Logging

14. Traditional Logging vs Serverless Logging
Traditional Application Logging Serverless Logging
Single application focused Multi-microservices focused
Logs coming from single source Logs coming from many distributed
microservices
Predictable logs based on user traffic Unpredictable logs based on growing
serverless footprint
Nice assembly line of logs Spaghetti mess of logs
Typically written to a file or the console Typically written to a centralized logging
tool/system

15. Traditional Logging Architecture

16. Serverless Logging Architecture

17. ● Many distributed microservices that
communicate with each other
● Log scaling
● Minimal logging frameworks that work
well for serverless implementations
Serverless Logging Pain
Points

18. ● Centralized logging target
● Reference ID to reference across
microservices
● Execution ID to reference the individual
AWS execution
● Metadata about the process
Serverless Logging
Framework Must Haves

19. ● CloudWatch and CloudWatch Insights
● Splunk, Loggly, ELK, Elastic, Graylog
● Database
● X-Ray
● Many options in the marketplace
Serverless Logging
Options

20. The Modern Logging
Architecture

21. ● An iPaas implementation with a
serverless implementation
● Legacy system combined with AWS
Lambda
● EC2 combined with AWS Lambda
● Infinite number of combinations
Modern Environment
Might Have...

22. Modern Environment Logging Architecture

23. © 2020, Amazon Web Services, Inc. or its Affiliates.
Josh DeMuth
Partner Solutions Architect
Deeper Dive into Lambda

24. © 2020, Amazon Web Services, Inc. or its Affiliates.
What We’ll Cover
• AWS Lambda Function Anatomy
• Reducing redundancy with Lambda Layers
• Extend Lambda service with Lambda Extensions
• Monitor Performance with Lambda Insights
• Recent Lambda-related Announcements

25. © 2020, Amazon Web Services, Inc. or its Affiliates.
Lambda Function Anatomy

26. © 2020, Amazon Web Services, Inc. or its Affiliates.
Anatomy of an AWS Lambda function
Handler() function
Function to be executed
upon invocation
Event object
Data sent during
Lambda function
invocation
Context object
Methods available to
interact with runtime
information (request ID,
log group, more)
import json
def lambda_handler(event, context):
# TODO implement
return {
'statusCode': 200,
'body': json.dumps('Hello World!')
}

27. © 2020, Amazon Web Services, Inc. or its Affiliates.
Anatomy of an AWS Lambda function
Function myhandler(event, context) {
<Event handling logic> {
result = SubfunctionA()
} else {
result = SubfunctionB()
return result;
}
Function subFunctionA(thing){
## logic here
}
Function subFunctionB(thing){
## logic here
}

28. © 2020, Amazon Web Services, Inc. or its Affiliates.
Anatomy of an AWS Lambda function
Function myhandler(event, context) {
<Event handling logic> {
result = SubfunctionA()
} else {
result = SubfunctionB()
return result;
}
Function subFunctionA(thing){
## logic here
}
Function subFunctionB(thing){
## logic here
}
Functions will then grow in complexity with business
logic sub-functions

29. © 2020, Amazon Web Services, Inc. or its Affiliates.
Anatomy of an AWS Lambda function
Import sdk
Import http-lib
Import ham-sandwich
Pre-handler-secret-getter()
Pre-handler-db-connect()
Function myhandler(event, context) {
<Event handling logic> {
result = SubfunctionA()
}else {
result = SubfunctionB()
return result;
}
Function Pre-handler-secret-getter() {
}
Function Pre-handler-db-connect(){
}
Function subFunctionA(thing){
## logic here
}
Function subFunctionB(thing){
## logic here
}
Business logic sub-functions

30. © 2020, Amazon Web Services, Inc. or its Affiliates.
Import sdk
Import http-lib
Import ham-sandwich
Pre-handler-secret-getter()
Pre-handler-db-connect()
Function myhandler(event, context) {
<Event handling logic> {
result = SubfunctionA()
}else {
result = SubfunctionB()
return result;
}
Function Pre-handler-secret-getter() {
}
Function Pre-handler-db-connect(){
}
Function subFunctionA(thing){
## logic here
}
Function subFunctionB(thing){
## logic here
}
Anatomy of an AWS Lambda function
Dependencies, configuration information, common helper functions
Common helper functions
Business logic sub-functions

31. © 2020, Amazon Web Services, Inc. or its Affiliates.
Anatomy of a serverless application
/orders
/forums
/search
/lists
/user
/...
Now, assume
that I have an
API-based
workload
Amazon API
Gateway

32. © 2020, Amazon Web Services, Inc. or its Affiliates.
Anatomy of a serverless application
It needs to read/write
to a database and get
keys and configuration
from external services
AWS Secrets
Manager/
Parameter Store
Amazon
DynamoDB

33. © 2020, Amazon Web Services, Inc. or its Affiliates.
Anatomy of a serverless application
Import sdk
Import http-lib
Import ham-sandwich
Pre-handler-secret-getter()
Pre-handler-db-connect()
Function myhandler(event, context) {
<Event handling logic> {
result = SubfunctionA()
}else {
result = SubfunctionB()
return result;
}
Function Pre-handler-secret-getter() {
}
Function Pre-handler-db-connect(){
}
Function subFunctionA(thing){
## logic here
}
Function subFunctionA(thing){
## logic here
}
Dependencies, configuration information,
common helper functions
Common helper functions
Business logic sub-functions
/orders
/forums
/search
/lists
/user
/...
Import sdk
Import http-lib
Import ham-sandwich
Pre-handler-secret-getter()
Pre-handler-db-connect()
Function myhandler(event, context) {
<Event handling logic> {
result = SubfunctionA()
}else {
result = SubfunctionB()
return result;
}
Function Pre-handler-secret-getter() {
}
Function Pre-handler-db-connect(){
}
Function subFunctionA(thing){
## logic here
}
Function subFunctionA(thing){
## logic here
}
Dependencies, configuration information,
common helper functions
Common helper functions
Business logic sub-functions
Import sdk
Import http-lib
Import ham-sandwich
Pre-handler-secret-getter()
Pre-handler-db-connect()
Function myhandler(event, context) {
<Event handling logic> {
result = SubfunctionA()
}else {
result = SubfunctionB()
return result;
}
Function Pre-handler-secret-getter() {
}
Function Pre-handler-db-connect(){
}
Function subFunctionA(thing){
## logic here
}
Function subFunctionA(thing){
## logic here
}
Dependencies, configuration information,
common helper functions
Common helper functions
Business logic sub-functions
Import sdk
Import http-lib
Import ham-sandwich
Pre-handler-secret-getter()
Pre-handler-db-connect()
Function myhandler(event, context) {
<Event handling logic> {
result = SubfunctionA()
}else {
result = SubfunctionB()
return result;
}
Function Pre-handler-secret-getter() {
}
Function Pre-handler-db-connect(){
}
Function subFunctionA(thing){
## logic here
}
Function subFunctionA(thing){
## logic here
}
Dependencies, configuration information,
common helper functions
Common helper functions
Business logic sub-functions
Import sdk
Import http-lib
Import ham-sandwich
Pre-handler-secret-getter()
Pre-handler-db-connect()
Function myhandler(event, context) {
<Event handling logic> {
result = SubfunctionA()
}else {
result = SubfunctionB()
return result;
}
Function Pre-handler-secret-getter() {
}
Function Pre-handler-db-connect(){
}
Function subFunctionA(thing){
## logic here
}
Function subFunctionA(thing){
## logic here
}
Dependencies, configuration information,
common helper functions
Common helper functions
Business logic sub-functions
Import sdk
Import http-lib
Import ham-sandwich
Pre-handler-secret-getter()
Pre-handler-db-connect()
Function myhandler(event, context) {
<Event handling logic> {
result = SubfunctionA()
}else {
result = SubfunctionB()
return result;
}
Function Pre-handler-secret-getter() {
}
Function Pre-handler-db-connect(){
}
Function subFunctionA(thing){
## logic here
}
Function subFunctionA(thing){
## logic here
}
Dependencies, configuration information,
common helper functions
Common helper functions
Business logic sub-functions
Amazon API
Gateway
AWS Secrets
Manager/
Parameter Store
Amazon
DynamoDB

34. © 2020, Amazon Web Services, Inc. or its Affiliates.
Anatomy of a serverless application
Import sdk
Import http-lib
Import ham-sandwich
Pre-handler-secret-getter()
Pre-handler-db-connect()
Function myhandler(event, context) {
<Event handling logic> {
result = SubfunctionA()
}else {
result = SubfunctionB()
return result;
}
Function Pre-handler-secret-getter() {
}
Function Pre-handler-db-connect(){
}
Function subFunctionA(thing){
## logic here
}
Function subFunctionA(thing){
## logic here
}
Dependencies, configuration information,
common helper functions
Common helper functions
Business logic sub-functions
/orders
/forums
/search
/lists
/user
/...
Import sdk
Import http-lib
Import ham-sandwich
Pre-handler-secret-getter()
Pre-handler-db-connect()
Function myhandler(event, context) {
<Event handling logic> {
result = SubfunctionA()
}else {
result = SubfunctionB()
return result;
}
Function Pre-handler-secret-getter() {
}
Function Pre-handler-db-connect(){
}
Function subFunctionA(thing){
## logic here
}
Function subFunctionA(thing){
## logic here
}
Dependencies, configuration information,
common helper functions
Common helper functions
Business logic sub-functions
Import sdk
Import http-lib
Import ham-sandwich
Pre-handler-secret-getter()
Pre-handler-db-connect()
Function myhandler(event, context) {
<Event handling logic> {
result = SubfunctionA()
}else {
result = SubfunctionB()
return result;
}
Function Pre-handler-secret-getter() {
}
Function Pre-handler-db-connect(){
}
Function subFunctionA(thing){
## logic here
}
Function subFunctionA(thing){
## logic here
}
Dependencies, configuration information,
common helper functions
Common helper functions
Business logic sub-functions
Import sdk
Import http-lib
Import ham-sandwich
Pre-handler-secret-getter()
Pre-handler-db-connect()
Function myhandler(event, context) {
<Event handling logic> {
result = SubfunctionA()
}else {
result = SubfunctionB()
return result;
}
Function Pre-handler-secret-getter() {
}
Function Pre-handler-db-connect(){
}
Function subFunctionA(thing){
## logic here
}
Function subFunctionA(thing){
## logic here
}
Dependencies, configuration information,
common helper functions
Common helper functions
Business logic sub-functions
Import sdk
Import http-lib
Import ham-sandwich
Pre-handler-secret-getter()
Pre-handler-db-connect()
Function myhandler(event, context) {
<Event handling logic> {
result = SubfunctionA()
}else {
result = SubfunctionB()
return result;
}
Function Pre-handler-secret-getter() {
}
Function Pre-handler-db-connect(){
}
Function subFunctionA(thing){
## logic here
}
Function subFunctionA(thing){
## logic here
}
Dependencies, configuration information,
common helper functions
Common helper functions
Business logic sub-functions
Import sdk
Import http-lib
Import ham-sandwich
Pre-handler-secret-getter()
Pre-handler-db-connect()
Function myhandler(event, context) {
<Event handling logic> {
result = SubfunctionA()
}else {
result = SubfunctionB()
return result;
}
Function Pre-handler-secret-getter() {
}
Function Pre-handler-db-connect(){
}
Function subFunctionA(thing){
## logic here
}
Function subFunctionA(thing){
## logic here
}
Dependencies, configuration information,
common helper functions
Common helper functions
Business logic sub-functions
Amazon API
Gateway
AWS Secrets
Manager/
Parameter Store
Amazon
DynamoDB
There could be a lot of
duplicated code here!

35. © 2020, Amazon Web Services, Inc. or its Affiliates.
Anatomy of a serverless application
/orders
/forums
/search
/lists
/user
/...
Amazon API Gateway
Import sdk
Import http-lib
Import ham-sandwich
Pre-handler-secret-getter()
Pre-handler-db-connect()
Function myhandler(event, context) {
<Event handling logic> {
result = SubfunctionA()
}else {
result
= SubfunctionB()
return result;
}
Function
Import sdk
Import http-lib
Import ham-sandwich
Pre-handler-secret-getter()
Pre-handler-db-connect()
Function myhandler(event, context) {
<Event handling logic> {
result = SubfunctionA()
}else {
result
= SubfunctionB()
return result;
}
Function
Import sdk
Import http-lib
Import ham-sandwich
Pre-handler-secret-getter()
Pre-handler-db-connect()
Function myhandler(event, context) {
<Event handling logic> {
result = SubfunctionA()
}else {
result
= SubfunctionB()
return result;
}
Function
Import sdk
Import http-lib
Import ham-sandwich
Pre-handler-secret-getter()
Pre-handler-db-connect()
Function myhandler(event, context) {
<Event handling logic> {
result = SubfunctionA()
}else {
result
= SubfunctionB()
return result;
}
Function Import sdk
Import http-lib
Import ham-sandwich
Pre-handler-secret-getter()
Pre-handler-db-connect()
Function myhandler(event, context) {
<Event handling logic> {
result = SubfunctionA()
}else {
result
= SubfunctionB()
return result;
}
Function
Import sdk
Import http-lib
Import ham-sandwich
Pre-handler-secret-getter()
Pre-handler-db-connect()
Function myhandler(event, context) {
<Event handling logic> {
result = SubfunctionA()
}else {
result
= SubfunctionB()
return result;
}
Function
Dependencies, configuration information
Common helper functions
Business logic sub-functions
We want something
more like this
AWS Secrets
Manager/
Parameter Store
Amazon DynamoDB

36. © 2020, Amazon Web Services, Inc. or its Affiliates.
Lambda Layers
Lets functions easily share code:
Upload layer once, reference within
any function
Layer can be anything: dependencies,
training data, configuration files, etc
Promote separation of responsibilities,
lets developers iterate faster on
writing business logic
Built-in support for secure sharing by
ecosystem

37. © 2020, Amazon Web Services, Inc. or its Affiliates.
Lambda Extensions

38. © 2020, Amazon Web Services, Inc. or its Affiliates.
Lambda Extensions

39. © 2020, Amazon Web Services, Inc. or its Affiliates.
Lambda Extensions

40. © 2020, Amazon Web Services, Inc. or its Affiliates.
Lambda Extensions Use Cases
• capturing diagnostic information before, during,
and after function invocation
• automatically instrumenting your code without
needing code changes
• fetching configuration settings or secrets before
the function invocation
• detecting and alerting on function activity
through hardened security agents, which can run
as separate processes from the function

41. © 2020, Amazon Web Services, Inc. or its Affiliates.
CloudWatch Lambda Insights

42. © 2020, Amazon Web Services, Inc. or its Affiliates.
Use Cases
• Identify function issues such as
memory leaks
• Identify high-cost functions
• Identify performance changes
caused by new function versions
• Understand latency drivers in
functions
Features
• Deep linking and correlations across metrics, logs, and traces
• Curated automated dashboards summarizing the performance and
health of your Lambda functions
• Deeper integration with CloudWatch Logs for deeper analysis of log
data through Logs Insights and CloudWatch ServiceLens to analyze
trace dependencies
• New performance metrics from AWS Lambda with observable
context for each running function
What is Lambda Insights?
Monitor, troubleshoot, and optimize the performance of AWS Lambda functions

43. © 2020, Amazon Web Services, Inc. or its Affiliates.
Lambda Insights
• Two pre-defined set of automated dashboards, including new AWS
Lambda performance metrics and curated widgets:
• Multi-Function Dashboard
• Aggregated view across multiple Lambda functions
• Provide visibility into issues
• Single-Function Dashboard
• View of one single Lambda function
• Easily troubleshoot root causes of a Lambda issue

44. © 2020, Amazon Web Services, Inc. or its Affiliates.
Multi-Function Dashboard
• Aggregated view across multiple
Lambda functions
• Search by “name” (“tag” upcoming) to
focus on a specific or a subset of
Lambda functions
• Understand how compute, memory
allocation, and function duration
changes over a period of time to help
optimize Lambda function utilization
• Integration to CloudWatch
Logs Insights to analyze
function logs

45. © 2020, Amazon Web Services, Inc. or its Affiliates.
Single-Function Dashboard
• View of one single Lambda function
• Predefined performance metrics

46. © 2020, Amazon Web Services, Inc. or its Affiliates.
Logs sent to CloudWatch Logs
{
"_aws": {
"Timestamp": 1605034324256,
…
"event_type": "performance",
"function_name": "cpu-intensive",
"version": "Blue",
"request_id": "12345678-8bcc-42f7-b1de-123456789012",
"trace_id": "1-5faae118-12345678901234567890",
"duration": 45191,
"billed_duration": 45200,
"billed_mb_ms": 11571200,
"cold_start": true,
"init_duration": 130,
"tmp_free": 538329088,
"tmp_max": 551346176,
"threads_max": 11,
"used_memory_max": 63,
"total_memory": 256,
"memory_utilization": 24,
"cpu_user_time": 6640,
"cpu_system_time": 50,
"cpu_total_time": 6690,
"fd_use": 416,
"fd_max": 32642,
"tx_bytes": 4434,
"rx_bytes": 6911,
"timeout": true,
"shutdown_reason": "Timeout",
"total_network": 11345,
"agent_version": "1.0.72.0",
"agent_memory_avg": 10,
"agent_memory_max": 10
}

47. © 2020, Amazon Web Services, Inc. or its Affiliates.
Recent Announcements

48. © 2020, Amazon Web Services, Inc. or its Affiliates.
Four Exciting Announcements!

49. Demo
Serverless logging framework logging to ELK

50. Next Steps

51. ● Clone the Big Compass Logging
Framework on the Big Compass
GitHub
● For the repository
● Make pull requests
● Give us your feedback
● Reach out for an in depth demo or
implementation help
Serverless Logging
Options

52. Questions?