Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Deep Visibility: Logging From Distributed Microservices
Similar to Deep Visibility: Logging From Distributed Microservices (20)
More from AaronLieberman5
More from AaronLieberman5 (18)
Recently uploaded
Recently uploaded (20)
Deep Visibility: Logging From Distributed Microservices
- 1. Deep Visibility: Logging From Distributed Microservices September 2020
- 2. 1. The importance of logging 2. Logging common pitfalls 3. Logging best practices 4. Comparison of logging from a traditional application vs. a serverless implementation 5. Serverless logging Topics: Logging and Beyond
- 3. ● Consulting - specializing in integrations and related technologies ● AWS partners ● We build connections between systems, applications, people, and ideas Big Compass
- 5. Quiz Question What is the difference between traceability and visibility? 01
- 6. ● Visibility ● Traceability ● Supportability ● Debugging Why Is Logging So Important?
- 7. Logging Pitfalls and Best Practices 01
- 8. ● No design for logging ● Lack of standards ● Hard coding ● Lack of request identifiers ● Logging payloads or sensitive data ● Logging too much ● Logging too little Logging Common Pitfalls Organize Standardize Stabilize Optimize
- 9. Logging Common Pitfalls Example No identifier or metadata logged and sensitive information in plaintext logged
- 10. ● Standards, standards, standards ● Informative messages ● Request identifier ● Dynamic error messages ● Persisting logs to an easily accessible system ● Logging at appropriate levels ● Logging appropriate amount of data Logging Best Practices Organize Standardize Stabilize Optimize
- 11. Logging Best Practices Example All messages for a single transaction readily available
- 12. Quiz Question Can you name 5 commonly used logging levels? 01
- 13. Traditional Logging Vs. Serverless Logging 01
- 14. Traditional Logging vs Serverless Logging Traditional Application Logging Serverless Logging Single application focused Multi-microservices focused Logs coming from single source Logs coming from many distributed microservices Predictable logs based on user traffic Unpredictable logs based on growing serverless footprint Nice assembly line of logs Spaghetti mess of logs Typically written to a file or the console Typically written to a centralized logging tool/system
- 17. ● Many distributed microservices that communicate with each other ● Log scaling ● Default target for AWS Lambda is CloudWatch ● Minimal logging frameworks that work well for serverless implementations Serverless Logging Pain Points
- 18. ● Centralized logging target ● Reference ID to reference across microservices ● Execution ID to reference the individual AWS execution ● Metadata about the process Serverless Logging Framework Must Haves
- 19. ● CloudWatch ● Splunk, Loggly, ELK, Elastic, Graylog ● Database ● X-Ray ● Many options in the marketplace Serverless Logging Options
- 20. Traditional Logging Vs. Serverless Logging 01
- 21. ● An iPaas implementation with a serverless implementation ● Legacy system combined with AWS Lambda ● EC2 combined with AWS Lambda ● Infinite number of combinations Modern Environment Might Have...
- 22. Modern Environment Logging Architecture
- 23. Demo Serverless logging framework logging to ELK
- 24. Next Steps 01
- 25. Questions? 01
Editor's Notes
- Traceability relies on visibility. Visibility allows you to see into your system, whereas traceability allow you to trace a transaction from end to end
- Logging is so important for a variety of reasons First of all, can you imagine a system without logging? I like to present this scenario because it would be one that would be almost impossible to debug, support, etc I like to think of logging as giving you 3 pillars of the “ilities” Logging gives you visibility into your system Visibility is the backbone to any system You need the visibility into your system to be able to tell what happened at any given moment Logging the right amount of information to make the system visible to the user Traceability Traceability builds on visibility in that traceability allows you to trace one transactions or request from end to end Traceability relies on visibility to give you the right amount of logging, but traceability also adds more Request identifiers Logging across applications Being able to easily put in a unique ID and come up with all logs for that request Supportability Supportability relies on vis and traceability Debugging Also relies on vis and traceability
- No design for logging There are 2 key items in an implementation that often times go forgotten. One is API security, and the other we are talking about today, logging Lack of standards Hard coding Lack of connective traceability Logging too much Logging payloads or sensitive data Logging too little The outcome of filtering out these common mistakes is that you can start to organize Organization leads to standards and that is ultimately what you are after because that unlocks potential
- Can you note the common mistakes made here in the top picture? First picture Seems ok at first But looking into it, you have no idea what request was made besides the metadata of getting a 200 or 404 response and the time it was made at That forces you to guess You need more info to be able to actually tell what happened to the request Second picture Good metadata logged and good JSON logged, BUT sensitive information is logged This is SO common, so beware! This one is pretty obvious, but it happens all the time
- •Standards, standards, standards –JSON –Key-value pairs –Other formats •Informative messages –Visibility into the system –Enhance debugging –Before and after external system calls •Request identifier –Trace transaction from end to end –Ability to calculate latency •Dynamic error messages •Persisting logs to a system –Maintain historical logs –Allows non-technical users to access and support applications •Not too much logging, not too little Standards is the message here Standards unlock potential They unlock the ability to stabilize a system if you know what’s going wrong with a system They allow you to optimize your system if you know exactly what is occurring in your system
- Trace Debug Info Warn Error
- So all of that is great, but we’re here to talk about serverless logging, and how serverless logging presents unique challenges
- Let’s compare serverless logging against traditional logging from a legacy app or maybe even an iPaas. This would assume that the app has minimal or no microservices… it’s a monolith
- This is clean and simple Relatively straightforward to implement
- This is messy. Serverless logging gets messy and can be hard to maintain those best practices if you’re not careful
- Many distributed microservices that communicate with each other Could run into code replication We recommend to add a logging framework as a dependency so you’re minimizing replication Otherwise this is what can lead to code replication Log scaling log /data production Being able to batch or take one-off logs Can produce a massive amount of data Default target for AWS Lambda is CloudWatch Good for console logs Not so great for many different microservices and searching Minimal logging frameworks that work well for serverless implementations You need to find one for you specific coding language We like NodeJS here at Big Compass, and there weren’t any we could find that worked well with serverless
- The best practices from above are only amplified with serverless You need to take time to properly log from each AWS Lambda and put extra focus on it
- There are pros and cons of each approach The main con to CloudWatch is it’s either difficult or costly to search through so we don’t recommend using just CloudWatch CloudWatch might be used as a backup or for lower level logs than your centralized target logging system like Splunk
- So all of that is great, but we’re here to talk about serverless logging, and how serverless logging presents unique challenges
- A modern system has many components and can be messy It’s rare that a modern system is 100% serverless… at least in the integration world
- A modern system might look like this You have 80% serverless, but you also have EC2 and MuleSoft Can log from serverless/ serverful It can get complicated! You need to think about the producing side and replicating that to all your apps if you want a centralized system and you need to think about the consuming side and how you will scale/throttle and pump logs in a guaranteed manner to the destination, or even multiple destinations So what we set out to do for our projects is to create a framework that can be used for serverless and legacy/iPaas solutions also because we use a lot of those So we created a logging framework that can be used for internal or external apps Producer portion Does not care about the consumer side - it is decoupled from the consumer Dependency in producing apps Or you can just make API requests if your app doesn’t handle dependencies well The producer can allow you to send logs to any number of arbitrary targets as we’ll see in the demo Consumer portion Does not care about the producer side - it is decoupled from the producer And remember, this is just one consumer… it is a reusable consumer for our purposes, but you might have other consumers like a direct ingestion point from Splunk 2 inbound endpoints API Gateway SQS SQS ensures guaranteed delivery of our log messages to the logging target
- Lager will be available at some point by EOY. We would love for anyone to fork it, make it better, and let us know how it goes!
- Reach out if you would like to talk more. Don’t hesitate to email me or connect with me on LinkedIn