To Log or Not to Log: Proven Best Practices for Instrumentation - via @Loggly

963 views

Published on

To log or not to log? This is an age-old question for developers. Logging everything can be great because you have plenty of data to work from when you have a problem. But it’s not so great if you have to grep and inspect it all yourself. In my mind, developers should instead be thinking about logging the right events in the right format for the right consumer.

Instrumentation is one of the most valuable sources of data you can log. At Loggly, we instrument like crazy in our code because we need to understand what our system is doing and where we can improve it. When you’re dealing with billions of events per day, the ability to quickly identify hot-spots is invaluable. However, there are lots of different opinions about what you should instrument and concerns about the performance impact of that instrumentation. What follows are the guidelines and best practices that we’ve developed to help ourselves do a better job of instrumentation. Hopefully they’ll help you too…

Read the full blog post on the Loggly blog → http://bit.ly/BestLoggingPractices

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
963
On SlideShare
0
From Embeds
0
Number of Embeds
19
Actions
Shares
0
Downloads
11
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

To Log or Not to Log: Proven Best Practices for Instrumentation - via @Loggly

  1. 1. To Log or Not to Log: Proven Best Practices for Instrumentation Apache Storm Jon Gifford Founder and Chief Search Officer January 2014 | Log management as a service Simplify Log Management
  2. 2. About Me 15+ years of search experience § Lots of different search products § Lots and lots of machines § Lots and lots of logs... Bias: § Lucene, Java, *nix, Distributed, Realtime, § Measure, Measure, Measure | Log management as a service Simplify Log Management
  3. 3. Loggly: Addressing the first big data problem every company faces § Centralized logging and archival § Real-time processing, analysis and visualization § Monitoring, alerting and troubleshooting | Log management as a service Simplify Log Management
  4. 4. To Log or Not to Log? + - yourself Lots of data to solve operational problems faster! Incredibly painful to grep and inspect | Log management as a service Simplify Log Management
  5. 5. Instrumentation: Your Most Valuable Log Data For us, instrumentation is the act of gathering performance data about your application and surfacing it via logs. § Provides a low-impact mechanism for gathering production system performance data § Visualization is key to understanding this data | Log management as a service Simplify Log Management
  6. 6. Best Practices for Instrumentation | Log management as a service Simplify Log Management
  7. 7. Best Practice #1 Treat instrumentation as an ongoing, iterative process. § Start by logging everything at a high level § Instrument deeper every time you ask, “What happened here?” § Instrument deeper in areas that are more variable or expensive than you thought | Log management as a service Simplify Log Management
  8. 8. Best Practice #2 Always instrument anything that goes out of process. § Can be indicative of larger problems, like the canary in the coal mine § Example: Instrument calls to your database, even though they should always be fast § Distributed system problems are not well behaved – by definition | Log management as a service Simplify Log Management
  9. 9. Best Practice #3 Always log unacceptable performance. § Log “soft exceptions” (e.g. slow queries) § “Trust but verify” | Log management as a service Simplify Log Management
  10. 10. Best Practice #4 If possible, log enough context so you can get the complete picture of what happened from a single log event. Elapsed time Process Log this … not this! End Start End | Log management as a service Simplify Log Management
  11. 11. Best Practice #5 View machines, not people, as your logs’ consumer. | Log management as a service Simplify Log Management
  12. 12. Best Practice #6 Trends tell the story better than data points. § Graph performance over time to see daily fluctuations § Use rollups where it’s not practical to log every single event | Log management as a service Simplify Log Management
  13. 13. Best Practice #7 Instrumentation is NOT a substitute for profiling, and vice versa. Instrumentation Profiling Your production environment is not the same as development! | Log management as a service Simplify Log Management
  14. 14. Best Practice #8 Flying slower is better than flying blind. § “Speed advantage” is illusory if you can’t understand system performance § Our debate: not whether to instrument but how much | Log management as a service Simplify Log Management Photo by Eric Kilby
  15. 15. Implement Instrumentation Best Practices Faster and Better with Loggly. Try Loggly for Free! → http://bit.ly/BestLoggingPractices About Us: Loggly is the world’s most popular cloud-based log management solution, used by more than 5,000 happy customers to effortlessly spot problems in real-time, easily pinpoint root causes and resolve issues faster to ensure application success. Visit us at loggly.com or follow @loggly on Twitter. | Log management as a service Simplify Log Management
  16. 16. Proven and Trusted by 5,000+ your peers… “If we didn’t have Loggly, there would be days where our revenue would be affected by as much as 70 percent.” – Albert Ho, Executive Producer/ Platform Product Manager, Rumble Entertainment | Log management as a service Simplify Log Management

×