Graylog2 use cases for distributed web applications

3,954 views

Published on

Published in: Technology, Art & Photos

Graylog2 use cases for distributed web applications

  1. 1. Graylog2 use cases for distributed web applications Manage your logs in the dark and have lasers going and make it look like you're from space Lennart Koopmann, 2010 www.lennartkoopmann.net / www.graylog2.org
  2. 2. It's a DevOps thing. Compose meaningful and structured log messages to allow easy analysis and searching.
  3. 3. Bad: - Could not repair image foo.jpg - Could not repair image bar.jpg - Could not repair image baz.jpg – Invalid header checksum. - Missing POST param 'creditcardnumber' - Payment of John Doe did not succeeed.
  4. 4. Good: - [runner][repair-broken-images]Could not repair image foo.jpg – File not found. - [runner][repair-broken-images] Could not repair image bar.jpg – File not found. - [runner][repair-broken-images] Could not repair image baz.jpg – Invalid header checksum. - [payment][checkout] Missing POST param 'creditcartnumber' CUSTOMER #1337 - [payment][backend] Payment of CUSTOMER #1337 did not succeeed.
  5. 5. Which images were broken? repair-broken-images.+repair images(.+.jpg)s.s(.+) foo.jpg File not found. bar.jpg File not found. baz.jpg Invalid header checksum.
  6. 6. Why did the payment fail in the backend? payment].+CUSTOMER #1337 [payment][checkout] Missing POST param 'creditcartnumber' CUSTOMER #1337 [payment][backend] Payment of CUSTOMER #1337 did not succeeed.
  7. 7. Message type distribution payment runner payment-backend payment-checkout runner-image
  8. 8. Define log guidelines Just like your usual coding guidelines. (slap everybody who does not follow them with a large trout )
  9. 9. Use case 0: The usual stuff. Use Graylog2 to monitor your applications from the inside. Analyze your logs, see if something goes wrong, receive warnings when messages rates climb over a given level. Check the logs regularly to identify problems.
  10. 10. Use case 1: Developer logs. Use GELF and give every developer his own hostname like yourapp-johndoe – Now create a stream for every developer. Voilá: No more tail -f debug.log and Graylog2 sugar from the beginning of your development cycle.
  11. 11. Use case 2: Important messages Imagine you do some kind of domain registration for customers. This stuff likes to fail and you want to be informed when it does and why it did. Create a stream that fetches all failed domain registrations and subscribe to it by email (released in v. 0.9.4) to be notified instantly.
  12. 12. Use case 3: Streams of certain application parts. You have some scripts searching for broken images, deleting or repairing them that are running the whole day. Create a stream that fetches all messages from a runner and get a live output of what it is doing right now. You could also create a blacklist instead of a stream if you don't want to bug others with the messages. Get warnings like in use case 2 when something goes wrong.
  13. 13. Use case 4: Live tail at release. You are releasing a new version of your application today. Start the live tail (released in v. 0.9.4) to see what is happening in your system in real time.
  14. 14. Use case 5: Activity log. A user blames the support that you deleted all his content. How to debug this? Would be not such a big problem if you had logged every activity of your users to Graylog2. Blacklist [activitylog] and Log messages like [activitylog] USER #45262 DELETED image25526. Search for what you need with blacklist disabled. (released in v. 0.9.4)
  15. 15. Important: Use structured and meaningful messages. Have logging guidelines. (and follow them) Choose severity with care: You might be called in the night once that EMERG message arrives. Don't log useless messages. That will be the clutter that ruins your analysis, statistics and warning levels. Already think of what to log in your problem analysis steps.

×