Managing the logs of your (Rails) applications Lennart Koopmann,  RailsWayCon 2011 www.lennartkoopmann.net / @_lennart
About me 23 years old Living in Hamburg Rails developer at XING AG Developer of Graylog2
What is this log management stuff? Even grepping over flat files can be log management.
Log Management Maturity Scale Log management has different levels – Raffael Marty set up a scale for that.
Level 0 Do not collect logs at all.
Level 1 Collect logs. Mostly simple log files from email or HTTP servers.
Level 2 Use the logs for forensics and troubleshooting. Why was that email not sent out? Why was that HTTP 500 thrown?
Level 3 Save searches. The most basic case would be to save a grep command you used.
Level 4 Share searches. Store that search command somewhere so co-workers can find and use it to solve the same problem.
Level 5 Reporting.
Level 6 Alerting. Automate some of your troubleshooting tasks. Be warned automatically instead of waiting for a user to co...
Level 7 Collect more logs! We may need more sources for some use cases – Like multi-line application logs, firewall logs o...
Level 8 Correlation. Manual analysis of all this new data may take too long – Correlate different sources.
Level 9 Visual analysis.
Next levels Pattern detection, interactive visualization, dynamic queries, anomaly detection, more sharing.
Collecting logs Two different types.
Type 1 Logs automatically generated from a service. For example apache2.log or mail.log – Usually huge amount of structure...
Type 2 Logs sent directly from within your application. Triggered for example by a log.error() call or an Exception catche...
How to send your logs Don't store the logs in flat files. Send them somewhere to get more value out of them.
Syslog Syslog adapters for Rails are available and work pretty good.
GELF Graylog extended log format – Let's you structure your logs. Also check out structured syslog. Ruby library, Rack exc...
{ ' message ':'[pay] ZOMG credit card invalid', ' full_message ':'Stacktrace.nSome env vars', ' host ':'www19', ' file ':'...
AMQP Guaranteed and ordered delivery. Very flexible. Easily subscribe to the flow. Use routing keys to structure origin of...
Throw the messages out of your app like a hot potato Loose coupling! Your logs should always leave the application without...
Add more value to your logs For example pre-generate geo information for IP addresses or integrate the time_bandits gem.
https://github.com/skaes/time_bandits Completed in 680.378ms (View: 28.488, DB: 5.111(2,0), MC: 5.382(6r,0m), GC: 120.100(...
Where to send your logs There are a lot of tools available.
Hosted services: Loggly www.loggly.com Dynamic pricing based on your usage Free for 200MB/day with 1 week retention time U...
 
 
Hosted services: Splunk www.splunk.com Two license types: Free / Enterprise Supports any raw input
Two more hosted services: www.papertrailapp.com www.logentries.com
Open source tools: Logstash www.logstash.net Collect, parse and store logs for later use Input -> Filter -> Output Plays v...
Logstash inputs For example: AMQP, file, redis, stdin, syslog, tcp, stomp, twitter
Logstash filters For example: date, field, grep, grok, multiline
Logstash outputs For example: amqp, elasticsearch, gelf, mongodb, redis, stdout, tcp, websocket
 
Open source tools: Graylog2 www.graylog2.org Accepts syslog (TCP/UDP) and GELF (+ AMQP) Rails web interface for filtering,...
 
 
 
 
 
Log management use case: API consumer monitoring Something different from the usual alerting, monitoring and reporting.
Pre-processor script (or Logstash) parses raw access log (possibly via AMQP), combines multi line log messages of API engi...
oauth_consumer_key, severity, http_status_code, processed (controller#action)
Pre-processor sends the extracted value including the raw message to Graylog2.
n.notify { :severity => 4, :short_message => “UsersController#show [500]”, :full_message => full_msg, :_oauth_consumer_key...
Now use Graylog2 and the MongoDB shell to answer questions like...
What consumers are still using the deprecated find user by email call?
What errors are caused by the iPhone application?
Which applications keep causing errors?
Which consumers are inactive?
How many calls are done by the iPhone application and how many were it a month ago?
Extract everything you might need from the message in a structured format you can easily parse and query later. You alread...
Q & A @_lennart www.lennartkoopmann.net
Upcoming SlideShare
Loading in...5
×

Managing the logs of your (Rails) applications - RailsWayCon 2011

8,460

Published on

Published in: Technology
0 Comments
17 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
8,460
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
81
Comments
0
Likes
17
Embeds 0
No embeds

No notes for slide

Managing the logs of your (Rails) applications - RailsWayCon 2011

  1. 1. Managing the logs of your (Rails) applications Lennart Koopmann, RailsWayCon 2011 www.lennartkoopmann.net / @_lennart
  2. 2. About me 23 years old Living in Hamburg Rails developer at XING AG Developer of Graylog2
  3. 3. What is this log management stuff? Even grepping over flat files can be log management.
  4. 4. Log Management Maturity Scale Log management has different levels – Raffael Marty set up a scale for that.
  5. 5. Level 0 Do not collect logs at all.
  6. 6. Level 1 Collect logs. Mostly simple log files from email or HTTP servers.
  7. 7. Level 2 Use the logs for forensics and troubleshooting. Why was that email not sent out? Why was that HTTP 500 thrown?
  8. 8. Level 3 Save searches. The most basic case would be to save a grep command you used.
  9. 9. Level 4 Share searches. Store that search command somewhere so co-workers can find and use it to solve the same problem.
  10. 10. Level 5 Reporting.
  11. 11. Level 6 Alerting. Automate some of your troubleshooting tasks. Be warned automatically instead of waiting for a user to complain.
  12. 12. Level 7 Collect more logs! We may need more sources for some use cases – Like multi-line application logs, firewall logs or even physical access logs.
  13. 13. Level 8 Correlation. Manual analysis of all this new data may take too long – Correlate different sources.
  14. 14. Level 9 Visual analysis.
  15. 15. Next levels Pattern detection, interactive visualization, dynamic queries, anomaly detection, more sharing.
  16. 16. Collecting logs Two different types.
  17. 17. Type 1 Logs automatically generated from a service. For example apache2.log or mail.log – Usually huge amount of structured, but raw data. jira.graylog2.org:80 x.x.x.x - - [29/May/2011:01:47:38 +0200] "GET /browse/WEBINTERFACE-21?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel HTTP/1.1" 200 7639 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
  18. 18. Type 2 Logs sent directly from within your application. Triggered for example by a log.error() call or an Exception catcher. - Possible to send structured via for example GELF 2011-05-29 18:55:51 +0200 [payment] Could not validate credit card: Got HTTP 404 from example.org
  19. 19. How to send your logs Don't store the logs in flat files. Send them somewhere to get more value out of them.
  20. 20. Syslog Syslog adapters for Rails are available and work pretty good.
  21. 21. GELF Graylog extended log format – Let's you structure your logs. Also check out structured syslog. Ruby library, Rack exception notifier and Ruby logger available. ( www.graylog2.org )
  22. 22. { ' message ':'[pay] ZOMG credit card invalid', ' full_message ':'Stacktrace.nSome env vars', ' host ':'www19', ' file ':'/var/www/app.rb', ' line ':2638, ' level ':1, ' _something ':'foo', ' _something_else ':'bar' }
  23. 23. AMQP Guaranteed and ordered delivery. Very flexible. Easily subscribe to the flow. Use routing keys to structure origin of the logs. Hell yeah, use this if you have an AMQP bus available. (or build one) Check out https://github.com/paukul/amqp_logging
  24. 24. Throw the messages out of your app like a hot potato Loose coupling! Your logs should always leave the application without interfering it! Prefer UDP over TCP, decouple AMQP log transports. Catch all exceptions and get back into the app flow.
  25. 25. Add more value to your logs For example pre-generate geo information for IP addresses or integrate the time_bandits gem.
  26. 26. https://github.com/skaes/time_bandits Completed in 680.378ms (View: 28.488, DB: 5.111(2,0), MC: 5.382(6r,0m), GC: 120.100(1), HP: 0(2000000,546468,18682541,934967)) | 200 OK [ http://127.0.0.1/jobs/info ] Can generate a deep insight view of your application performance when used with LogJam: https://github.com/alpinegizmo/logjam
  27. 27. Where to send your logs There are a lot of tools available.
  28. 28. Hosted services: Loggly www.loggly.com Dynamic pricing based on your usage Free for 200MB/day with 1 week retention time UDP/TCP/HTTP API as input for syslog
  29. 31. Hosted services: Splunk www.splunk.com Two license types: Free / Enterprise Supports any raw input
  30. 32. Two more hosted services: www.papertrailapp.com www.logentries.com
  31. 33. Open source tools: Logstash www.logstash.net Collect, parse and store logs for later use Input -> Filter -> Output Plays very well with Graylog2
  32. 34. Logstash inputs For example: AMQP, file, redis, stdin, syslog, tcp, stomp, twitter
  33. 35. Logstash filters For example: date, field, grep, grok, multiline
  34. 36. Logstash outputs For example: amqp, elasticsearch, gelf, mongodb, redis, stdout, tcp, websocket
  35. 38. Open source tools: Graylog2 www.graylog2.org Accepts syslog (TCP/UDP) and GELF (+ AMQP) Rails web interface for filtering, analytics, alerting, reporting, … Stores in MongoDB
  36. 44. Log management use case: API consumer monitoring Something different from the usual alerting, monitoring and reporting.
  37. 45. Pre-processor script (or Logstash) parses raw access log (possibly via AMQP), combines multi line log messages of API engine and extracts value.
  38. 46. oauth_consumer_key, severity, http_status_code, processed (controller#action)
  39. 47. Pre-processor sends the extracted value including the raw message to Graylog2.
  40. 48. n.notify { :severity => 4, :short_message => “UsersController#show [500]”, :full_message => full_msg, :_oauth_consumer_key => “foo”, :_processed => “UsersController#show”, :_http_status_code => 500, ... ... }
  41. 49. Now use Graylog2 and the MongoDB shell to answer questions like...
  42. 50. What consumers are still using the deprecated find user by email call?
  43. 51. What errors are caused by the iPhone application?
  44. 52. Which applications keep causing errors?
  45. 53. Which consumers are inactive?
  46. 54. How many calls are done by the iPhone application and how many were it a month ago?
  47. 55. Extract everything you might need from the message in a structured format you can easily parse and query later. You already have all the data you need!
  48. 56. Q & A @_lennart www.lennartkoopmann.net
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×