Successfully reported this slideshow.
Your SlideShare is downloading. ×

Application Logging With The ELK Stack

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 42 Ad

More Related Content

Slideshows for you (20)

Advertisement

Similar to Application Logging With The ELK Stack (20)

Advertisement

Application Logging With The ELK Stack

  1. 1. Application Logging With The ELK Stack @bwaine - #DPC15 Monday, 29 June 15
  2. 2. 2 Ben Andersen-Waine Software Engineer Contractor Deployed ELK To Prod Numerous Times Monday, 29 June 15
  3. 3. Logging? Monday, 29 June 15
  4. 4. System Logs Monday, 29 June 15
  5. 5. 5 Monday, 29 June 15
  6. 6. Application Log Monday, 29 June 15
  7. 7. Debug Information - Errors (connections, uncaught exceptions, resource exhaustion) Narrative Information - Methods Calls, Event Triggers Business Events - Purchases, Logins, Registrations, Unsubscribes 7 Application Log Monday, 29 June 15
  8. 8. ssh webserver@mydomain.net tail -f /var/log/nginx/my-site.access.log tail -f /var/log/my.application.log ssh data@mydomain.net tail -f /var/log/mysql/mysql.log ssh q@mydomain.net tail -f /var/log/rabbitmq/nodename.log 8 Keeping Track Of All This.... Monday, 29 June 15
  9. 9. 9 The Elk Stack Monday, 29 June 15
  10. 10. Monday, 29 June 15
  11. 11. 1) Monolog 2) Everything else.... 11 PHP Logging Tools Monday, 29 June 15
  12. 12. 1) Monolog: Loggers And Handlers 2) Monolog:Tags & Formatters 3) Logging business events 12 Basic Logging Examples Monday, 29 June 15
  13. 13. use MonologLogger; use MonologHandlerFingersCrossedHandler; use MonologHandlerStreamHandler; $logEnv = getenv('LOG_LEVEL'); $level = empty($logLevel) ? $logEnv : Logger::WARNING; $appLog = new Logger('AppLog'); $strHandler = new StreamHandler('/var/log/app.log', Logger::DEBUG); $fcHandler = new FingersCrossedHandler($strHandler, $level); $appLog−>pushHandler($fcHandler); $appLog−>debug('LOGGING!'); EG1: Loggers And Handlers 13 Monday, 29 June 15
  14. 14. // Set A Log Level $logEnv = getenv('LOG_LEVEL'); $level = empty($logLevel) ? $logEnv : Logger::WARNING; // Create A Logger $appLog = new Logger('AppLog'); 14 Monday, 29 June 15
  15. 15. $strHandler = new StreamHandler('/var/log/app.log', Logger::DEBUG); $fcHandler = new FingersCrossedHandler($strHandler, $level); // Create Handlers $appLog−>pushHandler($fcHandler); $appLog−>debug('Start Logging!'); $appLog−>emergency('Something Terrible Happened'); // Push The Handler And Start Logging 15 Monday, 29 June 15
  16. 16. EG 2:Tagging Formatting $appLog = new Logger('AppLog'); $strHandler = new StreamHandler('/var/lg.lg', $level); $formatter = new LogstashFormatter("helloapp", "application"); $strHandler−>setFormatter($formatter); $appLog−>pushHandler($strHandler)); $id = $_SERVER('X_VARNISH'); $tag = new TagProcessor(['request−id' => $id]) $appLog−>pushProcessor($tag); $appLog−>debug("LOGGING!"); 16 Monday, 29 June 15
  17. 17. // Create A Logger $appLog = new Logger('AppLog'); $strHandler = new StreamHandler('/var/lg.lg', $level); $formatter = new LogstashFormatter("helloapp", "app"); // Create A Handler & Formatter // Set Formatter Onto Handler $strHandler−>setFormatter($formatter); $appLog−>pushHandler($strHandler)); //Push Handler Onto Logger 17 Monday, 29 June 15
  18. 18. $id = $_SERVER('X_VARNISH'); $tag = new TagProcessor(['request−id' => $id]) $appLog−>pushProcessor($tag); $appLog−>debug("LOGGING!"); // Capture A Unique Id, Create A Tag Processor, Push 18 Monday, 29 June 15
  19. 19. 2009 - RFC 5424 - Syslog Protocol Code / Severity 0 Emergency: system is unusable 1 Alert: action must be taken immediately 2 Critical: critical conditions 3 Error: error conditions 4 Warning: warning conditions 5 Notice: normal but significant condition 6 Informational: informational messages 7 Debug: debug-level messages https://tools.ietf.org/html/rfc5424 19 Log Levels Monday, 29 June 15
  20. 20. 2013 - PSR03 - PHP Logging Interface Standard http://www.php-fig.org/psr/psr-3/ 20 PSR3 Monday, 29 June 15
  21. 21. EG 3: Event Logging use MonologLogger; use SymfonyComponentEventDispatcherEventDispatcher; $dispatcher = new EventDispatcher(); $dispatcher−>addListener( "business.registration.post", function () use ($busLog) { $busLog−>info("Customer registered"); } ); $dispatcher−>dispatch("business.registration.post"); Monday, 29 June 15
  22. 22. Logstash Architecture 1. Logstash Shipper ships logs to logstash 2. Logstash processes them 3. Logstash Inserts Into Elastic Search 4. Kibana exposes a web interface to Elastic Search data Monday, 29 June 15
  23. 23. Logstash Architecture Monday, 29 June 15
  24. 24. Why not rate the talk now BEFORE the demo? 24 https://joind.in/talk/view/14235 Monday, 29 June 15
  25. 25. ELK Demo 25 1) Discover Data (search / diagnose) 2)Visualize Data 3) Produce A Dashboard 4) Demonstrate ‘the new hotness’ of Kibana 4 Monday, 29 June 15
  26. 26. 26 https://github.com/LoveSoftware/ getting-started-with-the-elk-stack Monday, 29 June 15
  27. 27. Monday, 29 June 15
  28. 28. Monday, 29 June 15
  29. 29. Monday, 29 June 15
  30. 30. Monday, 29 June 15
  31. 31. Logstash Config 31 Monday, 29 June 15
  32. 32. Logstash Collecting { "network": { "servers": [ "logs.logstashdemo.com:5000" ], "timeout": 15, "ssl ca": "/etc/pki/tls/certs/logstash−forwarder.crt" }, "files": [ { "paths": [ "/var/log/nginx/helloapp.access.log" ], "fields": { "type": "nginx−access" } } ] } 32 Monday, 29 June 15
  33. 33. Logstash Processing input { lumberjack { port => 5000 ssl_certificate => "/etc/pki/tls/certs/logstash−forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash−forwarder.key" } } Input 33 Monday, 29 June 15
  34. 34. Logstash Processing Filtering filter { if [type] == "nginx−access" { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] } date { match => [ "logdate", "dd/MMM/yyyy:HH:mm:ss Z" ] } } } 34 Monday, 29 June 15
  35. 35. Logstash Processing Output output { elasticsearch { host => localhost } } 35 Monday, 29 June 15
  36. 36. Groking grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } https://github.com/elasticsearch/logstash/blob/v1.4.2/patterns/grok-patterns http://grokdebug.herokuapp.com/ 55.3.244.1 GET /index.html 15824 0.043 %{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration} Monday, 29 June 15
  37. 37. 37 Hey Ben.... Have you got time for that gratuitously flashy geo data demo? Monday, 29 June 15
  38. 38. Monday, 29 June 15
  39. 39. Logging Ideas Release Marker Error rates of various applications over time Latency in various percentiles of each application tier HTTP Responses: 400 series responses HTTP Responses: 500 series responses Auto git blame production errors Auth and Syslogs 39 Monday, 29 June 15
  40. 40. Go Forth And Log.... BUT Remember log rotation Beware running out of space Beware file logging on NFS 40 Monday, 29 June 15
  41. 41. Questions? 41 Monday, 29 June 15
  42. 42. https://joind.in/talk/view/14235 42 Monday, 29 June 15

×