Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)

1,876 views

Published on

Logging is an absolute must for any API or web application, but when starting out, questions such as "how can we do it without disrupting everything else" and "what is the easiest way to log" often come up. We’re going to examine a tried and tested method to carry out high-performance, low-latency logging using the power of RabbitMQ to ensure minimal impact to the performance of your runtime application. The talk will show you that a really great logging architecture is a low-cost investment in your application that will definitely pay off in the long run.

Published in: Software
  • Be the first to comment

Low Latency Logging with RabbitMQ (PHP London - 4th Sep 2014)

  1. 1. Low Latency Logging with RabbitMQ PHP London 4th September 2014
  2. 2. Who is this guy? James Titcumb www.jamestitcumb.com www.protected.co.uk www.phphants.co.uk @asgrim
  3. 3. Let’s go back to basics...
  4. 4. Errors
  5. 5. source: http://www.dpaddbags.com/blog/episode-119-technology-sucks/
  6. 6. error_reporting(0);
  7. 7. They look rubbish!
  8. 8. @
  9. 9. Log to file only
  10. 10. Ways Around // Will raise E_WARNING fopen($somefile, 'r');
  11. 11. Ways Around // No error! :) if (file_exists($somefile)) { fopen($somefile, 'r'); } else { // nice handling... // maybe throw exception... }
  12. 12. Exceptions
  13. 13. Jargon Buster ● throw Triggering ● try Try to run ● catch Handle it ● finally Run code after try/catch
  14. 14. source: http://www.dpaddbags.com/blog/episode-119-technology-sucks/
  15. 15. C♯/C++
  16. 16. Catchable
  17. 17. Turn into fatal errors if not caught
  18. 18. Classy
  19. 19. Descriptive exceptions
  20. 20. Example (exception class) class DivisionByZeroException extends LogicException { }
  21. 21. Example (throw) class Mathematics { public function divide($a, $b) { if ($b == 0) { $msg = sprintf(“Divide %s by zero”, $a); throw new DivisionByZeroException($msg); } return ($a / $b); } }
  22. 22. Example (catch) $maths = new Mathematics(); try { $result = $maths->divide(5, 0); } catch (DivisionByZeroException $exception) { $logger->warning($exception->getMessage()); }
  23. 23. Logging
  24. 24. What is “logging”?
  25. 25. What is “logging”? Keeping a record of all events...
  26. 26. What is “logging”? Keeping a record of all events... exceptions, errors, warnings, info, debug
  27. 27. Logging is a paper trail
  28. 28. Log like you just don’t care
  29. 29. Log wherever you like
  30. 30. What about Apache’s error_log? source: http://up-ship.com/blog/?p=20903
  31. 31. Doin’ it rightwrong… // Magic file that makes your entire project work perfectly @ini_set('display_errors', 0); @error_reporting(0); function __globalErrorHandler() { return true; } @set_error_handler('__globalErrorHandler'); @set_exception_handler('__globalErrorHandler'); @register_shutdown_function(function() { if(error_get_last()) echo "Script executed successfully!"; }); https://github.com/webarto/boostrap.php
  32. 32. Requirements (for everyone) ● Fire & forget ● Minimum or zero latency ● Highly available ● Log everything ● PSR-3 compatible
  33. 33. PSR-3
  34. 34. Logging Options ● monolog (PSR-3) ● Drupal - PSR-3 Watchdog ● phpconsole ● log4php ● RavenPHP + Sentry ● FirePHP (dev environment) ● Roll your own
  35. 35. source: http://mirror-sg-wv1.gallery.hd.org/_c/natural-science/cogs-with-meshed-teeth-AJHD.jpg.html
  36. 36. Capturing Logging Use “capture methods”, send to $logger ● set_exception_handler() ○ Handles all uncaught exceptions ● set_error_handler() ○ Handles most errors ● register_shutdown_function() ○ Handles fatal errors
  37. 37. Sending Log Messages ● Handler/Adapter translates ● However you want… ● Monolog has loads: ○ syslog-compatible / error_log ○ Email, HipChat ○ AMQP, Sentry, Zend Monitor, Graylog2 ○ Redis, MongoDB, CouchDB
  38. 38. Typical PSR-3 Compatible Design Capture Method Logger (PSR-3) Handler / Adapter Data Storage
  39. 39. MonologErrorHandler ->handleException() MonologLogger ->log() MonologHandler ->handle() Monolog
  40. 40. Low Latency High Performance
  41. 41. Slow Logging Browser Application Log Server HTTP request Send log message to database Error! Acknowledge message HTTP response to client
  42. 42. Zero Latency Logging (ideal) Browser Application Log Server HTTP request Send log message to database Error! HTTP response to client
  43. 43. UDP?
  44. 44. Low Latency Logging (balance) Browser Application Log Server HTTP request Send log message to database Error! HTTP response to client
  45. 45. …so how?
  46. 46. Say hello to RabbitMQ
  47. 47. RabbitMQ - Basic test_queue 1 2 3 4 5 Producer Consumer source: http://www.rabbitmq.com/tutorials/tutorial-one-php.html
  48. 48. RabbitMQ - Exchanges Exchange Consumer test_queue 1 2 3 4 5 source: http://www.rabbitmq.com/tutorials/tutorial-three-php.html Consumer Consumer Consumer Consumer
  49. 49. Using Queues === Fast! Add RabbitMQ to logging architecture...
  50. 50. Low Latency (using AMQP) EdLogHandlerErrorHandler ->handleException() EdLogLogger ->log() EdLogPublisherAmqpPublisher ->publish() RabbitMQ JSON payload Logging Server
  51. 51. Low Latency Logging (with AMQP) Browser Application Log Server Fetch message HTTP request JSON via AMQP Error! HTTP response RabbitMQ
  52. 52. Why bother? ● Scalability RabbitMQ Application A Application B Application C Log Worker Log Worker Log Worker Log Worker
  53. 53. Single Point of Failure... ● RabbitMQ can do HA or clustering RabbitMQ Node 1 RabbitMQ Node 3 RabbitMQ Node 2 RabbitMQ Node 4 RabbitMQ Node 6 RabbitMQ Node 5
  54. 54. Questions?
  55. 55. Thanks for watching! James Titcumb www.jamestitcumb.com www.protected.co.uk www.phphants.co.uk @asgrim

×