Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Low Latency Logging 
with RabbitMQ 
BrnoPHP Conference 2014
Who is this guy? 
James Titcumb 
www.jamestitcumb.com 
www.protected.co.uk 
www.phphants.co.uk 
@asgrim
Let’s go back to basics...
Errors
source: http://www.dpaddbags.com/blog/episode-119-technology-sucks/
error_reporting(0);
They look rubbish!
@
Log to file only
Ways Around 
// Will raise E_WARNING 
fopen($somefile, 'r');
Ways Around 
// No error! :) 
if (file_exists($somefile)) { 
fopen($somefile, 'r'); 
} else { 
// nice handling... 
// may...
Exceptions
Jargon Buster 
● throw 
Triggering 
● try 
Try to run 
● catch 
Handle it 
● finally 
Run code after try/catch
source: http://www.dpaddbags.com/blog/episode-119-technology-sucks/
C♯/C++
Catchable
Turn into fatal errors 
if not caught
Classy
Descriptive exceptions
Example (exception class) 
class DivisionByZeroException 
extends LogicException 
{ 
}
Example (throw) 
class Mathematics { 
public function divide($a, $b) { 
if ($b == 0) { 
$msg = sprintf(“Divide %s by zero”...
Example (catch) 
$maths = new Mathematics(); 
try 
{ 
$result = $maths->divide(5, 0); 
} 
catch (DivisionByZeroException $...
Throw Descriptively
Logging
What is “logging”?
What is “logging”? 
Keeping a record of all events...
What is “logging”? 
Keeping a record of all events... 
exceptions, errors, warnings, info, debug
Paper Trail
Log like you just don’t care
Log like you just don’t care
Log wherever you like
What about Apache’s error_log? 
source: http://up-ship.com/blog/?p=20903
Doin’ it rightwrong… 
// Magic file that makes your entire project work perfectly 
@ini_set('display_errors', 0); 
@error_...
Fire & Forget
Low Latency
Highly Available
PSR-3
Logging Options 
● monolog (PSR-3) 
● Drupal - PSR-3 Watchdog 
● phpconsole 
● log4php 
● RavenPHP + Sentry 
● FirePHP (de...
source: http://mirror-sg-wv1.gallery.hd.org/_c/natural-science/cogs-with-meshed-teeth-AJHD.jpg.html
Capturing Messages
Capturing Messages 
set_error_handler()
Capturing Messages 
set_exception_handler()
Capturing Messages 
register_shutdown_function()
Capturing Messages 
Using the Logger Directly
Adapter / Handler
Typical PSR-3 Compatible Design 
Capture Method 
Logger (PSR-3) 
Handler / Adapter 
Data Storage
MonologErrorHandler 
->handleException() 
MonologLogger 
->log() 
MonologHandler 
->handle() 
Monolog
Low Latency Logging
Make it easy
Fire & Forget
Minimum Impact
Slow Logging 
Browser Application Log Server 
HTTP request 
Send log message to log server 
Error! 
Acknowledge message 
H...
Zero Latency Logging (ideal) 
Browser Application Log Server 
HTTP request 
Send log message to log server 
Error! 
HTTP r...
UDP?
We need a balance.
Low Latency Logging (balance) 
Browser Application Log Server 
HTTP request 
Send log message to log server 
Error! 
HTTP ...
…so how?
Disclaimer...
Say hello to RabbitMQ
RabbitMQ - Basic 
test_queue 
1 2 3 4 5 
Producer Consumer 
source: http://www.rabbitmq.com/tutorials/tutorial-one-php.htm...
RabbitMQ - Exchanges 
Exchange 
Consumer test_queue 
1 2 3 4 5 
source: http://www.rabbitmq.com/tutorials/tutorial-three-p...
Exchanges are flexible
RabbitMQ === Fast!
Low Latency (using AMQP) 
EdLogHandlerErrorHandler 
->handleException() 
EdLogLogger 
->log() 
EdLogPublisherAmqpPublisher...
Low Latency Logging (with AMQP) 
Browser Application Log Server 
Fetch message 
HTTP request 
JSON via AMQP 
Error! 
HTTP ...
Why bother? 
● Scalability 
RabbitMQ 
Application A 
Application B 
Application C 
Log Worker 
Log Worker 
Log Worker 
Log...
Single Point of Failure... 
RabbitMQ 
Node 1 
RabbitMQ 
Node 3 
RabbitMQ 
Node 2 
RabbitMQ 
Node 4 
RabbitMQ 
Node 6 
Rabb...
To recap...
Scalable.
Easy.
Fast.
Future proof.
Questions?
Thanks for watching! 
James Titcumb 
@asgrim
Low Latency Logging with RabbitMQ (Brno PHP, CZ - 20th Sep 2014)
Low Latency Logging with RabbitMQ (Brno PHP, CZ - 20th Sep 2014)
Low Latency Logging with RabbitMQ (Brno PHP, CZ - 20th Sep 2014)
Low Latency Logging with RabbitMQ (Brno PHP, CZ - 20th Sep 2014)
Low Latency Logging with RabbitMQ (Brno PHP, CZ - 20th Sep 2014)
Low Latency Logging with RabbitMQ (Brno PHP, CZ - 20th Sep 2014)
Low Latency Logging with RabbitMQ (Brno PHP, CZ - 20th Sep 2014)
Upcoming SlideShare
Loading in …5
×

Low Latency Logging with RabbitMQ (Brno PHP, CZ - 20th Sep 2014)

1,764 views

Published on

Logging is an absolute must for any API or web application, but when starting out, questions such as "how can we do it without disrupting everything else" and "what is the easiest way to log" often come up. We’re going to examine a tried and tested method to carry out high-performance, low-latency logging using the power of RabbitMQ to ensure minimal impact to the performance of your runtime application. The talk will show you that a really great logging architecture is a low-cost investment in your application that will definitely pay off in the long run.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Low Latency Logging with RabbitMQ (Brno PHP, CZ - 20th Sep 2014)

  1. 1. Low Latency Logging with RabbitMQ BrnoPHP Conference 2014
  2. 2. Who is this guy? James Titcumb www.jamestitcumb.com www.protected.co.uk www.phphants.co.uk @asgrim
  3. 3. Let’s go back to basics...
  4. 4. Errors
  5. 5. source: http://www.dpaddbags.com/blog/episode-119-technology-sucks/
  6. 6. error_reporting(0);
  7. 7. They look rubbish!
  8. 8. @
  9. 9. Log to file only
  10. 10. Ways Around // Will raise E_WARNING fopen($somefile, 'r');
  11. 11. Ways Around // No error! :) if (file_exists($somefile)) { fopen($somefile, 'r'); } else { // nice handling... // maybe throw exception... }
  12. 12. Exceptions
  13. 13. Jargon Buster ● throw Triggering ● try Try to run ● catch Handle it ● finally Run code after try/catch
  14. 14. source: http://www.dpaddbags.com/blog/episode-119-technology-sucks/
  15. 15. C♯/C++
  16. 16. Catchable
  17. 17. Turn into fatal errors if not caught
  18. 18. Classy
  19. 19. Descriptive exceptions
  20. 20. Example (exception class) class DivisionByZeroException extends LogicException { }
  21. 21. Example (throw) class Mathematics { public function divide($a, $b) { if ($b == 0) { $msg = sprintf(“Divide %s by zero”, $a); throw new DivisionByZeroException($msg); } return ($a / $b); } }
  22. 22. Example (catch) $maths = new Mathematics(); try { $result = $maths->divide(5, 0); } catch (DivisionByZeroException $exception) { $logger->warning($exception->getMessage()); }
  23. 23. Throw Descriptively
  24. 24. Logging
  25. 25. What is “logging”?
  26. 26. What is “logging”? Keeping a record of all events...
  27. 27. What is “logging”? Keeping a record of all events... exceptions, errors, warnings, info, debug
  28. 28. Paper Trail
  29. 29. Log like you just don’t care
  30. 30. Log like you just don’t care
  31. 31. Log wherever you like
  32. 32. What about Apache’s error_log? source: http://up-ship.com/blog/?p=20903
  33. 33. Doin’ it rightwrong… // Magic file that makes your entire project work perfectly @ini_set('display_errors', 0); @error_reporting(0); function __globalErrorHandler() { return true; } @set_error_handler('__globalErrorHandler'); @set_exception_handler('__globalErrorHandler'); @register_shutdown_function(function() { if(error_get_last()) echo "Script executed successfully!"; }); https://github.com/webarto/boostrap.php
  34. 34. Fire & Forget
  35. 35. Low Latency
  36. 36. Highly Available
  37. 37. PSR-3
  38. 38. Logging Options ● monolog (PSR-3) ● Drupal - PSR-3 Watchdog ● phpconsole ● log4php ● RavenPHP + Sentry ● FirePHP (dev environment) ● Roll your own
  39. 39. source: http://mirror-sg-wv1.gallery.hd.org/_c/natural-science/cogs-with-meshed-teeth-AJHD.jpg.html
  40. 40. Capturing Messages
  41. 41. Capturing Messages set_error_handler()
  42. 42. Capturing Messages set_exception_handler()
  43. 43. Capturing Messages register_shutdown_function()
  44. 44. Capturing Messages Using the Logger Directly
  45. 45. Adapter / Handler
  46. 46. Typical PSR-3 Compatible Design Capture Method Logger (PSR-3) Handler / Adapter Data Storage
  47. 47. MonologErrorHandler ->handleException() MonologLogger ->log() MonologHandler ->handle() Monolog
  48. 48. Low Latency Logging
  49. 49. Make it easy
  50. 50. Fire & Forget
  51. 51. Minimum Impact
  52. 52. Slow Logging Browser Application Log Server HTTP request Send log message to log server Error! Acknowledge message HTTP response to client
  53. 53. Zero Latency Logging (ideal) Browser Application Log Server HTTP request Send log message to log server Error! HTTP response to client
  54. 54. UDP?
  55. 55. We need a balance.
  56. 56. Low Latency Logging (balance) Browser Application Log Server HTTP request Send log message to log server Error! HTTP response to client
  57. 57. …so how?
  58. 58. Disclaimer...
  59. 59. Say hello to RabbitMQ
  60. 60. RabbitMQ - Basic test_queue 1 2 3 4 5 Producer Consumer source: http://www.rabbitmq.com/tutorials/tutorial-one-php.html
  61. 61. RabbitMQ - Exchanges Exchange Consumer test_queue 1 2 3 4 5 source: http://www.rabbitmq.com/tutorials/tutorial-three-php.html Consumer Producer Producer Producer
  62. 62. Exchanges are flexible
  63. 63. RabbitMQ === Fast!
  64. 64. Low Latency (using AMQP) EdLogHandlerErrorHandler ->handleException() EdLogLogger ->log() EdLogPublisherAmqpPublisher ->publish() RabbitMQ JSON payload Logging Server
  65. 65. Low Latency Logging (with AMQP) Browser Application Log Server Fetch message HTTP request JSON via AMQP Error! HTTP response RabbitMQ
  66. 66. Why bother? ● Scalability RabbitMQ Application A Application B Application C Log Worker Log Worker Log Worker Log Worker
  67. 67. Single Point of Failure... RabbitMQ Node 1 RabbitMQ Node 3 RabbitMQ Node 2 RabbitMQ Node 4 RabbitMQ Node 6 RabbitMQ Node 5
  68. 68. To recap...
  69. 69. Scalable.
  70. 70. Easy.
  71. 71. Fast.
  72. 72. Future proof.
  73. 73. Questions?
  74. 74. Thanks for watching! James Titcumb @asgrim

×