Low Latency Logging
at BrightonPHP
James Titcumb

@asgrim
github.com/asgrim
● Started on Amiga 500
● PHP for 11 years
● ZCE PHP 5.3
●
●
●
●
●

Run PHP Hampshire user group
Lead dev on browscap.ini
F...
Hello! :)

@asgrim
github.com/asgrim
Errors
What are errors?
● Something broke… :)
● e.g.
○ Can’t connect to MySQL (mysqli_connect)
○ No such file or directory (fopen...
Problems?

source: http://www.dpaddbags.com/blog/episode-119-technology-sucks/
Problems.
●
●
●
●
●

error_reporting setting
Errors look ugly
“@”
Limited options
Not very “OO”
Ways Around
// Will raise E_NOTICE
fopen($somefile, 'r');
Ways Around
// No error! :)
if (file_exists($somefile)) {
fopen($somefile, 'r');
} else {
// nice handling...
// maybe thr...
Exceptions
Jargon Buster
● throw
Triggering
● try
Try to run
● catch
Handle it
● finally
Run code after try/catch
What are exceptions?
●
●
●
●
●

Something still broke
OO
Catchable
Fatal errors
They are classes
SPL Exceptions
● Built in to PHP
● More descriptive than just “Exception”, e.g.:
○
○
○
○
○

InvalidArgumentException
Logic...
Example (exception class)
class DivisionByZeroException
extends LogicException
{
}
Example (throw)
class Mathematics
{
public function divide($a, $b)
{
if ($b == 0) {
$msg = sprintf(‘Tried to divide %s by ...
Example (catch)
$maths = new Mathematics();
try
{
$result = $maths->divide(5, 0);
}
catch (DivisionByZeroException $except...
Logging
What is “logging”?
Keeping a record of all events...
exceptions, errors, warnings, info, debug
Why use logging?
●
●
●
●

Easier to find problems
More detail
“paper trail” for code
Log where you want
What about Apache’s error_log?

source: http://up-ship.com/blog/?p=20903
Why?
● error_log is too basic
● Reading / parsing
● error_reporting (again)
Doin’ it right wrong…
/************************************************************
*

Magic file that makes your entire p...
Requirements (for everyone)
●
●
●
●

Fire & forget
Minimum or zero latency
Highly available
Log everything:
○
○
○
○

Excep...
PSR-3
● Common logging interface
→ LoggerInterface
● RFC-5424 Levels
(debug, info, notice, warning, error, critical, alert...
Logging Options
●
●
●
●
●
●
●

monolog (PSR-3)
Drupal - PSR-3 Watchdog
phpconsole
log4php
RavenPHP + Sentry
FirePHP (dev e...
How they work...

source: http://mirror-sg-wv1.gallery.hd.org/_c/natural-science/cogs-with-meshed-teeth-AJHD.jpg.html
Capturing Logging
Use “capture methods”, send to $logger
● set_exception_handler()
○ Handles all uncaught exceptions

● se...
Sending Log Messages
● Handler/Adapter translates
● However you want…
● Monolog has loads:
○
○
○
○

syslog-compatible / er...
Typical PSR-3 Compatible Design
Capture Method

Logger (PSR-3)

Handler / Adapter

Data Storage
Monolog
MonologErrorHandler
->handleException()

MonologLogger
->log()

MonologHandler
->handle()
Low Latency
High Performance
What do I mean?
● Easy to add
● Fire & Forget
● Minimum impact to request
Slow Logging
Browser

Application

Log Server

HTTP request

Error!

Send log message to database

Acknowledge message
HTT...
Zero Latency Logging (ideal)
Browser

Application

Log Server

HTTP request

Error!

HTTP response to client

Send log mes...
What about UDP?
● Yes… Zero latency, but….
● Do you even care about your logs?
(UDP means log messages may get lost...)
● ...
Low Latency Logging (balance)
Browser

Application

Log Server

HTTP request

Error!

HTTP response to client

Send log me...
So how?
Say hello to RabbitMQ
What is RabbitMQ?
●
●
●
●

Robust messaging for applications
Easy to use
Runs on all major operating systems
Supports a hu...
RabbitMQ - Basic

queue
“Consumer”

“Publisher”

source: http://www.rabbitmq.com/tutorials/tutorial-one-php.html
RabbitMQ - Exchanges
“Publisher”

“Publisher”

queue 1
“Consumer 1”

“Publisher”

“Exchange”

queue 2
“Consumer 2”

“Publi...
Using Queues === Fast!
Add RabbitMQ to logging architecture...
Low Latency (using AMQP)
EdLogHandlerErrorHandler
->handleException()

EdLogLogger
->log()

EdLogPublisherAmqpPublisher
->...
Low Latency Logging (with AMQP)
Browser

Application

RabbitMQ

Log Server

HTTP request

Error!

JSON via AMQP
HTTP respo...
Why bother?
● Scalability
Log Worker
Application A

Log Worker
Application B

RabbitMQ
Log Worker

Application C
Log Worke...
Single Point of Failure...
● RabbitMQ can do HA
RabbitMQ
Node 6
RabbitMQ
Node 1

RabbitMQ
Node 2
RabbitMQ
Node 5

RabbitMQ...
Live demo
(pre-recorded… meh)
Questions?
Feedback please...
https://joind.in/9928
Thanks!
https://joind.in/9928

@asgrim
github.com/asgrim
Low latency Logging (BrightonPHP - 18th Nov 2013)
Upcoming SlideShare
Loading in...5
×

Low latency Logging (BrightonPHP - 18th Nov 2013)

3,263

Published on

Logging is an absolute must for any API or web application, but when starting out, questions such as "how can we do it without disrupting everything else" and "what is the easiest way to log" often come out. I'm going to explore a couple of infrastructure ideas to carry out what I call "high-performance, low-latency" logging to ensure minimal impact to the performance of the runtime application. The talk will show you that a really great logging architecture is a low-cost investment in your application that will definitely pay off in the long run.

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,263
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
16
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Low latency Logging (BrightonPHP - 18th Nov 2013)"

  1. 1. Low Latency Logging at BrightonPHP
  2. 2. James Titcumb @asgrim github.com/asgrim
  3. 3. ● Started on Amiga 500 ● PHP for 11 years ● ZCE PHP 5.3 ● ● ● ● ● Run PHP Hampshire user group Lead dev on browscap.ini First open source GoDeploy Development Manager at Protected.co.uk Mostly a ZF2 developer @asgrim github.com/asgrim
  4. 4. Hello! :) @asgrim github.com/asgrim
  5. 5. Errors
  6. 6. What are errors? ● Something broke… :) ● e.g. ○ Can’t connect to MySQL (mysqli_connect) ○ No such file or directory (fopen) ● Usually PHP core ● Sometimes “fatal”
  7. 7. Problems? source: http://www.dpaddbags.com/blog/episode-119-technology-sucks/
  8. 8. Problems. ● ● ● ● ● error_reporting setting Errors look ugly “@” Limited options Not very “OO”
  9. 9. Ways Around // Will raise E_NOTICE fopen($somefile, 'r');
  10. 10. Ways Around // No error! :) if (file_exists($somefile)) { fopen($somefile, 'r'); } else { // nice handling... // maybe throw exception... }
  11. 11. Exceptions
  12. 12. Jargon Buster ● throw Triggering ● try Try to run ● catch Handle it ● finally Run code after try/catch
  13. 13. What are exceptions? ● ● ● ● ● Something still broke OO Catchable Fatal errors They are classes
  14. 14. SPL Exceptions ● Built in to PHP ● More descriptive than just “Exception”, e.g.: ○ ○ ○ ○ ○ InvalidArgumentException LogicException OutOfBoundsException RuntimeException see PHP manual for more
  15. 15. Example (exception class) class DivisionByZeroException extends LogicException { }
  16. 16. Example (throw) class Mathematics { public function divide($a, $b) { if ($b == 0) { $msg = sprintf(‘Tried to divide %s by zero”, $a); throw new DivisionByZeroException($msg); } return ($a / $b); } }
  17. 17. Example (catch) $maths = new Mathematics(); try { $result = $maths->divide(5, 0); } catch (DivisionByZeroException $exception) { $logger->warning($exception>getMessage()); }
  18. 18. Logging
  19. 19. What is “logging”? Keeping a record of all events... exceptions, errors, warnings, info, debug
  20. 20. Why use logging? ● ● ● ● Easier to find problems More detail “paper trail” for code Log where you want
  21. 21. What about Apache’s error_log? source: http://up-ship.com/blog/?p=20903
  22. 22. Why? ● error_log is too basic ● Reading / parsing ● error_reporting (again)
  23. 23. Doin’ it right wrong… /************************************************************ * Magic file that makes your entire project work perfectly * ************************************************************/ @ini_set('display_errors', 0); @error_reporting(0); function __globalErrorHandler() { return true; } @set_error_handler('__globalErrorHandler'); @set_exception_handler('__globalErrorHandler'); @register_shutdown_function(function() { if(error_get_last()) { echo "Script executed successfully!"; } }); https://github.com/webarto/boostrap.php
  24. 24. Requirements (for everyone) ● ● ● ● Fire & forget Minimum or zero latency Highly available Log everything: ○ ○ ○ ○ Exceptions Errors Fatal Errors Debug & info ● PSR-3 compatible
  25. 25. PSR-3 ● Common logging interface → LoggerInterface ● RFC-5424 Levels (debug, info, notice, warning, error, critical, alert, emergency) ● Interoperability ● Reusability
  26. 26. Logging Options ● ● ● ● ● ● ● monolog (PSR-3) Drupal - PSR-3 Watchdog phpconsole log4php RavenPHP + Sentry FirePHP (dev environment) Roll your own
  27. 27. How they work... source: http://mirror-sg-wv1.gallery.hd.org/_c/natural-science/cogs-with-meshed-teeth-AJHD.jpg.html
  28. 28. Capturing Logging Use “capture methods”, send to $logger ● set_exception_handler() ○ Handles all uncaught exceptions ● set_error_handler() ○ Handles most errors ● register_shutdown_function() ○ Handles fatal errors
  29. 29. Sending Log Messages ● Handler/Adapter translates ● However you want… ● Monolog has loads: ○ ○ ○ ○ syslog-compatible / error_log Email, HipChat AMQP, Sentry, Zend Monitor, Graylog2 Redis, MongoDB, CouchDB
  30. 30. Typical PSR-3 Compatible Design Capture Method Logger (PSR-3) Handler / Adapter Data Storage
  31. 31. Monolog MonologErrorHandler ->handleException() MonologLogger ->log() MonologHandler ->handle()
  32. 32. Low Latency High Performance
  33. 33. What do I mean? ● Easy to add ● Fire & Forget ● Minimum impact to request
  34. 34. Slow Logging Browser Application Log Server HTTP request Error! Send log message to database Acknowledge message HTTP response to client
  35. 35. Zero Latency Logging (ideal) Browser Application Log Server HTTP request Error! HTTP response to client Send log message to database
  36. 36. What about UDP? ● Yes… Zero latency, but…. ● Do you even care about your logs? (UDP means log messages may get lost...) ● TCP means guaranteed network delivery ● Any non-blocking fails
  37. 37. Low Latency Logging (balance) Browser Application Log Server HTTP request Error! HTTP response to client Send log message to database
  38. 38. So how?
  39. 39. Say hello to RabbitMQ
  40. 40. What is RabbitMQ? ● ● ● ● Robust messaging for applications Easy to use Runs on all major operating systems Supports a huge number of developer platforms ● Open source and commercially supported www.rabbitmq.com
  41. 41. RabbitMQ - Basic queue “Consumer” “Publisher” source: http://www.rabbitmq.com/tutorials/tutorial-one-php.html
  42. 42. RabbitMQ - Exchanges “Publisher” “Publisher” queue 1 “Consumer 1” “Publisher” “Exchange” queue 2 “Consumer 2” “Publisher” “Publisher” source: http://www.rabbitmq.com/tutorials/tutorial-three-php.html
  43. 43. Using Queues === Fast! Add RabbitMQ to logging architecture...
  44. 44. Low Latency (using AMQP) EdLogHandlerErrorHandler ->handleException() EdLogLogger ->log() EdLogPublisherAmqpPublisher ->publish() RabbitMQ Logging Server JSON payload
  45. 45. Low Latency Logging (with AMQP) Browser Application RabbitMQ Log Server HTTP request Error! JSON via AMQP HTTP response Fetch message
  46. 46. Why bother? ● Scalability Log Worker Application A Log Worker Application B RabbitMQ Log Worker Application C Log Worker
  47. 47. Single Point of Failure... ● RabbitMQ can do HA RabbitMQ Node 6 RabbitMQ Node 1 RabbitMQ Node 2 RabbitMQ Node 5 RabbitMQ Node 3 RabbitMQ Node 4
  48. 48. Live demo (pre-recorded… meh)
  49. 49. Questions?
  50. 50. Feedback please... https://joind.in/9928
  51. 51. Thanks! https://joind.in/9928 @asgrim github.com/asgrim
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×