In this talk, I will talk about why log files are horrible, logging log lines, and more structured performance metrics from large scale production applications as well as building reliable, scaleable and flexible large scale software systems in multiple languages. Why (almost) all log formats are horrible will be explained, and why JSON is a good solution for logging will be discussed, along with a number of message queuing, middleware and network transport technologies, including STOMP, AMQP and ZeroMQ. The Message::Passing framework will be introduced, along with the logstash.net project which the perl code is interoperable with. These are pluggable frameworks in ruby/java/jruby and perl with pre-written sets of inputs, filters and outputs for many many different systems, message formats and transports. They were initially designed to be aggregators and filters of data for logging. However they are flexible enough to be used as part of your messaging middleware, or even as a replacement for centralised message queuing systems. You can have your cake and eat it too - an architecture which is flexible, extensible, scaleable and distributed. Build discrete, loosely coupled components which just pass messages to each other easily. Integrate and interoperate with your existing code and code bases easily, consume from or publish to any existing message queue, logging or performance metrics system you have installed. Simple examples using common input and output classes will be demonstrated using the framework, as will easily adding your own custom filters. A number of common messaging middleware patterns will be shown to be trivial to implement. Some higher level use-cases will also be explored, demonstrating log indexing in ElasticSearch and how to build a responsive platform API using webhooks. Interoperability is also an important goal for messaging middleware. The logstash.net project will be highlighted and we'll discuss crossing the single language barrier, allowing us to have full integration between java, ruby and perl components, and to easily write bindings into libraries we want to reuse in any of those languages.

1. Messaging,
interoperability and log
aggregation - a new
framework
Tomas Doran (t0m) <bobtfish@bobtfish.net>

2. Who are you?
• Perl Developer
• Been paid to write perl code for ~14 years
• Open Source hacker
• Catalyst core team
• >160 CPAN dists
• Also C, Javascript, ruby, etc..

3. Sponsored by
• state51
• Pb of mogilefs, 100+ boxes.
• > 4 million tracks on-demand via API
• > 400 reqs/s per server, >1Gb peak from backhaul
• Suretec VOIP Systems
• UK voice over IP provider
• Extensive API, including WebHooks for notifications
• TIM Group
• International Financial apps
• Java / ruby / puppet

4. What?
• This talk is about my new perl library:
Message::Passing

5. Why?
• I’d better stop, and explain a specific
problem.
• The solution that grew out of this is more
generic.
• But it illustrates my concerns and design
choices well.
• And everyone likes a story, right?

6. Once upon a time...
• I was bored of tailing log files across dozens
of servers
• splunk was amazing, but unaffordable

7. Logstash

8. Centralised logging
• Syslog isn’t good enough
• UDP is lossy, TCP not much better
• Limited fields
• No structure to actual message
• RFC3164 - “This document describes the
observed behaviour of the syslog protocol”

9. Centralised logging
• Syslog isn’t good enough
• Structured app logging
• We want to log data, rather than text
from our application
• E.g. HTTP request - vhost, path, time to
generate, N db queries etc..

10. Centralised logging
• Syslog isn’t good enough
• Structured app logging

11. Centralised logging
• Syslog isn’t good enough
• Structured app logging
• Post-process log files to re-structure
• We can do this in cases we don’t control
• Apache logs, etc..
• SO MANY DATE FORMATS. ARGHH!!

12. Centralised logging
• Syslog isn’t good enough
• Structured app logging
• Post-process log files to re-structure
• Publish logs as JSON to a message queue
• JSON is fast, and widely supported
• Great for arbitrary structured data!

13. Message queue
• Flattens load spikes!
• Only have to keep up with average message
volume, not peak volume.
• Logs are bursty! (Peak rate 1000x average.)
• Easy to scale - just add more consumers
• Allows smart routing
• Great as a common integration point.

14. elasticsearch
• Just tip JSON documents into it
• Figures out type for each field, indexes
appropriately.
• Free sharding and replication
• Histograms!

15. Histograms!
• elasticsearch does ‘big data’, not just text
search.
• Ask arbitrary questions
• Get back aggregate metrics / counts
• Very powerful.

16. Logstash
In JRuby, by Jordan Sissel
Input
Simple: Filter
Output
Flexible
Extensible
Plays well with others
Nice web interface

18. Logstash

19. Logstash
INPUT
FILTER
OUTPUT

20. Logstash
INPUT
FILTER
OUTPUT

21. Logstash

22. Logstash

23. Logstash
IS
MASSIVE

24. 440Mb
IDLE!

25. 2+Gb
working

26. 440Mb
IDLE!

27. OH HAI
JVM

28. Java (JRuby) decoding
AMQP is, however
much much faster than
perl doing that...
JVM+-

29. Logstash on each host
is totally out...
• Running it on elasticsearch servers which
are already dedicated to this is fine..
• I’d still like to reuse all of it’s parsing

30. This talk
• Is about my new library: Message::Passing
• The clue is in the name...
• Hopefully really simple
• Maybe even useful!

31. Wait a second!
• My app logs are already structured!
• Why don’t I just publish AMQP from the
app

32. Good question!
• I tried that.
• App logging relies on RabbitMQ being up
• Adds a single point of failure.
• Logging isn’t that important!
• ZeroMQ to the rescue

33. ZeroMQ has the
correct semantics
• Pub/Sub sockets
• Never, ever blocking
• Lossy! (If needed)
• Buffer sizes / locations configureable
• Arbitrary message size
• IO done in a background thread

34. On host log collector
• ZeroMQ SUB socket
• App logs - pre structured
• Syslog listener
• Forward rsyslogd
• Log file tailer
• Ship to AMQP

35. On host log collector

36. Lets make it generic!
• So, I wanted a log shipper
• I ended up with a framework for messaging
interoperability
• Whoops!
• Got sick of writing scripts..

37. Events - my model for
message passing
• a hash {}
• Output consumes events:
• method consume ($event) { ...
• Input produces events:
• has output_to => (..
• Filter does both

38. Simplifying assumption
$self->output_to->consume($message)

39. Events

40. That’s it.
• No, really - that’s all the complexity you
have to care about!
• Except for the complexity introduced by
the inputs and outputs you use.
• Unified attribute names / reconnection
model, etc.. This helps, somewhat..

41. Inputs and outputs
• ZeroMQ In / Out
• AMQP (RabbitMQ) In / Out
• STOMP (ActiveMQ) In / Out
• elasticsearch Out
• Redis PubSub In/Out
• Syslog In
• HTTP POST (“WebHooks”) Out

42. DSL
• Building more complex
chains easy!
• Multiple inputs
• Multiple outputs
• Multiple independent chains

43. CLI
• 1 Input
• 1 Output
• 1 Filter (default Null)
• For simple use, or testing.

44. CLI
• Encode / Decode step is just a Filter
• JSON by default

45. Questions?

46. Questions?
I can build my log shipper, without using 1/2
Gb of RAM.

47. Questions?
I built my log shipper.

48. Questions?
24Mb

49. Demo?

50. Demo?

51. Does this actually
work?
• YES - In production at two sites.

52. Does this actually
work?
• YES - In production at two sites.
• Some of the adaptors are partially
complete

53. Does this actually
work?
• YES - In production at two sites.
• Some of the adaptors are partially
complete
• Dumber than logstash - no multiple
threads/cores

54. Does this actually
work?
• YES - In production at two sites.
• Some of the adaptors are partially
complete
• Dumber than logstash - no multiple
threads/cores
• ZeroMQ is insanely fast

55. Other people are using
it in production!
Two people I know of already writing adaptors!

56. What about logstash?
• Use my lightweight code on end nodes.
• Use ‘proper’ logstash for parsing/filtering
on the dedicated hardware (elasticsearch
boxes)
• Filter to change my hashes to logstash
compatible hashes
• For use with MooseX::Storage and/or
Log::Message::Structured

57. Other applications
• Anywhere an asynchronous event stream is
useful!
• Monitoring
• Metrics transport
• Queued jobs

58. Other applications
(Web stuff)
• User activity (ajax ‘what are your users
doing’)
• WebSockets / MXHR
• HTTP Push notifications - “WebHooks”

59. WebHooks
• HTTP PUSH notification
• E.g. Paypal IPN
• Shopify API

60. Messaging patterns
• Pub / Sub (AMQP / STOMP / Redis / ZMQ)
• Round robin (AMQP / STOMP / Redis /
ZMQ)
• Partial subscribe - ‘routing keys’
• AMQP - Best at this, wildcards anywhere
• Redis - wildcards as suffix
• ZMQ - Exact match

61. Demo?

62. Jenga?

63. Jenga!

64. Demo?
• The last demo wasn’t silly enough!
• How could I top that?
• Plan - Re-invent mongrel2
• Badly

65. PSGI
• PSGI $env is basically just a hash.
• (With a little fiddling), you can serialize it as
JSON
• PSGI response is just an array.
• Ignore streaming responses!

66. PUSH socket does fan
out between multiple
handlers.
Reply to address
embedded in request

67. Code
• https://metacpan.org/module/
Message::Passing
• https://github.com/suretec/Message-Passing
• #message-passing on irc.perl.org
• Demo examples:
• git://gist.github.com/2941747.git