This document discusses using message queues and AMQP for real-time system performance monitoring and job queuing. It describes how message queues allow producers and consumers to communicate asynchronously. The author details their experience building a job server using RabbitMQ, Catalyst, and Web::Hippie to queue and monitor long-running jobs. While the system is still a work in progress, all the major components are there and it is in use in production.

1. Real time system
performance monitoring
AMQP and
CatalystX::JobServer
Tomas (t0m) Doran
São Paulo.pm perl workshop 2010

2. Aka the “my code
doesn’t work yet” talk
• I wrote half the slides for this at 6am Friday
(São Paulo time)
• The title is a bit misleading, as I wrote the
talk abstract in March, then wrote entirely
different software.
• Sorry about that.

3. Message queueing
• Lets not talk about the (not working) code
just yet...
• This talk is about Message Queueing.
• What is Message Queueing?
• Why would I want to use it?

4. What is message
queueing.
• In it’s simplest form, it’s just a list.
• 1 (or more) ‘producers’ (writers)
• 1 (or more) ‘consumers’ (readers)
• Queue if rate of production > rate of
consumption

5. Why do I want it?
• Decouples producers and consumers
(probably across the network).
• Lets you manage load, and spikes in load
(only n consumers).
• In a web environment - lets you serve
more pages, quicker.

6. Other cases
• View as a list a little simplistic.
• Depending on your application, you may
want:
• One to one
• One to many
• One to many (broadcast)
• Many to all of the above
• More complex topology?

7. AMQP
• Queuing protocol
• Fast
• Interoperable (maybe?)

8. AMQP
AMQP enables complete
interoperability for messaging
middleware, both the networking
protocol and the semantics of broker
services are defined in AMQP.

9. AMQP
Say what?

10. AMQP
• Full abstraction of all the mechanics of
messaging across the network.
• Serialization and framing of messages.
• Wiring of message routing is part of the
protocol.

11. AMQP
• So all your clients know (at least half) of
the wiring.
• Different topologies depending on routing
configuration.
• Can specify other options such as durability
• Nice when your server dies - no ‘current
config’

12. AMQP Concepts
RabbitMQ
vhost
Publisher Exchange
Queue
Consumer

13. Concepts - Exchanges
• Named
• Messages are published (sent) to one
exchange
• Can be durable (lasts till deleted), or auto-
deleted

14. Queues
• Queues can be named or (or named by the
server - private queues).
• Queues are bound to one (or more)
exchanges.
• Queues can have 0 or more clients
• Queues may persist with 0 clients or not.
• FIFO (if you have 1 consumer)
• Message never delivered to > 1 client

15. Bindings
• Binding is what joins a queue and an
exchange.

16. Traditional queue
• Named exchange
• Bound to 1 Named queue
• 0 or more listeners get round-robin
messages
• Messages queue when nobody listens / if
consumers are slow

17. Broadcast
• Named exchange
• Each client creates an anonymous
ephermeral queue
• Client binds the queue to that exchange
• All clients get all messages
• Messages go to /dev/null if no clients

18. Others
• Different exchange types - direct, topic,
fanout
• You can do a lot using just these and a mix
of named and anonymous queues
• More complex topologies possible

19. I accidentally wrote
software
• I try to avoid doing this
• I’m not very good at it ;)

20. I blame Net::RabbitFoot
• It’s written using AnyEvent and Coro
• Which I hadn’t used before, but looked
good for this type of thing.
• I felt my way around writing simple things
standalone.
• Got all the web servers logging into Rabbit

21. Logging into message
queuing
• Good example of broadcast
• Want to aggregate logs to files
• And be able to ‘tail’ them
• Logging directly from the application
• Also tailing (normal) log files to message
queue

22. Getting the logs back

23. Ok, there is some setup

24. Ok, there is some setup

25. Dumping them to a file
• That’s pretty simple after that..
• Except:
• Log rotation
• Not flushing to disk once per message
• etc...

26. Viewing them live
• Someone wrote an AMQP client in flash
• AMQP security model not useful publicly
• Cute prototype
• (Sorry, no demo - refuses to work locally)

27. Queueing Jobs
• Next application, simple job queue.
• 1 worker, serialized.. (Crap, good enough)
• Log JSON into RabbitMQ (again), inflate,
call the ->run method.
• Works!

28. Job Server
• I now have several scripts, all doing bits
with queuing. All duplicating code.
• I want to aggregate messages
• Send messages of aggregates at regular
intervals
• Need something more generic

29. Job server
• Wrote a simple abstraction on getting
channels, making exchanges and queues, etc
• Was still going to end up with a load of
scripts/jobs that needed running and
managing.Yuk.
• Inspecting the state of each job ‘service’
hard / boring..

30. But wait
• Each ‘thing’ (timer, listener, logger, emitter,
worker etc) can be an instance of a class
with attributes for the state.
• Construct a set from some config file.
• Lets me aggregate jobs together / share
code (Can use AnyEvent/Coro for threads).
• Less processes to manage. Working out the
state just got harder still.

31. But wait
• Make all the classes use MooseX::Storage
• Catalyst makes instances of things from
config...
• So, I could adapt all these instances into
Catalyst models, write a controller, done!
• Erm, but Catalyst, and message queueing,
and what?!?!?

32. I blame Miyagawa
• For Plack, and Catalyst::Engine::Plack, and
Corona (the Coro PSGI server).
• I had this INSANE idea one night.
• I tried it.
• IT WORKED.
• WOAH.

33. I blame clkao
• Next step - browser kicks off a job.
• Status via long poll?
• He did it already!
• With long hair!

34. Hippie
• Async pipe to the browser.
• Abstracts all the nasty ajax details (also
does long poll, or websockets)
• Applications:
• Automatically updated monitor of app
components
• Start a job and then get an AJAX update
screen for the job progress.

35. Useable?
• The basics work.
• Running it production at work
• (Since WedneThursday....)
• If you need a high volume, simple,
production ready job scheduler right
now, use Gearman.

36. Useable?
• Needs my github versions of
Catalyst::Engine::PSGI and Web::Hippie
• Running jobs from the web app doesn’t
work/exist yet.
• All the pieces are there - next week?

37. Demo?

38. Next steps?
• Production battle scars ;)
• Hippie for watching jobs run, not just
component status updates.
• Docs would be good...
• Hippe::Pipe for bidirectional
• More traits / plugins / components

39. What to go look at
• http://www.rabbitmq.com
• Docs, setup info, FAQ
• Slideshare - lots of good and more detailed
AMQP introductions.

40. What to go look at
(CPAN)
• Net::RabbitFoot
• App::RabbitTail
• Plack
• Web::Hippie
• CatalystX::JobServer (github)

41. Bonus slide!
• I forgot to mention this originally...
• DO NOT FORK AFTER USING
Net::RabbitFoot (and then try to use it in
the child process)
• This will go horribly horribly wrong.

42. Other solutions
• Gearman
• STOMP
• ActiveMQ (or Rabbit has a STOMP
adaptor)
• Catalyst::Engine::STOMP

43. Thanks!
• Questions?
• Anyone interested in playing, grab me on irc
• Happy to hand-hold as I need more people
using this.
• I’ll love you forever if you write docs.