Real time system_performance_mon


Published on

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

  • Real time system_performance_mon

    1. 1. Real time system performance monitoring AMQP and CatalystX::JobServer Tomas (t0m) Doran São perl workshop 2010
    2. 2. Aka the “my code doesn’t work yet” talk • I wrote half the slides for this at 6am Friday (São Paulo time) • The title is a bit misleading, as I wrote the talk abstract in March, then wrote entirely different software. • Sorry about that.
    3. 3. Message queueing • Lets not talk about the (not working) code just yet... • This talk is about Message Queueing. • What is Message Queueing? • Why would I want to use it?
    4. 4. What is message queueing. • In it’s simplest form, it’s just a list. • 1 (or more) ‘producers’ (writers) • 1 (or more) ‘consumers’ (readers) • Queue if rate of production > rate of consumption
    5. 5. Why do I want it? • Decouples producers and consumers (probably across the network). • Lets you manage load, and spikes in load (only n consumers). • In a web environment - lets you serve more pages, quicker.
    6. 6. Other cases • View as a list a little simplistic. • Depending on your application, you may want: • One to one • One to many • One to many (broadcast) • Many to all of the above • More complex topology?
    7. 7. AMQP • Queuing protocol • Fast • Interoperable (maybe?)
    8. 8. AMQP AMQP enables complete interoperability for messaging middleware, both the networking protocol and the semantics of broker services are defined in AMQP.
    9. 9. AMQP Say what?
    10. 10. AMQP • Full abstraction of all the mechanics of messaging across the network. • Serialization and framing of messages. • Wiring of message routing is part of the protocol.
    11. 11. AMQP • So all your clients know (at least half) of the wiring. • Different topologies depending on routing configuration. • Can specify other options such as durability • Nice when your server dies - no ‘current config’
    12. 12. AMQP Concepts RabbitMQ vhost Publisher Exchange Queue Consumer
    13. 13. Concepts - Exchanges • Named • Messages are published (sent) to one exchange • Can be durable (lasts till deleted), or auto- deleted
    14. 14. Queues • Queues can be named or (or named by the server - private queues). • Queues are bound to one (or more) exchanges. • Queues can have 0 or more clients • Queues may persist with 0 clients or not. • FIFO (if you have 1 consumer) • Message never delivered to > 1 client
    15. 15. Bindings • Binding is what joins a queue and an exchange.
    16. 16. Traditional queue • Named exchange • Bound to 1 Named queue • 0 or more listeners get round-robin messages • Messages queue when nobody listens / if consumers are slow
    17. 17. Broadcast • Named exchange • Each client creates an anonymous ephermeral queue • Client binds the queue to that exchange • All clients get all messages • Messages go to /dev/null if no clients
    18. 18. Others • Different exchange types - direct, topic, fanout • You can do a lot using just these and a mix of named and anonymous queues • More complex topologies possible
    19. 19. I accidentally wrote software • I try to avoid doing this • I’m not very good at it ;)
    20. 20. I blame Net::RabbitFoot • It’s written using AnyEvent and Coro • Which I hadn’t used before, but looked good for this type of thing. • I felt my way around writing simple things standalone. • Got all the web servers logging into Rabbit
    21. 21. Logging into message queuing • Good example of broadcast • Want to aggregate logs to files • And be able to ‘tail’ them • Logging directly from the application • Also tailing (normal) log files to message queue
    22. 22. Getting the logs back
    23. 23. Ok, there is some setup
    24. 24. Ok, there is some setup
    25. 25. Dumping them to a file • That’s pretty simple after that.. • Except: • Log rotation • Not flushing to disk once per message • etc...
    26. 26. Viewing them live • Someone wrote an AMQP client in flash • AMQP security model not useful publicly • Cute prototype • (Sorry, no demo - refuses to work locally)
    27. 27. Queueing Jobs • Next application, simple job queue. • 1 worker, serialized.. (Crap, good enough) • Log JSON into RabbitMQ (again), inflate, call the ->run method. • Works!
    28. 28. Job Server • I now have several scripts, all doing bits with queuing. All duplicating code. • I want to aggregate messages • Send messages of aggregates at regular intervals • Need something more generic
    29. 29. Job server • Wrote a simple abstraction on getting channels, making exchanges and queues, etc • Was still going to end up with a load of scripts/jobs that needed running and managing.Yuk. • Inspecting the state of each job ‘service’ hard / boring..
    30. 30. But wait • Each ‘thing’ (timer, listener, logger, emitter, worker etc) can be an instance of a class with attributes for the state. • Construct a set from some config file. • Lets me aggregate jobs together / share code (Can use AnyEvent/Coro for threads). • Less processes to manage. Working out the state just got harder still.
    31. 31. But wait • Make all the classes use MooseX::Storage • Catalyst makes instances of things from config... • So, I could adapt all these instances into Catalyst models, write a controller, done! • Erm, but Catalyst, and message queueing, and what?!?!?
    32. 32. I blame Miyagawa • For Plack, and Catalyst::Engine::Plack, and Corona (the Coro PSGI server). • I had this INSANE idea one night. • I tried it. • IT WORKED. • WOAH.
    33. 33. I blame clkao • Next step - browser kicks off a job. • Status via long poll? • He did it already! • With long hair!
    34. 34. Hippie • Async pipe to the browser. • Abstracts all the nasty ajax details (also does long poll, or websockets) • Applications: • Automatically updated monitor of app components • Start a job and then get an AJAX update screen for the job progress.
    35. 35. Useable? • The basics work. • Running it production at work • (Since WedneThursday....) • If you need a high volume, simple, production ready job scheduler right now, use Gearman.
    36. 36. Useable? • Needs my github versions of Catalyst::Engine::PSGI and Web::Hippie • Running jobs from the web app doesn’t work/exist yet. • All the pieces are there - next week?
    37. 37. Demo?
    38. 38. Next steps? • Production battle scars ;) • Hippie for watching jobs run, not just component status updates. • Docs would be good... • Hippe::Pipe for bidirectional • More traits / plugins / components
    39. 39. What to go look at • • Docs, setup info, FAQ • Slideshare - lots of good and more detailed AMQP introductions.
    40. 40. What to go look at (CPAN) • Net::RabbitFoot • App::RabbitTail • Plack • Web::Hippie • CatalystX::JobServer (github)
    41. 41. Bonus slide! • I forgot to mention this originally... • DO NOT FORK AFTER USING Net::RabbitFoot (and then try to use it in the child process) • This will go horribly horribly wrong.
    42. 42. Other solutions • Gearman • STOMP • ActiveMQ (or Rabbit has a STOMP adaptor) • Catalyst::Engine::STOMP
    43. 43. Thanks! • Questions? • Anyone interested in playing, grab me on irc • Happy to hand-hold as I need more people using this. • I’ll love you forever if you write docs.