Real time system_performance_mon

  • 5,095 views
Uploaded on

 

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
5,095
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
62
Comments
0
Likes
4

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide











































Transcript

  • 1. Real time system performance monitoring AMQP and CatalystX::JobServer Tomas (t0m) Doran São Paulo.pm perl workshop 2010
  • 2. Aka the “my code doesn’t work yet” talk • I wrote half the slides for this at 6am Friday (São Paulo time) • The title is a bit misleading, as I wrote the talk abstract in March, then wrote entirely different software. • Sorry about that.
  • 3. Message queueing • Lets not talk about the (not working) code just yet... • This talk is about Message Queueing. • What is Message Queueing? • Why would I want to use it?
  • 4. What is message queueing. • In it’s simplest form, it’s just a list. • 1 (or more) ‘producers’ (writers) • 1 (or more) ‘consumers’ (readers) • Queue if rate of production > rate of consumption
  • 5. Why do I want it? • Decouples producers and consumers (probably across the network). • Lets you manage load, and spikes in load (only n consumers). • In a web environment - lets you serve more pages, quicker.
  • 6. Other cases • View as a list a little simplistic. • Depending on your application, you may want: • One to one • One to many • One to many (broadcast) • Many to all of the above • More complex topology?
  • 7. AMQP • Queuing protocol • Fast • Interoperable (maybe?)
  • 8. AMQP AMQP enables complete interoperability for messaging middleware, both the networking protocol and the semantics of broker services are defined in AMQP.
  • 9. AMQP Say what?
  • 10. AMQP • Full abstraction of all the mechanics of messaging across the network. • Serialization and framing of messages. • Wiring of message routing is part of the protocol.
  • 11. AMQP • So all your clients know (at least half) of the wiring. • Different topologies depending on routing configuration. • Can specify other options such as durability • Nice when your server dies - no ‘current config’
  • 12. AMQP Concepts RabbitMQ vhost Publisher Exchange Queue Consumer
  • 13. Concepts - Exchanges • Named • Messages are published (sent) to one exchange • Can be durable (lasts till deleted), or auto- deleted
  • 14. Queues • Queues can be named or (or named by the server - private queues). • Queues are bound to one (or more) exchanges. • Queues can have 0 or more clients • Queues may persist with 0 clients or not. • FIFO (if you have 1 consumer) • Message never delivered to > 1 client
  • 15. Bindings • Binding is what joins a queue and an exchange.
  • 16. Traditional queue • Named exchange • Bound to 1 Named queue • 0 or more listeners get round-robin messages • Messages queue when nobody listens / if consumers are slow
  • 17. Broadcast • Named exchange • Each client creates an anonymous ephermeral queue • Client binds the queue to that exchange • All clients get all messages • Messages go to /dev/null if no clients
  • 18. Others • Different exchange types - direct, topic, fanout • You can do a lot using just these and a mix of named and anonymous queues • More complex topologies possible
  • 19. I accidentally wrote software • I try to avoid doing this • I’m not very good at it ;)
  • 20. I blame Net::RabbitFoot • It’s written using AnyEvent and Coro • Which I hadn’t used before, but looked good for this type of thing. • I felt my way around writing simple things standalone. • Got all the web servers logging into Rabbit
  • 21. Logging into message queuing • Good example of broadcast • Want to aggregate logs to files • And be able to ‘tail’ them • Logging directly from the application • Also tailing (normal) log files to message queue
  • 22. Getting the logs back
  • 23. Ok, there is some setup
  • 24. Ok, there is some setup
  • 25. Dumping them to a file • That’s pretty simple after that.. • Except: • Log rotation • Not flushing to disk once per message • etc...
  • 26. Viewing them live • Someone wrote an AMQP client in flash • AMQP security model not useful publicly • Cute prototype • (Sorry, no demo - refuses to work locally)
  • 27. Queueing Jobs • Next application, simple job queue. • 1 worker, serialized.. (Crap, good enough) • Log JSON into RabbitMQ (again), inflate, call the ->run method. • Works!
  • 28. Job Server • I now have several scripts, all doing bits with queuing. All duplicating code. • I want to aggregate messages • Send messages of aggregates at regular intervals • Need something more generic
  • 29. Job server • Wrote a simple abstraction on getting channels, making exchanges and queues, etc • Was still going to end up with a load of scripts/jobs that needed running and managing.Yuk. • Inspecting the state of each job ‘service’ hard / boring..
  • 30. But wait • Each ‘thing’ (timer, listener, logger, emitter, worker etc) can be an instance of a class with attributes for the state. • Construct a set from some config file. • Lets me aggregate jobs together / share code (Can use AnyEvent/Coro for threads). • Less processes to manage. Working out the state just got harder still.
  • 31. But wait • Make all the classes use MooseX::Storage • Catalyst makes instances of things from config... • So, I could adapt all these instances into Catalyst models, write a controller, done! • Erm, but Catalyst, and message queueing, and what?!?!?
  • 32. I blame Miyagawa • For Plack, and Catalyst::Engine::Plack, and Corona (the Coro PSGI server). • I had this INSANE idea one night. • I tried it. • IT WORKED. • WOAH.
  • 33. I blame clkao • Next step - browser kicks off a job. • Status via long poll? • He did it already! • With long hair!
  • 34. Hippie • Async pipe to the browser. • Abstracts all the nasty ajax details (also does long poll, or websockets) • Applications: • Automatically updated monitor of app components • Start a job and then get an AJAX update screen for the job progress.
  • 35. Useable? • The basics work. • Running it production at work • (Since WedneThursday....) • If you need a high volume, simple, production ready job scheduler right now, use Gearman.
  • 36. Useable? • Needs my github versions of Catalyst::Engine::PSGI and Web::Hippie • Running jobs from the web app doesn’t work/exist yet. • All the pieces are there - next week?
  • 37. Demo?
  • 38. Next steps? • Production battle scars ;) • Hippie for watching jobs run, not just component status updates. • Docs would be good... • Hippe::Pipe for bidirectional • More traits / plugins / components
  • 39. What to go look at • http://www.rabbitmq.com • Docs, setup info, FAQ • Slideshare - lots of good and more detailed AMQP introductions.
  • 40. What to go look at (CPAN) • Net::RabbitFoot • App::RabbitTail • Plack • Web::Hippie • CatalystX::JobServer (github)
  • 41. Bonus slide! • I forgot to mention this originally... • DO NOT FORK AFTER USING Net::RabbitFoot (and then try to use it in the child process) • This will go horribly horribly wrong.
  • 42. Other solutions • Gearman • STOMP • ActiveMQ (or Rabbit has a STOMP adaptor) • Catalyst::Engine::STOMP
  • 43. Thanks! • Questions? • Anyone interested in playing, grab me on irc • Happy to hand-hold as I need more people using this. • I’ll love you forever if you write docs.