Your SlideShare is downloading. ×
0
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Real time system_performance_mon
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Real time system_performance_mon

5,412

Published on

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
5,412
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
63
Comments
0
Likes
4
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide











































  • Transcript

    • 1. Real time system performance monitoring AMQP and CatalystX::JobServer Tomas (t0m) Doran São Paulo.pm perl workshop 2010
    • 2. Aka the “my code doesn’t work yet” talk • I wrote half the slides for this at 6am Friday (São Paulo time) • The title is a bit misleading, as I wrote the talk abstract in March, then wrote entirely different software. • Sorry about that.
    • 3. Message queueing • Lets not talk about the (not working) code just yet... • This talk is about Message Queueing. • What is Message Queueing? • Why would I want to use it?
    • 4. What is message queueing. • In it’s simplest form, it’s just a list. • 1 (or more) ‘producers’ (writers) • 1 (or more) ‘consumers’ (readers) • Queue if rate of production > rate of consumption
    • 5. Why do I want it? • Decouples producers and consumers (probably across the network). • Lets you manage load, and spikes in load (only n consumers). • In a web environment - lets you serve more pages, quicker.
    • 6. Other cases • View as a list a little simplistic. • Depending on your application, you may want: • One to one • One to many • One to many (broadcast) • Many to all of the above • More complex topology?
    • 7. AMQP • Queuing protocol • Fast • Interoperable (maybe?)
    • 8. AMQP AMQP enables complete interoperability for messaging middleware, both the networking protocol and the semantics of broker services are defined in AMQP.
    • 9. AMQP Say what?
    • 10. AMQP • Full abstraction of all the mechanics of messaging across the network. • Serialization and framing of messages. • Wiring of message routing is part of the protocol.
    • 11. AMQP • So all your clients know (at least half) of the wiring. • Different topologies depending on routing configuration. • Can specify other options such as durability • Nice when your server dies - no ‘current config’
    • 12. AMQP Concepts RabbitMQ vhost Publisher Exchange Queue Consumer
    • 13. Concepts - Exchanges • Named • Messages are published (sent) to one exchange • Can be durable (lasts till deleted), or auto- deleted
    • 14. Queues • Queues can be named or (or named by the server - private queues). • Queues are bound to one (or more) exchanges. • Queues can have 0 or more clients • Queues may persist with 0 clients or not. • FIFO (if you have 1 consumer) • Message never delivered to > 1 client
    • 15. Bindings • Binding is what joins a queue and an exchange.
    • 16. Traditional queue • Named exchange • Bound to 1 Named queue • 0 or more listeners get round-robin messages • Messages queue when nobody listens / if consumers are slow
    • 17. Broadcast • Named exchange • Each client creates an anonymous ephermeral queue • Client binds the queue to that exchange • All clients get all messages • Messages go to /dev/null if no clients
    • 18. Others • Different exchange types - direct, topic, fanout • You can do a lot using just these and a mix of named and anonymous queues • More complex topologies possible
    • 19. I accidentally wrote software • I try to avoid doing this • I’m not very good at it ;)
    • 20. I blame Net::RabbitFoot • It’s written using AnyEvent and Coro • Which I hadn’t used before, but looked good for this type of thing. • I felt my way around writing simple things standalone. • Got all the web servers logging into Rabbit
    • 21. Logging into message queuing • Good example of broadcast • Want to aggregate logs to files • And be able to ‘tail’ them • Logging directly from the application • Also tailing (normal) log files to message queue
    • 22. Getting the logs back
    • 23. Ok, there is some setup
    • 24. Ok, there is some setup
    • 25. Dumping them to a file • That’s pretty simple after that.. • Except: • Log rotation • Not flushing to disk once per message • etc...
    • 26. Viewing them live • Someone wrote an AMQP client in flash • AMQP security model not useful publicly • Cute prototype • (Sorry, no demo - refuses to work locally)
    • 27. Queueing Jobs • Next application, simple job queue. • 1 worker, serialized.. (Crap, good enough) • Log JSON into RabbitMQ (again), inflate, call the ->run method. • Works!
    • 28. Job Server • I now have several scripts, all doing bits with queuing. All duplicating code. • I want to aggregate messages • Send messages of aggregates at regular intervals • Need something more generic
    • 29. Job server • Wrote a simple abstraction on getting channels, making exchanges and queues, etc • Was still going to end up with a load of scripts/jobs that needed running and managing.Yuk. • Inspecting the state of each job ‘service’ hard / boring..
    • 30. But wait • Each ‘thing’ (timer, listener, logger, emitter, worker etc) can be an instance of a class with attributes for the state. • Construct a set from some config file. • Lets me aggregate jobs together / share code (Can use AnyEvent/Coro for threads). • Less processes to manage. Working out the state just got harder still.
    • 31. But wait • Make all the classes use MooseX::Storage • Catalyst makes instances of things from config... • So, I could adapt all these instances into Catalyst models, write a controller, done! • Erm, but Catalyst, and message queueing, and what?!?!?
    • 32. I blame Miyagawa • For Plack, and Catalyst::Engine::Plack, and Corona (the Coro PSGI server). • I had this INSANE idea one night. • I tried it. • IT WORKED. • WOAH.
    • 33. I blame clkao • Next step - browser kicks off a job. • Status via long poll? • He did it already! • With long hair!
    • 34. Hippie • Async pipe to the browser. • Abstracts all the nasty ajax details (also does long poll, or websockets) • Applications: • Automatically updated monitor of app components • Start a job and then get an AJAX update screen for the job progress.
    • 35. Useable? • The basics work. • Running it production at work • (Since WedneThursday....) • If you need a high volume, simple, production ready job scheduler right now, use Gearman.
    • 36. Useable? • Needs my github versions of Catalyst::Engine::PSGI and Web::Hippie • Running jobs from the web app doesn’t work/exist yet. • All the pieces are there - next week?
    • 37. Demo?
    • 38. Next steps? • Production battle scars ;) • Hippie for watching jobs run, not just component status updates. • Docs would be good... • Hippe::Pipe for bidirectional • More traits / plugins / components
    • 39. What to go look at • http://www.rabbitmq.com • Docs, setup info, FAQ • Slideshare - lots of good and more detailed AMQP introductions.
    • 40. What to go look at (CPAN) • Net::RabbitFoot • App::RabbitTail • Plack • Web::Hippie • CatalystX::JobServer (github)
    • 41. Bonus slide! • I forgot to mention this originally... • DO NOT FORK AFTER USING Net::RabbitFoot (and then try to use it in the child process) • This will go horribly horribly wrong.
    • 42. Other solutions • Gearman • STOMP • ActiveMQ (or Rabbit has a STOMP adaptor) • Catalyst::Engine::STOMP
    • 43. Thanks! • Questions? • Anyone interested in playing, grab me on irc • Happy to hand-hold as I need more people using this. • I’ll love you forever if you write docs.

    ×