Cooking a rabbit pie

1,799 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,799
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
17
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Cooking a rabbit pie

    1. 1. Real time system performance monitoring AMQP and CatalystX::JobServer Cooking a Rabbit pie Tomas (t0m) Doran São Paulo.pm perl workshop 2010 London perl workshop 2010
    2. 2. I accidentally wrote software• I try to avoid doing this• I’m not very good at it ;)
    3. 3. Aka the “my code stilldoesn’t work yet” talk• This is the 2nd time I’ve talked about AMQP stuff this year.• Half the code still isn’t properly production ready.• It’s an order of magnitude better than last time round ;)
    4. 4. Message queueing• Lets not talk about the code just yet...• This talk is about Message Queueing and specifically AMQP.• I’m going to assume if you knew nothing, you went to the earlier talk - not going to duplicate too much.
    5. 5. AMQP Concepts RabbitMQ vhost Publisher Exchange Queue Consumer
    6. 6. AMQP• All your clients know (at least half) of the wiring.• Different topologies depending on routing configuration.• Can specify other options such as durability• Nice when your server dies - no ‘current config’
    7. 7. Routing keys• Each message sent to an exchange has a routing key.• Each queue can be bound to exchanges with routing keys• I.e. you can subscribe to thingsilike.meat and thingsilike.beer on an exchange, but not subscribe to thingsilike.classicalmusic• Wildcards - # and *
    8. 8. AMQP Delivery modes• Two modes of consuming messages • Get - Gets a single message from the queue • Subscribe - server sends messages from the queue to the client until cancelled• Durability - exchange and queue must agree• Explicit ACK possible but not required
    9. 9. Non-trivial message queueing• Flexible topologies.• Each queue can bind to multiple exchanges, with multiple routing keys.• Routing can be dynamic.• E.g. one client can ‘tail’ a log, but then re- bind with a different routing key to get a different subset of messages.
    10. 10. Custom Exchanges• RabbitMQ allows pluggable exchange types• Simplest and most useful example is the ‘emit last message on bind’ exchange• New consumers get the last message seen on the exchange
    11. 11. I wanted to play with RabbitMQ• I blame Net::RabbitFoot• It’s written using AnyEvent• Which I hadn’t used before, but looked good for this type of thing.• I felt my way around writing simple things standalone.• Got all the web servers logging into Rabbit
    12. 12. Logging into message queuing• Good example of broadcast• Want to aggregate logs to files• And be able to ‘tail’ them• Logging directly from the application• Also tailing (normal) log files to message queue
    13. 13. Practical example:Getting the logs back
    14. 14. Ok, there is some setup
    15. 15. Ok, there is some setup
    16. 16. Dumping them to a file• That’s pretty simple after that..• Except: • Log rotation • Not flushing to disk once per message • etc...
    17. 17. Viewing them live• Someone wrote an AMQP client in flash• AMQP security model not useful publicly• Cute prototype• (Sorry no live demo - it hated me when I tried to make it work again)
    18. 18. Queueing Jobs• New skool scripts - MX::Getopt and a - >run method• Add MooseX::Storage• You can flatten a script as JSON, send it over the wire, re-inflate it, call the ->run method.
    19. 19. Message Queueing Framework?• I now have several scripts, all doing bits with queuing. All duplicating code.• I want to run batch jobs• I want to aggregate log messages (e.g. average web requests per min).• I want to log messages to file(s)• I want to broadcast (and log) aggregates• Need something more generic
    20. 20. Most urgent problem: Job Server• Wrote a simple abstraction on getting channels, making exchanges and queues, etc• Was still going to end up with a load of scripts/jobs that needed running and managing.Yuk.• Inspecting the state of each job ‘service’ hard / boring..
    21. 21. But wait• Each ‘thing’ (timer, listener, logger, emitter, worker etc) an instance with state in attributes - traits => [‘Serialize’]• Construct a instances from config file.• One process managing each machine’s tasks.• Less processes to manage, win. Working out the state just got harder still, FAIL.
    22. 22. But wait• Make all the classes use MooseX::Storage• Catalyst makes instances of things from config...• Build Catalyst app from config file at boot• 2 Trivial controllers introspect entire app• State monitoring (nagios, munin, debugging) now trivial
    23. 23. Really really trivial
    24. 24. Hubris• I could have just used Gearman for my jobs.• I could have come up with a simpler solution for logging and aggregation.• I was having fun.• Until I tried running it in production.
    25. 25. Hubris• I could have just used Gearman for my jobs.• I could have come up with a simpler solution for logging and aggregation.• I was having fun.• Until I tried running it in production.
    26. 26. Hubris• I could have just used Gearman for my jobs.• I could have come up with a simpler solution for logging and aggregation.• I was having fun.• It crashed and burned in production.• sys programming, async, pluggable, FUUUU
    27. 27. Hubris• I could have just used Gearman for my jobs.• I could have come up with a simpler solution for logging and aggregation.• I was having fun.• It crashed and burned in production.• sys programming, async, pluggable, FUUUU
    28. 28. I learnt a lot• I knew a lot in theory before. Doing it is somewhat harder :)• It doesn’t crash any more.• Or leak fds• Still not perfect (e.g. doesn’t reconnect right if mq falls over)
    29. 29. Why laziness didn’t win.• Brad’s code is great, until you try not being Livejournal.• My day job is nothing like Livejournal.• I was still very excited about AMQP• I had a talk to write for a conference (and I could avoid writing it by writing software)• Hippie rocked my world
    30. 30. Web 2.0• Lots of Javascript, update pages dynamically• Messages already JSON from MX::Storage• comet - long poll, multipart xhr• Joose.Storage - inflate your objects in Javascript• Present data from message queues to the user as it becomes available.• Hippie - painless comet - rocked my world
    31. 31. Web::Hippie• Async pipe to the browser.• Abstracts all the nasty ajax details (also does long poll, or websockets)• Applications (I had a practical use for): • Interactive log tail. Realtime systems graphs. • Instant feedback from long running batch jobs • ‘Social’ features by broadcasting/aggregating data
    32. 32. Job Statuses• We inflate JSON data to $job• $job->run($status_cb);• $status_cb->( CompletionEstimate- >new( percent => 50 ) );
    33. 33. Job Statuses• All jobs have a UUID• Job statuses - ‘Running/Complete/StatusLine/ CompletionEstimate/RunJob’• Ask for a pipe to some UUID(s)• Draw nice progress indicators• Further jobs can be included (RunJob magic)• Custom statuses trivial - perl class with attributes, Javascript class with display logic
    34. 34. Wait a second• I don’t want the unwashed masses making HTTP connections to machines running jobs.• Ergo: Send all the job statuses to an exchange, use UUID as routing key.• Optional ‘Hippies’ controller - client produces a set of keys, these sprintf => routing keys.• Hippie pipe => one queue per-client queue bound to keys they want.• ‘RunJob’ messages automatically binds extra key, so you see things triggered by things you are watching.
    35. 35. How it all hangs together
    36. 36. Useable?• Running jobs works.• Running it in production at work. It doesn’t crash any more.• Status pipe stuff not deployed to clients yet.• If you need a high volume, simple, production ready job scheduler, right now, use Gearman.
    37. 37. Demo?
    38. 38. Demo 1• Simple job server• Enqueues 10 jobs at start• 1 worker process• JSON status of app• Add workers dynamically
    39. 39. Demo 2• Publish component status to queue regularly• Simple CLI script tailing queue• Jobs indexed by UUID - allows Hippie
    40. 40. Demo 3• Status updates from ‘Job Server’ published.• 2nd ‘/hippies’ process binds queue to exchange for some UUIDs.• Gets ‘RunJob’ notifications, and statuses when they run.
    41. 41. Next steps?• Reliability - recover from errors better.• Expose more stats about MQ use.• Better (some!) logging.• Docs would be good...• Hippe::Pipe for bidirectional• More traits / plugins / components• More / better stock Javascript
    42. 42. What to go look at• http://www.rabbitmq.com• Docs, setup info, FAQ• Slideshare - lots of good and more detailed AMQP introductions.
    43. 43. What to go look at (CPAN)• Net::RabbitFoot• App::RabbitTail• Plack / Twiggy• Web::Hippie• CatalystX::JobServer (github.com/bobtfish/ CatalystX-JobServer)
    44. 44. Other solutions• Gearman• STOMP • ActiveMQ (or Rabbit has a STOMP adaptor - good luck) • Catalyst::Engine::STOMP
    45. 45. Thanks!• Questions?• Anyone interested in playing, feel free to grab me on irc (t0m @ magnet & Freenode)• Happy to hand-hold as I need more people using this.• I’ll love you forever if you write docs.

    ×