Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

RabbitMQ Plugins Talk

4,749 views

Published on

Writing Erlang plugins for RabbitMQ.

  • Be the first to comment

  • Be the first to like this

RabbitMQ Plugins Talk

  1. 1. Writing Plugins With RabbitMQ<br />Noah Gift and Michael Vierling<br />
  2. 2. Noah’s Background<br />Film<br />Commercial Data Center Automation Software<br />Los Angeles, Atlanta, New Zealand, San Francisco<br />I like programming in ANY language that is innovative<br />
  3. 3. How & Why<br />Why AMQP?<br />Why Erlang?<br />Why OTP?<br />Why RabbitMQ?<br />How – Machinery<br />
  4. 4. Why AMQP<br />Industry Message Standard<br />Cisco, Microsoft, Red Hat, VMWare, JPMorgan, Chase, Goldman Sachs<br />Binary wire protocal<br />
  5. 5. Why AMQP: Continued<br />Key component in Distributed Architecture<br />Enables Multi-OS, Multi-Language<br />Reliable, Transactional, Flow-Control, Fault-tolerant<br />
  6. 6. Downsides of AMQP<br />AMQP 1.0 is almost final<br />Support resources<br />Upgrade glitches<br />Monitoring issues<br />Investment in the future<br />
  7. 7. Why Erlang?<br />Functional Language<br />Immutable Data<br />Scalable<br />Reliable, Fault-tolerant<br />
  8. 8. Why RabbitMQ<br />Built with Erlang<br />10K messages per second – transient mode<br />3-5K messages per second – persistent mode<br />
  9. 9. Why RabbitMQ<br />Transactions<br />Open Source<br />Maintained by VMWare<br />
  10. 10. Broker Infrastructure<br />
  11. 11. Broker Infrastructure<br />Repeatable Install<br />Reliable storage<br />Redundant hardware<br />
  12. 12. Broker Infrastructure: Continued<br />Cluster of nodes<br />Disaster Recovery<br />Logging (such as Splunk) and monitoring<br />Operations & Support<br />
  13. 13. Broker Infrastructure: Use Escript<br />
  14. 14. Broker InfrastructureFuture?<br />Integrate Rebar<br />Continuous Integration Testing<br />QuickCheck<br />
  15. 15. RabbitMQ Clients<br />Any AMQP Clients (Apache)<br />Ruby<br />Python<br />C#<br />F# (Wrap the C# dll)<br />
  16. 16. Consume.py Part: A<br />
  17. 17. Consume.py Part: B<br />
  18. 18. Consume.py Part: C<br />
  19. 19. Send.py Part: A<br />
  20. 20. Send.py Part: B<br />
  21. 21. Writing RabbitMQ Plugin: WHY<br />Access internal RabbitMQ functionality<br />Running in same Erlang VM as broker can increase performance <br />Leverage your existing infrastructure<br />
  22. 22. Writing RabbitMQ Plugin: WHY NOT<br />Developers don’t know Erlang<br />Poorly written plugin could crash broker<br />You could lock yourself to internal API<br />
  23. 23. StableRabbitMQ Plugin<br />Rabbitmq-management (replaced Alice)<br />Rabbitmq-shovel<br />Rabbitmq-stomp<br />Rabbitmq-erlang-client<br />
  24. 24. ExperimentalRabbitMQ Plugin Ideas<br />Rabbitmq-auth-backend-ldap<br />Rabbitmq-auth-mechanism-ssl<br />Rabbitmq-jsonrpc-channel<br />Rabbitmq-xmpp<br />
  25. 25. CrazyRabbitMQ Plugin Ideas<br />Distributed Data Structure<br />Artificial Intelligence<br />Data Mining<br />Create a protocol on top<br />Event Processing<br />
  26. 26. Michael Background<br />Apple 5 Years<br />Ascend/Lucent 4 Years<br />Various Startups<br />
  27. 27. Supervision Tree<br />
  28. 28. Plugin Supervisor<br />
  29. 29. Rabbit RSS Supervisor<br />
  30. 30. Rabbit RSS Worker: Slide A<br />
  31. 31. Rabbit RSS Worker: Slide B<br />
  32. 32. Rabbit RSS Worker: Slide C<br />
  33. 33. Rabbit RSS Worker: Slide D<br />
  34. 34. Rabbit RSS Worker: Slide E<br />
  35. 35. Rabbit RSS Worker: Slide F<br />
  36. 36. XML Parsing<br />
  37. 37. RSS.ERL: Slide A<br />
  38. 38. RSS.ERL: Slide B<br />
  39. 39. RSS.ERL: Slide C<br />
  40. 40. RSS.ERL: Slide D<br />
  41. 41. Checkout source: HG clone<br />
  42. 42. Make: Be Patient….<br />
  43. 43. Build<br />
  44. 44. Skeleton<br />
  45. 45. Symlink Plugin<br />
  46. 46. Run Broker<br />$ make run<br />(rabbit@rabbit9)1><br />(rabbit@rabbit9)1>application:which_applications().<br />{rabbit_rss,"Embedded Rabbit RSS Reader","0.01"},<br />
  47. 47. Test Broker<br />$ mkdirtest<br />$ vi test/rabbit_rss_tests.erl<br />$ vi Makefile<br />TEST_APPS=amqp_clientrabbit_rss<br />TEST_COMMANDS=rabbit_rss_tests:test() START_RABBIT_IN_TESTS=true<br />$ make test<br />
  48. 48. Package Plugin<br />
  49. 49. Plugin Supporting Machinery<br />Unit Tests<br />Source Code<br />Logger<br />Printf<br />AMQP Clients<br />
  50. 50. How To Build a Plugin<br />RabbitMQ docs<br />Source<br />Examples<br />
  51. 51. Use the Source!<br />Documentation is hit/miss<br />Google “erlang lists”<br />http://www.erlang.org/doc/apps/stdlib/<br />Or skip the docs and go directly to source<br />./lib/stdlib/src/lists.erl<br />
  52. 52. Lists.erl<br />
  53. 53. Logger<br />Plugin executes in its own process<br />No plugin stdout<br />No plugin execution debugger<br />
  54. 54. Logger: Continued<br />Solution: Erlang’s logger<br />Output to /var/log/rabbitmq/<hostname<br />Nothing to configure<br />
  55. 55. RSS.erl<br />
  56. 56. OTP Design Principles<br />Supervision Tree<br />Behaviors<br />Applications<br />Releases<br />Release Handling<br />
  57. 57. Machine Learning<br />The next step for the RSS plugin.<br />A potential use case for RabbitMQ plugins.<br />
  58. 58. Fisher Classifier For RSS Feeds<br />Data mining<br />Classification<br />
  59. 59. Fisher Classifier: Slide A<br />
  60. 60. Fisher Classifier: Slide B<br />
  61. 61. Fisher Classifier: Slide C<br />
  62. 62. Fisher Classifier: Slide D<br />
  63. 63. Fisher Classifier: Slide E<br />
  64. 64. Fisher Classifier: Slide E<br />
  65. 65. Fisher Classifier: Slide F<br />
  66. 66. Fisher Classifier: Slide G<br />
  67. 67. Conclusion<br />AMQP<br />Erlang<br />Python<br />RSS<br />
  68. 68. Questions<br />Code: https://github.com/mikev/rabbitmq_rss_plugin<br />Contact: “Noah Gift” noah.gift@gmail.com, “Michael Vierling” mvierling@gmail.com<br />

×