RabbitMQ Plugins Talk

4,389 views
4,237 views

Published on

Writing Erlang plugins for RabbitMQ.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
4,389
On SlideShare
0
From Embeds
0
Number of Embeds
747
Actions
Shares
0
Downloads
32
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

RabbitMQ Plugins Talk

  1. 1. Writing Plugins With RabbitMQ<br />Noah Gift and Michael Vierling<br />
  2. 2. Noah’s Background<br />Film<br />Commercial Data Center Automation Software<br />Los Angeles, Atlanta, New Zealand, San Francisco<br />I like programming in ANY language that is innovative<br />
  3. 3. How & Why<br />Why AMQP?<br />Why Erlang?<br />Why OTP?<br />Why RabbitMQ?<br />How – Machinery<br />
  4. 4. Why AMQP<br />Industry Message Standard<br />Cisco, Microsoft, Red Hat, VMWare, JPMorgan, Chase, Goldman Sachs<br />Binary wire protocal<br />
  5. 5. Why AMQP: Continued<br />Key component in Distributed Architecture<br />Enables Multi-OS, Multi-Language<br />Reliable, Transactional, Flow-Control, Fault-tolerant<br />
  6. 6. Downsides of AMQP<br />AMQP 1.0 is almost final<br />Support resources<br />Upgrade glitches<br />Monitoring issues<br />Investment in the future<br />
  7. 7. Why Erlang?<br />Functional Language<br />Immutable Data<br />Scalable<br />Reliable, Fault-tolerant<br />
  8. 8. Why RabbitMQ<br />Built with Erlang<br />10K messages per second – transient mode<br />3-5K messages per second – persistent mode<br />
  9. 9. Why RabbitMQ<br />Transactions<br />Open Source<br />Maintained by VMWare<br />
  10. 10. Broker Infrastructure<br />
  11. 11. Broker Infrastructure<br />Repeatable Install<br />Reliable storage<br />Redundant hardware<br />
  12. 12. Broker Infrastructure: Continued<br />Cluster of nodes<br />Disaster Recovery<br />Logging (such as Splunk) and monitoring<br />Operations & Support<br />
  13. 13. Broker Infrastructure: Use Escript<br />
  14. 14. Broker InfrastructureFuture?<br />Integrate Rebar<br />Continuous Integration Testing<br />QuickCheck<br />
  15. 15. RabbitMQ Clients<br />Any AMQP Clients (Apache)<br />Ruby<br />Python<br />C#<br />F# (Wrap the C# dll)<br />
  16. 16. Consume.py Part: A<br />
  17. 17. Consume.py Part: B<br />
  18. 18. Consume.py Part: C<br />
  19. 19. Send.py Part: A<br />
  20. 20. Send.py Part: B<br />
  21. 21. Writing RabbitMQ Plugin: WHY<br />Access internal RabbitMQ functionality<br />Running in same Erlang VM as broker can increase performance <br />Leverage your existing infrastructure<br />
  22. 22. Writing RabbitMQ Plugin: WHY NOT<br />Developers don’t know Erlang<br />Poorly written plugin could crash broker<br />You could lock yourself to internal API<br />
  23. 23. StableRabbitMQ Plugin<br />Rabbitmq-management (replaced Alice)<br />Rabbitmq-shovel<br />Rabbitmq-stomp<br />Rabbitmq-erlang-client<br />
  24. 24. ExperimentalRabbitMQ Plugin Ideas<br />Rabbitmq-auth-backend-ldap<br />Rabbitmq-auth-mechanism-ssl<br />Rabbitmq-jsonrpc-channel<br />Rabbitmq-xmpp<br />
  25. 25. CrazyRabbitMQ Plugin Ideas<br />Distributed Data Structure<br />Artificial Intelligence<br />Data Mining<br />Create a protocol on top<br />Event Processing<br />
  26. 26. Michael Background<br />Apple 5 Years<br />Ascend/Lucent 4 Years<br />Various Startups<br />
  27. 27. Supervision Tree<br />
  28. 28. Plugin Supervisor<br />
  29. 29. Rabbit RSS Supervisor<br />
  30. 30. Rabbit RSS Worker: Slide A<br />
  31. 31. Rabbit RSS Worker: Slide B<br />
  32. 32. Rabbit RSS Worker: Slide C<br />
  33. 33. Rabbit RSS Worker: Slide D<br />
  34. 34. Rabbit RSS Worker: Slide E<br />
  35. 35. Rabbit RSS Worker: Slide F<br />
  36. 36. XML Parsing<br />
  37. 37. RSS.ERL: Slide A<br />
  38. 38. RSS.ERL: Slide B<br />
  39. 39. RSS.ERL: Slide C<br />
  40. 40. RSS.ERL: Slide D<br />
  41. 41. Checkout source: HG clone<br />
  42. 42. Make: Be Patient….<br />
  43. 43. Build<br />
  44. 44. Skeleton<br />
  45. 45. Symlink Plugin<br />
  46. 46. Run Broker<br />$ make run<br />(rabbit@rabbit9)1><br />(rabbit@rabbit9)1>application:which_applications().<br />{rabbit_rss,"Embedded Rabbit RSS Reader","0.01"},<br />
  47. 47. Test Broker<br />$ mkdirtest<br />$ vi test/rabbit_rss_tests.erl<br />$ vi Makefile<br />TEST_APPS=amqp_clientrabbit_rss<br />TEST_COMMANDS=rabbit_rss_tests:test() START_RABBIT_IN_TESTS=true<br />$ make test<br />
  48. 48. Package Plugin<br />
  49. 49. Plugin Supporting Machinery<br />Unit Tests<br />Source Code<br />Logger<br />Printf<br />AMQP Clients<br />
  50. 50. How To Build a Plugin<br />RabbitMQ docs<br />Source<br />Examples<br />
  51. 51. Use the Source!<br />Documentation is hit/miss<br />Google “erlang lists”<br />http://www.erlang.org/doc/apps/stdlib/<br />Or skip the docs and go directly to source<br />./lib/stdlib/src/lists.erl<br />
  52. 52. Lists.erl<br />
  53. 53. Logger<br />Plugin executes in its own process<br />No plugin stdout<br />No plugin execution debugger<br />
  54. 54. Logger: Continued<br />Solution: Erlang’s logger<br />Output to /var/log/rabbitmq/<hostname<br />Nothing to configure<br />
  55. 55. RSS.erl<br />
  56. 56. OTP Design Principles<br />Supervision Tree<br />Behaviors<br />Applications<br />Releases<br />Release Handling<br />
  57. 57. Machine Learning<br />The next step for the RSS plugin.<br />A potential use case for RabbitMQ plugins.<br />
  58. 58. Fisher Classifier For RSS Feeds<br />Data mining<br />Classification<br />
  59. 59. Fisher Classifier: Slide A<br />
  60. 60. Fisher Classifier: Slide B<br />
  61. 61. Fisher Classifier: Slide C<br />
  62. 62. Fisher Classifier: Slide D<br />
  63. 63. Fisher Classifier: Slide E<br />
  64. 64. Fisher Classifier: Slide E<br />
  65. 65. Fisher Classifier: Slide F<br />
  66. 66. Fisher Classifier: Slide G<br />
  67. 67. Conclusion<br />AMQP<br />Erlang<br />Python<br />RSS<br />
  68. 68. Questions<br />Code: https://github.com/mikev/rabbitmq_rss_plugin<br />Contact: “Noah Gift” noah.gift@gmail.com, “Michael Vierling” mvierling@gmail.com<br />

×