Scaling RabbitMQ to 11

8,020
-1

Published on

Published in: Technology, Business
0 Comments
11 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
8,020
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
69
Comments
0
Likes
11
Embeds 0
No embeds

No notes for slide

Scaling RabbitMQ to 11

  1. 1. Scaling RabbitMQ to 11Gavin M. Roy @crad - Surge 2011
  2. 2. Scaling RabbitMQ to 11 INTERACTIVE POLLS • http://pollev.com/gmr • SMS: 223332 Gavin M. Roy @crad - Surge 2011
  3. 3. myYearbook.com• #1 Comscore Teen Destination, Top Trafficked Sites in US• Use of RabbitMQ dates back to 2009• Multiple clusters for different purposes • Browser <-> Backend Message Bus • Messaging, Event Processing 3 Gavin M. Roy @crad - Surge 2011
  4. 4. DISASTER PORN Gavin M. Roy @crad - Surge 2011
  5. 5. ABOUT AMPQ• Platform, Vendor Neutral Messaging format• 0.9.1 finalized in November 2008 • JP Morgan Chase, Red Hat, Rabbit Technologies, iMatrix, IONA Technologies, Cisco Systems, Envoy Technologies, Twist Process Innovations• Members now include Bank of America, Barclays Banks, Microsoft Corporation, Novell, VMWare• 1.0 Final: October 2011 5 Gavin M. Roy @crad - Surge 2011
  6. 6. ABOUT RABBITMQ• Born in 2006, Released in 2007, Acquired in 2010• Written in Erlang/OTP• Talks AMQP 0.8 and 0.9.1• Features include:• Clustering, Active/Active HA Queues• Management UI and API• Plugin Architecture 6 Gavin M. Roy @crad - Surge 2011
  7. 7. ABOUT YOU & RABBITMQ Gavin M. Roy @crad - Surge 2011
  8. 8. RABBITMQ& AMQPCONCEPTS Gavin M. Roy @crad - Surge 2011
  9. 9. PUBLISHERS Gavin M. Roy @crad - Surge 2011
  10. 10. CONSUMERS Gavin M. Roy @crad - Surge 2011
  11. 11. QUEUES• Durability: Queue survives reboot• Auto-Delete: Delete the queue when consumer goes away• Exclusive: Only one consumer may consume• x-expires: Auto-Delete after given duration• x-message-ttl: Auto-discard a message after ttl 11 Gavin M. Roy @crad - Surge 2011
  12. 12. MESSAGES• Have client specified properties: • Content type, Encoding, Timestamp, App-Id, User-Id, Headers • Delivery Mode: • 1: Non-Persistent • 2: Persistent 12 Gavin M. Roy @crad - Surge 2011
  13. 13. MESSAGEROUTINGIt’s not just about Queues. Gavin M. Roy @crad - Surge 2011
  14. 14. ROUTING KEYIn addition to the exchange, determines what is done with amessage. Gavin M. Roy @crad - Surge 2011
  15. 15. CC & BCCMore routing options per message, but violates AMQP. Gavin M. Roy @crad - Surge 2011
  16. 16. HEADERSEXCHANGEMore on this later. Gavin M. Roy @crad - Surge 2011
  17. 17. EXCHANGES Gavin M. Roy @crad - Surge 2011
  18. 18. FANOUT• 1:N Message Delivery Pattern• No Routing Keys Involved• Queues bound to a fanout exchange get all messages sent to the exchange 18 Gavin M. Roy @crad - Surge 2011
  19. 19. DIRECT• 1:1 Message Delivery Pattern• Routing Keys are direct, no wildcarding• All queues bound to a routing key get the message 19 Gavin M. Roy @crad - Surge 2011
  20. 20. TOPIC EXCHANGES• Pattern Matching in Routing Keys• Publish one message type to Surge.Sessions, another to Surge.Mentions• Period delimiter, * stays within the period scope, # does not• Example patterns: *.Mentions, Surge.*, #• Slower than Fanout and Direct 20 Gavin M. Roy @crad - Surge 2011
  21. 21. HEADERS EXCHANGES• Matches on Basic.Properties headers table values to the arguments table specified when binding a queue• Use x-match when declaring exchange to specify any or all headers must match• Slow implementation in Erlang 21 Gavin M. Roy @crad - Surge 2011
  22. 22. PLUGIN EXCHANGES• Consistent Hash Exchange: Round-robin distribution to queues• External: RPC Exchange Plugin• Script: Similar in intent to External• Riak• Last-Value Cache• Recent History Exchange: Keeps last 20 messages routed• Global Fanout: Sends to all queues, regardless of routing key 22 Gavin M. Roy @crad - Surge 2011
  23. 23. ONE MORETHING...Exchange-to-Exchange Bindings Gavin M. Roy @crad - Surge 2011
  24. 24. CLUSTERS Gavin M. Roy @crad - Surge 2011
  25. 25. STATS, DISKAND RAMNODES Gavin M. Roy @crad - Surge 2011
  26. 26. WHERE DOESTHE QUEUELIVE? Gavin M. Roy @crad - Surge 2011
  27. 27. CROSS-NODEPUBLISHING&CONSUMING Gavin M. Roy @crad - Surge 2011
  28. 28. MAX CLUSTERSIZE~32 notes is a realistic maximum node count for a cluster Gavin M. Roy @crad - Surge 2011
  29. 29. SCALING ISSUES? Gavin M. Roy @crad - Surge 2011
  30. 30. Scaling RabbitMQ to 11Gavin M. Roy @crad - Surge 2011
  31. 31. WHAT TO WATCH Gavin M. Roy @crad - Surge 2011
  32. 32. Scaling RabbitMQ to 11 RABBITMQY U NO TAKE MESSAGE? Gavin M. Roy @crad - Surge 2011
  33. 33. TCPBACKPRESSUREMUCH? Gavin M. Roy @crad - Surge 2011
  34. 34. MEMORYMANAGEMENT Gavin M. Roy @crad - Surge 2011
  35. 35. [{RABBIT, [{VM_MEMORY_HIGH_WATERMARK, 0.4}]}]. 40% Gavin M. Roy @crad - Surge 2011
  36. 36. MESSAGEPERSISTENCE50-70% performance penalty compared to non-persistedmessages. * http://www.rabbitmq.com/faq.html#performance-persistent Gavin M. Roy @crad - Surge 2011
  37. 37. CONSUMINGAck, Nack and Reject vs Basic.Consume(no_ack=True) Gavin M. Roy @crad - Surge 2011
  38. 38. Scaling RabbitMQ to 11Server ClientBasic.Deliver Basic.[Ack/Nack/Reject]Basic.[Ack/Nack/Reject]Ok Gavin M. Roy @crad - Surge 2011
  39. 39. Gavin M. Roy @crad - Surge 2011
  40. 40. BASIC.QOS • Prefetch Size and Count • How many messages the client wants the RabbitMQ to pre-deliver to the client • Doesn’t work in no_ack Gavin M. Roy @crad - Surge 2011
  41. 41. TOO MANYSHORT LIVEDCONNECTIONS Gavin M. Roy @crad - Surge 2011
  42. 42. APACHE +MOD_PHP Gavin M. Roy @crad - Surge 2011
  43. 43. CONNECTIONPERSISTENCE Gavin M. Roy @crad - Surge 2011
  44. 44. 100 SERVERS125 BACKENDS1 REQUEST/S12,500 CONN/S Gavin M. Roy @crad - Surge 2011
  45. 45. AMQP CONNECTION SEQUENCEClient ServerProtocol Header Frame Connection.StartConnection.StartOk Connection.TuneConnection.TuneOk Connection.OpenConnection.OpenOkChannel.Open Channel.OpenOk Gavin M. Roy @crad - Surge 2011
  46. 46. 685 BYTES PER8.16 MB/S INSTARTUP @12,500 CONN/S Gavin M. Roy @crad - Surge 2011
  47. 47. VORPAL BUNNY Gavin M. Roy @crad - Surge 2011
  48. 48. VORPAL BUNNY• Light-weight PHP client for RabbitMQ• Used the experimental JSON-RPC Channel plugin• Handles the overhead of the HTTP, JSON-RPC to AMQP analog• Fire and forget message publishing• Can not be transparently load balanced 48 Gavin M. Roy @crad - Surge 2011
  49. 49. TOO MANYCONNECTIONS Gavin M. Roy @crad - Surge 2011
  50. 50. VELOCITY TOOHIGH FOR ONENODE? Gavin M. Roy @crad - Surge 2011
  51. 51. MEMORYUTILIZATIONDISPROPORTIONATETO QUEUE DEPTHS Gavin M. Roy @crad - Surge 2011
  52. 52. CLUSTERNODEFAILURES Gavin M. Roy @crad - Surge 2011
  53. 53. UP TO 60SECONDS TODETECT Gavin M. Roy @crad - Surge 2011
  54. 54. MESSAGESDROPPED ONTHE FLOOR* Gavin M. Roy @crad - Surge 2011
  55. 55. * 2.6.0 ADDEDACTIVE/ACTIVEQUEUES INCLUSTERS Gavin M. Roy @crad - Surge 2011
  56. 56. ACTIVE/PASSIVEFAILOVER?Recommendation is to use Pacemaker and DRBD or sharedstorage for Mnesia databases. Gavin M. Roy @crad - Surge 2011
  57. 57. RECOVERY?Gavin M. Roy @crad - Surge 2011
  58. 58. CLUSTERING ALTERNATIVES Gavin M. Roy @crad - Surge 2011
  59. 59. Scaling RabbitMQ to 11FEDERATION • Exchange Plugin • Provides ability to route all messages to a x-federation exchange to another broker Gavin M. Roy @crad - Surge 2011
  60. 60. SHOVEL PLUGIN• From Queue on one broker to an exchange on another broker.• Effective for cross-cluster data broadcasting Gavin M. Roy @crad - Surge 2011
  61. 61. TRENDING & MONITORING• RabbitMQ Message Velocity Rates Gotcha• Management API Plugin exposes RESTful API including monitoring data points• 2.6.1 allows monitoring tag for users to allow data access without full admin access• Queue.Declare(passive=True) gets queue depths as well• Plug in to Graphite, Nagios,or your favorite tool 61 Gavin M. Roy @crad - Surge 2011
  62. 62. PHOTO CREDITS• “skateboard kids 7” © 2007 Volker Neumann http://flickr.com/prawnpie• “methodcn contest” ©2009 Erin O’Hara http://flickr.com/erinohara73/• USPS Truck Accident Valley City Fire Department http://www.valleycityfire.com 62 Gavin M. Roy @crad - Surge 2011
  63. 63. Scaling RabbitMQ to 11QUESTIONS? Gavin M. Roy @crad - Surge 2011

×