Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

An update from the RabbitMQ team - Michael Klishin


Published on

Watch the full lecture on YouTube:

Curious about what the RabbitMQ team has been up to since 3.7.0 shipped in November 2017? In this talk we will cover what's coming in 3.8.0 and beyond.


The first RabbitMQ Summit connected RabbitMQ users and developers from around the world in London on November 12, 2018. Learn what's happening in and around RabbitMQ, and how top companies utilize RabbitMQ to power their services.

RabbitMQ Summit was organized by:
- Erlang Solutions, offering world-leading RabbitMQ Consultancy, Support, Health Checks & Tuning solutions
- CloudAMQP, offering fully managed RabbitMQ clusters

RabbitMQ Summit 2018 was sponsored by the following companies.

Platinum sponsors:

Gold sponors:

Silver sponsor:
Cogin Queue Explorer

Published in: Technology
  • Be the first to comment

  • Be the first to like this

An update from the RabbitMQ team - Michael Klishin

  1. 1. MONDAY 12TH NOV CodeNode •London• ORGANISERS
  2. 2. © Copyright 2018 Pivotal Software, Inc. All rights Reserved. Version 1.0 November 2018 An update from the RabbitMQ team
  3. 3. Who am I? Michael Klishin ! Senior Engineer at Pivotal ! RabbitMQ contributor since 2010 ! @michaelklishin on GitHub and most other places
  4. 4. Cover w/ Image Agenda ■ The state of RabbitMQ 3.7.x ■ Future directions ■ What’s coming in 3.8 and beyond
  5. 5. The state of 3.7.x An update from the RabbitMQ team
  6. 6. RabbitMQ 3.7.0 3.7.0 release themes ! Deployment automation friendliness ! Operator friendliness ! Stability ! Setting up stage for larger changes
  7. 7. RabbitMQ 3.7.0 Operator friendliness ! Distributed stats storage ! Per-vhost limits ! Operator policies ! New logging subsystem ! Proxy protocol Deployment automation friendliness ! Peer discovery subsystem ! New, easy to generate configuration format ! Extensible command line tools
  8. 8. RabbitMQ 3.7.x What have we been up to in 3.7.x? ! Best practices ! Documentation ! Non-breaking refinements ! Ecosystem improvements
  9. 9. RabbitMQ 3.7.x Documentation ! Upgrades guide ! Monitoring guide ! Production checklist updates ! Runtime tuning guide Guidance & best practices ! Blue/green deployment upgrades ! Workload-specific runtime tuning
  10. 10. RabbitMQ 3.7.x Non-breaking refinements ! Erlang 21 support ! OAuth 2 support ! New metrics ! New CLI commands ! Greater adoption of the new config format in plugins ! Usability improvements ! Kubernetes deployment example ! Improved default runtime flags* ! LDAP Ecosystem Improvements ! Client libraries (primarily Java, .NET, Python) ! Erlang packaging
  11. 11. Future directions An update from the RabbitMQ team
  12. 12. Future directions General directions ! Improved reliability and operator friendliness ! Improved scalability ! Simplify upgrades ! Continue improving and expanding the ecosystem ! Move away from home grown distsys algorithms 🤦 " ! Improved correctness ! Repay technical debt
  13. 13. What problems are we addressing? Scalability ! Mirroring uses a ring topology which is linear ! Excessive bandwidth usage ! No DC awareness ! The core is not protocol-agnostic which leads to overhead Reliability ! Recovery from failures is not always predictable (or easy to reason about) ! Fails to pass certain resiliency tests in some scenarios ! Queue sync after failure can take too long and cause a thundering herd ! Schema data store (Mnesia) is overly opinionated and isn’t a great fit ! Lack of unified WAL/oplog makes backups hard
  14. 14. Future directions Currently in progress ! Quorum queues (mirrored queues 2.0) based on Raft ! OAuth 2.0 support ! Mixed version clusters ! Mnevis: a new schema data store ! Protocol-agnostic core
  15. 15. Quorum queues An update from the RabbitMQ team
  16. 16. How does Raft-based replication compare? Improved scalability ! No ring topology, parallel replication ! More reasonable bandwidth usage* ! Opens the door to cross-DC replication Improved reliability ! Recovery from failures is predictable and well defined ! Queue sync after failure transfers as little data as possible ! Passes more resiliency tests ! Opens the door to a unified WAL/oplog to simplify backups
  17. 17. What is Raft? An update from the RabbitMQ team
  18. 18. What is Raft? ! A group of algorithms for reaching consensus in a distributed system ! Implementer-focused ! Proven ! Multiple implementations ! Industry use ! TLA+ specification
  19. 19. Raft: an oversimplified explanation Raft Follower Leader Follower
  20. 20. Raft: an oversimplified explanation Raft Follower Leader Follower Commit index = 0 Enqueue Append entry Append entry Log = [] Log = [] Log = []
  21. 21. Raft: an oversimplified explanation Raft Follower Leader Follower Commit index = 1 Success Success Success Log = [{1, Enqueue}] Log = [{1, Enqueue}] Log = [{1, Enqueue}]
  22. 22. Quorum queues The State of Quorum Queues ! Reasonably polished ! Throughput beats mirrored queues ! Passes a [slightly modified] Jepsen test ! Available in RabbitMQ 3.8.0-beta.1 today
  23. 23. Quorum queues Implementing Quorum Queues ! Powered by our own from scratch Raft implementation ! ! Heavily geared towards but not specific to RabbitMQ ! Deviates from Raft a little bit to reduce network and disk I/O ! Does not sacrifice correctness ! Includes a peer unavailability detection library ! Optimized for throughput, adapts I/O for latency or throughput based on load
  24. 24. Quorum queues The State of Quorum Queues ! Has limitations ! Doesn’t support some features, e.g. TTL won’t be supported ! Memory management is still a hard problem ! As any new major feature, will take time to mature (you can help by giving it a try!)
  25. 25. Queue types An update from the RabbitMQ team
  26. 26. One queue type to rule them all? What are the chances that all these features can work efficiently or be easy to reason about when they are combined? ! Durable ! Auto-delete ! Mirrored ! Lazy ! With TTL ! With message TTL ! With length limit ! With dead-lettering ! With failure recovery settings ! With exclusive consumers ! Affected by plugins ! …
  27. 27. One queue type per workload? Or is there a better way? ! Durable queues with consensus replication ! Transient queues, auto-delete with TTL and/or length limit ! “Infinite queues” (durable, aggressive paging to disk) ! Lower latency in-memory queues
  28. 28. OAuth 2.0 Support An update from the RabbitMQ team
  29. 29. OAuth 2.0 support How does it work? ! Implemented as a plugin, rabbitmq_auth_backend_oauth2 ! OAuth 2.0/JWT token scopes that follow naming conventions are translated to RabbitMQ permissions ! Clients can use any OAuth 2.0 authorization code flow ! Management UI will use the authorization code flow ! Officially supported clients will simplify token renewal
  30. 30. OAuth 2.0 support The State of OAuth 2.0 Support ! Plugin is done, open source and currently targets RabbitMQ 3.8 ! Will ship in 3.8.0 Milestone 2 ! Management UI needs work ! Targets UAA and AD
  31. 31. Next gen schema storage An update from the RabbitMQ team
  32. 32. How does next gen schema storage compare? Improved scalability ! Mnesia does reasonably well here (for schema storage) ! Opens the door to DC awareness ! No long-entrenched Mnesia limitations ! Higher rate of development iteration Improved reliability ! Recovery from failures that works for RabbitMQ users (and maintainers) ! Nodes no longer have to erase themselves in order to re-sync ! Can integrate with the unified WAL/oplog
  33. 33. Next gen schema storage The State of Mnevis ! Currently an area of active research ! Mnesia will be used in a node-local way ! Mnesia is quite extensible ! Raft is not necessarily a great fit for transaction log propagation due to [relatively] high latency
  34. 34. Mixed version clusters An update from the RabbitMQ team
  35. 35. Mixed version clusters How do we get there? ! Relaxing overly conservative restrictions ! Capability testing instead of version testing ! Feature flags ! Extensive internal refactoring ! Somewhat funny APIs to work around Erlang record limitations
  36. 36. Mixed version clusters The State of Mixed Version Clusters ! Feature flag experiment with promising results so far ! Meant to simplify upgrades ! Long running mixed clusters is not a goal ! Cannot guarantee safety for every possible breaking change ! Needs more feedback
  37. 37. RabbitMQ 3.8 An update from the RabbitMQ team
  38. 38. RabbitMQ 3.8 What’s coming in 3.8? ! Quorum queues ! OAuth 2.0 ! Initial mixed version cluster support ! Minor usability improvements
  39. 39. RabbitMQ 3.9 An update from the RabbitMQ team
  40. 40. RabbitMQ 3.9 What’s coming in 3.9? ! Protocol-agnostic core ! 🤔
  41. 41. RabbitMQ 4.0 An update from the RabbitMQ team
  42. 42. RabbitMQ 4.0 What’s coming in 4.0? ! Next generation schema databas ! Transport beam? Food replicators? Tricorders? 🤔 ! All the breaking changes
  43. 43. Thank you! An update from the RabbitMQ team
  44. 44. Transforming How The World Builds Software © Copyright 2018 Pivotal Software, Inc. All rights Reserved.