Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Eugene Shakhsuvarov - Improving enterprise store scalability using AMQP and Asynchronous Operations


Published on

We has discussed ways to scale an enterprise store leveraging asynchronous capabilities introduced in Magento EE via AMQP and Asynchronous Operations modules.
On the example of mass update in a background task, we has covered API for queuing tasks to multiple workers and ways to handle recoverable and non-recoverable errors.

  • Login to see the comments

  • Be the first to like this

Eugene Shakhsuvarov - Improving enterprise store scalability using AMQP and Asynchronous Operations

  1. 1. © 2016 Magento, Inc. Page | 1
  2. 2. © 2016 Magento, Inc. Page | 2 Improving Enterprise Store Scalability with Queues
  3. 3. © 2016 Magento, Inc. Page | 3 Message Queue Framework in Magento EE Overview of capabilities
  4. 4. © 2016 Magento, Inc. Page | 4 What are Queue advantages? • Execute tasks in background process • Asynchronously, with multiple workers • Take advantage of RabbitMQ benefits: – High Availability – Scalability – Load Balancing – Postponed execution – Platform Agnostic – Open Source
  5. 5. © 2016 Magento, Inc. Page | 5 What does Magento EE add to this? • Provides support for queuing with the simple API • Transparent workflow • Takes care of topology with just xml configurations • Great way to eliminate blocking requests • Fallback queue solution based solely on MqSQL for testing and specific applications
  6. 6. © 2016 Magento, Inc. Page | 6 Basic Concepts • Message is an atomic unit of processing • Message exists in queue until explicitly deleted • Never sent more then once • Always sent with specific topic • Consumers get appropriate messages by subscribing to the desired topic
  7. 7. © 2016 Magento, Inc. Page | 7 Basic Workflow
  8. 8. © 2016 Magento, Inc. Page | 8 Publisher • Publishes message data to the specific exchange • Data may be represented as an arbitrary array or DTO • Configured with xml configuration • Simple interface
  9. 9. © 2016 Magento, Inc. Page | 9 Consumer • Consumes message from the queue • Passes it to the configured handler class for processing • Again, simple xml configuration
  10. 10. © 2016 Magento, Inc. Page | 10 Batch Consumer • For the cases where lots of messages are involved • Consumes a batch of multiple messages of configured size • Allows handler to process whole batch in a loop
  11. 11. © 2016 Magento, Inc. Page | 11 Configuration • Queues configuration happens in a few files inside the module’s “etc” directory. • “communication.xml” is responsible for aspects of the message queue system that all communication types have in common. AMQP and Database connections. Values may be overridden for deployment in “env.php”. • “queue.xml” is responsible for defining the broker that processes the topic. • “queue_consumer.xml” responsible for configuring a consumer, allowing to set it’s type and handler for messages. • “queue_publisher.xml” configures publisher connection to the message broker.
  12. 12. © 2016 Magento, Inc. Page | 12 Are there any disadvantages? • Concurrent execution of similar actions may lead to locks in specific scenarios • No guarantee on the timeframe, when the task will be processed • Generally, using a single worker decreases overall performance: – Using Multiple workers improves performance, but only up to a certain grade.
  13. 13. © 2016 Magento, Inc. Page | 13 Asynchronous Operations Module Yes, it gets even better.
  14. 14. © 2016 Magento, Inc. Page | 14 Asynchronous Operations in Magento EE • Significantly improves “Look and Feel” for the merchant • Allows Store Admin to send entity updates to the queue for background processing eliminating long blocking requests • Provides a UI for tracking status of the execution and logging • Allows to retry certain failed operations • Built on top of the Message Queue Framework
  15. 15. © 2016 Magento, Inc. Page | 15 Asynchronous Operations
  16. 16. © 2016 Magento, Inc. Page | 16 Magento Asynchronous Operations Workflow • Create Operation per entity to be processed • Form bulk of operations • Post messages to the Queue • Handle operations and their statuses from the Queue Operation Operation Operation Queue Handler UI
  17. 17. © 2016 Magento, Inc. Page | 17 Operation • Operation is a single unit of work to be processed • Framework interprets it as a simple DTO containing: – Operation UUID – Name of the destination topic in Queue – Serialized data or DTO for processing – Operation Status and Status Message • Recoverably Failed operations may be retried in a separate bulk
  18. 18. © 2016 Magento, Inc. Page | 18 Operation Statuses • Operations can be in one of the 4 possible statuses: – Open: still in queue – Complete: processed successfully without errors – Failed Recoverably: based on exception type it is possible to determine if operation is worth retrying. For example a connection, wait or lock exceptions, which, in some cases, may happen when dealing with the large number of similar items – Failed Non Recoverably: for cases when operation is failed due to reasons which require manual interaction, for example validation errors • Operation status is accompanied by Result Message, visible to the store admin, which may contain more detailed information on the problem
  19. 19. © 2016 Magento, Inc. Page | 19 Bulk Management • Bulk is an array of operations, identified by Unique ID • Visible as a separate entry to the admin user who created it, along with the count and status of operations within it • Every Operation is published as a separate message under a desired topic name • Bulk Management provides a simple API to publish Operations into the Queue
  20. 20. © 2016 Magento, Inc. Page | 20 How does it look for a store admin?
  21. 21. © 2016 Magento, Inc. Page | 21 Quick Recap
  22. 22. © 2016 Magento, Inc. Page | 22 Quick recap • Queue is good for performing long operations in background • Great way to avoid blocking requests • Scales well with the number of workers, however has its limits • Simple API on Magento's side
  23. 23. © 2016 Magento, Inc. Page | 23 Example Module • You may take a look at the example module, which using some of the discussed features updates Catalog Product Stock Quantities • •
  24. 24. © 2016 Magento, Inc. Page | 24 Q&A For more questions: @ishakhsuvarov (twitter, github or email)