Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

[OSC2016] マイクロサービスを支える MQ を考える

3,698 views

Published on

OSC2016 発表資料

Published in: Technology
  • Be the first to comment

[OSC2016] マイクロサービスを支える MQ を考える

  1. 1. Think MQ(Message Queue) for micro-services Hiroyasu OHYAMA DMM.com Labo/department of Infrastructure OSC2016 Tokyo / Fall
  2. 2. SO, WHAT IS MQ ? 2
  3. 3. 3 PROCESS A BROKER PROCESS B PUBLISHER SUBSCRIBER B SUBSCRIBER A SUBSCRIBER C BROKER What is MQ? A mechanism that supports message passing between application processes. Queuing model Pub/Sub model
  4. 4. 4 PROCESS A PROCESS BBROKER Queuing model What is MQ? supports transporting data between processes asynchronously - These processes might not be woken at all time.
  5. 5. 5 supports publishing data to processes that share the ‘topic’ - There is no need for Publisher to know the existance of Subscriber(s). Pub/Sub model What is MQ? PUBLISHER SUBSCRIBER B SUBSCRIBER A SUBSCRIBER C BROKER
  6. 6. WHY MQ ? 6
  7. 7. 7 Imagine a system without MQ ...
  8. 8. SERVICE A SERVICE B SERVICE E SERVICE D REQUEST SERVICE C 8 A system without MQ Introduction
  9. 9. SERVICE A SERVICE B REQUEST SERVICE E SERVICE D SERVICE C DATA REQUEST ”REQUEST ” 9Introduction A system without MQ
  10. 10. SERVICE A SERVICE B REQUEST SERVICE E SERVICE D SERVICE C DATA IF THIS IS STOPPED... WHAT SHOULD IT DO ..? REQUEST ”REQUEST ” 10Introduction A system without MQ
  11. 11. SERVICE A SERVICE B REQUEST SERVICE E SERVICE D SERVICE C DATA SERVICE F A NEW SERVICE IS APPEARED !! REQUEST ”REQUEST ” WHAT SHOULD IT DO ..?? 11Introduction A system without MQ
  12. 12. 12 Imagine a system with MQ
  13. 13. SERVICE A SERVICE B MQ SERVICE CSERVICE E SERVICE D REQUEST REQUEST 13 A system with MQ Introduction
  14. 14. SERVICE A SERVICE B MQ SERVICE E SERVICE D SERVICE C DATA REQUEST ”REQUEST ” REQUEST REQUEST REQUEST ” 14Introduction A system with MQ
  15. 15. SERVICE A SERVICE B MQ SERVICE E SERVICE D SERVICE C 15 DATA REQUEST REQUEST REQUEST ” IF THIS IS STOPPED... REQUEST ”REQUEST ” OK, NO PROBLEM Introduction A system with MQ
  16. 16. SERVICE A SERVICE B MQ SERVICE E SERVICE D SERVICE C 16 DATA REQUEST ”REQUEST ” REQUEST REQUEST REQUEST ” SERVICE F REQUEST ” A NEW SERVICE IS APPEARED !! OK, NO PROBLEM Introduction A system with MQ
  17. 17. MQ simplify the system model of distributed system. - It could make a system HA(High Available) - It could make a system Scalable 17Introduction A system with MQ
  18. 18. 18 OVERVIEW
  19. 19. Comparison with each protocols and implementations 19 Messaging Pub/Sub AMQP (RabbitMQ) MQTT (ActiveMQ) STOMP (NewtMQ) Kafka ZeroMQ … Overview - There are many distinctive protocols/implementations.
  20. 20. 20 HOW THEY ARE USED?
  21. 21. CASE: SENSU 21 - Scalability for Pub/Sub - Load leveling for Queuing (reference:SENSU | SENSU Architecture) How they are used?
  22. 22. CASE: YAHOO! JAPAN 22 Very long queue (many millions of messages) and Scalability (reference:ストリーム処理プラットフォームにおけるKafka導入事例 #kafkajp) How they are used?
  23. 23. CASE: IBM 23 Less Traffic & Scalability for IoT Cloud Platform IBM IoT Foundation (MQTT Frontend) IBM Bluemix (PaaS) APPLICATION Devices THE INTERNET APPLICATION WEBSPHERE MQ APPLICATION MQTT (reference:IoT クラウドで広がる Bluemix の世界 - IBM Bluemix) How they are used?
  24. 24. 24 WHAT’S THE BEST ONE ?
  25. 25. 25 There is no MQ that satisfies all the system requirements, I guess. What’s the best one ?
  26. 26. 26What’s the best one ? If you only know yourself, but not your opponent, you may win or may lose. (reference: The art of war, Sun Tzu) There is no MQ that satisfies all the system requirements, I guess.
  27. 27. 27 FEATURE OF AMQP(V0.9.1)
  28. 28. AMQP SERVER 28 SUBSCRIBER PUBLISHER SUBSCRIBER SUBSCRIBER EXCHANGE [Normal Pub/Sub message passing] Flexible message routing Feature of AMQP (v0.9.1) QUEUE QUEUE QUEUE
  29. 29. A MESSAGE BROKER AMQP SERVER 29 SUBSCRIBER PUBLISHER SUBSCRIBER SUBSCRIBER EXCHANGE [Normal Pub/Sub message passing] Feature of AMQP (v0.9.1) Flexible message routing QUEUE QUEUE QUEUE
  30. 30. AMQP SERVER 30 SUBSCRIBER PUBLISHER SUBSCRIBER SUBSCRIBER EXCHANGE QUEUE QUEUE QUEUE Flexible message routing [Normal Pub/Sub message passing] Feature of AMQP (v0.9.1)
  31. 31. AMQP SERVER 31 QUEUE QUEUE QUEUE SUBSCRIBER PUBLISHER SUBSCRIBER SUBSCRIBER EXCHANGE Flexible message routing [Normal Pub/Sub message passing] Feature of AMQP (v0.9.1)
  32. 32. AMQP SERVER 32 SUBSCRIBER PUBLISHER SUBSCRIBER SUBSCRIBER EXCHANGE QUEUE QUEUE QUEUE Flexible message routing [Normal Pub/Sub message passing] Feature of AMQP (v0.9.1)
  33. 33. 33 QUEUE QUEUE QUEUE SUBSCRIBER PUBLISHER SUBSCRIBERQUEUE EXCHANGE EXCHANGE SUBSCRIBER SUBSCRIBER AMQP SERVER AMQP SERVER CRITICAL.* *.SERVICE-A *.SERVICE-B # Flexible message routing [Advanced Pub/Sub message passing] Feature of AMQP (v0.9.1)
  34. 34. 34 QUEUE QUEUE QUEUE SUBSCRIBER PUBLISHER SUBSCRIBERQUEUE EXCHANGE EXCHANGE SUBSCRIBER SUBSCRIBER AMQP SERVER AMQP SERVER ROUTE OF MESSAGE CRITICAL.* *.SERVICE-A *.SERVICE-B # Flexible message routing [Advanced Pub/Sub message passing] Feature of AMQP (v0.9.1)
  35. 35. 35 QUEUE QUEUE QUEUE SUBSCRIBER PUBLISHER SUBSCRIBERQUEUE EXCHANGE EXCHANGE SUBSCRIBER SUBSCRIBER AMQP SERVER AMQP SERVER CRITICAL.* *.SERVICE-A *.SERVICE-B # Flexible message routing CRITIAL.SERVICE-A [Advanced Pub/Sub message passing] Feature of AMQP (v0.9.1)
  36. 36. 36 QUEUE QUEUE QUEUE SUBSCRIBER PUBLISHER SUBSCRIBERQUEUE EXCHANGE EXCHANGE SUBSCRIBER SUBSCRIBER AMQP SERVER AMQP SERVER *.SERVICE-B Flexible message routing CRITICAL.SERVICE-A [Advanced Pub/Sub message passing] CRITICAL.SERVICE-A Feature of AMQP (v0.9.1) CRITICAL.SERVICE-A # CRITICAL.* *.SERVICE-A
  37. 37. 37 FEATURE OF MQTT
  38. 38. MQTT 38 Lightweight message protocol Three qualities of service for message delivery - “At most once", where messages are delivered according to 
 the best efforts of the operating environment. - "At least once", where messages are assured to arrive
 but duplicates can occur. - "Exactly once", where message are assured to arrive exactly once. Feature of MQTT
  39. 39. MQTT 39 Lightweight message protocol - The header-size comparison of MQTT and HTTP. (publish data to the topic ‘hoge’) (Fixed Header) (Variable Header) (POST data to ‘http://hoge.fuga.com/hoge') POST /hoge HTTP/1.1n Host: hoge.fuga.comn Content-Type: application/octet-streamn Content-Length: 100n n Feature of MQTT
  40. 40. MQTT 40 Lightweight message protocol - The header-size comparison of MQTT and HTTP. (publish data to the topic ‘hoge’) (Fixed Header) (Variable Header) (POST data to ‘http://hoge.fuga.com/hoge') POST /hoge HTTP/1.1n Host: hoge.fuga.comn Content-Type: application/octet-streamn Content-Length: 100n n 10 BYTES 101 BYTES< Feature of MQTT
  41. 41. 41 FEATURE OF KAFKA
  42. 42. KAFKA 42 - provide both ordering guarantees and load balancing - relies on the filesystem for storing and caching messages (reference: Documentation - Apache Kafka) Feature of Kafka
  43. 43. What essentials of MQ for a micro-service is?
  44. 44. An example of micro service is What essentials of MQ for a micro-service is? A cloud operating system - that controls large pools of compute, storage,and networking resources (reference: What is OpenStack? / OpenStack)
  45. 45. (reference: OpenStack Docs: Logical architecture) What essentials of MQ for a micro-service is? Logical architecture of OpenStack
  46. 46. Logical architecture of OpenStack What essentials of MQ for a micro-service is? (reference: OpenStack Docs: Logical architecture)
  47. 47. What essentials of MQ for a micro-service is? How OpenStack use MQ?
  48. 48. What essentials of MQ for a micro-service is? How OpenStack use MQ? Use Oslo/Messaging that provides - RPC for implementing client-server remote procedure calls - Notify for emitting and handling event notifications (reference: Oslo/Messaging - OpenStack)
  49. 49. What essentials of MQ for a micro-service is? How OpenStack use MQ? Use-case of Oslo/Messaging in Nova (reference: Nova System Architecture)
  50. 50. What essentials of MQ for a micro-service is? What are essentials of MQ for OpenStack?
  51. 51. What essentials of MQ for a micro-service is? The essentials of MQ for OpenStack are Order-guarantee High Availability Performance (Scalability)
  52. 52. What essentials of MQ for a micro-service is? The non-essentials of MQ for OpenStack are Transaction
 - increase throughput by compromising message lost on the node 
 which is failure Encryption
 - performance increase without encoding/decoding processing
  53. 53. Performance oriented MQ
  54. 54. A yet another brokered-MQ which specialize in throughput 54 (reference: ツチノコブログ) NewtMQ Performance oriented MQ
  55. 55. NATS Performance oriented MQ A simple, high performance messaging system for
 cloud native applications and microservices architectures (reference: About NATS | The best messaging system for native cloud application developement)
  56. 56. NewtMQ vs NATS
  57. 57. NewtMQ vs NATS Throughput Benchmark tool: https://github.com/userlocalhost2000/rpc-bench
  58. 58. NewtMQ vs NATS Functionality NewtMQ NATS RabbitMQ Kafka Topic Request-Reply Transaction Persistance Clustering
  59. 59. Conclusion
  60. 60. Conclusion NATS is better for micro-service applications
 - If you can permit these restrictions There are many MQ with a functional character
 - choose it which confirms to your application
  61. 61. Thank you!

×