Q4M a high-performance message queue for MySQL Cybozu Labs, Inc. Kazuho Oku
<ul><li>What is a Message Queue? </li></ul><ul><li>from Distributed Systems (Tanenbaum / Van Steen) </li></ul>
What is a Message Queue? <ul><li>Middleware for persistent asynchronous communication </li></ul><ul><ul><ul><li>持続的な非同期コミュ...
Message Queue (cont.) <ul><li>MQ is an intermediate storage </li></ul><ul><ul><ul><li>MQ  は中間的なストレージ </li></ul></ul></ul><...
Minimal Configuration of a MQ ( 最小構成 ) <ul><li>Senders and receivers access a single queue </li></ul><ul><ul><ul><li>送信プロセ...
MQ and Relays <ul><li>Separate queue for sender and receiver </li></ul><ul><ul><ul><li>送信プロセスと受信プロセスが別個にキューをもつ </li></ul><...
Merits of Message Relays ( リレーの意義 ) <ul><li>Destination can be easily changed </li></ul><ul><ul><ul><li>宛先の変更が容易 </li></ul...
Message Brokers <ul><li>Transform (filter) messages within the relay agent </li></ul><ul><ul><ul><li>リレーエージェント内でメッセージの変換を行...
Q4M <ul><li>Q4M (can) support all models </li></ul><ul><ul><ul><li>Q4M  は全モデルに対応  ( 可能 ) </li></ul></ul></ul><ul><li>Messa...
<ul><li>Architecture of Q4M </li></ul>
File Format <ul><li>Log-based file format </li></ul><ul><ul><ul><li>ログベースのファイルフォーマット </li></ul></ul></ul><ul><li>Each row ...
Row Header <ul><li>Row Types: 3bits </li></ul><ul><ul><ul><li>Normal row </li></ul></ul></ul><ul><ul><ul><li>Removed row <...
Data Consistency <ul><li>INSERTs group-commited using writev </li></ul><ul><ul><ul><li>INSERT  は  writev  を利用してグループコミット </...
Data Consistency (cont.) <ul><li>DELETEs are modifs. of row headers </li></ul><ul><ul><li>DELETE  は行ヘッダの型変更 </li></ul></ul...
Row IDs <ul><li>64bit Int </li></ul><ul><li>Starts from one and incremented per each row </li></ul><ul><ul><li>1  から始まって1行...
Restritions in SQL of Q4M <ul><li>No indexes </li></ul><ul><li>No UPDATE </li></ul><ul><ul><li>Why need UPDATE a queue row...
Why no Indexes Support? <ul><li>There is generally no reason to support indexes with mysql-based queues, since mysql canno...
Why no Indexes Support? <ul><li>Instead use conditional subscription </li></ul><ul><ul><ul><li>Conditional subscription  機...
<ul><li>Tutorial </li></ul>
Sending a Message <ul><li>INSERT into queue values (1,’Hello World!’); </li></ul>
Receiving a Message <ul><li>while (1) { </li></ul><ul><li>SELECT queue_wait(‘queue’);; </li></ul><ul><li>my @row = SELECT ...
OWNER Mode and NON-OWNER Mode
OWNER Mode and NON-OWNER Mode (cont.) <ul><li>Within OWNER mode, only the owned row are visible </li></ul><ul><ul><ul><li>...
Function for Entering OWNER Mode <ul><li>queue_wait </li></ul><ul><ul><li>Used to enter OWNER mode </li></ul></ul><ul><ul>...
queue_wait <ul><li>Two forms </li></ul><ul><ul><ul><li>queue_wait(tbl_cond) </li></ul></ul></ul><ul><ul><ul><li>queue_wait...
queue_wait(tbl_cond) <ul><li>Wait max. 60 seconds until any data becomes available on given table under given condition </...
queue_wait(tbl_cond) <ul><li>Table name (and optionally condition) should be specified </li></ul><ul><ul><ul><li>テーブル名と  (...
queue_wait(tbl_cond) (cont.) <ul><li>SELECT queue_wait(‘table:v<3’); </li></ul>
queue_wait(tbl_cond,[tbl_cond…,timeout]) <ul><li>Accepts multiple tables and timeout </li></ul><ul><ul><li>複数のテーブルとタイムアウトを...
Functions for Exiting OWNER Mode <ul><li>queue_end </li></ul><ul><ul><li>Deletes the owned row and exits OWNER mode </li><...
Receiving a Message (revisited) <ul><li>while (1) { </li></ul><ul><li>SELECT queue_wait(‘queue’);; </li></ul><ul><li>my @r...
<ul><li>Applications of Q4M </li></ul>
Connecting Distant Servers <ul><li>Pathtraq uses Q4M queues and a relay to communicate with its content analysis service r...
User Notifications <ul><li>For sending notifications from web services </li></ul><ul><ul><ul><li>ウェブサービスでユーザ通知送信のために使用可能 <...
Asynchronous Updates <ul><li>Asynchronous updates leads to faster response of web services </li></ul><ul><ul><ul><li>非同期更新...
Scheduling Web Crawlers <ul><li>Web crawlers with retry-on-error </li></ul><ul><ul><ul><li>リトライ機能つき  Web  クローラー </li></ul>...
<ul><li>Current Limitations and the Future of Q4M </li></ul>
Current Limitations <ul><li>Table compactions is a blocking operation </li></ul><ul><ul><ul><li>テーブルコンパクションが他の処理をブロックする </...
Future of Q4M <ul><li>API for consuming multiple rows at once </li></ul><ul><ul><ul><li>複数行を一度に受信処理するモード </li></ul></ul></...
Configuration Options of Q4M <ul><li>--with-sync=no|fsync|fdatasync|fcntl </li></ul><ul><ul><li>Controls synchronization t...
Upcoming SlideShare
Loading in …5
×

Q4M - a high-performance message queue for MySQL

11,887
-1

Published on

Explains the design, usage, and user senarios of Q4M.

Published in: Technology, Business
1 Comment
9 Likes
Statistics
Notes
No Downloads
Views
Total Views
11,887
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
106
Comments
1
Likes
9
Embeds 0
No embeds

No notes for slide

Q4M - a high-performance message queue for MySQL

  1. 1. Q4M a high-performance message queue for MySQL Cybozu Labs, Inc. Kazuho Oku
  2. 2. <ul><li>What is a Message Queue? </li></ul><ul><li>from Distributed Systems (Tanenbaum / Van Steen) </li></ul>
  3. 3. What is a Message Queue? <ul><li>Middleware for persistent asynchronous communication </li></ul><ul><ul><ul><li>持続的な非同期コミュニケーションのためのミドルウェア </li></ul></ul></ul><ul><li>A.k.a. Message-Oriented Middleware </li></ul><ul><ul><ul><li>別名 : MOM </li></ul></ul></ul>
  4. 4. Message Queue (cont.) <ul><li>MQ is an intermediate storage </li></ul><ul><ul><ul><li>MQ は中間的なストレージ </li></ul></ul></ul><ul><ul><li>vs. RDBMS (long-term persistent storage) </li></ul></ul><ul><ul><ul><li>vs. RDBMS ( 永続的なストレージ ) </li></ul></ul></ul><ul><li>Senders and/or receivers may go down </li></ul><ul><ul><li>送受信プロセスが任意に停止可能 </li></ul></ul>
  5. 5. Minimal Configuration of a MQ ( 最小構成 ) <ul><li>Senders and receivers access a single queue </li></ul><ul><ul><ul><li>送信プロセスと受信プロセスが同一のキューにアクセス </li></ul></ul></ul>Sender Receiver Queue
  6. 6. MQ and Relays <ul><li>Separate queue for sender and receiver </li></ul><ul><ul><ul><li>送信プロセスと受信プロセスが別個にキューをもつ </li></ul></ul></ul><ul><li>Messages relayed between queues </li></ul><ul><ul><ul><li>メッセージリレーでキュー間を接続 </li></ul></ul></ul>Sender Receiver Relay Queue Queue
  7. 7. Merits of Message Relays ( リレーの意義 ) <ul><li>Destination can be easily changed </li></ul><ul><ul><ul><li>宛先の変更が容易 </li></ul></ul></ul><ul><ul><li>Relays may transfer messages to different locations depending on the header </li></ul></ul><ul><ul><ul><li>ヘッダを見て転送先を変えるとか </li></ul></ul></ul><ul><li>Robustness against network failure </li></ul><ul><ul><ul><li>ネットワーク障害の影響を受けない </li></ul></ul></ul><ul><li>Logging and Multicasting, etc. </li></ul><ul><ul><ul><li>ロギングやマルチキャストが可能 </li></ul></ul></ul>
  8. 8. Message Brokers <ul><li>Transform (filter) messages within the relay agent </li></ul><ul><ul><ul><li>リレーエージェント内でメッセージの変換を行う </li></ul></ul></ul><ul><li>Publish / subscribe model </li></ul><ul><ul><li>Separation between logic and data transfer </li></ul></ul><ul><ul><ul><li>ロジックとデータ転送の分離 </li></ul></ul></ul>
  9. 9. Q4M <ul><li>Q4M (can) support all models </li></ul><ul><ul><ul><li>Q4M は全モデルに対応 ( 可能 ) </li></ul></ul></ul><ul><li>Message transfer by q4m-forward </li></ul><ul><ul><ul><li>メッセージ転送は q4m-forward で </li></ul></ul></ul><ul><li>APIs for routers and message brokers </li></ul><ul><ul><ul><li>ルータやメッセージブローカ用の API </li></ul></ul></ul>
  10. 10. <ul><li>Architecture of Q4M </li></ul>
  11. 11. File Format <ul><li>Log-based file format </li></ul><ul><ul><ul><li>ログベースのファイルフォーマット </li></ul></ul></ul><ul><li>Each row ( queue_row_t ) consist of: </li></ul><ul><ul><ul><li>Row Header (4 bytes) -- row type and body size </li></ul></ul></ul><ul><ul><ul><li>Source Row ID -- optional, for message transfer </li></ul></ul></ul><ul><ul><ul><li>Body -- < 512MB </li></ul></ul></ul>
  12. 12. Row Header <ul><li>Row Types: 3bits </li></ul><ul><ul><ul><li>Normal row </li></ul></ul></ul><ul><ul><ul><li>Removed row </li></ul></ul></ul><ul><ul><ul><li>Normal row with source rowid </li></ul></ul></ul><ul><ul><ul><li>Removed row with source rowid </li></ul></ul></ul><ul><ul><ul><li>Checksum </li></ul></ul></ul><ul><ul><ul><ul><li>Used to recovering successful INSERTs </li></ul></ul></ul></ul><ul><ul><ul><ul><li>29-bit adler32 </li></ul></ul></ul></ul><ul><ul><ul><li>Number of Rows Removed </li></ul></ul></ul><ul><ul><ul><ul><li>Used for recovering rowids </li></ul></ul></ul></ul><ul><li>Body Size: 29bits </li></ul>
  13. 13. Data Consistency <ul><li>INSERTs group-commited using writev </li></ul><ul><ul><ul><li>INSERT は writev を利用してグループコミット </li></ul></ul></ul><ul><ul><li>Each group has a checksum prepended </li></ul></ul><ul><ul><ul><li>各グループの先頭にチェックサムが存在 </li></ul></ul></ul><ul><ul><li>Crash recovery is an iteration through the logfile until reaching a block with an invalid checksum </li></ul></ul><ul><ul><ul><li>クラッシュリカバリの際は、チェックサム比較に失敗するブロックに到達するまでログを舐める </li></ul></ul></ul>
  14. 14. Data Consistency (cont.) <ul><li>DELETEs are modifs. of row headers </li></ul><ul><ul><li>DELETE は行ヘッダの型変更 </li></ul></ul><ul><ul><ul><ul><li>Normal row -> Removed row </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Normal row w. source rowid -> Removed row w. source rowid </li></ul></ul></ul></ul><ul><ul><li>Either by pwrite or mmap(PROT_WRITE) + msync </li></ul></ul><ul><ul><ul><ul><li>pwrite あるいは mmap(PROT_WRITE) + msync で削除 </li></ul></ul></ul></ul><ul><ul><li>Multirow DELETEs aren’t atomic </li></ul></ul><ul><ul><ul><ul><li>複数行 DELETE は非アトミック </li></ul></ul></ul></ul><ul><ul><ul><li>Not a problem; no API to consume multiple rows at once </li></ul></ul></ul><ul><ul><ul><ul><li>複数行を一度に消費する API がないので問題ではない </li></ul></ul></ul></ul>
  15. 15. Row IDs <ul><li>64bit Int </li></ul><ul><li>Starts from one and incremented per each row </li></ul><ul><ul><li>1 から始まって1行毎にインクリメント </li></ul></ul><ul><li>Used to detect and block duplicates by relays </li></ul><ul><ul><ul><li>メッセージリレーが重複を検出・除外するために使用 </li></ul></ul></ul>
  16. 16. Restritions in SQL of Q4M <ul><li>No indexes </li></ul><ul><li>No UPDATE </li></ul><ul><ul><li>Why need UPDATE a queue row? </li></ul></ul><ul><ul><ul><li>キューのデータを UPDATE する必要なんてない </li></ul></ul></ul><ul><li>SELECT and DELETE are supported for queue administration </li></ul><ul><ul><ul><li>SELECT DELETE を使ってキューを管理可能 </li></ul></ul></ul><ul><ul><li>ex. select count(*) from queue; </li></ul></ul>
  17. 17. Why no Indexes Support? <ul><li>There is generally no reason to support indexes with mysql-based queues, since mysql cannot handle a large number of connections requesting data under various conditions </li></ul><ul><ul><ul><li>インデックスが必要になるほど多様な条件を指定して同時に購読できるほど、 mysql の同時接続数は大きくない </li></ul></ul></ul>
  18. 18. Why no Indexes Support? <ul><li>Instead use conditional subscription </li></ul><ul><ul><ul><li>Conditional subscription 機能を使うべき </li></ul></ul></ul><ul><li>Or route data after reading from Q4M </li></ul><ul><ul><ul><li>あるいは Q4M からデータを読んだ後にルーティングすべき </li></ul></ul></ul>
  19. 19. <ul><li>Tutorial </li></ul>
  20. 20. Sending a Message <ul><li>INSERT into queue values (1,’Hello World!’); </li></ul>
  21. 21. Receiving a Message <ul><li>while (1) { </li></ul><ul><li>SELECT queue_wait(‘queue’);; </li></ul><ul><li>my @row = SELECT ROW * FROM queue; </li></ul><ul><li>or next; </li></ul><ul><li>… </li></ul><ul><li>} </li></ul><ul><li>SELECT queue_end(‘queue’);; </li></ul>
  22. 22. OWNER Mode and NON-OWNER Mode
  23. 23. OWNER Mode and NON-OWNER Mode (cont.) <ul><li>Within OWNER mode, only the owned row are visible </li></ul><ul><ul><ul><li>OWNER モードでは、所有中の行のみが見える </li></ul></ul></ul><ul><li>Within NON-OWNER mode, rows owned by other connections are invisible </li></ul><ul><ul><ul><li>NON-OWNER モードでは、他の接続が所有している行は見えない </li></ul></ul></ul>
  24. 24. Function for Entering OWNER Mode <ul><li>queue_wait </li></ul><ul><ul><li>Used to enter OWNER mode </li></ul></ul><ul><ul><ul><li>OWNER モードに ( 再 ) 切替するために使用 </li></ul></ul></ul><ul><ul><li>When called within OWNER mode, the owned row is deleted </li></ul></ul>
  25. 25. queue_wait <ul><li>Two forms </li></ul><ul><ul><ul><li>queue_wait(tbl_cond) </li></ul></ul></ul><ul><ul><ul><li>queue_wait(tbl_cond,[tbl_cond…,timeout]) </li></ul></ul></ul>
  26. 26. queue_wait(tbl_cond) <ul><li>Wait max. 60 seconds until any data becomes available on given table under given condition </li></ul><ul><ul><ul><li>最大 60 秒間、指定された条件に合致する行が指定されたテーブルに登録されるまでブロック </li></ul></ul></ul><ul><li>Returns 1 if successful, 0 if no data </li></ul><ul><ul><ul><li>成功すれば 1 、データがなければ 0 </li></ul></ul></ul><ul><li>Enters OWNER mode even if no data </li></ul><ul><ul><ul><li>データがない場合でも OWNER モードへ切り替わる </li></ul></ul></ul>
  27. 27. queue_wait(tbl_cond) <ul><li>Table name (and optionally condition) should be specified </li></ul><ul><ul><ul><li>テーブル名と ( オプションで条件 ) を指定 </li></ul></ul></ul><ul><li>Only numeric columns may be used within condition </li></ul><ul><ul><ul><li>条件節では数値型のカラムのみ使用可能 </li></ul></ul></ul><ul><ul><li>See queue_share_t::init_fixed_fields </li></ul></ul><ul><ul><li>tbl_cond ::= table [ “:” cond] </li></ul></ul>
  28. 28. queue_wait(tbl_cond) (cont.) <ul><li>SELECT queue_wait(‘table:v<3’); </li></ul>
  29. 29. queue_wait(tbl_cond,[tbl_cond…,timeout]) <ul><li>Accepts multiple tables and timeout </li></ul><ul><ul><li>複数のテーブルとタイムアウトを指定可能 </li></ul></ul><ul><li>Data searched from leftmost table to right </li></ul><ul><ul><ul><li>データは左端のテーブルから右方向に探索 </li></ul></ul></ul><ul><li>Returns table index (the leftmost table is 1) of the newly owned row </li></ul><ul><ul><ul><li>獲得した行の属するテーブルのインデックス ( 左端のテーブルが 1) を返す </li></ul></ul></ul>
  30. 30. Functions for Exiting OWNER Mode <ul><li>queue_end </li></ul><ul><ul><li>Deletes the owned row and exits OWNER mode </li></ul></ul><ul><ul><ul><li>所有中の行を削除して OWNER モードを脱出 </li></ul></ul></ul><ul><li>queue_abort </li></ul><ul><ul><li>Releases (instead of deleting) the owned row and exits OWNER mode </li></ul></ul><ul><ul><ul><li>所有中の行を非所有状態に変更して OWNER モードを脱出 </li></ul></ul></ul><ul><ul><li>Close of a MySQL connection does the same thing </li></ul></ul><ul><ul><ul><li>MySQL の接続切断時も同様 </li></ul></ul></ul>
  31. 31. Receiving a Message (revisited) <ul><li>while (1) { </li></ul><ul><li>SELECT queue_wait(‘queue’);; </li></ul><ul><li>my @row = SELECT ROW * FROM queue; </li></ul><ul><li>or next; </li></ul><ul><li>if (consume_row(@row) != SUCCESS) { </li></ul><ul><li>exit(1); </li></ul><ul><li>} </li></ul><ul><li>} </li></ul>
  32. 32. <ul><li>Applications of Q4M </li></ul>
  33. 33. Connecting Distant Servers <ul><li>Pathtraq uses Q4M queues and a relay to communicate with its content analysis service running in a different iDC </li></ul><ul><ul><ul><li>Pathtraq では、 Q4M キューとメッセージリレーを使用して、別の iDC にあるコンテンツ分析サービスと接続している </li></ul></ul></ul>Content Analysis Service Pathtraq DB MySQL conn. over SSL,gzip Queue Queue
  34. 34. User Notifications <ul><li>For sending notifications from web services </li></ul><ul><ul><ul><li>ウェブサービスでユーザ通知送信のために使用可能 </li></ul></ul></ul>DB App. Logic SMTP Agent IM Agent Queue(s)
  35. 35. Asynchronous Updates <ul><li>Asynchronous updates leads to faster response of web services </li></ul><ul><ul><ul><li>非同期更新に利用することでウェブサービスの応答性を向上可能 </li></ul></ul></ul>DB App. Logic DB Queue
  36. 36. Scheduling Web Crawlers <ul><li>Web crawlers with retry-on-error </li></ul><ul><ul><ul><li>リトライ機能つき Web クローラー </li></ul></ul></ul>URL DB Spiders Re- scheduler Store Result Read URL If failed to fetch, store URL in retry queue Request Queue Retry Queue
  37. 37. <ul><li>Current Limitations and the Future of Q4M </li></ul>
  38. 38. Current Limitations <ul><li>Table compactions is a blocking operation </li></ul><ul><ul><ul><li>テーブルコンパクションが他の処理をブロックする </li></ul></ul></ul><ul><ul><li>Runs when live data becomes <25% of log file </li></ul></ul><ul><ul><ul><li>ライブデータが 25% 以下になると実行 </li></ul></ul></ul><ul><li>Relays are slow </li></ul><ul><ul><li>Not a problem for Cybozu Labs :-p </li></ul></ul>
  39. 39. Future of Q4M <ul><li>API for consuming multiple rows at once </li></ul><ul><ul><ul><li>複数行を一度に受信処理するモード </li></ul></ul></ul><ul><ul><li>Necessary for speeding up relays </li></ul></ul><ul><ul><ul><li>リレーの高速化に必要 </li></ul></ul></ul><ul><li>Simple “delete-on-SELECT” mode for novices </li></ul><ul><ul><ul><li>初心者向けの「 SELECT 時に削除」モード </li></ul></ul></ul><ul><li>Support for transaction w. InnoDB? </li></ul><ul><ul><ul><li>InnoDB とのトランザクション処理 </li></ul></ul></ul>
  40. 40. Configuration Options of Q4M <ul><li>--with-sync=no|fsync|fdatasync|fcntl </li></ul><ul><ul><li>Controls synchronization to disk, see --help for detail </li></ul></ul><ul><li>--enable-mmap </li></ul><ul><ul><li>Mmap’ed reads lead to higher throughput </li></ul></ul><ul><li>--enable-mmap-writes </li></ul><ul><ul><li>DELETEs commited using mmap(PROT_WRITE) and msync, recommended on linux>=2.6.20 if you need really high performance </li></ul></ul>
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×