Q4M Microblogcon

Q4M a high-performance message queue for MySQL Cybozu Labs, Inc. Kazuho Oku

Introduction of Q4M ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Design Goals of Q4M ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

[object Object],[object Object]

What is a Message Queue? ,[object Object],[object Object],[object Object],[object Object]

Message Queue (cont.) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Minimal Configuration of a MQ ( 最小構成 ) ,[object Object],[object Object],Sender Receiver Queue

MQ and Relays ,[object Object],[object Object],[object Object],[object Object],Sender Receiver Relay Queue Queue

Merits of Message Relays ( リレーの意義 ) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Message Brokers ,[object Object],[object Object],[object Object],[object Object],[object Object]

Q4M as a Message Queue (MQ としての Q4M) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Internal Architecture of Q4M ,[object Object],[object Object],[object Object],[object Object]

Benchmark ,[object Object],[object Object],[object Object],[object Object],$ VAR_LENGTH=504 CONCURRENCY=50 MESSAGES=100000 DBI='dbi:mysql:test;mysql_socket=/tmp/mysql51.sock' DBI_USER=root MYSQL_SOCKET='/tmp/mysql51.sock' USE_C_CLIENT=1 t/05-multirw.t 1..4 ok 1 - check number of messages ok 2 - min value of received message ok 3 - max value of received message ok 4 - should have no rows in table Multi-reader-writer benchmark result: Number of messages: 100000 Number of readers: 50 Elapsed: 13.063 seconds Throughput: 7655.070 mess./sec.

Benchmark (cont.) ,[object Object],[object Object],[object Object]

Sending a Message ,[object Object]

Receiving a Message ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

OWNER Mode and NON-OWNER Mode (cont.) ,[object Object],[object Object],[object Object],[object Object]

Function for Entering OWNER Mode ,[object Object],[object Object],[object Object],[object Object]

queue_wait ,[object Object],[object Object],[object Object]

queue_wait(tbl_cond) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

queue_wait(tbl_cond) (cont.) ,[object Object]

queue_wait(tbl_cond,[tbl_cond…,timeout]) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Functions for Exiting OWNER Mode ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Receiving a Message (revisited) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Restrictions in SQL of Q4M ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Why no Indexes Support? ,[object Object],[object Object]

Why no Indexes Support? ,[object Object],[object Object],[object Object],[object Object]

Configuration Options of Q4M ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

My Build Options ,[object Object],[object Object]

Connecting Distant Servers ,[object Object],[object Object],Content Analysis Service Pathtraq DB MySQL conn. over SSL,gzip Queue Queue

User Notifications ,[object Object],[object Object],DB App. Logic SMTP Agent IM Agent Queue(s)

Asynchronous Updates ,[object Object],[object Object],DB App. Logic DB Queue

Scheduling Web Crawlers ,[object Object],[object Object],[object Object],URL DB Spiders Re- scheduler Store Result Read URL If failed to fetch, store URL in retry queue Request Queue Retry Queue

Prioritized Task Scheduling ,[object Object],[object Object],[object Object],Content & Keyword DB Request Queue Keyword Analyzer Store Keywords Read URL ID Read Content URL ID Prio 0 1 0

Prioritized Task Scheduling (cont.) ,[object Object],[object Object],[object Object],Content & Keyword DB Request Queue Keyword Analyzer Store Keywords Read URL ID Read Content URL ID Prio 0 1 0

Q4M Microblogcon

Recommended

Recommended

More Related Content

What's hot

What's hot (7)

Viewers also liked

Viewers also liked (20)

Similar to Q4M Microblogcon

Similar to Q4M Microblogcon (20)

More from Kazuho Oku

More from Kazuho Oku (20)

Recently uploaded

Recently uploaded (14)

Q4M Microblogcon