SlideShare a Scribd company logo
1 of 53
Download to read offline
Using Q4M
a message queue storage engine for MySQL


           Cybozu Labs, Inc.
             Kazuho Oku
Background




                            2
Apr 22 2009     Using Q4M
Who am I?

   Name: Kazuho Oku (奥一穂)
   Original Developer of Palmscape / Xiino
        The oldest web browser for Palm OS
   Working at Cybozu Labs since 2005
        Research subsidiary of Cybozu, Inc. in Japan




                                                        3
Apr 22 2009                 Using Q4M
About Cybozu, Inc.

   Japan’s largest groupware vendor
        Mostly provides as software products, not as
         services
        Some of our apps bundle MySQL as storage




                                                        4
Apr 22 2009                 Using Q4M
About Pathtraq

   Started in Aug. 2007
   Web ranking service
        One of Japan’s largest
        like Alexa, but semi-realtime, and per-page
        running on MySQL
   Need for a fast and reliable message
    relay
        for communication between the main server and
         content analysis server(s)
                                                         5
Apr 22 2009                 Using Q4M
Design Goals of Q4M

   Robust
        Do not lose data on OS crash or power failure
   Fast
        Transfer thousands of messages per second
   Easy to Use
        Use SQL for access / maintenance
        Integration into MySQL
              no more separate daemons to take care of


                                                          6
Apr 22 2009                      Using Q4M
What is a Message Queue?




                                         7
Apr 22 2009            Using Q4M
What is a Message Queue?

   Middleware for persistent asynchronous
    communication
        communicate between fixed pairs (parties)
        a.k.a. Message Oriented Middleware
   MQ is intermediate storage
        RDBMS is persistent storage
   Senders / receivers may go down


                                                     8
Apr 22 2009                 Using Q4M
Minimal Configuration of a MQ

   Senders and receivers access a single
    queue




               Sender                Receiver

                         Queue



                                                9
Apr 22 2009              Using Q4M
MQ and Relays

   Separate queue for sender and receiver
   Messages relayed between queues



                                Relay
              Sender                               Receiver

                       Queue               Queue



                                                              10
Apr 22 2009                    Using Q4M
Merits of Message Relays

   Destination can be changed easily
        Relays may transfer messages to different
         locations depending on their headers
   Robustness against network failure
        no loss or duplicates when the relay fails
   Logging and Multicasting, etc.



                                                      11
Apr 22 2009                  Using Q4M
Message Brokers

   Publish / subscribe model
        Separation between components and their
         integration
        Components read / write to predefined queues
        Integration is definition of routing rules between
         the message queues
        Messages are often transformed (filtered) within
         the relay agent



                                                              12
Apr 22 2009                  Using Q4M
What about Q4M?

   Q4M itself is a message queue
   Can connect Q4M instances to create a
    message relay
   Provides API for creating message relays
    and brokers




                                           13
Apr 22 2009             Using Q4M
Performance of Q4M

   over 7,000 mess/sec.
        message size: avg. 512 bytes
        syncing to disk
   Outperforming most needs
        if you need more, just scale out
        Can coexist with other storage engines without
         sacrificing their performance


        see http://labs.cybozu.co.jp/blog/kazuhoatwork/2008/06/q4m_06_release_and_benchmarks.php
                                                                                                   14
Apr 22 2009                                   Using Q4M
Applications of Q4M




                                    15
Apr 22 2009          Using Q4M
Asynchronous Updates

   Mixi (Japan's one of the largest SNS)
    uses Q4M to buffer writes to DB, to
    offload peak demands
                         from http://alpha.mixi.co.jp/blog/?p=272




                                                               16
Apr 22 2009            Using Q4M
Connecting Distant Servers

   Pathtraq uses Q4M to create a relay
    between its database and content
    analysis processes
                         → Contents to be analyzed →



                                                        Content
                                MySQL conn.             Analysis
              Pathtraq
                                over SSL,gzip          Processes
                DB



                         ← Results of the analysis ←

                                                                   17
Apr 22 2009                        Using Q4M
To Prefetch Data

   livedoor Reader (web-based feed
    aggregator) uses Q4M to prefetch data
    from database to memcached
   uses Q4M for scheduling web crawlers
    as well
                 from http://d.hatena.ne.jp/mala/20081212/1229074359




                                                                  18
Apr 22 2009                 Using Q4M
Scheduling Web Crawlers

   Web crawlers with retry-on-error
   Sample code included in Q4M dist.
                                       If failed to fetch, store URL in retry queue
              Store Result

                                       Read URL
                             Spiders
              URL
              DB

                                        Request Queue              Retry Queue
                                                           Re-
                                                        scheduler



                                                                                      19
Apr 22 2009                            Using Q4M
Delayed Content Generation

   Hatetter(RSS feed-to-twitter-API
    gateway) uses Q4M to delay content
    generation
        Source code: github.com/yappo/website-hatetter




                                                          20
Apr 22 2009                Using Q4M
User Notifications

   For sending notifications from web
    services


                              DB


              App. Logic                SMTP Agent




                           Queue(s)
                                         IM Agent


                                                     21
Apr 22 2009                 Using Q4M
Installing Q4M




                               22
Apr 22 2009       Using Q4M
Installing Q4M

   Compatible with MySQL 5.1
   Download from q4m.31tools.com
        Binary releases available for some platforms
   Installing from source:
        requires source code of MySQL
        ./configure && make && make install
        run support-files/install.sql



                                                        23
Apr 22 2009                 Using Q4M
Configuration Options of Q4M

   --with-sync=no|fsync|fdatasync|fcntl
        Controls synchronization to disk
        default: fdatasync on linux
   --enable-mmap
        Mmap’ed reads lead to higher throughput
        default: yes
   --with-delete=pwrite|msync
        msyncrecommended on linux>=2.6.20 if you
         need really high performance
                                                    24
Apr 22 2009                  Using Q4M
Q4M Basics




                            25
Apr 22 2009     Using Q4M
The Model

        Various publishers write to queue
        Set of subscribers consume the entries in queue



                        Publisher




                                    Q4M table
         Publisher


                                                 Subscribers

                     Publisher
                                                               26
Apr 22 2009                          Using Q4M
Creating a Q4M Table

    ENGINE=QUEUE creates
                                mysql> CREATE TABLE qt (
                                  -> id int(10) unsigned NOT NULL,

     a Q4M table
                                  -> message varchar(255) NOT NULL
                                  -> ) ENGINE=QUEUE;
                                Query OK, 0 rows affected (0.42 sec)
    No primary keys or
     indexes
    Sorted by insertion
     order (it’s a queue)




                                                                       27
Apr 22 2009             Using Q4M
Modifying Data on a Q4M Table

    No restrictions for
                                mysql> INSERT INTO qt (id,message)
                                  -> VALUES

     INSERT and DELETE
                                  -> (1,'Hello'),
                                  -> (2,'Bonjour'),
                                  -> (3,'Hola');
    No support for UPDATE      Query OK, 3 rows affected (0.02 sec)

                                mysql> SELECT * FROM qt;
                                +----+---------+
                                | id | message |
                                +----+---------+
                                | 1 | Hello |
                                | 2 | Bonjour |
                                | 3 | Hola |
                                +----+---------+
                                3 rows in set (0.00 sec)




                                                                       28
Apr 22 2009             Using Q4M
SELECT from a Q4M Table

   Works the same as
                                  mysql> SELECT * FROM qt;
                                  +----+---------+
                                  | id | message |
    other storage                 +----+---------+
                                  | 1 | Hello |
    engines                       | 2 | Bonjour |
                                  | 3 | Hola |
                                  +----+---------+
   SELECT COUNT(*) is            3 rows in set (0.00 sec)


    cached                        mysql> SELECT COUNT(*) FROM qt;
                                  +----------+
                                  | COUNT(*) |
                                  +----------+
                                  |      3|
                                  +----------+
                                  1 row in set (0.00 sec)




              How to subscribe to a queue?
                                                                    29
Apr 22 2009               Using Q4M
Calling queue_wait()

    After calling, only one
                                       mysql> SELECT * FROM qt;
                                       +----+---------+
                                       | id | message |
     row becomes visible               +----+---------+
                                       | 1 | Hello |

     from the connection               | 2 | Bonjour |
                                       | 3 | Hola |
                                       +----+---------+
                                       3 rows in set (0.00 sec)

                                       mysql> SELECT queue_wait('qt');
                                       +------------------+
                                       | queue_wait('qt') |
                                       +------------------+
                                       |             1|
                                       +------------------+
                                       1 row in set (0.00 sec)

                                       mysql> SELECT * FROM qt;
                                       +----+---------+
                                       | id | message |
                                       +----+---------+
                                       | 1 | Hello |
                                       +----+---------+
                                       1 row in set (0.00 sec)



                                                                         30
Apr 22 2009                    Using Q4M
OWNER Mode and NON-OWNER Mode

   In OWNER mode, only the OWNED row
    is visible
              OWNED row becomes invisible from other connections
              rows of other storage engines are visible


    NON-OWNER Mode                            OWNER Mode
                              queue_wait()
    1,'Hello'                                 1,'Hello'
    2,'Bonjour'
                              queue_end()
    3,'Hola'
                             queue_abort()

                                                                31
Apr 22 2009                      Using Q4M
Returning to NON-OWNER mode

    By calling
                                     mysql> SELECT QUEUE_ABORT();
                                     +---------------+

     queue_abort, the
                                     | QUEUE_ABORT() |
                                     +---------------+

     connection returns to
                                     |          1|
                                     +---------------+

     NON-OWNER mode
                                     1 row in set (0.00 sec)

                                     mysql> SELECT * FROM qt;
                                     +----+---------+
                                     | id | message |
                                     +----+---------+
                                     | 1 | Hello |
                                     | 2 | Bonjour |
                                     | 3 | Hola |
                                     +----+---------+
                                     3 rows in set (0.01 sec)




                                                                    32
Apr 22 2009                  Using Q4M
Consuming a Row

    By calling
                                mysql> SELECT queue_wait('qt');
                                (snip)
                                mysql> SELECT * FROM qt;
     queue_end, the OWNED       +----+---------+
                                | id | message |

     row is deleted, and        +----+---------+
                                | 1 | Hello |

     connection returns to
                                +----+---------+
                                1 row in set (0.01 sec)

     NON-OWNER mode             mysql> SELECT queue_end();
                                +-------------+
                                | queue_end() |
                                +-------------+
                                |        1|
                                +-------------+
                                1 row in set (0.01 sec)

                                mysql> SELECT * FROM qt;
                                +----+---------+
                                | id | message |
                                +----+---------+
                                | 2 | Bonjour |
                                | 3 | Hola |
                                +----+---------+
                                2 rows in set (0.00 sec)


                                                                  33
Apr 22 2009             Using Q4M
Writing a Subscriber

    Call two functions: queue_wait, queue_end
    Multiple subscribers can be run concurrently
         each row in the queue is consumed only once


   while (true) {
     SELECT queue_wait('qt'); # switch to owner mode
     rows := SELECT * FROM qt; # obtain data
     if (count(rows) != 0) # if we have any data, then
       handle_row(rows[0]); # consume the row
     SELECT queue_end();         # erase the row from queue
   }

                                                              34
Apr 22 2009                         Using Q4M
Writing a Subscriber (cont'd)

   Or call queue_wait as a condition
        Warning: conflicts with trigger-based insertions



   while (true) {
     rows := SELECT * FROM qt WHERE queue_wait('qt');
     if (count(rows) != 0)
       handle_row(rows[0]);
     SELECT queue_end();
   }


                                                            35
Apr 22 2009                      Using Q4M
The Model – with code

                    INSERT INTO queue ...



                            Publisher
                                                         while (true) {
                                                           rows := SELECT * FROM qt
                                                               WHERE queue_wait('qt');
                                                           if (count(rows) != 0)
                                                             handle_row(rows[0]);
    INSERT INTO queue ...
                                                           SELECT queue_end();
                                                         }
                                            Q4M table
              Publisher

                                                                       Subscribers
                INSERT INTO queue ...


                      Publisher

                                                                                         36
Apr 22 2009                                  Using Q4M
Three Functions in Detail




                                          37
Apr 22 2009            Using Q4M
queue_wait(table)

   Enters OWNER mode
   0〜1 row becomes OWNED
        Enters OWNER mode even if no rows were
         available
        Default timeout: 60 seconds
        Returns 1 if a row is OWNED (0 on timeout)
   If called within OWNER mode, the
    owned row is deleted

                                                      38
Apr 22 2009                 Using Q4M
Revisiting Subscriber Code

   Calls to queue_end just before
    queue_wait can be omitted


   while (true) {
     rows := SELECT * FROM qt WHERE queue_wait('qt');
     if (count(rows) != 0)
       handle_row(rows[0]);
     SELECT queue_end();
   }



                                                        39
Apr 22 2009                      Using Q4M
Conditional queue_wait()

   Consume rows of certain condition
        Rows that do not match will be left untouched
        Only numeric columns can be checked
        Fast - condition tested once per each row

   examples:
    SELECT queue_wait('table:(col_a*3)+col_b<col_c');
    SELECT queue_wait('table:retry_count<5');




                                                         40
Apr 22 2009                       Using Q4M
queue_wait(tbl_cond,[tbl_cond…,timeout])

   Accepts multiple tables and timeout
   Data searched from leftmost table to
    right
   Returns table index (the leftmost table is
    1) of the newly owned row
        Returns zero if no rows are being owned

   example:
    SELECT queue_wait('table_A','table_B',60);

                                                   41
Apr 22 2009                        Using Q4M
Functions for Exiting OWNER Mode

   queue_end
        Deletes the owned row and exits OWNER mode
   queue_abort
        Releases (instead of deleting) the owned row and
         exits OWNER mode
        Close of a MySQL connection does the same thing




                                                        42
Apr 22 2009                 Using Q4M
Relaying and Routing Messages




                                              43
Apr 22 2009               Using Q4M
The Problem

   Relay (or router) consists of more than 3
    processes, 2 conns
   No losses, no duplicates on crash or
    disconnection



              Q4M Table                   Q4M Table
                          Relay Program
               (source)                    (dest.)


                                                      44
Apr 22 2009                 Using Q4M
Internal Row ID

   Every row have a internal row ID
        invisible from Q4M table definition
        monotonically increasing 64-bit integer
   Used for detecting duplicates
        Use two functions to skip duplicates
        Data loss prevented by using queue_wait /
         queue_end



                                                     45
Apr 22 2009                  Using Q4M
queue_rowid()

   Returns row ID of the OWNED row (if
    any)
        Returns NULL if no row is OWNED
   Call when retrieving data from source




                                            46
Apr 22 2009                Using Q4M
queue_set_srcid(src_tbl_id, mode, src_row_id)

   Call before inserting a row to destination
    table
        Checks if the row is already inserted into the
         table, and ignores next INSERT if true
   Parameters:
        src_tbl_id - id to determine source table (0〜63)
        mode - quot;aquot; to drop duplicates, quot;wquot; to reset
        src_row_id - row ID obtained from source table


                                                            47
Apr 22 2009                  Using Q4M
Pseudo Code

   Relays data from src_tbl to dest_tbl
   while (true) {
    # wait for data
    SELECT queue_wait(src_tbl) =>src_db;



    # read row and rowid
    row := (SELECT * FROM src_tbl =>src_db);
   rowid := (SELECT queue_rowid() =>src_db);



    # insert the row after setting srcid
    SELECT queue_set_srcid(src_tbl_id, 'a', rowid) =>dest_db;
    INSERT INTO dest_tbl (row) =>dest_db;
                                                                48
Apr 22 2009                            Using Q4M
q4m-forward

   Simple forwarder script
        installed into mysql-dir/bin

   usage: q4m-forward [options] src_addrdest_addr
   example:
    % support-files/q4m-forward 
     quot;dbi:mysql:database=db1;table=tbl1;user=foo;password=XXXquot; 
     quot;dbi:mysql:database=db2;table=tbl2;host=bar;user=fooquot;
   options:
    --reset    reset duplicate check info.
    --sender=idx slot no. used for checking duplicates (0..63, default: 0)
    --help




                                                                             49
Apr 22 2009                                         Using Q4M
Limitations and the Future of Q4M




                                                  50
Apr 22 2009                 Using Q4M
Things that Need to be Fixed

   Table compactions is a blocking
    operation
        runs when live data becomes <25% of log file
        very bad, though not as bad as it seems
              it's fast since it's a sequential write operation

   Relays are slow
        since transfer is done row-by-row
   Binlog does not work
        since MQ replication should be synchronous
                                                                   51
Apr 22 2009                          Using Q4M
Future of Q4M

   2-phase commit with other storage
    engines (maybe)
        queue consumption and InnoDB updates can
         become atomic operation




                                                    52
Apr 22 2009               Using Q4M
Thank you

              http://q4m.31tools.com/




                                        53
Apr 22 2009          Using Q4M

More Related Content

What's hot

Hyper-V VMM ile Cloud computing
Hyper-V VMM ile Cloud computingHyper-V VMM ile Cloud computing
Hyper-V VMM ile Cloud computingAhmet Mutlu
 
Rdma presentation-kisti-v2
Rdma presentation-kisti-v2Rdma presentation-kisti-v2
Rdma presentation-kisti-v2balmanme
 
Munich 2016 - Z011601 Martin Packer - Parallel Sysplex Performance Topics topics
Munich 2016 - Z011601 Martin Packer - Parallel Sysplex Performance Topics topicsMunich 2016 - Z011601 Martin Packer - Parallel Sysplex Performance Topics topics
Munich 2016 - Z011601 Martin Packer - Parallel Sysplex Performance Topics topicsMartin Packer
 
Introduction to AtomPub Web Services
Introduction to AtomPub Web ServicesIntroduction to AtomPub Web Services
Introduction to AtomPub Web ServicesBen Ramsey
 
High speed networks and Java (Ryan Sciampacone)
High speed networks and Java (Ryan Sciampacone)High speed networks and Java (Ryan Sciampacone)
High speed networks and Java (Ryan Sciampacone)Chris Bailey
 
ソフトウェアでのパケット処理あれこれ〜何故我々はロードバランサを自作するに至ったのか〜
ソフトウェアでのパケット処理あれこれ〜何故我々はロードバランサを自作するに至ったのか〜ソフトウェアでのパケット処理あれこれ〜何故我々はロードバランサを自作するに至ったのか〜
ソフトウェアでのパケット処理あれこれ〜何故我々はロードバランサを自作するに至ったのか〜LINE Corporation
 
Tungsten University: Setup and Operate Tungsten Replicators
Tungsten University: Setup and Operate Tungsten ReplicatorsTungsten University: Setup and Operate Tungsten Replicators
Tungsten University: Setup and Operate Tungsten ReplicatorsContinuent
 
MOW2010: Under the Hood of Oracle Clusterware by Alex Gorbachev, Pythian
MOW2010: Under the Hood of Oracle Clusterware by Alex Gorbachev, PythianMOW2010: Under the Hood of Oracle Clusterware by Alex Gorbachev, Pythian
MOW2010: Under the Hood of Oracle Clusterware by Alex Gorbachev, PythianAlex Gorbachev
 
CloudStack for Java User Group
CloudStack for Java User GroupCloudStack for Java User Group
CloudStack for Java User GroupSebastien Goasguen
 
HA Clustering of PostgreSQL(replication)@2012.9.29 PG Study.
HA Clustering of PostgreSQL(replication)@2012.9.29 PG Study.HA Clustering of PostgreSQL(replication)@2012.9.29 PG Study.
HA Clustering of PostgreSQL(replication)@2012.9.29 PG Study.Takatoshi Matsuo
 
Windows offloaded data_transfer_steve_olsson
Windows offloaded data_transfer_steve_olssonWindows offloaded data_transfer_steve_olsson
Windows offloaded data_transfer_steve_olssonscsibeast
 
Optimize Performance of I/O-intensive Java applications Using Zero Copy
Optimize Performance of I/O-intensive Java applications Using Zero CopyOptimize Performance of I/O-intensive Java applications Using Zero Copy
Optimize Performance of I/O-intensive Java applications Using Zero CopyIndicThreads
 
OSGi Cloud Ecosystems (OSGi Users Forum Germany)
OSGi Cloud Ecosystems (OSGi Users Forum Germany)OSGi Cloud Ecosystems (OSGi Users Forum Germany)
OSGi Cloud Ecosystems (OSGi Users Forum Germany)David Bosschaert
 
Living the Easy Life with Rules-Based Autonomic Database Clusters
Living the Easy Life with Rules-Based Autonomic Database ClustersLiving the Easy Life with Rules-Based Autonomic Database Clusters
Living the Easy Life with Rules-Based Autonomic Database ClustersLinas Virbalas
 
Rabbit MQ introduction
Rabbit MQ introductionRabbit MQ introduction
Rabbit MQ introductionSitg Yao
 

What's hot (20)

Hyper-V VMM ile Cloud computing
Hyper-V VMM ile Cloud computingHyper-V VMM ile Cloud computing
Hyper-V VMM ile Cloud computing
 
Rdma presentation-kisti-v2
Rdma presentation-kisti-v2Rdma presentation-kisti-v2
Rdma presentation-kisti-v2
 
OSGi Cloud Ecosystems
OSGi Cloud EcosystemsOSGi Cloud Ecosystems
OSGi Cloud Ecosystems
 
Munich 2016 - Z011601 Martin Packer - Parallel Sysplex Performance Topics topics
Munich 2016 - Z011601 Martin Packer - Parallel Sysplex Performance Topics topicsMunich 2016 - Z011601 Martin Packer - Parallel Sysplex Performance Topics topics
Munich 2016 - Z011601 Martin Packer - Parallel Sysplex Performance Topics topics
 
Introduction to AtomPub Web Services
Introduction to AtomPub Web ServicesIntroduction to AtomPub Web Services
Introduction to AtomPub Web Services
 
High speed networks and Java (Ryan Sciampacone)
High speed networks and Java (Ryan Sciampacone)High speed networks and Java (Ryan Sciampacone)
High speed networks and Java (Ryan Sciampacone)
 
ソフトウェアでのパケット処理あれこれ〜何故我々はロードバランサを自作するに至ったのか〜
ソフトウェアでのパケット処理あれこれ〜何故我々はロードバランサを自作するに至ったのか〜ソフトウェアでのパケット処理あれこれ〜何故我々はロードバランサを自作するに至ったのか〜
ソフトウェアでのパケット処理あれこれ〜何故我々はロードバランサを自作するに至ったのか〜
 
cosbench-openstack.pdf
cosbench-openstack.pdfcosbench-openstack.pdf
cosbench-openstack.pdf
 
Tungsten University: Setup and Operate Tungsten Replicators
Tungsten University: Setup and Operate Tungsten ReplicatorsTungsten University: Setup and Operate Tungsten Replicators
Tungsten University: Setup and Operate Tungsten Replicators
 
Apache CloudStack AlpesJUG
Apache CloudStack AlpesJUGApache CloudStack AlpesJUG
Apache CloudStack AlpesJUG
 
MOW2010: Under the Hood of Oracle Clusterware by Alex Gorbachev, Pythian
MOW2010: Under the Hood of Oracle Clusterware by Alex Gorbachev, PythianMOW2010: Under the Hood of Oracle Clusterware by Alex Gorbachev, Pythian
MOW2010: Under the Hood of Oracle Clusterware by Alex Gorbachev, Pythian
 
Front-End Optimization (FEO)
Front-End Optimization (FEO)Front-End Optimization (FEO)
Front-End Optimization (FEO)
 
CloudStack for Java User Group
CloudStack for Java User GroupCloudStack for Java User Group
CloudStack for Java User Group
 
Cosbench apac
Cosbench apacCosbench apac
Cosbench apac
 
HA Clustering of PostgreSQL(replication)@2012.9.29 PG Study.
HA Clustering of PostgreSQL(replication)@2012.9.29 PG Study.HA Clustering of PostgreSQL(replication)@2012.9.29 PG Study.
HA Clustering of PostgreSQL(replication)@2012.9.29 PG Study.
 
Windows offloaded data_transfer_steve_olsson
Windows offloaded data_transfer_steve_olssonWindows offloaded data_transfer_steve_olsson
Windows offloaded data_transfer_steve_olsson
 
Optimize Performance of I/O-intensive Java applications Using Zero Copy
Optimize Performance of I/O-intensive Java applications Using Zero CopyOptimize Performance of I/O-intensive Java applications Using Zero Copy
Optimize Performance of I/O-intensive Java applications Using Zero Copy
 
OSGi Cloud Ecosystems (OSGi Users Forum Germany)
OSGi Cloud Ecosystems (OSGi Users Forum Germany)OSGi Cloud Ecosystems (OSGi Users Forum Germany)
OSGi Cloud Ecosystems (OSGi Users Forum Germany)
 
Living the Easy Life with Rules-Based Autonomic Database Clusters
Living the Easy Life with Rules-Based Autonomic Database ClustersLiving the Easy Life with Rules-Based Autonomic Database Clusters
Living the Easy Life with Rules-Based Autonomic Database Clusters
 
Rabbit MQ introduction
Rabbit MQ introductionRabbit MQ introduction
Rabbit MQ introduction
 

Viewers also liked

17) 11 (may, 2003) squid master this proxy server
17) 11 (may, 2003)   squid master this proxy server17) 11 (may, 2003)   squid master this proxy server
17) 11 (may, 2003) squid master this proxy serverswarup1435
 
MessagePack(msgpack): Compact and Fast Serialization Library
MessagePack(msgpack): Compact and Fast Serialization LibraryMessagePack(msgpack): Compact and Fast Serialization Library
MessagePack(msgpack): Compact and Fast Serialization LibraryTakatoshi Kondo
 
Phoenix Energy Performance Contracting Presentation
Phoenix   Energy Performance Contracting PresentationPhoenix   Energy Performance Contracting Presentation
Phoenix Energy Performance Contracting PresentationICF_HCD
 
25et_Bulgaria
25et_Bulgaria25et_Bulgaria
25et_BulgariaGavranica
 
Christmas in croatia eng
Christmas in croatia engChristmas in croatia eng
Christmas in croatia engGavranica
 
Inbound Marketing Cookbook
Inbound Marketing CookbookInbound Marketing Cookbook
Inbound Marketing CookbookEddie Choi
 
Zima sisljavic
Zima sisljavicZima sisljavic
Zima sisljavicGavranica
 
Story Telling By Eddie Choi
Story Telling By Eddie ChoiStory Telling By Eddie Choi
Story Telling By Eddie ChoiEddie Choi
 
Laws of marketing presentation
Laws of marketing presentationLaws of marketing presentation
Laws of marketing presentationDIYMarketers
 
Standardisierte Medizinische Übergaben - Wie lernen, lehren und implementiere...
Standardisierte Medizinische Übergaben - Wie lernen, lehren und implementiere...Standardisierte Medizinische Übergaben - Wie lernen, lehren und implementiere...
Standardisierte Medizinische Übergaben - Wie lernen, lehren und implementiere...Hendrik Drachsler
 
Do Hierarchical Positions Influence Participant’s Network Behaviour within C...
Do Hierarchical Positions Influence Participant’s  Network Behaviour within C...Do Hierarchical Positions Influence Participant’s  Network Behaviour within C...
Do Hierarchical Positions Influence Participant’s Network Behaviour within C...Martin Rehm
 
Coleg Sir Gar HE Conference 030714
Coleg Sir Gar HE Conference 030714Coleg Sir Gar HE Conference 030714
Coleg Sir Gar HE Conference 030714Lis Parcell
 
Managing Communities of Learning: The Impact and Role of Facilitators
Managing Communities of Learning: The Impact and Role of FacilitatorsManaging Communities of Learning: The Impact and Role of Facilitators
Managing Communities of Learning: The Impact and Role of FacilitatorsMartin Rehm
 
Issues and Considerations regarding Sharable Data Sets for Recommender System...
Issues and Considerations regarding Sharable Data Sets for Recommender System...Issues and Considerations regarding Sharable Data Sets for Recommender System...
Issues and Considerations regarding Sharable Data Sets for Recommender System...Hendrik Drachsler
 

Viewers also liked (20)

17) 11 (may, 2003) squid master this proxy server
17) 11 (may, 2003)   squid master this proxy server17) 11 (may, 2003)   squid master this proxy server
17) 11 (may, 2003) squid master this proxy server
 
MessagePack(msgpack): Compact and Fast Serialization Library
MessagePack(msgpack): Compact and Fast Serialization LibraryMessagePack(msgpack): Compact and Fast Serialization Library
MessagePack(msgpack): Compact and Fast Serialization Library
 
Phoenix Energy Performance Contracting Presentation
Phoenix   Energy Performance Contracting PresentationPhoenix   Energy Performance Contracting Presentation
Phoenix Energy Performance Contracting Presentation
 
25et_Bulgaria
25et_Bulgaria25et_Bulgaria
25et_Bulgaria
 
Christmas in croatia eng
Christmas in croatia engChristmas in croatia eng
Christmas in croatia eng
 
Inbound Marketing Cookbook
Inbound Marketing CookbookInbound Marketing Cookbook
Inbound Marketing Cookbook
 
Zima sisljavic
Zima sisljavicZima sisljavic
Zima sisljavic
 
Story Telling By Eddie Choi
Story Telling By Eddie ChoiStory Telling By Eddie Choi
Story Telling By Eddie Choi
 
Laws of marketing presentation
Laws of marketing presentationLaws of marketing presentation
Laws of marketing presentation
 
Standardisierte Medizinische Übergaben - Wie lernen, lehren und implementiere...
Standardisierte Medizinische Übergaben - Wie lernen, lehren und implementiere...Standardisierte Medizinische Übergaben - Wie lernen, lehren und implementiere...
Standardisierte Medizinische Übergaben - Wie lernen, lehren und implementiere...
 
數位電視
數位電視數位電視
數位電視
 
Do Hierarchical Positions Influence Participant’s Network Behaviour within C...
Do Hierarchical Positions Influence Participant’s  Network Behaviour within C...Do Hierarchical Positions Influence Participant’s  Network Behaviour within C...
Do Hierarchical Positions Influence Participant’s Network Behaviour within C...
 
Wine Of Austria
Wine Of AustriaWine Of Austria
Wine Of Austria
 
Weblin20080826
Weblin20080826Weblin20080826
Weblin20080826
 
Tugas Prof. Yusuf
Tugas Prof. YusufTugas Prof. Yusuf
Tugas Prof. Yusuf
 
Laying a Strong Foundation for Agile Transformation
Laying a Strong Foundation for Agile TransformationLaying a Strong Foundation for Agile Transformation
Laying a Strong Foundation for Agile Transformation
 
Coleg Sir Gar HE Conference 030714
Coleg Sir Gar HE Conference 030714Coleg Sir Gar HE Conference 030714
Coleg Sir Gar HE Conference 030714
 
Managing Communities of Learning: The Impact and Role of Facilitators
Managing Communities of Learning: The Impact and Role of FacilitatorsManaging Communities of Learning: The Impact and Role of Facilitators
Managing Communities of Learning: The Impact and Role of Facilitators
 
Issues and Considerations regarding Sharable Data Sets for Recommender System...
Issues and Considerations regarding Sharable Data Sets for Recommender System...Issues and Considerations regarding Sharable Data Sets for Recommender System...
Issues and Considerations regarding Sharable Data Sets for Recommender System...
 
Friends Calendar 2010
Friends Calendar 2010Friends Calendar 2010
Friends Calendar 2010
 

Similar to Using Q4M - a message queue storage engine for MySQL

Cloud Native Camel Design Patterns
Cloud Native Camel Design PatternsCloud Native Camel Design Patterns
Cloud Native Camel Design PatternsBilgin Ibryam
 
Life in a Queue - Using Message Queue with django
Life in a Queue - Using Message Queue with djangoLife in a Queue - Using Message Queue with django
Life in a Queue - Using Message Queue with djangoTareque Hossain
 
Integration and Batch Processing on Cloud Foundry
Integration and Batch Processing on Cloud FoundryIntegration and Batch Processing on Cloud Foundry
Integration and Batch Processing on Cloud FoundryJoshua Long
 
Introduction to First Commercial Memcached Service for Cloud
Introduction to First Commercial Memcached Service for CloudIntroduction to First Commercial Memcached Service for Cloud
Introduction to First Commercial Memcached Service for CloudGear6
 
Five cool ways the JVM can run Apache Spark faster
Five cool ways the JVM can run Apache Spark fasterFive cool ways the JVM can run Apache Spark faster
Five cool ways the JVM can run Apache Spark fasterTim Ellison
 
Memcached, presented to LCA2010
Memcached, presented to LCA2010Memcached, presented to LCA2010
Memcached, presented to LCA2010Mark Atwood
 
Verification of the QorIQ Communication Platform Containing CoreNet Fabric wi...
Verification of the QorIQ Communication Platform Containing CoreNet Fabric wi...Verification of the QorIQ Communication Platform Containing CoreNet Fabric wi...
Verification of the QorIQ Communication Platform Containing CoreNet Fabric wi...DVClub
 
Event Detection Pipelines with Apache Kafka
Event Detection Pipelines with Apache KafkaEvent Detection Pipelines with Apache Kafka
Event Detection Pipelines with Apache KafkaDataWorks Summit
 
CNPM: Private NPM for Company / 企業級私有NPM
CNPM: Private NPM for Company / 企業級私有NPMCNPM: Private NPM for Company / 企業級私有NPM
CNPM: Private NPM for Company / 企業級私有NPMFeng Yuan
 
JITServerTalk.pdf
JITServerTalk.pdfJITServerTalk.pdf
JITServerTalk.pdfRichHagarty
 
Java-light-speed NebraskaCode.pdf
Java-light-speed NebraskaCode.pdfJava-light-speed NebraskaCode.pdf
Java-light-speed NebraskaCode.pdfRichHagarty
 
JITServerTalk JCON World 2023.pdf
JITServerTalk JCON World 2023.pdfJITServerTalk JCON World 2023.pdf
JITServerTalk JCON World 2023.pdfRichHagarty
 
Couchbase Performance Benchmarking
Couchbase Performance BenchmarkingCouchbase Performance Benchmarking
Couchbase Performance BenchmarkingRenat Khasanshyn
 

Similar to Using Q4M - a message queue storage engine for MySQL (20)

MPI n OpenMP
MPI n OpenMPMPI n OpenMP
MPI n OpenMP
 
Cloud Native Camel Design Patterns
Cloud Native Camel Design PatternsCloud Native Camel Design Patterns
Cloud Native Camel Design Patterns
 
Life in a Queue - Using Message Queue with django
Life in a Queue - Using Message Queue with djangoLife in a Queue - Using Message Queue with django
Life in a Queue - Using Message Queue with django
 
Integration and Batch Processing on Cloud Foundry
Integration and Batch Processing on Cloud FoundryIntegration and Batch Processing on Cloud Foundry
Integration and Batch Processing on Cloud Foundry
 
Introduction to First Commercial Memcached Service for Cloud
Introduction to First Commercial Memcached Service for CloudIntroduction to First Commercial Memcached Service for Cloud
Introduction to First Commercial Memcached Service for Cloud
 
Sakar jain
Sakar jainSakar jain
Sakar jain
 
The bigrabbit
The bigrabbitThe bigrabbit
The bigrabbit
 
Five cool ways the JVM can run Apache Spark faster
Five cool ways the JVM can run Apache Spark fasterFive cool ways the JVM can run Apache Spark faster
Five cool ways the JVM can run Apache Spark faster
 
EVOLVE'16 | Deploy | Abhishek Dwevedi | Understanding a Typical AEM Deployment
EVOLVE'16 | Deploy | Abhishek Dwevedi | Understanding a Typical AEM DeploymentEVOLVE'16 | Deploy | Abhishek Dwevedi | Understanding a Typical AEM Deployment
EVOLVE'16 | Deploy | Abhishek Dwevedi | Understanding a Typical AEM Deployment
 
Fastest Servlets in the West
Fastest Servlets in the WestFastest Servlets in the West
Fastest Servlets in the West
 
re7jweiss
re7jweissre7jweiss
re7jweiss
 
Memcached, presented to LCA2010
Memcached, presented to LCA2010Memcached, presented to LCA2010
Memcached, presented to LCA2010
 
Verification of the QorIQ Communication Platform Containing CoreNet Fabric wi...
Verification of the QorIQ Communication Platform Containing CoreNet Fabric wi...Verification of the QorIQ Communication Platform Containing CoreNet Fabric wi...
Verification of the QorIQ Communication Platform Containing CoreNet Fabric wi...
 
Event Detection Pipelines with Apache Kafka
Event Detection Pipelines with Apache KafkaEvent Detection Pipelines with Apache Kafka
Event Detection Pipelines with Apache Kafka
 
reBuy on Kubernetes
reBuy on KubernetesreBuy on Kubernetes
reBuy on Kubernetes
 
CNPM: Private NPM for Company / 企業級私有NPM
CNPM: Private NPM for Company / 企業級私有NPMCNPM: Private NPM for Company / 企業級私有NPM
CNPM: Private NPM for Company / 企業級私有NPM
 
JITServerTalk.pdf
JITServerTalk.pdfJITServerTalk.pdf
JITServerTalk.pdf
 
Java-light-speed NebraskaCode.pdf
Java-light-speed NebraskaCode.pdfJava-light-speed NebraskaCode.pdf
Java-light-speed NebraskaCode.pdf
 
JITServerTalk JCON World 2023.pdf
JITServerTalk JCON World 2023.pdfJITServerTalk JCON World 2023.pdf
JITServerTalk JCON World 2023.pdf
 
Couchbase Performance Benchmarking
Couchbase Performance BenchmarkingCouchbase Performance Benchmarking
Couchbase Performance Benchmarking
 

More from Kazuho Oku

HTTP/2で 速くなるとき ならないとき
HTTP/2で 速くなるとき ならないときHTTP/2で 速くなるとき ならないとき
HTTP/2で 速くなるとき ならないときKazuho Oku
 
QUIC標準化動向 〜2017/7
QUIC標準化動向 〜2017/7QUIC標準化動向 〜2017/7
QUIC標準化動向 〜2017/7Kazuho Oku
 
HTTP/2の課題と将来
HTTP/2の課題と将来HTTP/2の課題と将来
HTTP/2の課題と将来Kazuho Oku
 
TLS 1.3 と 0-RTT のこわ〜い話
TLS 1.3 と 0-RTT のこわ〜い話TLS 1.3 と 0-RTT のこわ〜い話
TLS 1.3 と 0-RTT のこわ〜い話Kazuho Oku
 
Reorganizing Website Architecture for HTTP/2 and Beyond
Reorganizing Website Architecture for HTTP/2 and BeyondReorganizing Website Architecture for HTTP/2 and Beyond
Reorganizing Website Architecture for HTTP/2 and BeyondKazuho Oku
 
Recent Advances in HTTP, controlling them using ruby
Recent Advances in HTTP, controlling them using rubyRecent Advances in HTTP, controlling them using ruby
Recent Advances in HTTP, controlling them using rubyKazuho Oku
 
Programming TCP for responsiveness
Programming TCP for responsivenessProgramming TCP for responsiveness
Programming TCP for responsivenessKazuho Oku
 
Programming TCP for responsiveness
Programming TCP for responsivenessProgramming TCP for responsiveness
Programming TCP for responsivenessKazuho Oku
 
Developing the fastest HTTP/2 server
Developing the fastest HTTP/2 serverDeveloping the fastest HTTP/2 server
Developing the fastest HTTP/2 serverKazuho Oku
 
TLS & LURK @ IETF 95
TLS & LURK @ IETF 95TLS & LURK @ IETF 95
TLS & LURK @ IETF 95Kazuho Oku
 
HTTPとサーバ技術の最新動向
HTTPとサーバ技術の最新動向HTTPとサーバ技術の最新動向
HTTPとサーバ技術の最新動向Kazuho Oku
 
ウェブを速くするためにDeNAがやっていること - HTTP/2と、さらにその先
ウェブを速くするためにDeNAがやっていること - HTTP/2と、さらにその先ウェブを速くするためにDeNAがやっていること - HTTP/2と、さらにその先
ウェブを速くするためにDeNAがやっていること - HTTP/2と、さらにその先Kazuho Oku
 
Cache aware-server-push in H2O version 1.5
Cache aware-server-push in H2O version 1.5Cache aware-server-push in H2O version 1.5
Cache aware-server-push in H2O version 1.5Kazuho Oku
 
HTTP/2時代のウェブサイト設計
HTTP/2時代のウェブサイト設計HTTP/2時代のウェブサイト設計
HTTP/2時代のウェブサイト設計Kazuho Oku
 
H2O - making the Web faster
H2O - making the Web fasterH2O - making the Web faster
H2O - making the Web fasterKazuho Oku
 
H2O - making HTTP better
H2O - making HTTP betterH2O - making HTTP better
H2O - making HTTP betterKazuho Oku
 
H2O - the optimized HTTP server
H2O - the optimized HTTP serverH2O - the optimized HTTP server
H2O - the optimized HTTP serverKazuho Oku
 
JSON SQL Injection and the Lessons Learned
JSON SQL Injection and the Lessons LearnedJSON SQL Injection and the Lessons Learned
JSON SQL Injection and the Lessons LearnedKazuho Oku
 
JSX 速さの秘密 - 高速なJavaScriptを書く方法
JSX 速さの秘密 - 高速なJavaScriptを書く方法JSX 速さの秘密 - 高速なJavaScriptを書く方法
JSX 速さの秘密 - 高速なJavaScriptを書く方法Kazuho Oku
 
JSX の現在と未来 - Oct 26 2013
JSX の現在と未来 - Oct 26 2013JSX の現在と未来 - Oct 26 2013
JSX の現在と未来 - Oct 26 2013Kazuho Oku
 

More from Kazuho Oku (20)

HTTP/2で 速くなるとき ならないとき
HTTP/2で 速くなるとき ならないときHTTP/2で 速くなるとき ならないとき
HTTP/2で 速くなるとき ならないとき
 
QUIC標準化動向 〜2017/7
QUIC標準化動向 〜2017/7QUIC標準化動向 〜2017/7
QUIC標準化動向 〜2017/7
 
HTTP/2の課題と将来
HTTP/2の課題と将来HTTP/2の課題と将来
HTTP/2の課題と将来
 
TLS 1.3 と 0-RTT のこわ〜い話
TLS 1.3 と 0-RTT のこわ〜い話TLS 1.3 と 0-RTT のこわ〜い話
TLS 1.3 と 0-RTT のこわ〜い話
 
Reorganizing Website Architecture for HTTP/2 and Beyond
Reorganizing Website Architecture for HTTP/2 and BeyondReorganizing Website Architecture for HTTP/2 and Beyond
Reorganizing Website Architecture for HTTP/2 and Beyond
 
Recent Advances in HTTP, controlling them using ruby
Recent Advances in HTTP, controlling them using rubyRecent Advances in HTTP, controlling them using ruby
Recent Advances in HTTP, controlling them using ruby
 
Programming TCP for responsiveness
Programming TCP for responsivenessProgramming TCP for responsiveness
Programming TCP for responsiveness
 
Programming TCP for responsiveness
Programming TCP for responsivenessProgramming TCP for responsiveness
Programming TCP for responsiveness
 
Developing the fastest HTTP/2 server
Developing the fastest HTTP/2 serverDeveloping the fastest HTTP/2 server
Developing the fastest HTTP/2 server
 
TLS & LURK @ IETF 95
TLS & LURK @ IETF 95TLS & LURK @ IETF 95
TLS & LURK @ IETF 95
 
HTTPとサーバ技術の最新動向
HTTPとサーバ技術の最新動向HTTPとサーバ技術の最新動向
HTTPとサーバ技術の最新動向
 
ウェブを速くするためにDeNAがやっていること - HTTP/2と、さらにその先
ウェブを速くするためにDeNAがやっていること - HTTP/2と、さらにその先ウェブを速くするためにDeNAがやっていること - HTTP/2と、さらにその先
ウェブを速くするためにDeNAがやっていること - HTTP/2と、さらにその先
 
Cache aware-server-push in H2O version 1.5
Cache aware-server-push in H2O version 1.5Cache aware-server-push in H2O version 1.5
Cache aware-server-push in H2O version 1.5
 
HTTP/2時代のウェブサイト設計
HTTP/2時代のウェブサイト設計HTTP/2時代のウェブサイト設計
HTTP/2時代のウェブサイト設計
 
H2O - making the Web faster
H2O - making the Web fasterH2O - making the Web faster
H2O - making the Web faster
 
H2O - making HTTP better
H2O - making HTTP betterH2O - making HTTP better
H2O - making HTTP better
 
H2O - the optimized HTTP server
H2O - the optimized HTTP serverH2O - the optimized HTTP server
H2O - the optimized HTTP server
 
JSON SQL Injection and the Lessons Learned
JSON SQL Injection and the Lessons LearnedJSON SQL Injection and the Lessons Learned
JSON SQL Injection and the Lessons Learned
 
JSX 速さの秘密 - 高速なJavaScriptを書く方法
JSX 速さの秘密 - 高速なJavaScriptを書く方法JSX 速さの秘密 - 高速なJavaScriptを書く方法
JSX 速さの秘密 - 高速なJavaScriptを書く方法
 
JSX の現在と未来 - Oct 26 2013
JSX の現在と未来 - Oct 26 2013JSX の現在と未来 - Oct 26 2013
JSX の現在と未来 - Oct 26 2013
 

Recently uploaded

Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataSafe Software
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncGenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncObject Automation
 
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServicePicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServiceRenan Moreira de Oliveira
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIUdaiappa Ramachandran
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum ComputingGDSC PJATK
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 

Recently uploaded (20)

Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataCloud Revolution: Exploring the New Wave of Serverless Spatial Data
Cloud Revolution: Exploring the New Wave of Serverless Spatial Data
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
GenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation IncGenAI and AI GCC State of AI_Object Automation Inc
GenAI and AI GCC State of AI_Object Automation Inc
 
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServicePicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AI
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
Introduction to Quantum Computing
Introduction to Quantum ComputingIntroduction to Quantum Computing
Introduction to Quantum Computing
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 

Using Q4M - a message queue storage engine for MySQL

  • 1. Using Q4M a message queue storage engine for MySQL Cybozu Labs, Inc. Kazuho Oku
  • 2. Background 2 Apr 22 2009 Using Q4M
  • 3. Who am I? Name: Kazuho Oku (奥一穂) Original Developer of Palmscape / Xiino The oldest web browser for Palm OS Working at Cybozu Labs since 2005 Research subsidiary of Cybozu, Inc. in Japan 3 Apr 22 2009 Using Q4M
  • 4. About Cybozu, Inc. Japan’s largest groupware vendor Mostly provides as software products, not as services Some of our apps bundle MySQL as storage 4 Apr 22 2009 Using Q4M
  • 5. About Pathtraq Started in Aug. 2007 Web ranking service One of Japan’s largest like Alexa, but semi-realtime, and per-page running on MySQL Need for a fast and reliable message relay for communication between the main server and content analysis server(s) 5 Apr 22 2009 Using Q4M
  • 6. Design Goals of Q4M Robust Do not lose data on OS crash or power failure Fast Transfer thousands of messages per second Easy to Use Use SQL for access / maintenance Integration into MySQL no more separate daemons to take care of 6 Apr 22 2009 Using Q4M
  • 7. What is a Message Queue? 7 Apr 22 2009 Using Q4M
  • 8. What is a Message Queue? Middleware for persistent asynchronous communication communicate between fixed pairs (parties) a.k.a. Message Oriented Middleware MQ is intermediate storage RDBMS is persistent storage Senders / receivers may go down 8 Apr 22 2009 Using Q4M
  • 9. Minimal Configuration of a MQ Senders and receivers access a single queue Sender Receiver Queue 9 Apr 22 2009 Using Q4M
  • 10. MQ and Relays Separate queue for sender and receiver Messages relayed between queues Relay Sender Receiver Queue Queue 10 Apr 22 2009 Using Q4M
  • 11. Merits of Message Relays Destination can be changed easily Relays may transfer messages to different locations depending on their headers Robustness against network failure no loss or duplicates when the relay fails Logging and Multicasting, etc. 11 Apr 22 2009 Using Q4M
  • 12. Message Brokers Publish / subscribe model Separation between components and their integration Components read / write to predefined queues Integration is definition of routing rules between the message queues Messages are often transformed (filtered) within the relay agent 12 Apr 22 2009 Using Q4M
  • 13. What about Q4M? Q4M itself is a message queue Can connect Q4M instances to create a message relay Provides API for creating message relays and brokers 13 Apr 22 2009 Using Q4M
  • 14. Performance of Q4M over 7,000 mess/sec. message size: avg. 512 bytes syncing to disk Outperforming most needs if you need more, just scale out Can coexist with other storage engines without sacrificing their performance see http://labs.cybozu.co.jp/blog/kazuhoatwork/2008/06/q4m_06_release_and_benchmarks.php 14 Apr 22 2009 Using Q4M
  • 15. Applications of Q4M 15 Apr 22 2009 Using Q4M
  • 16. Asynchronous Updates Mixi (Japan's one of the largest SNS) uses Q4M to buffer writes to DB, to offload peak demands from http://alpha.mixi.co.jp/blog/?p=272 16 Apr 22 2009 Using Q4M
  • 17. Connecting Distant Servers Pathtraq uses Q4M to create a relay between its database and content analysis processes → Contents to be analyzed → Content MySQL conn. Analysis Pathtraq over SSL,gzip Processes DB ← Results of the analysis ← 17 Apr 22 2009 Using Q4M
  • 18. To Prefetch Data livedoor Reader (web-based feed aggregator) uses Q4M to prefetch data from database to memcached uses Q4M for scheduling web crawlers as well from http://d.hatena.ne.jp/mala/20081212/1229074359 18 Apr 22 2009 Using Q4M
  • 19. Scheduling Web Crawlers Web crawlers with retry-on-error Sample code included in Q4M dist. If failed to fetch, store URL in retry queue Store Result Read URL Spiders URL DB Request Queue Retry Queue Re- scheduler 19 Apr 22 2009 Using Q4M
  • 20. Delayed Content Generation Hatetter(RSS feed-to-twitter-API gateway) uses Q4M to delay content generation Source code: github.com/yappo/website-hatetter 20 Apr 22 2009 Using Q4M
  • 21. User Notifications For sending notifications from web services DB App. Logic SMTP Agent Queue(s) IM Agent 21 Apr 22 2009 Using Q4M
  • 22. Installing Q4M 22 Apr 22 2009 Using Q4M
  • 23. Installing Q4M Compatible with MySQL 5.1 Download from q4m.31tools.com Binary releases available for some platforms Installing from source: requires source code of MySQL ./configure && make && make install run support-files/install.sql 23 Apr 22 2009 Using Q4M
  • 24. Configuration Options of Q4M --with-sync=no|fsync|fdatasync|fcntl Controls synchronization to disk default: fdatasync on linux --enable-mmap Mmap’ed reads lead to higher throughput default: yes --with-delete=pwrite|msync msyncrecommended on linux>=2.6.20 if you need really high performance 24 Apr 22 2009 Using Q4M
  • 25. Q4M Basics 25 Apr 22 2009 Using Q4M
  • 26. The Model Various publishers write to queue Set of subscribers consume the entries in queue Publisher Q4M table Publisher Subscribers Publisher 26 Apr 22 2009 Using Q4M
  • 27. Creating a Q4M Table  ENGINE=QUEUE creates mysql> CREATE TABLE qt ( -> id int(10) unsigned NOT NULL, a Q4M table -> message varchar(255) NOT NULL -> ) ENGINE=QUEUE; Query OK, 0 rows affected (0.42 sec)  No primary keys or indexes  Sorted by insertion order (it’s a queue) 27 Apr 22 2009 Using Q4M
  • 28. Modifying Data on a Q4M Table  No restrictions for mysql> INSERT INTO qt (id,message) -> VALUES INSERT and DELETE -> (1,'Hello'), -> (2,'Bonjour'), -> (3,'Hola');  No support for UPDATE Query OK, 3 rows affected (0.02 sec) mysql> SELECT * FROM qt; +----+---------+ | id | message | +----+---------+ | 1 | Hello | | 2 | Bonjour | | 3 | Hola | +----+---------+ 3 rows in set (0.00 sec) 28 Apr 22 2009 Using Q4M
  • 29. SELECT from a Q4M Table Works the same as mysql> SELECT * FROM qt; +----+---------+ | id | message | other storage +----+---------+ | 1 | Hello | engines | 2 | Bonjour | | 3 | Hola | +----+---------+ SELECT COUNT(*) is 3 rows in set (0.00 sec) cached mysql> SELECT COUNT(*) FROM qt; +----------+ | COUNT(*) | +----------+ | 3| +----------+ 1 row in set (0.00 sec) How to subscribe to a queue? 29 Apr 22 2009 Using Q4M
  • 30. Calling queue_wait()  After calling, only one mysql> SELECT * FROM qt; +----+---------+ | id | message | row becomes visible +----+---------+ | 1 | Hello | from the connection | 2 | Bonjour | | 3 | Hola | +----+---------+ 3 rows in set (0.00 sec) mysql> SELECT queue_wait('qt'); +------------------+ | queue_wait('qt') | +------------------+ | 1| +------------------+ 1 row in set (0.00 sec) mysql> SELECT * FROM qt; +----+---------+ | id | message | +----+---------+ | 1 | Hello | +----+---------+ 1 row in set (0.00 sec) 30 Apr 22 2009 Using Q4M
  • 31. OWNER Mode and NON-OWNER Mode In OWNER mode, only the OWNED row is visible OWNED row becomes invisible from other connections rows of other storage engines are visible NON-OWNER Mode OWNER Mode queue_wait() 1,'Hello' 1,'Hello' 2,'Bonjour' queue_end() 3,'Hola' queue_abort() 31 Apr 22 2009 Using Q4M
  • 32. Returning to NON-OWNER mode  By calling mysql> SELECT QUEUE_ABORT(); +---------------+ queue_abort, the | QUEUE_ABORT() | +---------------+ connection returns to | 1| +---------------+ NON-OWNER mode 1 row in set (0.00 sec) mysql> SELECT * FROM qt; +----+---------+ | id | message | +----+---------+ | 1 | Hello | | 2 | Bonjour | | 3 | Hola | +----+---------+ 3 rows in set (0.01 sec) 32 Apr 22 2009 Using Q4M
  • 33. Consuming a Row  By calling mysql> SELECT queue_wait('qt'); (snip) mysql> SELECT * FROM qt; queue_end, the OWNED +----+---------+ | id | message | row is deleted, and +----+---------+ | 1 | Hello | connection returns to +----+---------+ 1 row in set (0.01 sec) NON-OWNER mode mysql> SELECT queue_end(); +-------------+ | queue_end() | +-------------+ | 1| +-------------+ 1 row in set (0.01 sec) mysql> SELECT * FROM qt; +----+---------+ | id | message | +----+---------+ | 2 | Bonjour | | 3 | Hola | +----+---------+ 2 rows in set (0.00 sec) 33 Apr 22 2009 Using Q4M
  • 34. Writing a Subscriber  Call two functions: queue_wait, queue_end  Multiple subscribers can be run concurrently  each row in the queue is consumed only once while (true) { SELECT queue_wait('qt'); # switch to owner mode rows := SELECT * FROM qt; # obtain data if (count(rows) != 0) # if we have any data, then handle_row(rows[0]); # consume the row SELECT queue_end(); # erase the row from queue } 34 Apr 22 2009 Using Q4M
  • 35. Writing a Subscriber (cont'd) Or call queue_wait as a condition Warning: conflicts with trigger-based insertions while (true) { rows := SELECT * FROM qt WHERE queue_wait('qt'); if (count(rows) != 0) handle_row(rows[0]); SELECT queue_end(); } 35 Apr 22 2009 Using Q4M
  • 36. The Model – with code INSERT INTO queue ... Publisher while (true) { rows := SELECT * FROM qt WHERE queue_wait('qt'); if (count(rows) != 0) handle_row(rows[0]); INSERT INTO queue ... SELECT queue_end(); } Q4M table Publisher Subscribers INSERT INTO queue ... Publisher 36 Apr 22 2009 Using Q4M
  • 37. Three Functions in Detail 37 Apr 22 2009 Using Q4M
  • 38. queue_wait(table) Enters OWNER mode 0〜1 row becomes OWNED Enters OWNER mode even if no rows were available Default timeout: 60 seconds Returns 1 if a row is OWNED (0 on timeout) If called within OWNER mode, the owned row is deleted 38 Apr 22 2009 Using Q4M
  • 39. Revisiting Subscriber Code Calls to queue_end just before queue_wait can be omitted while (true) { rows := SELECT * FROM qt WHERE queue_wait('qt'); if (count(rows) != 0) handle_row(rows[0]); SELECT queue_end(); } 39 Apr 22 2009 Using Q4M
  • 40. Conditional queue_wait() Consume rows of certain condition Rows that do not match will be left untouched Only numeric columns can be checked Fast - condition tested once per each row examples: SELECT queue_wait('table:(col_a*3)+col_b<col_c'); SELECT queue_wait('table:retry_count<5'); 40 Apr 22 2009 Using Q4M
  • 41. queue_wait(tbl_cond,[tbl_cond…,timeout]) Accepts multiple tables and timeout Data searched from leftmost table to right Returns table index (the leftmost table is 1) of the newly owned row Returns zero if no rows are being owned example: SELECT queue_wait('table_A','table_B',60); 41 Apr 22 2009 Using Q4M
  • 42. Functions for Exiting OWNER Mode queue_end Deletes the owned row and exits OWNER mode queue_abort Releases (instead of deleting) the owned row and exits OWNER mode Close of a MySQL connection does the same thing 42 Apr 22 2009 Using Q4M
  • 43. Relaying and Routing Messages 43 Apr 22 2009 Using Q4M
  • 44. The Problem Relay (or router) consists of more than 3 processes, 2 conns No losses, no duplicates on crash or disconnection Q4M Table Q4M Table Relay Program (source) (dest.) 44 Apr 22 2009 Using Q4M
  • 45. Internal Row ID Every row have a internal row ID invisible from Q4M table definition monotonically increasing 64-bit integer Used for detecting duplicates Use two functions to skip duplicates Data loss prevented by using queue_wait / queue_end 45 Apr 22 2009 Using Q4M
  • 46. queue_rowid() Returns row ID of the OWNED row (if any) Returns NULL if no row is OWNED Call when retrieving data from source 46 Apr 22 2009 Using Q4M
  • 47. queue_set_srcid(src_tbl_id, mode, src_row_id) Call before inserting a row to destination table Checks if the row is already inserted into the table, and ignores next INSERT if true Parameters: src_tbl_id - id to determine source table (0〜63) mode - quot;aquot; to drop duplicates, quot;wquot; to reset src_row_id - row ID obtained from source table 47 Apr 22 2009 Using Q4M
  • 48. Pseudo Code Relays data from src_tbl to dest_tbl while (true) { # wait for data SELECT queue_wait(src_tbl) =>src_db; # read row and rowid row := (SELECT * FROM src_tbl =>src_db); rowid := (SELECT queue_rowid() =>src_db); # insert the row after setting srcid SELECT queue_set_srcid(src_tbl_id, 'a', rowid) =>dest_db; INSERT INTO dest_tbl (row) =>dest_db; 48 Apr 22 2009 Using Q4M
  • 49. q4m-forward Simple forwarder script installed into mysql-dir/bin usage: q4m-forward [options] src_addrdest_addr example: % support-files/q4m-forward quot;dbi:mysql:database=db1;table=tbl1;user=foo;password=XXXquot; quot;dbi:mysql:database=db2;table=tbl2;host=bar;user=fooquot; options: --reset reset duplicate check info. --sender=idx slot no. used for checking duplicates (0..63, default: 0) --help 49 Apr 22 2009 Using Q4M
  • 50. Limitations and the Future of Q4M 50 Apr 22 2009 Using Q4M
  • 51. Things that Need to be Fixed Table compactions is a blocking operation runs when live data becomes <25% of log file very bad, though not as bad as it seems it's fast since it's a sequential write operation Relays are slow since transfer is done row-by-row Binlog does not work since MQ replication should be synchronous 51 Apr 22 2009 Using Q4M
  • 52. Future of Q4M 2-phase commit with other storage engines (maybe) queue consumption and InnoDB updates can become atomic operation 52 Apr 22 2009 Using Q4M
  • 53. Thank you http://q4m.31tools.com/ 53 Apr 22 2009 Using Q4M