オープンソース開発者がDeNAを選ぶ理由

    CTO2011 7 16 2

 Palmscape  1997 Xiino  Palm OS ( )  IBM WorkPad, Sony CLIE  NTT DoCoMo  M.I.T. TR100/2002    Web  IPA 2004 –2011 7 16 3

(2)   :  2005 8 2010 11  Japanize  Web UI      Q4M   (MySQL ) 2011 7 16 4

(3)   :  2010 11   IT  2011 5 CTO 2011 7 16 5

  DeNA   OSS  DeNA OSS   2011 7 16 6

2011 7 16 7

Q4M 2011 7 16 8

What is Q4M?  A message queue  runs as a storage plugin of MySQL 5.1  Why is it a MySQL plugin?  accessible by using existing MySQL clients  no need for a new client library  administrable by using SQL  friendly to DB admins  First release: Jan 2008 2011 7 16 9

Design Goals of Q4M  Robust  Does not lose data on OS crash or power failure  necessary for Tokyo wo. nuclear power plants… orz  Fast  Transfer thousands of messages per second  Easy to Use  Use SQL for access / maintenance  Integration into MySQL  no more separate daemons to take care of 2011 7 16 10

Users of Q4M  Many leading web services in Japan  DeNA Co., Ltd.  livedoor Co., Ltd.  mixi, Inc.  Zynga Japan (formerly Unoh, Inc.)2011 7 16 11

What is a Message Queue? 2011 7 16 12

What is a Message Queue?  Middleware for persistent asynchronous communication  communicate between fixed pairs (parties)  a.k.a. Message Oriented Middleware  MQ is intermediate storage  RDBMS is persistent storage  Senders / receivers may go down2011 7 16 13

Minimal Configuration of a Message Queue  Senders and receivers access a single queue Sender Receiver Queue2011 7 16 14

Using MQ as a Relay  Separate queue for sender and receiver  Messages relayed between queues Relay Sender Receiver Queue Queue2011 7 16 15

Merits of using Message Relays  Destination can be changed easily  Relays may transfer messages to different locations depending on their headers  Robustness against network failure  no loss or duplicates when the relay fails  Logging and Multicasting, etc. 2011 7 16 16

Message Brokers  Publish / subscribe model  Separation between components and their integration  Components read / write to predefined queues  Integration is definition of routing rules between the message queues  Messages are often transformed (filtered) within the relay agent2011 7 16 17

What about Q4M?  Q4M itself is a message queue  Can connect Q4M instances to create a message relay  Provides API for creating message relays and brokers2011 7 16 18

Performance of Q4M  over 7,000 mess/sec.  message size: avg. 512 bytes  syncing to disk  Outperforming most needs  if you need more, just scale out  Can coexist with other storage engines without sacrificing their performance see http://labs.cybozu.co.jp/blog/kazuhoatwork/2008/06/q4m_06_release_and_benchmarks.php2011 7 16 19

Q4M in DeNA  uses Q4M for sending notifications asynchronously http://engineer.dena.jp/2010/03/dena-technical- seminar-1-2.html2011 7 16 20

Q4M in DeNA  Q4M  2,000 /   DeNA    2011 7 16 21

HandlerSocket Plugin 2011 7 16 22

Memcached    RDBMS /      RDBMS      Memcached RDBMS    primary storage2011 7 16 23

Memcached  RDBMS       +   :     SQL2011 7 16 24

SQL ( ) 2011 7 16 25

Mycached  2009 8  mysqld memcached   memcached get   get   SQL 2011 7 16 26

Mycached – mycached http://dev.mysql.com/doc/refman/5.1/en/pluggable-storage-overview.html 2011 7 16 27

Mycached –  1,200  JSON, msgpack,  Picoev (I/O )  MySQL ( )    get 2011 7 16 28

Mycached –  SQL 40,000qps 2011 7 16 29

Mycached    2011 7 16 30

HandlerSocket Plugin  DeNA  2010 8    MySQL Contributor Award 2011  MySQL SQL   qps   mycached2011 7 16 31

… 2011 7 16 32

…2011 7 16 33

… … 2011 7 16 34

… … … 2011 7 16 35

2011 7 16 36

           2011 7 16 37

HandlerSocket Plugin  mycached      FastEther  2011 7 16 38

2011 7 16 39

2011 7 16 40

–         ? 2011 7 16 41

–    Server::Starter –  Parallel::Prefork –  Starlet – PSGI2011 7 16 42

–    fork   : mod_bumpy_life (Apache)  by hirose31++  MaxRequestsPerChild     Parallel::Prefork2011 7 16 43

–   MaxRequestsPerChild   fork … 2011 7 16 44

– : Slow Restart     fork listen accept fork & exec SIGUSR1 accept logic app. spawn app. servers fork & exec app. logic SIGHUP 2011 7 16 45

– OSS   Perl github CPAN     … 2011 7 16 46

 Q. ? $SIG{HUP} = sub { $got_sig = 1 }; … while (! $got_sig) { sleep(1000); // }2011 7 16 47

 Q. ? $SIG{HUP} = sub { $got_sig = 1 }; … while (! $got_sig) { // 1000 sleep(1000); // }2011 7 16 48

  Parallel::Prefork     …   : pselect   FAQ   : Perl pselect   XS (Perl C )   Perl XS 2011 7 16 49

OSS DeNA 2011 7 16 50

 Perl …  2009 Plack  Perl WSGI or Rack  Furl ( HTTP )   DeNA id:xaicron    MySQL2011 7 16 51

    MySQL  YAPC  MySQL Conference       : MySQL Casual    2011 7 16 52

2011 7 16 53

 OSS     OSS     2011 7 16 54

http://blog.kazuhooku.com	6063
http://a0.twimg.com	59
http://twitter.com	8
http://nuevospowerpoints.blogspot.com	7
http://webcache.googleusercontent.com	3
http://orhowilearnedtolovetheperl.com	1
http://b.hatena.ne.jp	1
http://us-w1.rockmelt.com	1
http://translate.googleusercontent.com	1

オープンソース開発者がDeNAを選ぶ理由

by Kazuho Oku

Accessibility

Categories

Upload Details

Usage Rights

9 Embeds 6,144

Statistics

オープンソース開発者がDeNAを選ぶ理由 Presentation Transcript