オープンソース開発者がDeNAを選ぶ理由

14,928 views

Published on

Published in: Technology

オープンソース開発者がDeNAを選ぶ理由

  1. 1.     CTO2011 7 16 2
  2. 2.  Palmscape  1997 Xiino  Palm OS ( )  IBM WorkPad, Sony CLIE  NTT DoCoMo  M.I.T. TR100/2002    Web  IPA 2004 –2011 7 16 3
  3. 3. (2)   :  2005 8 2010 11  Japanize  Web UI      Q4M   (MySQL ) 2011 7 16 4
  4. 4. (3)   :  2010 11   IT  2011 5 CTO 2011 7 16 5
  5. 5.   DeNA   OSS  DeNA OSS   2011 7 16 6
  6. 6. 2011 7 16 7
  7. 7. Q4M 2011 7 16 8
  8. 8. What is Q4M?  A message queue  runs as a storage plugin of MySQL 5.1  Why is it a MySQL plugin?  accessible by using existing MySQL clients  no need for a new client library  administrable by using SQL  friendly to DB admins  First release: Jan 2008 2011 7 16 9
  9. 9. Design Goals of Q4M  Robust  Does not lose data on OS crash or power failure  necessary for Tokyo wo. nuclear power plants… orz  Fast  Transfer thousands of messages per second  Easy to Use  Use SQL for access / maintenance  Integration into MySQL  no more separate daemons to take care of 2011 7 16 10
  10. 10. Users of Q4M  Many leading web services in Japan  DeNA Co., Ltd.  livedoor Co., Ltd.  mixi, Inc.  Zynga Japan (formerly Unoh, Inc.)2011 7 16 11
  11. 11. What is a Message Queue? 2011 7 16 12
  12. 12. What is a Message Queue?  Middleware for persistent asynchronous communication  communicate between fixed pairs (parties)  a.k.a. Message Oriented Middleware  MQ is intermediate storage  RDBMS is persistent storage  Senders / receivers may go down2011 7 16 13
  13. 13. Minimal Configuration of a Message Queue  Senders and receivers access a single queue Sender Receiver Queue2011 7 16 14
  14. 14. Using MQ as a Relay  Separate queue for sender and receiver  Messages relayed between queues Relay Sender Receiver Queue Queue2011 7 16 15
  15. 15. Merits of using Message Relays  Destination can be changed easily  Relays may transfer messages to different locations depending on their headers  Robustness against network failure  no loss or duplicates when the relay fails  Logging and Multicasting, etc. 2011 7 16 16
  16. 16. Message Brokers  Publish / subscribe model  Separation between components and their integration  Components read / write to predefined queues  Integration is definition of routing rules between the message queues  Messages are often transformed (filtered) within the relay agent2011 7 16 17
  17. 17. What about Q4M?  Q4M itself is a message queue  Can connect Q4M instances to create a message relay  Provides API for creating message relays and brokers2011 7 16 18
  18. 18. Performance of Q4M  over 7,000 mess/sec.  message size: avg. 512 bytes  syncing to disk  Outperforming most needs  if you need more, just scale out  Can coexist with other storage engines without sacrificing their performance see http://labs.cybozu.co.jp/blog/kazuhoatwork/2008/06/q4m_06_release_and_benchmarks.php2011 7 16 19
  19. 19. Q4M in DeNA  uses Q4M for sending notifications asynchronously http://engineer.dena.jp/2010/03/dena-technical- seminar-1-2.html2011 7 16 20
  20. 20. Q4M in DeNA  Q4M  2,000 /   DeNA    2011 7 16 21
  21. 21. HandlerSocket Plugin 2011 7 16 22
  22. 22. Memcached    RDBMS /      RDBMS      Memcached RDBMS    primary storage2011 7 16 23
  23. 23. Memcached  RDBMS       +   :     SQL2011 7 16 24
  24. 24. SQL ( ) 2011 7 16 25
  25. 25. Mycached  2009 8  mysqld memcached   memcached get   get   SQL 2011 7 16 26
  26. 26. Mycached – mycached http://dev.mysql.com/doc/refman/5.1/en/pluggable-storage-overview.html 2011 7 16 27
  27. 27. Mycached –  1,200  JSON, msgpack,  Picoev (I/O )  MySQL ( )    get 2011 7 16 28
  28. 28. Mycached –  SQL 40,000qps 2011 7 16 29
  29. 29. Mycached    2011 7 16 30
  30. 30. HandlerSocket Plugin  DeNA  2010 8    MySQL Contributor Award 2011  MySQL SQL   qps   mycached2011 7 16 31
  31. 31. … 2011 7 16 32
  32. 32. …2011 7 16 33
  33. 33. … … 2011 7 16 34
  34. 34. … … … 2011 7 16 35
  35. 35. 2011 7 16 36
  36. 36.            2011 7 16 37
  37. 37. HandlerSocket Plugin  mycached      FastEther  2011 7 16 38
  38. 38. 2011 7 16 39
  39. 39. 2011 7 16 40
  40. 40. –         ? 2011 7 16 41
  41. 41. –    Server::Starter –  Parallel::Prefork –  Starlet – PSGI2011 7 16 42
  42. 42. –    fork   : mod_bumpy_life (Apache)  by hirose31++  MaxRequestsPerChild     Parallel::Prefork2011 7 16 43
  43. 43. –   MaxRequestsPerChild   fork … 2011 7 16 44
  44. 44. – : Slow Restart     fork listen accept fork & exec SIGUSR1 accept logic app. spawn app. servers fork & exec app. logic SIGHUP 2011 7 16 45
  45. 45. – OSS   Perl github CPAN     … 2011 7 16 46
  46. 46.  Q. ? $SIG{HUP} = sub { $got_sig = 1 }; … while (! $got_sig) { sleep(1000); // }2011 7 16 47
  47. 47.  Q. ? $SIG{HUP} = sub { $got_sig = 1 }; … while (! $got_sig) { // 1000 sleep(1000); // }2011 7 16 48
  48. 48.   Parallel::Prefork     …   : pselect   FAQ   : Perl pselect   XS (Perl C )   Perl XS 2011 7 16 49
  49. 49. OSS DeNA 2011 7 16 50
  50. 50.  Perl …  2009 Plack  Perl WSGI or Rack  Furl ( HTTP )   DeNA id:xaicron    MySQL2011 7 16 51
  51. 51.     MySQL  YAPC  MySQL Conference       : MySQL Casual    2011 7 16 52
  52. 52. 2011 7 16 53
  53. 53.  OSS     OSS     2011 7 16 54

×