• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Building High-Scalable Web Applications Using Q4M and Flare
 

Building High-Scalable Web Applications Using Q4M and Flare

on

  • 4,685 views

Q4MとFlareを使って スケーラブルなサービスを作る!

Q4MとFlareを使って スケーラブルなサービスを作る!

Building High-Scalable Web Applications
Using Q4M and Flare

Statistics

Views

Total Views
4,685
Views on SlideShare
3,816
Embed Views
869

Actions

Likes
6
Downloads
36
Comments
0

8 Embeds 869

http://blog.xole.net 826
http://jar2.tumblr.com 15
http://www.slideshare.net 11
http://webcache.googleusercontent.com 11
http://translate.googleusercontent.com 2
http://posterous.com 2
http://safe.tumblr.com 1
http://cptl.corp.yahoo.co.jp 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Building High-Scalable Web Applications Using Q4M and Flare Building High-Scalable Web Applications Using Q4M and Flare Presentation Transcript

    • Q4M Flare Building High-Scalable Web Applications Using Q4M and Flare Fillmoreadvisory,Inc. Yusuke Hata Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory About me •  Yusuke Hata ( ) ▫  blog: http://blog.xole.net/ •  Committer in Seasar.org for PHP framework development ▫  DI container and AOP Framework: S2Container.PHP5 ▫  DAO (Data-Access-Object) ORM Framework: S2Dao.PHP5 ▫  Port to PHP5 from Java   http://www.seasar.org/#php •  Working at Fillmore advisory since 2008 ▫  http://www.fillmoreadvisory.jp/ •  ( ) ▫  published from Shoeisha in Jan 2009 Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory vizoo We released “vizoo” h t t p : / / w w w. v i s u a l z o o . c o m / Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory About vizoo •  & •  ▫  •  YouTube •  •  PHP5.2.x + symfony Flash (flex) Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory vizoo Q4M Flare Talk about: Q4M and Flare Using in “vizoo” Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Agenda •  Q4M Flare ▫  About Q4M and Flare •  Web ( ) ▫  How to improve: High-load to Web Applications •  Web ( ) ▫  How to improve: Difficult to scaling in Web Applications •  Symfony ▫  Implementation in Symfony •  Q4M Flare ▫  (few) Advanced usage of Q4M and Flare Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Q4M •  Message Queue for MySQL ▫  MySQL •  MySQL Pluggable Storage Engine ▫  MySQL 5.1 Plugin •  ▫  http://q4m.31tools.com/ •  API •  MySQL Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory SQL Queue •  Q4M SQL •  Table •  CREATE TABLE Message CREATE TABLE phpcon_queue ( param_a int unsigned not null, param_b varchar(200), Message priority tinyint not null ) ENGINE=queue; MySQL Q4M queue Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory SQL Queue •  Queue INSERT INSERT INTO phpcon_queue ( enqueue param_a, param_b, priority ) VALUES ( param_a param_b priority 2009, ‘hello’, 9 2009 hello 9 ); 2009 world 5 INSERT INTO phpcon_queue ( param_a, param_b, priority ) VALUES ( enqueue 2009, ‘world’, 5 ); Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory SQL Queue •  Queue Q4M API •  Queue SELECT SELECT queue_wait(‘phpcon_queue’, 2); param_a param_b priority SELECT * FROM phpcon_queue; 2009 hello 9 queue_wait param_a param_b priority 2009 world 5 2009 hello 9 queue_end param_a param_b priority SELECT queue_end(); 2009 world 5 Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory SQL Queue •  queue_wait ( ) SELECT queue_wait(‘phpcon_queue’, 2); •  queue_end SELECT queue_end(); •  queue_abort SELECT queue_abort(); Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Q4M insert (enqueue) task1 insert (enqueue) task2 Q4M task3 insert (enqueue) task4 : : Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Q4M queue_wait task3 hit job task1 select task4 queue_end : Q4M : queue_wait job task2 queue_wait job Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory •  MySQL 5.1 ▫  MySQL •  •  API ▫  API OK Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Flare •  Memcached compatible KVS (key-value storage) •  Persistent storage (Tokyo Cabinet) •  Data Replication •  Dynamic re-construction •  Node monitoring •  GREE ▫  http://labs.gree.jp/Top/OpenSource/Flare.html ▫  http://labs.gree.jp/Top/OpenSource/Flare-en.html Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory master server index server •  checking node status •  switch master node monitoring •  switch proxy telnet slave servers Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Memcached memcache message set get flared apps Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Web Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Web •  Database read/write writes reads App Servers Database Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Web •  Database read/write writes reads   Database App Servers Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Web •  : ▫  Master / Slave Master writes replication reads Slave Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Web DB request write : : waits... : : response reply synchronous! Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Web •  : Q4M ▫  Q4M enqueue workers write write asynchronous! Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Web •  Web App server SMTP Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Web •  : database mail data app server every 5 min cron backend SMTP Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Web •  : event ▫  Q4M notify send mail backend request Q4M event notify excel download Q4M backend Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Web •  SQL heavy queries... Slave app server Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Web •  : backend every 6 hours summary read replication read read Slave app server Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Web •  : ▫  Q4M read only write only summary insert/update once row backend enqueue notify event Q4M app server Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Web Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Web .... Session! Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Web •  master/slave master slave ▫  DB read/write 1 Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Web •  ▫  Flare set xxxx Flare storage (protocol : memcache) Master replication get xxxx Slave (protocol : memcache) Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Web •  Flare flare-1000 memcache-1000 db-1000 db-100 0 50000 100000 150000 200000 Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Web •  Flare ( tx %) 100 1000 10000 (session ) 100 80 60 40 20 0 db memcache flare Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Symfony Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Q4M symfony task <?php class MyQueueTask extends sfTask { protected function execute($arguments = array(), $parameters = array()){ $iterator = new HermitQueue(‘MyQueueDao’, $timeout); foreach($iterator as $queue){ // infinite loop $row = $queue->get(); // // {{{ execute queue }}} // $queue->complete(); // or $queue->abort(); } } } shell> php symfony job:MyQueueTask –env=batch HermitQueue is Q4M wrapper; http://coderepos.org/share/wiki/Hermit Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Flare symfony storage <?php class fmFlareSessionStorage extends sfSessionStorage { public function initialize($opts = array(){ parent::initialize($opts); session_save_handler(array(.....)); } public function sessionRead($id){ $slave = fmMemcache::getInstance(‘slave’); if($slave->has($id)){ return $slave->get($id); } $master = fmMemcache::getInstance(‘master’); $master->set($id, ‘’); return ‘’; } public function sessionWrite($id, $data){ $master = fmMemcache::getInstance(‘master’); $master->set($id, $data); return true; } public function sessionDestory($id){ return fmMemcache::getInstance(‘master’)->remove($id); } } Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Flare symfony storage storage.yml dev: master: host: memcache.local port: 11211 slave: host: memcache.local port: 11211 prod: master: host: flare.master port: 12121 slave: host: flare.slare port: 1212 memcached flared Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Q4M Flare Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Q4M priority •  Priority job High Priority Workers tbl param_1 param_2 priority queue_wait(‘tbl:priority<=2’, 5) .... .... 1 .... .... 2 condition .... .... 1 Low Priority Workers .... .... 5 .... .... 9 queue_wait(‘tbl:priority>=5’, 10) Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Q4M multi queue •  Multi queue Queue a_event_queue b_event_queue c_event_queue param param param .... .... .... .... .... .... .... .... .... Event Workers queue_wait(‘a_event_queue’, ‘b_event_queue’, ‘c_event_queue’, 5) Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Q4M queue •  hoge_queue event_queue .... Hoge Workers big_task_1 .... big_task_2 .... big_task_3 small_task foo_queue .... Foo Workers Map Workers small_task .... .... bar_queue small_task .... Bar Workers .... .... Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Cron Q4M •  batch scheduled_queue task_id next_tstmp <?php $id = 8; 8 123456789 $tstmp = time() + 86400; $timeout = 5; 5 123331231 $stmt = $conn->prepare( 11 124534556 ‘select queue_wait(?,?)’ ); $stmt->bind(1, ‘scheduled_queue:task_id=‘ . $id . ‘&&next_tstmp=‘ . $tstmp Scheduled Workers ); $stmt->bind(2, $timeout); $stmt->execute(); ?> queue_wait( ‘scheduled_queue:task_id=8&&next_tstmp<=123456789’, 5 ) Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Q4M timeout 60 •  Job shutdown (= ) timeout MySQL shutdown •  60 Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory flare •  memcache Flare storage Master replication Slave Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Q4M DB Flare ( + + ) Q4M Flare Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Fillmoreadvisory,Inc. 2009.9.5
    • Fillmore Advisory Fillmoreadvisory,Inc. 2009.9.5