Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
The case file ofMobage Open Platform	  Mobage                    	 Toru Yamaguchi <zigorou@cpan.org>   http://d.hatena.ne....
!      !      ! !      ! 
!   Wikipedia     ! !      !      !          (Cause Analysis)     !          (Failure Prevention)     !          (Knowledg...
(   )
1 DeadLock                                     	!         API (T                 )                           Dead Lock    ...
(1)	!                   API   GET     !    totalResults             OpenSearch!             totalResults     SQL_CALC_FOUN...
(2)	!                    SHOW INNODB STATUS     !   Dead Lock     ! !      !                UPDATE          !             ...
BEGIN;SELECT id FROM somedata_group WHERE name = ?;INSERT INTO somedata(id, group_id, data, published_on)VALUES(?, ?, ?, ?...
(3)	!   Group     !      ! !      !                                     UPDATE               increment     !              ...
(4)	        Transaction	   group_id = 1	           Transaction	                        group_id = 2	                      ...
(1)	!      !            UPDATE                          w     !                       SELECT               totalResults   ...
(2)	/* API                */BEGIN;INSERT INTO somedata(id, group_id, data, published_on)VALUES(?, ?, ?, ?);INSERT INTO som...
(3)	/* Batch Worker       */BEGIN;SELECT id, group_id, affected_number FROMsomegroup_summary_queue ORDER BY published_on A...
(1)	!      !      !      !         InnoDB    QUEUE                 (      ) UPDATE          !                             ...
(2)	!      !    InnoDB            QUEUE              INDEX          !   INSERT, DELETE         enqueue     !              ...
2 INSERT vs DELETE	!      ! !          API (             T            )     !          1            INSERT     !        AP...
(   )
(1)	!              purge     !           MASTER      DELETE                 SLAVE            ww          !   DELETE       ...
Loop::Sustainable (1)	! ! https://github.com/zigorou/p5-loop-sustainable!      !      !      !                    wait    ...
Loop::Sustainable (2)	         process	                 process	                                               2sec	      ...
(2)	!                               w!   SET SESSION sql_log_bin = 0     !   MASTER      SLAVE             DELETE!        ...
(3)	! !   DELETE!   DELETE                    purge     !    partitioning    API!   STOP SLAVE       DB    Master     !   ...
(1)	!                   DELETE     !                                DELETE     !   DELETE              SELECT!            ...
(2)	!              DELETE                       INSERT     !               Partitioning     !           Sharding!      !  ...
(1)	!      !      !      !   PRIMARY KEY     !      !                 ALTER TABLE some_table ADD          KEY     !      !...
(2)	!      !      !              DML          !    incr/decr     !           ! 
3                    	! ! !      API    Push   friend timeline     !                               DB!                    ...
(1)	!      !      !      ! !                                       count,     startIndex (LIMIT/OFFSET      )!            ...
SELECT friend_user_id FROM friends WHERE user_id = ?;/*                       */SELECT user_id FROM user_app WHERE app_id ...
LIMIT 50                                 valid users    OFFSET 0	                                  500 users	             ...
(2)	!   friends       n:m, user_app     1:n, users        1:1!   friends                                                 (...
(1)	!      ! selectall_arrayref     ! !   SQL_CALC_FOUND_ROWS     !   LIMIT, OFFSET                         COUNT(*)     !...
(2)	!   prepare, fetchall_arrayref($max_rows)     !   DBI::st   fetchall_arrayref($max_rows)       $max_rows     !      ! ...
installed users                          valid users  friend                         installed users         750 users	   ...
(3)	! ! !   Iterator::GroupedRange     !   http://search.cpan.org/dist/Iterator-GroupedRange/!                            ...
(4)	! !   friends             DB                      1000    memcached        get/set!    user_app           mapping     ...
(1)	!      ! !      !           w!             kazuho
!      !      !      ! !      !           xaicron     ! !      ! 
Yapc asia 2011_zigorou
Yapc asia 2011_zigorou
Yapc asia 2011_zigorou
Yapc asia 2011_zigorou
Upcoming SlideShare
Loading in …5
×

Yapc asia 2011_zigorou

4,850 views

Published on

Trouble history of Mobage Platform

Published in: Technology
  • Be the first to comment

Yapc asia 2011_zigorou

  1. 1. The case file ofMobage Open Platform Mobage Toru Yamaguchi <zigorou@cpan.org> http://d.hatena.ne.jp/ZIGOROu/ DeNA co.,ltd. 2011/10/14 YAPC Asia 2011
  2. 2. !  !  ! !  ! 
  3. 3. !   Wikipedia ! !  !  !  (Cause Analysis) !  (Failure Prevention) !  (Knowledge Distribution)
  4. 4. ( )
  5. 5. 1 DeadLock !  API (T ) Dead Lock !  API Internal Server Error ! !  !   Internal Server Error
  6. 6. (1) !  API GET ! totalResults OpenSearch!  totalResults SQL_CALC_FOUND_ROWS !  SELECT!  totalResults !   INSERT, DELETE TRIGGER
  7. 7. (2) !  SHOW INNODB STATUS !   Dead Lock ! !  !  UPDATE !  UPDATE ! 
  8. 8. BEGIN;SELECT id FROM somedata_group WHERE name = ?;INSERT INTO somedata(id, group_id, data, published_on)VALUES(?, ?, ?, ?);UPDATE somedata_summary SET total_results = total_results+ 1, updated_on = ? WHERE group_id = ?;COMMIT;
  9. 9. (3) !   Group !  ! !  !  UPDATE increment !  MySQL Dead Lock
  10. 10. (4) Transaction group_id = 1 Transaction group_id = 2 group_id = 3 Transaction group_id = 3UPDATE group_id = 4 group_id = 5 Transaction
  11. 11. (1) !  !  UPDATE w !  SELECT totalResults !   INSERT incr/decr QUEUE UPDATE !  totalResults !  xaicron
  12. 12. (2) /* API */BEGIN;INSERT INTO somedata(id, group_id, data, published_on)VALUES(?, ?, ?, ?);INSERT INTO somedata_summary_queue(id, group_id,affected_number, published_on);COMMIT;
  13. 13. (3) /* Batch Worker */BEGIN;SELECT id, group_id, affected_number FROMsomegroup_summary_queue ORDER BY published_on ASC LIMIT 100;/* group UPDATE */UPDATE somedata_summary SET total_results = ?, updated_on = ?WHERE group_id = ?;…DELETE somegroup_summary_queue WHERE id IN (?, ?, …, ?);COMMIT;
  14. 14. (1) !  !  !  !  InnoDB QUEUE ( ) UPDATE ! 
  15. 15. (2) !  ! InnoDB QUEUE INDEX !   INSERT, DELETE enqueue !  queue WHERE id !  FOR UPDATE ! 
  16. 16. 2 INSERT vs DELETE !  ! !  API ( T ) !  1 INSERT !  API purge !  purge ( ) INSERT!  !  shard
  17. 17. ( )
  18. 18. (1) !  purge !  MASTER DELETE SLAVE ww !   DELETE !  redo SLAVE !  sleep wait DB wait !  wait!  wait !   SLAVE Seconds_Behind_Master wait !  wait
  19. 19. Loop::Sustainable (1) ! ! https://github.com/zigorou/p5-loop-sustainable!  !  !  !  wait !  !  Seconds_Behind_Master !  b ry
  20. 20. Loop::Sustainable (2) process process 2sec process process process 2sec process process 2sec process process 2sec Seconds_Behind_Master : 10 sec process 2sec
  21. 21. (2) !  w!   SET SESSION sql_log_bin = 0 !   MASTER SLAVE DELETE!  DELETE !  prefork DELETE xaicron ( )
  22. 22. (3) ! !   DELETE!   DELETE purge !  partitioning API!   STOP SLAVE DB Master !  ! 
  23. 23. (1) !  DELETE !  DELETE !   DELETE SELECT!  MySQL !  !  !  RDBMS ! !   SET SESSION sql_log_bin=0 !  DELETE
  24. 24. (2) !  DELETE INSERT !  Partitioning !  Sharding!  ! mysqldump –w WHERE
  25. 25. (1) !  !  !  !   PRIMARY KEY !  !  ALTER TABLE some_table ADD KEY !  !  M
  26. 26. (2) !  !  !  DML !  incr/decr !  ! 
  27. 27. 3 ! ! !  API Push friend timeline !  DB!  API
  28. 28. (1) !  !  !  ! !  count, startIndex (LIMIT/OFFSET )! 
  29. 29. SELECT friend_user_id FROM friends WHERE user_id = ?;/* */SELECT user_id FROM user_app WHERE app_id = ? ANDuser_id IN (?, ?, …, ?);/* */SELECT SQL_CALC_FOUND_ROWS user_id, nicknameFROM users WHERE user_id IN(?, ?, …, ?) LIMIT 50OFFSET 0;SELECT FOUND_ROWS();
  30. 30. LIMIT 50 valid users OFFSET 0 500 users installed users 750 users friend1000 users
  31. 31. (2) !   friends n:m, user_app 1:n, users 1:1!   friends ( 1000 ) user_app!   users SQL_CALC_FOUND_ROWS LIMIT, OFFSET!  friends !  Temporary Table INSERT users JOIN
  32. 32. (1) !  ! selectall_arrayref ! !   SQL_CALC_FOUND_ROWS !   LIMIT, OFFSET COUNT(*) ! !   Temporary Table !  !  !  user_id user_id LIMIT, OFFSET
  33. 33. (2) !   prepare, fetchall_arrayref($max_rows) !   DBI::st fetchall_arrayref($max_rows) $max_rows !  !  friends 1000!  1000 users user_app !   1000!  1000 users !  1000! 
  34. 34. installed users valid users friend installed users 750 users 750 users 1000 users 1000 users installed users valid users 750 users 750 users friend installed users1000 users 1000 users 100 users installed users 750 users installed users 250 users friend1000 users end iteration
  35. 35. (3) ! ! !   Iterator::GroupedRange ! http://search.cpan.org/dist/Iterator-GroupedRange/!  List::MoreUtils natatime I::GR (fetchall_arrayref($max_rows) )
  36. 36. (4) ! !   friends DB 1000 memcached get/set! user_app mapping memcached ( install )!  iteration DB DB iteration set!  3 !  xaicron
  37. 37. (1) !  ! !  !  w!  kazuho
  38. 38. !  !  !  ! !  !  xaicron ! !  ! 

×