Yapc asia 2011_zigorou

4,438 views
4,332 views

Published on

Trouble history of Mobage Platform

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,438
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
16
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

Yapc asia 2011_zigorou

  1. 1. The case file ofMobage Open Platform Mobage Toru Yamaguchi <zigorou@cpan.org> http://d.hatena.ne.jp/ZIGOROu/ DeNA co.,ltd. 2011/10/14 YAPC Asia 2011
  2. 2. !  !  ! !  ! 
  3. 3. !   Wikipedia ! !  !  !  (Cause Analysis) !  (Failure Prevention) !  (Knowledge Distribution)
  4. 4. ( )
  5. 5. 1 DeadLock !  API (T ) Dead Lock !  API Internal Server Error ! !  !   Internal Server Error
  6. 6. (1) !  API GET ! totalResults OpenSearch!  totalResults SQL_CALC_FOUND_ROWS !  SELECT!  totalResults !   INSERT, DELETE TRIGGER
  7. 7. (2) !  SHOW INNODB STATUS !   Dead Lock ! !  !  UPDATE !  UPDATE ! 
  8. 8. BEGIN;SELECT id FROM somedata_group WHERE name = ?;INSERT INTO somedata(id, group_id, data, published_on)VALUES(?, ?, ?, ?);UPDATE somedata_summary SET total_results = total_results+ 1, updated_on = ? WHERE group_id = ?;COMMIT;
  9. 9. (3) !   Group !  ! !  !  UPDATE increment !  MySQL Dead Lock
  10. 10. (4) Transaction group_id = 1 Transaction group_id = 2 group_id = 3 Transaction group_id = 3UPDATE group_id = 4 group_id = 5 Transaction
  11. 11. (1) !  !  UPDATE w !  SELECT totalResults !   INSERT incr/decr QUEUE UPDATE !  totalResults !  xaicron
  12. 12. (2) /* API */BEGIN;INSERT INTO somedata(id, group_id, data, published_on)VALUES(?, ?, ?, ?);INSERT INTO somedata_summary_queue(id, group_id,affected_number, published_on);COMMIT;
  13. 13. (3) /* Batch Worker */BEGIN;SELECT id, group_id, affected_number FROMsomegroup_summary_queue ORDER BY published_on ASC LIMIT 100;/* group UPDATE */UPDATE somedata_summary SET total_results = ?, updated_on = ?WHERE group_id = ?;…DELETE somegroup_summary_queue WHERE id IN (?, ?, …, ?);COMMIT;
  14. 14. (1) !  !  !  !  InnoDB QUEUE ( ) UPDATE ! 
  15. 15. (2) !  ! InnoDB QUEUE INDEX !   INSERT, DELETE enqueue !  queue WHERE id !  FOR UPDATE ! 
  16. 16. 2 INSERT vs DELETE !  ! !  API ( T ) !  1 INSERT !  API purge !  purge ( ) INSERT!  !  shard
  17. 17. ( )
  18. 18. (1) !  purge !  MASTER DELETE SLAVE ww !   DELETE !  redo SLAVE !  sleep wait DB wait !  wait!  wait !   SLAVE Seconds_Behind_Master wait !  wait
  19. 19. Loop::Sustainable (1) ! ! https://github.com/zigorou/p5-loop-sustainable!  !  !  !  wait !  !  Seconds_Behind_Master !  b ry
  20. 20. Loop::Sustainable (2) process process 2sec process process process 2sec process process 2sec process process 2sec Seconds_Behind_Master : 10 sec process 2sec
  21. 21. (2) !  w!   SET SESSION sql_log_bin = 0 !   MASTER SLAVE DELETE!  DELETE !  prefork DELETE xaicron ( )
  22. 22. (3) ! !   DELETE!   DELETE purge !  partitioning API!   STOP SLAVE DB Master !  ! 
  23. 23. (1) !  DELETE !  DELETE !   DELETE SELECT!  MySQL !  !  !  RDBMS ! !   SET SESSION sql_log_bin=0 !  DELETE
  24. 24. (2) !  DELETE INSERT !  Partitioning !  Sharding!  ! mysqldump –w WHERE
  25. 25. (1) !  !  !  !   PRIMARY KEY !  !  ALTER TABLE some_table ADD KEY !  !  M
  26. 26. (2) !  !  !  DML !  incr/decr !  ! 
  27. 27. 3 ! ! !  API Push friend timeline !  DB!  API
  28. 28. (1) !  !  !  ! !  count, startIndex (LIMIT/OFFSET )! 
  29. 29. SELECT friend_user_id FROM friends WHERE user_id = ?;/* */SELECT user_id FROM user_app WHERE app_id = ? ANDuser_id IN (?, ?, …, ?);/* */SELECT SQL_CALC_FOUND_ROWS user_id, nicknameFROM users WHERE user_id IN(?, ?, …, ?) LIMIT 50OFFSET 0;SELECT FOUND_ROWS();
  30. 30. LIMIT 50 valid users OFFSET 0 500 users installed users 750 users friend1000 users
  31. 31. (2) !   friends n:m, user_app 1:n, users 1:1!   friends ( 1000 ) user_app!   users SQL_CALC_FOUND_ROWS LIMIT, OFFSET!  friends !  Temporary Table INSERT users JOIN
  32. 32. (1) !  ! selectall_arrayref ! !   SQL_CALC_FOUND_ROWS !   LIMIT, OFFSET COUNT(*) ! !   Temporary Table !  !  !  user_id user_id LIMIT, OFFSET
  33. 33. (2) !   prepare, fetchall_arrayref($max_rows) !   DBI::st fetchall_arrayref($max_rows) $max_rows !  !  friends 1000!  1000 users user_app !   1000!  1000 users !  1000! 
  34. 34. installed users valid users friend installed users 750 users 750 users 1000 users 1000 users installed users valid users 750 users 750 users friend installed users1000 users 1000 users 100 users installed users 750 users installed users 250 users friend1000 users end iteration
  35. 35. (3) ! ! !   Iterator::GroupedRange ! http://search.cpan.org/dist/Iterator-GroupedRange/!  List::MoreUtils natatime I::GR (fetchall_arrayref($max_rows) )
  36. 36. (4) ! !   friends DB 1000 memcached get/set! user_app mapping memcached ( install )!  iteration DB DB iteration set!  3 !  xaicron
  37. 37. (1) !  ! !  !  w!  kazuho
  38. 38. !  !  !  ! !  !  xaicron ! !  ! 

×