Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Surpresas no usode Concorrência    em by Marcos Toledo        Rails
Marcos RenatoBaroni Toledo  @mtoledo
Socram O’taner
Socram O’tanerMarcos Renato
O’taner
Email
“I’m trying tounsubscribe butit says email has   already been      taken”!
validates_uniquen      ess_of   # id :integer(4)   # email :string(255)   class User < ActiveRecord::Base    validates_uni...
WTF ..  ?
Invalid Records!User.find_by_email(“angry_customer@email.com”).valid?# => false
Why?
Rails can’t guaranteeuniqueness!
Solutions?
Uniqueness           Constraint!class AddUniquenessIndexToUserEmail < ActiveRecord::Migration def self.up   add_index :use...
Success!
Another Email
Airbrake!
NoMethodError: undefined method `title for               nil:NilClass       “nil post”?
:dependent     => :destroyclass Comment < ActiveRecord::Base belongs_to :post validates_presence_of :postendclass Post < A...
WTF ..  ?
Orphan Models!Comment.all.reject(&:post).size# => 1
Why?
Rails can’t guaranteedependency   either!
Solutions?
Database Foreign     Keys!create_table :comments do |table| table.integer :post_id, :null => falseendActiveRecord::Base.co...
Success!
1 - Noob“Use database constraints”
O’taner
ruby 100% CPU!
Solutions?
ProfessionalDebuggingssh serverkillall ruby
Success!
WET Code
Infinite Loopclass Comment < ActiveRecord::Base after_create :update_comment_count def update_comment_count  post = Post.fin...
WTF ..  ?
TransactionIsolation Level  “I” in ACID
1 - READUNCOMMITTED2 - READCOMMITTED3 - REPEATABLEREAD
PostgreREAD-COMMITTED     MySQLREPEATABLE-READ
mysql> begin;Query OK, 0 rows affected (0,00 sec)mysql> select * from posts where id = 1;+----+-------+--------------+| id...
mysql> begin;Query OK, 0 rows affected (0,00 sec)mysql> select * from posts where id = 1;+----+-------+--------------+| id...
mysql> begin;Query OK, 0 rows affected (0,00 sec)mysql> select * from posts where id = 1;+----+-------+--------------+| id...
mysql> begin;Query OK, 0 rows affected (0,00 sec)mysql> select * from posts where id = 1;+----+-------+--------------+| id...
Solutions?
READ-    COMMITTED!              /etc/my.cnf[mysqld]transaction-isolation = READ-COMMITTED                                ...
mysql> begin;Query OK, 0 rows affected (0,00 sec)mysql> select * from posts where id = 1;+----+-------+--------------+| id...
Success!
2 - Beginner“Use READ-COMMITTED”
O’taner
Production             Console> u1 = User.first=> #<User id: 1, login: "marcos", options: {:active=>false}> u1.update_attri...
WTF ..  ?
# login :string(255)# options :textclass User < ActiveRecord::Base serialize :options def activate  self.options = {:activ...
No ‘:active =>    false’!
Why?
Dirty Checking> u1 = User.first=> #<User id: 1, login: "marcos", options: {:active=>false}> u2 = User.first=> #<User id: 1, ...
Serialize isALWAYS dirty!
// u1.update_attributes :options => {:active => true}UPDATE `users`SET `options` = --- n:active: truenWHERE (`users`.`id` ...
Solutions?
Options Model
# id         :integer(4)# user_id       :integer(4)# lock_version :integer(4)# options       :textclass UserOptions < Acti...
Success!
3 - Half Mouth“Use an Options     Model”
O’taner
production.logERROR 1213 (40001): Deadlock found when trying to get            lock;try restarting transaction
DAMN!Pessimistic Locks!
No pessimistic        locking!class Comment after_create :update_posts_count def update_posts_count  self.posts.increment ...
WTF ..  ?
Transac Transaction 1SELECT * FOR                         tion 2UPDATEFROM `comments`WHERE `id` = 1        SELECT * FOR   ...
Why?
Transactions lock   on update
Transac  Transaction 1UPDATE                             tion 2`comments`SET `title` = “Foo”WHERE `id` = 1                ...
Callbacks =Transactions
Solutions?
Update dependentsbefore parents!
No longer            deadlocks!class Comment after_create :update_posts_count def update_posts_count  self.posts.increment...
Avoid StatePropagation
Avoids updating all commentsclass Comment < ActiveRecord::Base def visible?   post.published? && self.published? endend
Success!
4 - Advanced“Acquire locks consistently”
O’taner
“There is no safe way to use                     Thread#kill or                    Thread#raise.”http://blog.headius.com/2...
Killing ruby (MRI)      threads - bluepill - god - monit - you and me
Simple      Transactionclass Order belongs_to :user after_create :mark_user_as_customer def mark_user_as_customer  user.cu...
What happened          to    my transaction?> order = Order.first=> #<Order id: 1, user_id: 1>> order.user.customer?=> fals...
WTF ..  ?
If you kill your      Ruby(MRI) Threads ..
ActiveRecord will     partially  commit your  transactions!
Why?
ActiveRecord    transaction (version for kids)def transaction yieldrescue Exception => e rollbackensure commit unless roll...
Main thread is               rescueddef print_if_rescued sleep 10rescue Exception puts rescuedensure puts ensuredendprint_...
Other Threads are                 NOT!def print_if_rescued sleep 10rescue Exception puts rescuedensure puts ensuredendt = ...
It commits!def transaction yield # thread killed inside yieldrescue Exception => e rollback # nothing is raisedensure comm...
Solutions?
DON’T Kill Your  Threads!           Headius Approv
If you do, don’tuse ActiveRecord+ Threads + MRI
5 Record Certified Enterprise Active         - ARCECA      “Don’t use       Concurrency ArchitectThread#kill/raise ”
Success!
Recap please!
1 - Noob“Use database constraints”
2 - Beginner“Use READ-COMMITTED”
3 - Half Mouth“Use an Options     Model”
4 - Advanced“Acquire locks consistently”
5 Record Certified Enterprise Active         - ARCECA      “Don’t use       Concurrency ArchitectThread#kill/raise ”
That’s all! Thank      You!
Questions?Slides @mtoledo
Rails Concurrency Gotchas
Upcoming SlideShare
Loading in …5
×

Rails Concurrency Gotchas

3,561 views

Published on

With the ever more frequent use of multiple servers and worker processes, issues which only occur when specific tasks are running in parallel become ever more likely to happen.
In this talk Marcos talks about some rails concurrency limitations you need to be aware of, including why you should never trust rails to check for uniqueness, why mysql can't do optimistic locking with retrying by default and how to fix it, how to properly choose between optimistic and pessimistic locking, how parallel processes can cause deadlocks even if you're not using locks, why you should never have serialized attributes in rows that might be edited in parallel, and why rails commits your transactions before they finish in case your multi threaded process is killed and how to prevent it from leaving your database on an inconsistent state.

Published in: Technology, Business
  • Be the first to comment

Rails Concurrency Gotchas

  1. 1. Surpresas no usode Concorrência em by Marcos Toledo Rails
  2. 2. Marcos RenatoBaroni Toledo @mtoledo
  3. 3. Socram O’taner
  4. 4. Socram O’tanerMarcos Renato
  5. 5. O’taner
  6. 6. Email
  7. 7. “I’m trying tounsubscribe butit says email has already been taken”!
  8. 8. validates_uniquen ess_of # id :integer(4) # email :string(255) class User < ActiveRecord::Base validates_uniqueness_of :email end
  9. 9. WTF .. ?
  10. 10. Invalid Records!User.find_by_email(“angry_customer@email.com”).valid?# => false
  11. 11. Why?
  12. 12. Rails can’t guaranteeuniqueness!
  13. 13. Solutions?
  14. 14. Uniqueness Constraint!class AddUniquenessIndexToUserEmail < ActiveRecord::Migration def self.up add_index :users, :email, :unique => true end def self.down remove_index :users, :email endend
  15. 15. Success!
  16. 16. Another Email
  17. 17. Airbrake!
  18. 18. NoMethodError: undefined method `title for nil:NilClass “nil post”?
  19. 19. :dependent => :destroyclass Comment < ActiveRecord::Base belongs_to :post validates_presence_of :postendclass Post < ActiveRecord::Base has_many :comments, :dependent => :destroyend
  20. 20. WTF .. ?
  21. 21. Orphan Models!Comment.all.reject(&:post).size# => 1
  22. 22. Why?
  23. 23. Rails can’t guaranteedependency either!
  24. 24. Solutions?
  25. 25. Database Foreign Keys!create_table :comments do |table| table.integer :post_id, :null => falseendActiveRecord::Base.connection.execute <<-EOS ALTER TABLE `comments` ADD CONSTRAINT FOREIGN KEY (post_id) REFERENCES posts(id) ON DELETE restrict ON UPDATE cascadeEOS
  26. 26. Success!
  27. 27. 1 - Noob“Use database constraints”
  28. 28. O’taner
  29. 29. ruby 100% CPU!
  30. 30. Solutions?
  31. 31. ProfessionalDebuggingssh serverkillall ruby
  32. 32. Success!
  33. 33. WET Code
  34. 34. Infinite Loopclass Comment < ActiveRecord::Base after_create :update_comment_count def update_comment_count post = Post.find post_id post.comment_count += 1 post.save rescue StaleObjectError retry endend
  35. 35. WTF .. ?
  36. 36. TransactionIsolation Level “I” in ACID
  37. 37. 1 - READUNCOMMITTED2 - READCOMMITTED3 - REPEATABLEREAD
  38. 38. PostgreREAD-COMMITTED MySQLREPEATABLE-READ
  39. 39. mysql> begin;Query OK, 0 rows affected (0,00 sec)mysql> select * from posts where id = 1;+----+-------+--------------+| id | title | lock_version |+----+-------+--------------+| 1 | Title | 0|+----+-------+--------------+1 row in set (0,00 sec)mysql> update posts set title = New Title where lock_version = 0;Query OK, 0 rows affected (0,00 sec)Rows matched: 0 Changed: 0 Warnings: 0mysql> select * from posts where id = 1;+----+-------+--------------+| id | title | lock_version |+----+-------+--------------+| 1 | Title | 0|+----+-------+--------------+1 row in set (0,00 sec)
  40. 40. mysql> begin;Query OK, 0 rows affected (0,00 sec)mysql> select * from posts where id = 1;+----+-------+--------------+| id | title | lock_version |+----+-------+--------------+| 1 | Title | 0|+----+-------+--------------+1 row in set (0,00 sec)mysql> update posts set title = New Title where lock_version = 0;Query OK, 0 rows affected (0,00 sec)Rows matched: 0 Changed: 0 Warnings: 0mysql> select * from posts where id = 1;+----+-------+--------------+| id | title | lock_version |+----+-------+--------------+| 1 | Title | 0|+----+-------+--------------+1 row in set (0,00 sec)
  41. 41. mysql> begin;Query OK, 0 rows affected (0,00 sec)mysql> select * from posts where id = 1;+----+-------+--------------+| id | title | lock_version |+----+-------+--------------+| 1 | Title | 0|+----+-------+--------------+1 row in set (0,00 sec)mysql> update posts set title = New Title where lock_version = 0;Query OK, 0 rows affected (0,00 sec)Rows matched: 0 Changed: 0 Warnings: 0 // raises StaleObjectErrormysql> select * from posts where id = 1;+----+-------+--------------+| id | title | lock_version |+----+-------+--------------+| 1 | Title | 0|+----+-------+--------------+1 row in set (0,00 sec)
  42. 42. mysql> begin;Query OK, 0 rows affected (0,00 sec)mysql> select * from posts where id = 1;+----+-------+--------------+| id | title | lock_version |+----+-------+--------------+| 1 | Title | 0|+----+-------+--------------+1 row in set (0,00 sec)mysql> update posts set title = New Title where lock_version = 0;Query OK, 0 rows affected (0,00 sec)Rows matched: 0 Changed: 0 Warnings: 0 // raises StaleObjectErrormysql> select * from posts where id = 1;+----+-------+--------------+| id | title | lock_version |+----+-------+--------------+| 1 | Title | 0|+----+-------+--------------+1 row in set (0,00 sec)
  43. 43. Solutions?
  44. 44. READ- COMMITTED! /etc/my.cnf[mysqld]transaction-isolation = READ-COMMITTED Just like postgre ;)
  45. 45. mysql> begin;Query OK, 0 rows affected (0,00 sec)mysql> select * from posts where id = 1;+----+-------+--------------+| id | title | lock_version |+----+-------+--------------+| 1 | Title | 0|+----+-------+--------------+1 row in set (0,00 sec)mysql> update posts set title = New Title where lock_version = 0;Query OK, 0 rows affected (0,00 sec)Rows matched: 0 Changed: 0 Warnings: 0 // raises StaleObjectErrormysql> select * from posts where id = 1;+----+-----------+--------------+| id | title | lock_version |+----+-------+--------------+| 1 | Title | 1|+----+-------+--------------+1 row in set (0,00 sec)
  46. 46. Success!
  47. 47. 2 - Beginner“Use READ-COMMITTED”
  48. 48. O’taner
  49. 49. Production Console> u1 = User.first=> #<User id: 1, login: "marcos", options: {:active=>false}> u1.update_attributes :options => {:active => true}=> true> u1.reload=> #<User id: 1, login: "marcos", options: {:active=>false}
  50. 50. WTF .. ?
  51. 51. # login :string(255)# options :textclass User < ActiveRecord::Base serialize :options def activate self.options = {:active => true} self.save endend
  52. 52. No ‘:active => false’!
  53. 53. Why?
  54. 54. Dirty Checking> u1 = User.first=> #<User id: 1, login: "marcos", options: {:active=>false}> u2 = User.first=> #<User id: 1, login: "marcos", options: {:active=>false}> u1.update_attributes :options => {:active => true}=> true> u2.update_attributes :login => "mtoledo"=> true> u1.reload=> #<User id: 1, login: "mtoledo", options: {:active=>false}
  55. 55. Serialize isALWAYS dirty!
  56. 56. // u1.update_attributes :options => {:active => true}UPDATE `users`SET `options` = --- n:active: truenWHERE (`users`.`id` = 1)// u2.update_attributes :login => "mtoledo"UPDATE `users`SET `login` = mtoledo, `options` = --- n:active: falsenWHERE (`users`.`id` = 1)
  57. 57. Solutions?
  58. 58. Options Model
  59. 59. # id :integer(4)# user_id :integer(4)# lock_version :integer(4)# options :textclass UserOptions < ActiveRecord::Base serialize :options belongs_to :userend
  60. 60. Success!
  61. 61. 3 - Half Mouth“Use an Options Model”
  62. 62. O’taner
  63. 63. production.logERROR 1213 (40001): Deadlock found when trying to get lock;try restarting transaction
  64. 64. DAMN!Pessimistic Locks!
  65. 65. No pessimistic locking!class Comment after_create :update_posts_count def update_posts_count self.posts.increment :comments_count endendclass Posts after_update :update_comments def update_comments comments.each {|c| c.published = self.published; c.save} endend
  66. 66. WTF .. ?
  67. 67. Transac Transaction 1SELECT * FOR tion 2UPDATEFROM `comments`WHERE `id` = 1 SELECT * FOR UPDATE locked.. FROM `posts` WHERE `id` = 1SELECT * FORUPDATEFROM `posts` DEADLOCK!WHERE `id` = 1 SELECT * FOR UPDATE FROM `comments` WHERE `id` = 1
  68. 68. Why?
  69. 69. Transactions lock on update
  70. 70. Transac Transaction 1UPDATE tion 2`comments`SET `title` = “Foo”WHERE `id` = 1 UPDATE `posts` SET `title` = “bar” locked.. WHERE `id` = 1UPDATE `posts`SET `title` = “bar”WHERE `id` = 1 DEADLOCK! UPDATE `comments` SET `title` = “foo” WHERE `id` = 1
  71. 71. Callbacks =Transactions
  72. 72. Solutions?
  73. 73. Update dependentsbefore parents!
  74. 74. No longer deadlocks!class Comment after_create :update_posts_count def update_posts_count self.posts.increment :comments_count endendclass Posts before_update :update_comments def update_comments comments.each {|c| c.published = self.published; c.save} endend
  75. 75. Avoid StatePropagation
  76. 76. Avoids updating all commentsclass Comment < ActiveRecord::Base def visible? post.published? && self.published? endend
  77. 77. Success!
  78. 78. 4 - Advanced“Acquire locks consistently”
  79. 79. O’taner
  80. 80. “There is no safe way to use Thread#kill or Thread#raise.”http://blog.headius.com/2008/02/rubys-threadraise-threadkill-timeoutrb.html
  81. 81. Killing ruby (MRI) threads - bluepill - god - monit - you and me
  82. 82. Simple Transactionclass Order belongs_to :user after_create :mark_user_as_customer def mark_user_as_customer user.customer = true user.save endend
  83. 83. What happened to my transaction?> order = Order.first=> #<Order id: 1, user_id: 1>> order.user.customer?=> false> order.user?=> #<User id: 1, login: "marcos", customer: false>
  84. 84. WTF .. ?
  85. 85. If you kill your Ruby(MRI) Threads ..
  86. 86. ActiveRecord will partially commit your transactions!
  87. 87. Why?
  88. 88. ActiveRecord transaction (version for kids)def transaction yieldrescue Exception => e rollbackensure commit unless rolledback?end
  89. 89. Main thread is rescueddef print_if_rescued sleep 10rescue Exception puts rescuedensure puts ensuredendprint_if_rescued# $ killall ruby# => rescued# => ensured
  90. 90. Other Threads are NOT!def print_if_rescued sleep 10rescue Exception puts rescuedensure puts ensuredendt = Thread.new { print_if_rescued }t.join# $ killall ruby# => ensured
  91. 91. It commits!def transaction yield # thread killed inside yieldrescue Exception => e rollback # nothing is raisedensure commit unless rolledback?end
  92. 92. Solutions?
  93. 93. DON’T Kill Your Threads! Headius Approv
  94. 94. If you do, don’tuse ActiveRecord+ Threads + MRI
  95. 95. 5 Record Certified Enterprise Active - ARCECA “Don’t use Concurrency ArchitectThread#kill/raise ”
  96. 96. Success!
  97. 97. Recap please!
  98. 98. 1 - Noob“Use database constraints”
  99. 99. 2 - Beginner“Use READ-COMMITTED”
  100. 100. 3 - Half Mouth“Use an Options Model”
  101. 101. 4 - Advanced“Acquire locks consistently”
  102. 102. 5 Record Certified Enterprise Active - ARCECA “Don’t use Concurrency ArchitectThread#kill/raise ”
  103. 103. That’s all! Thank You!
  104. 104. Questions?Slides @mtoledo

×