Rails Concurrency Gotchas
Upcoming SlideShare
Loading in...5
×
 

Rails Concurrency Gotchas

on

  • 2,056 views

With the ever more frequent use of multiple servers and worker processes, issues which only occur when specific tasks are running in parallel become ever more likely to happen....

With the ever more frequent use of multiple servers and worker processes, issues which only occur when specific tasks are running in parallel become ever more likely to happen.
In this talk Marcos talks about some rails concurrency limitations you need to be aware of, including why you should never trust rails to check for uniqueness, why mysql can't do optimistic locking with retrying by default and how to fix it, how to properly choose between optimistic and pessimistic locking, how parallel processes can cause deadlocks even if you're not using locks, why you should never have serialized attributes in rows that might be edited in parallel, and why rails commits your transactions before they finish in case your multi threaded process is killed and how to prevent it from leaving your database on an inconsistent state.

Statistics

Views

Total Views
2,056
Views on SlideShare
2,055
Embed Views
1

Actions

Likes
2
Downloads
4
Comments
0

1 Embed 1

http://www.slashdocs.com 1

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

Rails Concurrency Gotchas Rails Concurrency Gotchas Presentation Transcript

  • Surpresas no usode Concorrência em by Marcos Toledo Rails
  • Marcos RenatoBaroni Toledo @mtoledo
  • Socram O’taner
  • Socram O’tanerMarcos Renato
  • O’taner
  • Email
  • “I’m trying tounsubscribe butit says email has already been taken”!
  • validates_uniquen ess_of # id :integer(4) # email :string(255) class User < ActiveRecord::Base validates_uniqueness_of :email end
  • WTF .. ?
  • Invalid Records!User.find_by_email(“angry_customer@email.com”).valid?# => false
  • Why?
  • Rails can’t guaranteeuniqueness!
  • Solutions?
  • Uniqueness Constraint!class AddUniquenessIndexToUserEmail < ActiveRecord::Migration def self.up add_index :users, :email, :unique => true end def self.down remove_index :users, :email endend
  • Success!
  • Another Email
  • Airbrake!
  • NoMethodError: undefined method `title for nil:NilClass “nil post”?
  • :dependent => :destroyclass Comment < ActiveRecord::Base belongs_to :post validates_presence_of :postendclass Post < ActiveRecord::Base has_many :comments, :dependent => :destroyend
  • WTF .. ?
  • Orphan Models!Comment.all.reject(&:post).size# => 1
  • Why?
  • Rails can’t guaranteedependency either!
  • Solutions?
  • Database Foreign Keys!create_table :comments do |table| table.integer :post_id, :null => falseendActiveRecord::Base.connection.execute <<-EOS ALTER TABLE `comments` ADD CONSTRAINT FOREIGN KEY (post_id) REFERENCES posts(id) ON DELETE restrict ON UPDATE cascadeEOS
  • Success!
  • 1 - Noob“Use database constraints”
  • O’taner
  • ruby 100% CPU!
  • Solutions?
  • ProfessionalDebuggingssh serverkillall ruby
  • Success!
  • WET Code
  • Infinite Loopclass Comment < ActiveRecord::Base after_create :update_comment_count def update_comment_count post = Post.find post_id post.comment_count += 1 post.save rescue StaleObjectError retry endend
  • WTF .. ?
  • TransactionIsolation Level “I” in ACID
  • 1 - READUNCOMMITTED2 - READCOMMITTED3 - REPEATABLEREAD
  • PostgreREAD-COMMITTED MySQLREPEATABLE-READ
  • mysql> begin;Query OK, 0 rows affected (0,00 sec)mysql> select * from posts where id = 1;+----+-------+--------------+| id | title | lock_version |+----+-------+--------------+| 1 | Title | 0|+----+-------+--------------+1 row in set (0,00 sec)mysql> update posts set title = New Title where lock_version = 0;Query OK, 0 rows affected (0,00 sec)Rows matched: 0 Changed: 0 Warnings: 0mysql> select * from posts where id = 1;+----+-------+--------------+| id | title | lock_version |+----+-------+--------------+| 1 | Title | 0|+----+-------+--------------+1 row in set (0,00 sec)
  • mysql> begin;Query OK, 0 rows affected (0,00 sec)mysql> select * from posts where id = 1;+----+-------+--------------+| id | title | lock_version |+----+-------+--------------+| 1 | Title | 0|+----+-------+--------------+1 row in set (0,00 sec)mysql> update posts set title = New Title where lock_version = 0;Query OK, 0 rows affected (0,00 sec)Rows matched: 0 Changed: 0 Warnings: 0mysql> select * from posts where id = 1;+----+-------+--------------+| id | title | lock_version |+----+-------+--------------+| 1 | Title | 0|+----+-------+--------------+1 row in set (0,00 sec)
  • mysql> begin;Query OK, 0 rows affected (0,00 sec)mysql> select * from posts where id = 1;+----+-------+--------------+| id | title | lock_version |+----+-------+--------------+| 1 | Title | 0|+----+-------+--------------+1 row in set (0,00 sec)mysql> update posts set title = New Title where lock_version = 0;Query OK, 0 rows affected (0,00 sec)Rows matched: 0 Changed: 0 Warnings: 0 // raises StaleObjectErrormysql> select * from posts where id = 1;+----+-------+--------------+| id | title | lock_version |+----+-------+--------------+| 1 | Title | 0|+----+-------+--------------+1 row in set (0,00 sec)
  • mysql> begin;Query OK, 0 rows affected (0,00 sec)mysql> select * from posts where id = 1;+----+-------+--------------+| id | title | lock_version |+----+-------+--------------+| 1 | Title | 0|+----+-------+--------------+1 row in set (0,00 sec)mysql> update posts set title = New Title where lock_version = 0;Query OK, 0 rows affected (0,00 sec)Rows matched: 0 Changed: 0 Warnings: 0 // raises StaleObjectErrormysql> select * from posts where id = 1;+----+-------+--------------+| id | title | lock_version |+----+-------+--------------+| 1 | Title | 0|+----+-------+--------------+1 row in set (0,00 sec)
  • Solutions?
  • READ- COMMITTED! /etc/my.cnf[mysqld]transaction-isolation = READ-COMMITTED Just like postgre ;)
  • mysql> begin;Query OK, 0 rows affected (0,00 sec)mysql> select * from posts where id = 1;+----+-------+--------------+| id | title | lock_version |+----+-------+--------------+| 1 | Title | 0|+----+-------+--------------+1 row in set (0,00 sec)mysql> update posts set title = New Title where lock_version = 0;Query OK, 0 rows affected (0,00 sec)Rows matched: 0 Changed: 0 Warnings: 0 // raises StaleObjectErrormysql> select * from posts where id = 1;+----+-----------+--------------+| id | title | lock_version |+----+-------+--------------+| 1 | Title | 1|+----+-------+--------------+1 row in set (0,00 sec)
  • Success!
  • 2 - Beginner“Use READ-COMMITTED”
  • O’taner
  • Production Console> u1 = User.first=> #<User id: 1, login: "marcos", options: {:active=>false}> u1.update_attributes :options => {:active => true}=> true> u1.reload=> #<User id: 1, login: "marcos", options: {:active=>false}
  • WTF .. ?
  • # login :string(255)# options :textclass User < ActiveRecord::Base serialize :options def activate self.options = {:active => true} self.save endend
  • No ‘:active => false’!
  • Why?
  • Dirty Checking> u1 = User.first=> #<User id: 1, login: "marcos", options: {:active=>false}> u2 = User.first=> #<User id: 1, login: "marcos", options: {:active=>false}> u1.update_attributes :options => {:active => true}=> true> u2.update_attributes :login => "mtoledo"=> true> u1.reload=> #<User id: 1, login: "mtoledo", options: {:active=>false}
  • Serialize isALWAYS dirty!
  • // u1.update_attributes :options => {:active => true}UPDATE `users`SET `options` = --- n:active: truenWHERE (`users`.`id` = 1)// u2.update_attributes :login => "mtoledo"UPDATE `users`SET `login` = mtoledo, `options` = --- n:active: falsenWHERE (`users`.`id` = 1)
  • Solutions?
  • Options Model
  • # id :integer(4)# user_id :integer(4)# lock_version :integer(4)# options :textclass UserOptions < ActiveRecord::Base serialize :options belongs_to :userend
  • Success!
  • 3 - Half Mouth“Use an Options Model”
  • O’taner
  • production.logERROR 1213 (40001): Deadlock found when trying to get lock;try restarting transaction
  • DAMN!Pessimistic Locks!
  • No pessimistic locking!class Comment after_create :update_posts_count def update_posts_count self.posts.increment :comments_count endendclass Posts after_update :update_comments def update_comments comments.each {|c| c.published = self.published; c.save} endend
  • WTF .. ?
  • Transac Transaction 1SELECT * FOR tion 2UPDATEFROM `comments`WHERE `id` = 1 SELECT * FOR UPDATE locked.. FROM `posts` WHERE `id` = 1SELECT * FORUPDATEFROM `posts` DEADLOCK!WHERE `id` = 1 SELECT * FOR UPDATE FROM `comments` WHERE `id` = 1
  • Why?
  • Transactions lock on update
  • Transac Transaction 1UPDATE tion 2`comments`SET `title` = “Foo”WHERE `id` = 1 UPDATE `posts` SET `title` = “bar” locked.. WHERE `id` = 1UPDATE `posts`SET `title` = “bar”WHERE `id` = 1 DEADLOCK! UPDATE `comments` SET `title` = “foo” WHERE `id` = 1
  • Callbacks =Transactions
  • Solutions?
  • Update dependentsbefore parents!
  • No longer deadlocks!class Comment after_create :update_posts_count def update_posts_count self.posts.increment :comments_count endendclass Posts before_update :update_comments def update_comments comments.each {|c| c.published = self.published; c.save} endend
  • Avoid StatePropagation
  • Avoids updating all commentsclass Comment < ActiveRecord::Base def visible? post.published? && self.published? endend
  • Success!
  • 4 - Advanced“Acquire locks consistently”
  • O’taner
  • “There is no safe way to use Thread#kill or Thread#raise.”http://blog.headius.com/2008/02/rubys-threadraise-threadkill-timeoutrb.html
  • Killing ruby (MRI) threads - bluepill - god - monit - you and me
  • Simple Transactionclass Order belongs_to :user after_create :mark_user_as_customer def mark_user_as_customer user.customer = true user.save endend
  • What happened to my transaction?> order = Order.first=> #<Order id: 1, user_id: 1>> order.user.customer?=> false> order.user?=> #<User id: 1, login: "marcos", customer: false>
  • WTF .. ?
  • If you kill your Ruby(MRI) Threads ..
  • ActiveRecord will partially commit your transactions!
  • Why?
  • ActiveRecord transaction (version for kids)def transaction yieldrescue Exception => e rollbackensure commit unless rolledback?end
  • Main thread is rescueddef print_if_rescued sleep 10rescue Exception puts rescuedensure puts ensuredendprint_if_rescued# $ killall ruby# => rescued# => ensured
  • Other Threads are NOT!def print_if_rescued sleep 10rescue Exception puts rescuedensure puts ensuredendt = Thread.new { print_if_rescued }t.join# $ killall ruby# => ensured
  • It commits!def transaction yield # thread killed inside yieldrescue Exception => e rollback # nothing is raisedensure commit unless rolledback?end
  • Solutions?
  • DON’T Kill Your Threads! Headius Approv
  • If you do, don’tuse ActiveRecord+ Threads + MRI
  • 5 Record Certified Enterprise Active - ARCECA “Don’t use Concurrency ArchitectThread#kill/raise ”
  • Success!
  • Recap please!
  • 1 - Noob“Use database constraints”
  • 2 - Beginner“Use READ-COMMITTED”
  • 3 - Half Mouth“Use an Options Model”
  • 4 - Advanced“Acquire locks consistently”
  • 5 Record Certified Enterprise Active - ARCECA “Don’t use Concurrency ArchitectThread#kill/raise ”
  • That’s all! Thank You!
  • Questions?Slides @mtoledo