Zero downtime deploys for Rails apps

18,871 views
19,528 views

Published on

Published in: Technology, Education
2 Comments
49 Likes
Statistics
Notes
No Downloads
Views
Total views
18,871
On SlideShare
0
From Embeds
0
Number of Embeds
20
Actions
Shares
0
Downloads
117
Comments
2
Likes
49
Embeds 0
No embeds

No notes for slide

Zero downtime deploys for Rails apps

  1. 1. zero deploysdowntime for Rails photo by Chris http://www.flickr.com/people/chrisinplymouth/
  2. 2. class ApparentlyHarmlessMigration < ActiveRecord::Migration  def self.up    remove_column :users, :notes  endend
  3. 3. class CompletelySafeMigration < ActiveRecord::Migration  def self.up    remove_column :users, :notes  end  def self.down    add_column :users, :notes, :text  endend
  4. 4. class CompletelySafeMigration < ActiveRecord::Migration  def self.up    remove_column :users, :notes  end  def self.down    add_column :users, :notes, :text  endend
  5. 5. PGError: ERROR: column "notes" does not exist
  6. 6. INSERT INTO users(email, password, ..., notes)
  7. 7. Zero downtime?Challenge accepted
  8. 8. Agenda• Introduction• Development practices• Production environment setup• Future• Conclusion
  9. 9. Testable code + TDD Test coverage Easy maintenance DRYZero downtime deploys Hot compatibility
  10. 10. Testable code + TDD Test coverage Easy maintenance DRYZero downtime deploys Hot compatibility
  11. 11. Testable code + TDD Test coverage Easy maintenance DRYZero downtime deploys Hot compatibility
  12. 12. Hot compatibility• Run concurrent versions of your application• That are compatible between each other
  13. 13. Hot compatibility v1 v2
  14. 14. Hot compatibility v1 v2writes to `notes` drop column `notes`
  15. 15. Hot compatibility v1 v1’ v2writes to `notes` stop writing to `notes` drop column `notes`
  16. 16. Hot compatibility v1 v1’ v2writes to `notes` stop writing to `notes` drop column `notes`
  17. 17. Hot compatibility v1 v1’ v2writes to `notes` stop writing to `notes` drop column `notes`
  18. 18. Hot compatibility v1 v1’ v2writes to `notes` stop writing to `notes` drop column `notes`
  19. 19. Hot compatibility v1 v1’ v2writes to `notes` stop writing to `notes` drop column `notes` class User   def self.columns     super.reject { |c| c.name == "notes" }   end end
  20. 20. Self dependency
  21. 21. Self dependencyuser loads form
  22. 22. Self dependencynew form is deployed
  23. 23. Self dependency user submits the form POST /login username=bill@microsoft.com&password=q1w2e3
  24. 24. Self dependency server blows up
  25. 25. Self dependency• Make controllers compatible:
  26. 26. Self dependency• Make controllers compatible: class AuthController < ApplicationController   protected   def filtered_params     params.dup.tap do |p|       p.merge!(:email => p.delete(:username))     end   end end
  27. 27. Assets - Rails 2 Server 1GET /dashboard200GET /stylesheets/all.js200
  28. 28. Assets - Rails 2 Server 1 Server 2GET /dashboard200GET /stylesheets/all.js404
  29. 29. Assets - Rails 2• https://github.com/ddollar/asset-resource
  30. 30. Asset pipeline - Rails 3 v1 GET /dashboard 200 GET /assets/application-{v1}.js 200
  31. 31. Asset pipeline - Rails 3 v1 v2 GET /dashboard 200 GET /assets/application-{v1}.js 404
  32. 32. Asset pipeline - Rails 3
  33. 33. Asset pipeline - Rails 3• When updating assets, keep the existing version around
  34. 34. Asset pipeline - Rails 3• When updating assets, keep the existing version around• We’re still in need of tooling to help cleaning up old assets
  35. 35. Asset pipeline - Rails 3• When updating assets, keep the existing version around• We’re still in need of tooling to help cleaning up old assets rake assets:clean:outdated
  36. 36. Migration strategies
  37. 37. Migration strategies• Read-only models
  38. 38. Migration strategies• Read-only models class Role < ActiveRecord::Base   def readonly?     true   end end
  39. 39. Migration strategies• Read-only models• Ignore cached columns
  40. 40. Migration strategies• Read-only models• Ignore cached columns class User   def self.columns     super.reject { |c| c.name == "notes" }   end end
  41. 41. Migration strategies• Read-only models• Ignore cached columns• Know your database
  42. 42. Migration strategies• Read-only models• Ignore cached columns• Know your database • Beware of O(n) db locks!
  43. 43. MyISAM
  44. 44. MyISAM
  45. 45. InnoDB
  46. 46. InnoDB• O(n) locks on: • Adding, updating or removing columns • Adding or removing indexes
  47. 47. InnoDB1. Create a new table with the same structure2. Apply changes to new table3. Copy data over4. Acquire lock on original table5. Sync new/updated/deleted rows6. Swap tables7. Release lock
  48. 48. InnoDB1. Create a new table with the same structure2. Apply changes to new table3. Copy data over4. Acquire lock on original table5. Sync new/updated/deleted rows6. Swap tables7. Release lock
  49. 49. InnoDBhttps://github.com/freels/table_migrator
  50. 50. InnoDBhttps://github.com/freels/table_migratorclass AddStuffToMyBigTable < TableMigration  migrates :users  change_table do |t|    t.integer :foo, :null => false, :default => 0    t.string :bar    t.index :foo, :name => index_for_foo  endend
  51. 51. InnoDBhttps://github.com/freels/table_migrator• Requires acts_as_paranoid• Pretty old/stale codebase • Check forks for Rails 3 compatibility
  52. 52. Postgres• Table locks on: • Adding, updating or removing columns • Adding or removing indexes
  53. 53. Postgres• O(n) locks on: • Adding, updating or removing columns • Adding or removing indexes
  54. 54. Postgres• O(n) locks on: constant time if you avoid default values and constraints • Adding, updating or removing columns • Adding or removing indexes
  55. 55. Postgres• O(n) locks on: constant time • Adding, updating or removing columns • Adding or removing indexes
  56. 56. Postgres• O(n) locks on: • Adding, updating or removing columns • Adding or removing indexes can be done without any locking
  57. 57. Postgres• O(n) locks on: • Adding, updating or removing columns • Adding or removing indexes constant time
  58. 58. Postgres• Adding fields with defaults
  59. 59. Postgres• Adding fields with defaults before_save :assign_defaults def assign_defaults   self.admin ||= false end
  60. 60. Postgres• Adding fields with defaults• Create indexes concurrently
  61. 61. Postgres • Adding fields with defaults • Create indexes concurrentlyCREATE INDEX CONCURRENTLY users_email_index ON users(email);
  62. 62. Postgres• Adding fields with defaults• Create indexes concurrently • Outside a transaction!
  63. 63. Postgres • Adding fields with defaults • Create indexes concurrently • Outside a transaction!class AddIndexToUsers < ActiveRecord::Migration  def self.up    return if indexes("users").map(&:name).include?("users_email_index")    add_index :users, :email, :name => "users_email_index"  end  def self.down    remove_index :users, :name => "users_email_index"  endend
  64. 64. What about renaming? v1 v2 reads and writes reads and writes to `foo` to `bar`
  65. 65. What about renaming? v1 v1’ v1’’ v2reads and writes add column `bar` populate `bar` reads and writes to `foo` + where needed to `bar` reads from `foo` writes to both `foo` and `bar`
  66. 66. What about renaming? v2 v2’ v3 reads and writes ignore `foo` drop column `foo` to `bar`
  67. 67. NoSQL
  68. 68. NoSQL• Schema migrations / O(n) locks are just part of hot compatibility
  69. 69. NoSQL• Schema migrations / O(n) locks are just part of hot compatibility• You still need to put effort into data migration
  70. 70. Workflow
  71. 71. Workflow• Make sure pull requests are addressing hot compatibility
  72. 72. Workflow• Make sure pull requests are addressing hot compatibility• Work after code is merged
  73. 73. Meanwhile, in production...$ thin -C config/server.yml restartStopping server on 0.0.0.0:5000 ...Sending QUIT signal to process 3463 ...Stopping server on 0.0.0.0:5001 ...Sending QUIT signal to process 3464 ...Stopping server on 0.0.0.0:5002 ...Sending QUIT signal to process 3465 ...Starting server on 0.0.0.0:5000 ...Starting server on 0.0.0.0:5001 ...Starting server on 0.0.0.0:5002 ...
  74. 74. Meanwhile, in production...$ thin -C config/server.yml restartStopping server on 0.0.0.0:5000 ...Sending QUIT signal to process 3463 ...Stopping server on 0.0.0.0:5001 ...Sending QUIT signal to process 3464 ...Stopping server on 0.0.0.0:5002 ...Sending QUIT signal to process 3465 ...Starting server on 0.0.0.0:5000 ...Starting server on 0.0.0.0:5001 ...Starting server on 0.0.0.0:5002 ...
  75. 75. Meanwhile, in production...$ thin -C config/server.yml restartStopping server on 0.0.0.0:5000 ...Sending QUIT signal to process 3463 ...Stopping server on 0.0.0.0:5001 ...Sending QUIT signal to process 3464 ...Stopping server on 0.0.0.0:5002 ...Sending QUIT signal to process 3465 ...Starting server on 0.0.0.0:5000 ...Starting server on 0.0.0.0:5001 ...Starting server on 0.0.0.0:5002 ...
  76. 76. Meanwhile, in production...$ thin -C config/server.yml restartStopping server on 0.0.0.0:5000 ...Sending QUIT signal to process 3463 ...Stopping server on 0.0.0.0:5001 ...Sending QUIT signal to process 3464 ...Stopping server on 0.0.0.0:5002 ...Sending QUIT signal to process 3465 ...Starting server on 0.0.0.0:5000 ...Starting server on 0.0.0.0:5001 ...Starting server on 0.0.0.0:5002 ...
  77. 77. Meanwhile, in production...$ thin -C config/server.yml restartStopping server on 0.0.0.0:5000 ...Sending QUIT signal to process 3463 ...Stopping server on 0.0.0.0:5001 ...Sending QUIT signal to process 3464 ...Stopping server on 0.0.0.0:5002 ...Sending QUIT signal to process 3465 ...Starting server on 0.0.0.0:5000 ...Starting server on 0.0.0.0:5001 ...Starting server on 0.0.0.0:5002 ...
  78. 78. Agenda• Introduction• Development practices• Production environment setup• Future• Conclusion
  79. 79. Production
  80. 80. Production• Run multiple server processes
  81. 81. Production• Run multiple server processes• Cycle them individually, or in batches
  82. 82. Production• Run multiple server processes• Cycle them individually, or in batches • Watch out for capacity!
  83. 83. Production• Run multiple server processes• Cycle them individually, or in batches • Watch out for capacity!• Cycle them gracefully
  84. 84. Cycling techniques• Above the stack• Within the stack• Platform
  85. 85. Above the stack• Use your favorite web server• Setup a load balancer above them, route around servers that are cycling
  86. 86. Above the stack• PROTIP: default setup won’t work• Configure load balancer to poll for your application health: class HealthController < ApplicationController   def index     if File.exists?(Rails.root.join("MAINTENANCE"))       render :text => "Forget me!", :status => 503     else       render :text => "Im alive!"     end   end end
  87. 87. Cycling + Load balancer Server Load balancer touch MAINTENANCE bounce servers remove server from rotation rm MAINTENANCEnew servers come online add server to rotation
  88. 88. HAProxyoption httpchk GET /healthlisten myapp :9000 server myapp-1 1.0.0.1:5000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 1.0.0.2:5000 maxconn 1 check inter 2000 fall 1 rise 1
  89. 89. HAProxy tell HAProxy where to check for the healthoption httpchk GET /healthlisten myapp :9000 server myapp-1 1.0.0.1:5000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 1.0.0.2:5000 maxconn 1 check inter 2000 fall 1 rise 1
  90. 90. HAProxyoption httpchk GET /healthlisten myapp :9000 server myapp-1 1.0.0.1:5000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 1.0.0.2:5000 maxconn 1 check inter 2000 fall 1 rise 1 do not let your web server queue requests
  91. 91. HAProxyoption httpchk GET /healthlisten myapp :9000 server myapp-1 1.0.0.1:5000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 1.0.0.2:5000 maxconn 1 check inter 2000 fall 1 rise 1 tell HAProxy to run the health check specified above
  92. 92. HAProxyoption httpchk GET /healthlisten myapp :9000 server myapp-1 1.0.0.1:5000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 1.0.0.2:5000 maxconn 1 check inter 2000 fall 1 rise 1 every 2s
  93. 93. HAProxyoption httpchk GET /healthlisten myapp :9000 server myapp-1 1.0.0.1:5000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 1.0.0.2:5000 maxconn 1 check inter 2000 fall 1 rise 1 stop routing to it once the health check failed once
  94. 94. HAProxyoption httpchk GET /healthlisten myapp :9000 server myapp-1 1.0.0.1:5000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 1.0.0.2:5000 maxconn 1 check inter 2000 fall 1 rise 1 and bring it back once it worked once
  95. 95. ELB$ elb-configure-healthcheck MyLoadBalancer --target "HTTP:5000/health" --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
  96. 96. Above the stack
  97. 97. Above the stack• Minimal impact
  98. 98. Above the stack• Minimal impact• Extended availability comes for free
  99. 99. Above the stack• Minimal impact • Complicated setup• Extended availability comes for free
  100. 100. Above the stack• Minimal impact • Complicated setup• Extended availability • Another component you comes for free have to track and maintain
  101. 101. Within the stack
  102. 102. Within the stack
  103. 103. Unicorn• Replaces your web server• Bounces gracefully
  104. 104. Unicorn63480 0.0 1.4 2549796 unicorn_rails master -c config/unicorn.rb63489 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63490 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63491 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb
  105. 105. Unicorn$ kill -USR2 6348063480 0.0 1.4 2549796 unicorn_rails master -c config/unicorn.rb63489 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63490 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63491 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb
  106. 106. Unicorn$ kill -USR2 6348063480 0.0 1.4 2549796 unicorn_rails master (old) -c config/unicorn.rb63489 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63490 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63491 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb63521 0.0 1.4 2549796 unicorn_rails master -c config/unicorn.rb63535 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63536 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63537 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb
  107. 107. Unicorn63480 0.0 1.4 2549796 unicorn_rails master (old) -c config/unicorn.rb63489 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63490 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63491 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb63521 0.0 1.4 2549796 unicorn_rails master -c config/unicorn.rb63535 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63536 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63537 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb
  108. 108. Unicorn$ kill -QUIT 6348063480 0.0 1.4 2549796 unicorn_rails master (old) -c config/unicorn.rb63489 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63490 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63491 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb63521 0.0 1.4 2549796 unicorn_rails master -c config/unicorn.rb63535 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63536 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63537 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb
  109. 109. Unicorn$ kill -QUIT 6348063521 0.0 1.4 2549796 unicorn_rails master -c config/unicorn.rb63535 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63536 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63537 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb
  110. 110. Unicorn63521 0.0 1.4 2549796 unicorn_rails master -c config/unicorn.rb63535 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63536 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63537 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb
  111. 111. Within the stack
  112. 112. Within the stack• Easy to setup and deploy
  113. 113. Within the stack• Easy to setup and deploy• Optimal performance to cycle the server
  114. 114. Within the stack• Easy to setup and deploy • Changes your application stack• Optimal performance to cycle the server
  115. 115. Within the stack• Easy to setup and deploy • Changes your application stack• Optimal performance to cycle the server • Doesn’t address availability issues
  116. 116. Platform
  117. 117. HerokuProcess State Command------- --------------- -------------------------------web.1 up for 10h bundle exec thin start -p $PORT
  118. 118. Heroku$ git push heroku masterProcess State Command------- --------------- -------------------------------web.1 up for 10h bundle exec thin start -p $PORT
  119. 119. Heroku$ git push heroku masterProcess State Command------- --------------- -------------------------------web.1 starting for 1s bundle exec thin start -p $PORT
  120. 120. HerokuProcess State Command------- --------------- -------------------------------web.1 starting for 5s bundle exec thin start -p $PORT
  121. 121. HerokuProcess State Command------- --------------- -------------------------------web.1 up for 1s bundle exec thin start -p $PORT
  122. 122. Heroku BETAProcess State Command------- --------------- -------------------------------web.1 up for 10h bundle exec thin start -p $PORT
  123. 123. Heroku BETA$ heroku labs:enable prebootProcess State Command------- --------------- -------------------------------web.1 up for 10h bundle exec thin start -p $PORT
  124. 124. Heroku BETAProcess State Command------- --------------- -------------------------------web.1 up for 10h bundle exec thin start -p $PORT
  125. 125. Heroku BETA$ git push heroku masterProcess State Command------- --------------- -------------------------------web.1 up for 10h bundle exec thin start -p $PORT
  126. 126. Heroku BETA$ git push heroku masterProcess State Command------- --------------- -------------------------------web.1 up for 10h bundle exec thin start -p $PORTweb.1 starting for 5s bundle exec thin start -p $PORT
  127. 127. Heroku BETAProcess State Command------- --------------- -------------------------------web.1 up for 10h bundle exec thin start -p $PORTweb.1 up for 1s bundle exec thin start -p $PORT
  128. 128. Heroku BETAProcess State Command------- --------------- -------------------------------web.1 up for 5s bundle exec thin start -p $PORT
  129. 129. Heroku• Migrations
  130. 130. Heroku• Migrations $ git push pre-production master $ heroku run rake db:migrate --remote pre-production $ git push production master
  131. 131. Agenda• Introduction• Development practices• Production environment setup• Future• Conclusion
  132. 132. Future
  133. 133. Future• Asset pipeline
  134. 134. Future• Asset pipeline• Mysql tooling
  135. 135. Future• Asset pipeline• Mysql tooling• Can Rails help making the development of hot compatible applications easier?
  136. 136. Future• Asset pipeline• Mysql tooling• Can Rails help making the development of hot compatible applications easier?• Can databases?
  137. 137. Future
  138. 138. Future• Hot compatibility beyond zero downtime deploys
  139. 139. Conclusion
  140. 140. Conclusion• Zero downtime is a pain in the ass
  141. 141. Conclusion• Zero downtime is a pain in the ass• Hot compatibility to the rescue
  142. 142. Conclusion• Zero downtime is a pain in the ass• Hot compatibility to the rescue• Make sure your production env is ready
  143. 143. Thank you!http://pedro.heroku.com @ped

×