Successfully reported this slideshow.

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Zero downtime deploys for Rails apps

  1. 1. zero deploys downtime for Rails photo by Chris http://www.flickr.com/people/chrisinplymouth/
  2. 2. class ApparentlyHarmlessMigration < ActiveRecord::Migration   def self.up     remove_column :users, :notes   end end
  3. 3. class CompletelySafeMigration < ActiveRecord::Migration   def self.up     remove_column :users, :notes   end   def self.down     add_column :users, :notes, :text   end end
  4. 4. class CompletelySafeMigration < ActiveRecord::Migration   def self.up     remove_column :users, :notes   end   def self.down     add_column :users, :notes, :text   end end
  5. 5. PGError: ERROR: column "notes" does not exist
  6. 6. INSERT INTO users(email, password, ..., notes)
  7. 7. Zero downtime? Challenge accepted
  8. 8. Agenda • Introduction • Development practices • Production environment setup • Future • Conclusion
  9. 9. Testable code + TDD Test coverage Easy maintenance DRY Zero downtime deploys Hot compatibility
  10. 10. Testable code + TDD Test coverage Easy maintenance DRY Zero downtime deploys Hot compatibility
  11. 11. Testable code + TDD Test coverage Easy maintenance DRY Zero downtime deploys Hot compatibility
  12. 12. Hot compatibility • Run concurrent versions of your application • That are compatible between each other
  13. 13. Hot compatibility v1 v2
  14. 14. Hot compatibility v1 v2 writes to `notes` drop column `notes`
  15. 15. Hot compatibility v1 v1’ v2 writes to `notes` stop writing to `notes` drop column `notes`
  16. 16. Hot compatibility v1 v1’ v2 writes to `notes` stop writing to `notes` drop column `notes`
  17. 17. Hot compatibility v1 v1’ v2 writes to `notes` stop writing to `notes` drop column `notes`
  18. 18. Hot compatibility v1 v1’ v2 writes to `notes` stop writing to `notes` drop column `notes`
  19. 19. Hot compatibility v1 v1’ v2 writes to `notes` stop writing to `notes` drop column `notes` class User   def self.columns     super.reject { |c| c.name == "notes" }   end end
  20. 20. Self dependency
  21. 21. Self dependency user loads form
  22. 22. Self dependency new form is deployed
  23. 23. Self dependency user submits the form POST /login username=bill@microsoft.com&password=q1w2e3
  24. 24. Self dependency server blows up
  25. 25. Self dependency • Make controllers compatible:
  26. 26. Self dependency • Make controllers compatible: class AuthController < ApplicationController   protected   def filtered_params     params.dup.tap do |p|       p.merge!(:email => p.delete(:username))     end   end end
  27. 27. Assets - Rails 2 Server 1 GET /dashboard 200 GET /stylesheets/all.js 200
  28. 28. Assets - Rails 2 Server 1 Server 2 GET /dashboard 200 GET /stylesheets/all.js 404
  29. 29. Assets - Rails 2 • https://github.com/ddollar/asset-resource
  30. 30. Asset pipeline - Rails 3 v1 GET /dashboard 200 GET /assets/application-{v1}.js 200
  31. 31. Asset pipeline - Rails 3 v1 v2 GET /dashboard 200 GET /assets/application-{v1}.js 404
  32. 32. Asset pipeline - Rails 3
  33. 33. Asset pipeline - Rails 3 • When updating assets, keep the existing version around
  34. 34. Asset pipeline - Rails 3 • When updating assets, keep the existing version around • We’re still in need of tooling to help cleaning up old assets
  35. 35. Asset pipeline - Rails 3 • When updating assets, keep the existing version around • We’re still in need of tooling to help cleaning up old assets rake assets:clean:outdated
  36. 36. Migration strategies
  37. 37. Migration strategies • Read-only models
  38. 38. Migration strategies • Read-only models class Role < ActiveRecord::Base   def readonly?     true   end end
  39. 39. Migration strategies • Read-only models • Ignore cached columns
  40. 40. Migration strategies • Read-only models • Ignore cached columns class User   def self.columns     super.reject { |c| c.name == "notes" }   end end
  41. 41. Migration strategies • Read-only models • Ignore cached columns • Know your database
  42. 42. Migration strategies • Read-only models • Ignore cached columns • Know your database • Beware of O(n) db locks!
  43. 43. MyISAM
  44. 44. MyISAM
  45. 45. InnoDB
  46. 46. InnoDB • O(n) locks on: • Adding, updating or removing columns • Adding or removing indexes
  47. 47. InnoDB 1. Create a new table with the same structure 2. Apply changes to new table 3. Copy data over 4. Acquire lock on original table 5. Sync new/updated/deleted rows 6. Swap tables 7. Release lock
  48. 48. InnoDB 1. Create a new table with the same structure 2. Apply changes to new table 3. Copy data over 4. Acquire lock on original table 5. Sync new/updated/deleted rows 6. Swap tables 7. Release lock
  49. 49. InnoDB https://github.com/freels/table_migrator
  50. 50. InnoDB https://github.com/freels/table_migrator class AddStuffToMyBigTable < TableMigration   migrates :users   change_table do |t|     t.integer :foo, :null => false, :default => 0     t.string :bar     t.index :foo, :name => 'index_for_foo'   end end
  51. 51. InnoDB https://github.com/freels/table_migrator • Requires acts_as_paranoid • Pretty old/stale codebase • Check forks for Rails 3 compatibility
  52. 52. Postgres • Table locks on: • Adding, updating or removing columns • Adding or removing indexes
  53. 53. Postgres • O(n) locks on: • Adding, updating or removing columns • Adding or removing indexes
  54. 54. Postgres • O(n) locks on: constant time if you avoid default values and constraints • Adding, updating or removing columns • Adding or removing indexes
  55. 55. Postgres • O(n) locks on: constant time • Adding, updating or removing columns • Adding or removing indexes
  56. 56. Postgres • O(n) locks on: • Adding, updating or removing columns • Adding or removing indexes can be done without any locking
  57. 57. Postgres • O(n) locks on: • Adding, updating or removing columns • Adding or removing indexes constant time
  58. 58. Postgres • Adding fields with defaults
  59. 59. Postgres • Adding fields with defaults before_save :assign_defaults def assign_defaults   self.admin ||= false end
  60. 60. Postgres • Adding fields with defaults • Create indexes concurrently
  61. 61. Postgres • Adding fields with defaults • Create indexes concurrently CREATE INDEX CONCURRENTLY users_email_index ON users(email);
  62. 62. Postgres • Adding fields with defaults • Create indexes concurrently • Outside a transaction!
  63. 63. Postgres • Adding fields with defaults • Create indexes concurrently • Outside a transaction! class AddIndexToUsers < ActiveRecord::Migration   def self.up     return if indexes("users").map(&:name).include?("users_email_index")     add_index :users, :email, :name => "users_email_index"   end   def self.down     remove_index :users, :name => "users_email_index"   end end
  64. 64. What about renaming? v1 v2 reads and writes reads and writes to `foo` to `bar`
  65. 65. What about renaming? v1 v1’ v1’’ v2 reads and writes add column `bar` populate `bar` reads and writes to `foo` + where needed to `bar` reads from `foo` writes to both `foo` and `bar`
  66. 66. What about renaming? v2 v2’ v3 reads and writes ignore `foo` drop column `foo` to `bar`
  67. 67. NoSQL
  68. 68. NoSQL • Schema migrations / O(n) locks are just part of hot compatibility
  69. 69. NoSQL • Schema migrations / O(n) locks are just part of hot compatibility • You still need to put effort into data migration
  70. 70. Workflow
  71. 71. Workflow • Make sure pull requests are addressing hot compatibility
  72. 72. Workflow • Make sure pull requests are addressing hot compatibility • Work after code is merged
  73. 73. Meanwhile, in production... $ thin -C config/server.yml restart Stopping server on 0.0.0.0:5000 ... Sending QUIT signal to process 3463 ... Stopping server on 0.0.0.0:5001 ... Sending QUIT signal to process 3464 ... Stopping server on 0.0.0.0:5002 ... Sending QUIT signal to process 3465 ... Starting server on 0.0.0.0:5000 ... Starting server on 0.0.0.0:5001 ... Starting server on 0.0.0.0:5002 ...
  74. 74. Meanwhile, in production... $ thin -C config/server.yml restart Stopping server on 0.0.0.0:5000 ... Sending QUIT signal to process 3463 ... Stopping server on 0.0.0.0:5001 ... Sending QUIT signal to process 3464 ... Stopping server on 0.0.0.0:5002 ... Sending QUIT signal to process 3465 ... Starting server on 0.0.0.0:5000 ... Starting server on 0.0.0.0:5001 ... Starting server on 0.0.0.0:5002 ...
  75. 75. Meanwhile, in production... $ thin -C config/server.yml restart Stopping server on 0.0.0.0:5000 ... Sending QUIT signal to process 3463 ... Stopping server on 0.0.0.0:5001 ... Sending QUIT signal to process 3464 ... Stopping server on 0.0.0.0:5002 ... Sending QUIT signal to process 3465 ... Starting server on 0.0.0.0:5000 ... Starting server on 0.0.0.0:5001 ... Starting server on 0.0.0.0:5002 ...
  76. 76. Meanwhile, in production... $ thin -C config/server.yml restart Stopping server on 0.0.0.0:5000 ... Sending QUIT signal to process 3463 ... Stopping server on 0.0.0.0:5001 ... Sending QUIT signal to process 3464 ... Stopping server on 0.0.0.0:5002 ... Sending QUIT signal to process 3465 ... Starting server on 0.0.0.0:5000 ... Starting server on 0.0.0.0:5001 ... Starting server on 0.0.0.0:5002 ...
  77. 77. Meanwhile, in production... $ thin -C config/server.yml restart Stopping server on 0.0.0.0:5000 ... Sending QUIT signal to process 3463 ... Stopping server on 0.0.0.0:5001 ... Sending QUIT signal to process 3464 ... Stopping server on 0.0.0.0:5002 ... Sending QUIT signal to process 3465 ... Starting server on 0.0.0.0:5000 ... Starting server on 0.0.0.0:5001 ... Starting server on 0.0.0.0:5002 ...
  78. 78. Agenda • Introduction • Development practices • Production environment setup • Future • Conclusion
  79. 79. Production
  80. 80. Production • Run multiple server processes
  81. 81. Production • Run multiple server processes • Cycle them individually, or in batches
  82. 82. Production • Run multiple server processes • Cycle them individually, or in batches • Watch out for capacity!
  83. 83. Production • Run multiple server processes • Cycle them individually, or in batches • Watch out for capacity! • Cycle them gracefully
  84. 84. Cycling techniques • Above the stack • Within the stack • Platform
  85. 85. Above the stack • Use your favorite web server • Setup a load balancer above them, route around servers that are cycling
  86. 86. Above the stack • PROTIP: default setup won’t work • Configure load balancer to poll for your application health: class HealthController < ApplicationController   def index     if File.exists?(Rails.root.join("MAINTENANCE"))       render :text => "Forget me!", :status => 503     else       render :text => "I'm alive!"     end   end end
  87. 87. Cycling + Load balancer Server Load balancer touch MAINTENANCE bounce servers remove server from rotation rm MAINTENANCE new servers come online add server to rotation
  88. 88. HAProxy option httpchk GET /health listen myapp :9000 server myapp-1 1.0.0.1:5000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 1.0.0.2:5000 maxconn 1 check inter 2000 fall 1 rise 1
  89. 89. HAProxy tell HAProxy where to check for the health option httpchk GET /health listen myapp :9000 server myapp-1 1.0.0.1:5000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 1.0.0.2:5000 maxconn 1 check inter 2000 fall 1 rise 1
  90. 90. HAProxy option httpchk GET /health listen myapp :9000 server myapp-1 1.0.0.1:5000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 1.0.0.2:5000 maxconn 1 check inter 2000 fall 1 rise 1 do not let your web server queue requests
  91. 91. HAProxy option httpchk GET /health listen myapp :9000 server myapp-1 1.0.0.1:5000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 1.0.0.2:5000 maxconn 1 check inter 2000 fall 1 rise 1 tell HAProxy to run the health check specified above
  92. 92. HAProxy option httpchk GET /health listen myapp :9000 server myapp-1 1.0.0.1:5000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 1.0.0.2:5000 maxconn 1 check inter 2000 fall 1 rise 1 every 2s
  93. 93. HAProxy option httpchk GET /health listen myapp :9000 server myapp-1 1.0.0.1:5000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 1.0.0.2:5000 maxconn 1 check inter 2000 fall 1 rise 1 stop routing to it once the health check failed once
  94. 94. HAProxy option httpchk GET /health listen myapp :9000 server myapp-1 1.0.0.1:5000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 1.0.0.2:5000 maxconn 1 check inter 2000 fall 1 rise 1 and bring it back once it worked once
  95. 95. ELB $ elb-configure-healthcheck MyLoadBalancer --target "HTTP:5000/health" --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
  96. 96. Above the stack
  97. 97. Above the stack • Minimal impact
  98. 98. Above the stack • Minimal impact • Extended availability comes for free
  99. 99. Above the stack • Minimal impact • Complicated setup • Extended availability comes for free
  100. 100. Above the stack • Minimal impact • Complicated setup • Extended availability • Another component you comes for free have to track and maintain
  101. 101. Within the stack
  102. 102. Within the stack
  103. 103. Unicorn • Replaces your web server • Bounces gracefully
  104. 104. Unicorn 63480 0.0 1.4 2549796 unicorn_rails master -c config/unicorn.rb 63489 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb 63490 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb 63491 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb
  105. 105. Unicorn $ kill -USR2 63480 63480 0.0 1.4 2549796 unicorn_rails master -c config/unicorn.rb 63489 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb 63490 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb 63491 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb
  106. 106. Unicorn $ kill -USR2 63480 63480 0.0 1.4 2549796 unicorn_rails master (old) -c config/unicorn.rb 63489 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb 63490 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb 63491 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb 63521 0.0 1.4 2549796 unicorn_rails master -c config/unicorn.rb 63535 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb 63536 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb 63537 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb
  107. 107. Unicorn 63480 0.0 1.4 2549796 unicorn_rails master (old) -c config/unicorn.rb 63489 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb 63490 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb 63491 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb 63521 0.0 1.4 2549796 unicorn_rails master -c config/unicorn.rb 63535 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb 63536 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb 63537 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb
  108. 108. Unicorn $ kill -QUIT 63480 63480 0.0 1.4 2549796 unicorn_rails master (old) -c config/unicorn.rb 63489 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb 63490 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb 63491 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb 63521 0.0 1.4 2549796 unicorn_rails master -c config/unicorn.rb 63535 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb 63536 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb 63537 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb
  109. 109. Unicorn $ kill -QUIT 63480 63521 0.0 1.4 2549796 unicorn_rails master -c config/unicorn.rb 63535 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb 63536 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb 63537 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb
  110. 110. Unicorn 63521 0.0 1.4 2549796 unicorn_rails master -c config/unicorn.rb 63535 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb 63536 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb 63537 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb
  111. 111. Within the stack
  112. 112. Within the stack • Easy to setup and deploy
  113. 113. Within the stack • Easy to setup and deploy • Optimal performance to cycle the server
  114. 114. Within the stack • Easy to setup and deploy • Changes your application stack • Optimal performance to cycle the server
  115. 115. Within the stack • Easy to setup and deploy • Changes your application stack • Optimal performance to cycle the server • Doesn’t address availability issues
  116. 116. Platform
  117. 117. Heroku Process State Command ------- --------------- ------------------------------- web.1 up for 10h bundle exec thin start -p $PORT
  118. 118. Heroku $ git push heroku master Process State Command ------- --------------- ------------------------------- web.1 up for 10h bundle exec thin start -p $PORT
  119. 119. Heroku $ git push heroku master Process State Command ------- --------------- ------------------------------- web.1 starting for 1s bundle exec thin start -p $PORT
  120. 120. Heroku Process State Command ------- --------------- ------------------------------- web.1 starting for 5s bundle exec thin start -p $PORT
  121. 121. Heroku Process State Command ------- --------------- ------------------------------- web.1 up for 1s bundle exec thin start -p $PORT
  122. 122. Heroku BETA Process State Command ------- --------------- ------------------------------- web.1 up for 10h bundle exec thin start -p $PORT
  123. 123. Heroku BETA $ heroku labs:enable preboot Process State Command ------- --------------- ------------------------------- web.1 up for 10h bundle exec thin start -p $PORT
  124. 124. Heroku BETA Process State Command ------- --------------- ------------------------------- web.1 up for 10h bundle exec thin start -p $PORT
  125. 125. Heroku BETA $ git push heroku master Process State Command ------- --------------- ------------------------------- web.1 up for 10h bundle exec thin start -p $PORT
  126. 126. Heroku BETA $ git push heroku master Process State Command ------- --------------- ------------------------------- web.1 up for 10h bundle exec thin start -p $PORT web.1 starting for 5s bundle exec thin start -p $PORT
  127. 127. Heroku BETA Process State Command ------- --------------- ------------------------------- web.1 up for 10h bundle exec thin start -p $PORT web.1 up for 1s bundle exec thin start -p $PORT
  128. 128. Heroku BETA Process State Command ------- --------------- ------------------------------- web.1 up for 5s bundle exec thin start -p $PORT
  129. 129. Heroku • Migrations
  130. 130. Heroku • Migrations $ git push pre-production master $ heroku run rake db:migrate --remote pre-production $ git push production master
  131. 131. Agenda • Introduction • Development practices • Production environment setup • Future • Conclusion
  132. 132. Future
  133. 133. Future • Asset pipeline
  134. 134. Future • Asset pipeline • Mysql tooling
  135. 135. Future • Asset pipeline • Mysql tooling • Can Rails help making the development of hot compatible applications easier?
  136. 136. Future • Asset pipeline • Mysql tooling • Can Rails help making the development of hot compatible applications easier? • Can databases?
  137. 137. Future
  138. 138. Future • Hot compatibility beyond zero downtime deploys
  139. 139. Conclusion
  140. 140. Conclusion • Zero downtime is a pain in the ass
  141. 141. Conclusion • Zero downtime is a pain in the ass • Hot compatibility to the rescue
  142. 142. Conclusion • Zero downtime is a pain in the ass • Hot compatibility to the rescue • Make sure your production env is ready
  143. 143. Thank you! http://pedro.heroku.com @ped

×