Zero downtime deploys for Rails apps

  • 11,736 views
Uploaded on

 

More in: Technology , Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • Very nice and useful. Thanks.
    Are you sure you want to
    Your message goes here
  • cools
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
11,736
On Slideshare
0
From Embeds
0
Number of Embeds
15

Actions

Shares
Downloads
99
Comments
2
Likes
45

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. zero deploysdowntime for Rails photo by Chris http://www.flickr.com/people/chrisinplymouth/
  • 2. class ApparentlyHarmlessMigration < ActiveRecord::Migration  def self.up    remove_column :users, :notes  endend
  • 3. class CompletelySafeMigration < ActiveRecord::Migration  def self.up    remove_column :users, :notes  end  def self.down    add_column :users, :notes, :text  endend
  • 4. class CompletelySafeMigration < ActiveRecord::Migration  def self.up    remove_column :users, :notes  end  def self.down    add_column :users, :notes, :text  endend
  • 5. PGError: ERROR: column "notes" does not exist
  • 6. INSERT INTO users(email, password, ..., notes)
  • 7. Zero downtime?Challenge accepted
  • 8. Agenda• Introduction• Development practices• Production environment setup• Future• Conclusion
  • 9. Testable code + TDD Test coverage Easy maintenance DRYZero downtime deploys Hot compatibility
  • 10. Testable code + TDD Test coverage Easy maintenance DRYZero downtime deploys Hot compatibility
  • 11. Testable code + TDD Test coverage Easy maintenance DRYZero downtime deploys Hot compatibility
  • 12. Hot compatibility• Run concurrent versions of your application• That are compatible between each other
  • 13. Hot compatibility v1 v2
  • 14. Hot compatibility v1 v2writes to `notes` drop column `notes`
  • 15. Hot compatibility v1 v1’ v2writes to `notes` stop writing to `notes` drop column `notes`
  • 16. Hot compatibility v1 v1’ v2writes to `notes` stop writing to `notes` drop column `notes`
  • 17. Hot compatibility v1 v1’ v2writes to `notes` stop writing to `notes` drop column `notes`
  • 18. Hot compatibility v1 v1’ v2writes to `notes` stop writing to `notes` drop column `notes`
  • 19. Hot compatibility v1 v1’ v2writes to `notes` stop writing to `notes` drop column `notes` class User   def self.columns     super.reject { |c| c.name == "notes" }   end end
  • 20. Self dependency
  • 21. Self dependencyuser loads form
  • 22. Self dependencynew form is deployed
  • 23. Self dependency user submits the form POST /login username=bill@microsoft.com&password=q1w2e3
  • 24. Self dependency server blows up
  • 25. Self dependency• Make controllers compatible:
  • 26. Self dependency• Make controllers compatible: class AuthController < ApplicationController   protected   def filtered_params     params.dup.tap do |p|       p.merge!(:email => p.delete(:username))     end   end end
  • 27. Assets - Rails 2 Server 1GET /dashboard200GET /stylesheets/all.js200
  • 28. Assets - Rails 2 Server 1 Server 2GET /dashboard200GET /stylesheets/all.js404
  • 29. Assets - Rails 2• https://github.com/ddollar/asset-resource
  • 30. Asset pipeline - Rails 3 v1 GET /dashboard 200 GET /assets/application-{v1}.js 200
  • 31. Asset pipeline - Rails 3 v1 v2 GET /dashboard 200 GET /assets/application-{v1}.js 404
  • 32. Asset pipeline - Rails 3
  • 33. Asset pipeline - Rails 3• When updating assets, keep the existing version around
  • 34. Asset pipeline - Rails 3• When updating assets, keep the existing version around• We’re still in need of tooling to help cleaning up old assets
  • 35. Asset pipeline - Rails 3• When updating assets, keep the existing version around• We’re still in need of tooling to help cleaning up old assets rake assets:clean:outdated
  • 36. Migration strategies
  • 37. Migration strategies• Read-only models
  • 38. Migration strategies• Read-only models class Role < ActiveRecord::Base   def readonly?     true   end end
  • 39. Migration strategies• Read-only models• Ignore cached columns
  • 40. Migration strategies• Read-only models• Ignore cached columns class User   def self.columns     super.reject { |c| c.name == "notes" }   end end
  • 41. Migration strategies• Read-only models• Ignore cached columns• Know your database
  • 42. Migration strategies• Read-only models• Ignore cached columns• Know your database • Beware of O(n) db locks!
  • 43. MyISAM
  • 44. MyISAM
  • 45. InnoDB
  • 46. InnoDB• O(n) locks on: • Adding, updating or removing columns • Adding or removing indexes
  • 47. InnoDB1. Create a new table with the same structure2. Apply changes to new table3. Copy data over4. Acquire lock on original table5. Sync new/updated/deleted rows6. Swap tables7. Release lock
  • 48. InnoDB1. Create a new table with the same structure2. Apply changes to new table3. Copy data over4. Acquire lock on original table5. Sync new/updated/deleted rows6. Swap tables7. Release lock
  • 49. InnoDBhttps://github.com/freels/table_migrator
  • 50. InnoDBhttps://github.com/freels/table_migratorclass AddStuffToMyBigTable < TableMigration  migrates :users  change_table do |t|    t.integer :foo, :null => false, :default => 0    t.string :bar    t.index :foo, :name => index_for_foo  endend
  • 51. InnoDBhttps://github.com/freels/table_migrator• Requires acts_as_paranoid• Pretty old/stale codebase • Check forks for Rails 3 compatibility
  • 52. Postgres• Table locks on: • Adding, updating or removing columns • Adding or removing indexes
  • 53. Postgres• O(n) locks on: • Adding, updating or removing columns • Adding or removing indexes
  • 54. Postgres• O(n) locks on: constant time if you avoid default values and constraints • Adding, updating or removing columns • Adding or removing indexes
  • 55. Postgres• O(n) locks on: constant time • Adding, updating or removing columns • Adding or removing indexes
  • 56. Postgres• O(n) locks on: • Adding, updating or removing columns • Adding or removing indexes can be done without any locking
  • 57. Postgres• O(n) locks on: • Adding, updating or removing columns • Adding or removing indexes constant time
  • 58. Postgres• Adding fields with defaults
  • 59. Postgres• Adding fields with defaults before_save :assign_defaults def assign_defaults   self.admin ||= false end
  • 60. Postgres• Adding fields with defaults• Create indexes concurrently
  • 61. Postgres • Adding fields with defaults • Create indexes concurrentlyCREATE INDEX CONCURRENTLY users_email_index ON users(email);
  • 62. Postgres• Adding fields with defaults• Create indexes concurrently • Outside a transaction!
  • 63. Postgres • Adding fields with defaults • Create indexes concurrently • Outside a transaction!class AddIndexToUsers < ActiveRecord::Migration  def self.up    return if indexes("users").map(&:name).include?("users_email_index")    add_index :users, :email, :name => "users_email_index"  end  def self.down    remove_index :users, :name => "users_email_index"  endend
  • 64. What about renaming? v1 v2 reads and writes reads and writes to `foo` to `bar`
  • 65. What about renaming? v1 v1’ v1’’ v2reads and writes add column `bar` populate `bar` reads and writes to `foo` + where needed to `bar` reads from `foo` writes to both `foo` and `bar`
  • 66. What about renaming? v2 v2’ v3 reads and writes ignore `foo` drop column `foo` to `bar`
  • 67. NoSQL
  • 68. NoSQL• Schema migrations / O(n) locks are just part of hot compatibility
  • 69. NoSQL• Schema migrations / O(n) locks are just part of hot compatibility• You still need to put effort into data migration
  • 70. Workflow
  • 71. Workflow• Make sure pull requests are addressing hot compatibility
  • 72. Workflow• Make sure pull requests are addressing hot compatibility• Work after code is merged
  • 73. Meanwhile, in production...$ thin -C config/server.yml restartStopping server on 0.0.0.0:5000 ...Sending QUIT signal to process 3463 ...Stopping server on 0.0.0.0:5001 ...Sending QUIT signal to process 3464 ...Stopping server on 0.0.0.0:5002 ...Sending QUIT signal to process 3465 ...Starting server on 0.0.0.0:5000 ...Starting server on 0.0.0.0:5001 ...Starting server on 0.0.0.0:5002 ...
  • 74. Meanwhile, in production...$ thin -C config/server.yml restartStopping server on 0.0.0.0:5000 ...Sending QUIT signal to process 3463 ...Stopping server on 0.0.0.0:5001 ...Sending QUIT signal to process 3464 ...Stopping server on 0.0.0.0:5002 ...Sending QUIT signal to process 3465 ...Starting server on 0.0.0.0:5000 ...Starting server on 0.0.0.0:5001 ...Starting server on 0.0.0.0:5002 ...
  • 75. Meanwhile, in production...$ thin -C config/server.yml restartStopping server on 0.0.0.0:5000 ...Sending QUIT signal to process 3463 ...Stopping server on 0.0.0.0:5001 ...Sending QUIT signal to process 3464 ...Stopping server on 0.0.0.0:5002 ...Sending QUIT signal to process 3465 ...Starting server on 0.0.0.0:5000 ...Starting server on 0.0.0.0:5001 ...Starting server on 0.0.0.0:5002 ...
  • 76. Meanwhile, in production...$ thin -C config/server.yml restartStopping server on 0.0.0.0:5000 ...Sending QUIT signal to process 3463 ...Stopping server on 0.0.0.0:5001 ...Sending QUIT signal to process 3464 ...Stopping server on 0.0.0.0:5002 ...Sending QUIT signal to process 3465 ...Starting server on 0.0.0.0:5000 ...Starting server on 0.0.0.0:5001 ...Starting server on 0.0.0.0:5002 ...
  • 77. Meanwhile, in production...$ thin -C config/server.yml restartStopping server on 0.0.0.0:5000 ...Sending QUIT signal to process 3463 ...Stopping server on 0.0.0.0:5001 ...Sending QUIT signal to process 3464 ...Stopping server on 0.0.0.0:5002 ...Sending QUIT signal to process 3465 ...Starting server on 0.0.0.0:5000 ...Starting server on 0.0.0.0:5001 ...Starting server on 0.0.0.0:5002 ...
  • 78. Agenda• Introduction• Development practices• Production environment setup• Future• Conclusion
  • 79. Production
  • 80. Production• Run multiple server processes
  • 81. Production• Run multiple server processes• Cycle them individually, or in batches
  • 82. Production• Run multiple server processes• Cycle them individually, or in batches • Watch out for capacity!
  • 83. Production• Run multiple server processes• Cycle them individually, or in batches • Watch out for capacity!• Cycle them gracefully
  • 84. Cycling techniques• Above the stack• Within the stack• Platform
  • 85. Above the stack• Use your favorite web server• Setup a load balancer above them, route around servers that are cycling
  • 86. Above the stack• PROTIP: default setup won’t work• Configure load balancer to poll for your application health: class HealthController < ApplicationController   def index     if File.exists?(Rails.root.join("MAINTENANCE"))       render :text => "Forget me!", :status => 503     else       render :text => "Im alive!"     end   end end
  • 87. Cycling + Load balancer Server Load balancer touch MAINTENANCE bounce servers remove server from rotation rm MAINTENANCEnew servers come online add server to rotation
  • 88. HAProxyoption httpchk GET /healthlisten myapp :9000 server myapp-1 1.0.0.1:5000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 1.0.0.2:5000 maxconn 1 check inter 2000 fall 1 rise 1
  • 89. HAProxy tell HAProxy where to check for the healthoption httpchk GET /healthlisten myapp :9000 server myapp-1 1.0.0.1:5000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 1.0.0.2:5000 maxconn 1 check inter 2000 fall 1 rise 1
  • 90. HAProxyoption httpchk GET /healthlisten myapp :9000 server myapp-1 1.0.0.1:5000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 1.0.0.2:5000 maxconn 1 check inter 2000 fall 1 rise 1 do not let your web server queue requests
  • 91. HAProxyoption httpchk GET /healthlisten myapp :9000 server myapp-1 1.0.0.1:5000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 1.0.0.2:5000 maxconn 1 check inter 2000 fall 1 rise 1 tell HAProxy to run the health check specified above
  • 92. HAProxyoption httpchk GET /healthlisten myapp :9000 server myapp-1 1.0.0.1:5000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 1.0.0.2:5000 maxconn 1 check inter 2000 fall 1 rise 1 every 2s
  • 93. HAProxyoption httpchk GET /healthlisten myapp :9000 server myapp-1 1.0.0.1:5000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 1.0.0.2:5000 maxconn 1 check inter 2000 fall 1 rise 1 stop routing to it once the health check failed once
  • 94. HAProxyoption httpchk GET /healthlisten myapp :9000 server myapp-1 1.0.0.1:5000 maxconn 1 check inter 2000 fall 1 rise 1 server myapp-2 1.0.0.2:5000 maxconn 1 check inter 2000 fall 1 rise 1 and bring it back once it worked once
  • 95. ELB$ elb-configure-healthcheck MyLoadBalancer --target "HTTP:5000/health" --interval 2 --unhealthy-threshold 1 --healthy-threshold 1
  • 96. Above the stack
  • 97. Above the stack• Minimal impact
  • 98. Above the stack• Minimal impact• Extended availability comes for free
  • 99. Above the stack• Minimal impact • Complicated setup• Extended availability comes for free
  • 100. Above the stack• Minimal impact • Complicated setup• Extended availability • Another component you comes for free have to track and maintain
  • 101. Within the stack
  • 102. Within the stack
  • 103. Unicorn• Replaces your web server• Bounces gracefully
  • 104. Unicorn63480 0.0 1.4 2549796 unicorn_rails master -c config/unicorn.rb63489 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63490 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63491 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb
  • 105. Unicorn$ kill -USR2 6348063480 0.0 1.4 2549796 unicorn_rails master -c config/unicorn.rb63489 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63490 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63491 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb
  • 106. Unicorn$ kill -USR2 6348063480 0.0 1.4 2549796 unicorn_rails master (old) -c config/unicorn.rb63489 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63490 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63491 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb63521 0.0 1.4 2549796 unicorn_rails master -c config/unicorn.rb63535 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63536 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63537 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb
  • 107. Unicorn63480 0.0 1.4 2549796 unicorn_rails master (old) -c config/unicorn.rb63489 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63490 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63491 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb63521 0.0 1.4 2549796 unicorn_rails master -c config/unicorn.rb63535 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63536 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63537 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb
  • 108. Unicorn$ kill -QUIT 6348063480 0.0 1.4 2549796 unicorn_rails master (old) -c config/unicorn.rb63489 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63490 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63491 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb63521 0.0 1.4 2549796 unicorn_rails master -c config/unicorn.rb63535 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63536 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63537 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb
  • 109. Unicorn$ kill -QUIT 6348063521 0.0 1.4 2549796 unicorn_rails master -c config/unicorn.rb63535 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63536 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63537 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb
  • 110. Unicorn63521 0.0 1.4 2549796 unicorn_rails master -c config/unicorn.rb63535 0.0 0.2 2549796 unicorn_rails worker[0] -c config/unicorn.rb63536 0.0 0.2 2549796 unicorn_rails worker[1] -c config/unicorn.rb63537 0.0 0.2 2549796 unicorn_rails worker[2] -c config/unicorn.rb
  • 111. Within the stack
  • 112. Within the stack• Easy to setup and deploy
  • 113. Within the stack• Easy to setup and deploy• Optimal performance to cycle the server
  • 114. Within the stack• Easy to setup and deploy • Changes your application stack• Optimal performance to cycle the server
  • 115. Within the stack• Easy to setup and deploy • Changes your application stack• Optimal performance to cycle the server • Doesn’t address availability issues
  • 116. Platform
  • 117. HerokuProcess State Command------- --------------- -------------------------------web.1 up for 10h bundle exec thin start -p $PORT
  • 118. Heroku$ git push heroku masterProcess State Command------- --------------- -------------------------------web.1 up for 10h bundle exec thin start -p $PORT
  • 119. Heroku$ git push heroku masterProcess State Command------- --------------- -------------------------------web.1 starting for 1s bundle exec thin start -p $PORT
  • 120. HerokuProcess State Command------- --------------- -------------------------------web.1 starting for 5s bundle exec thin start -p $PORT
  • 121. HerokuProcess State Command------- --------------- -------------------------------web.1 up for 1s bundle exec thin start -p $PORT
  • 122. Heroku BETAProcess State Command------- --------------- -------------------------------web.1 up for 10h bundle exec thin start -p $PORT
  • 123. Heroku BETA$ heroku labs:enable prebootProcess State Command------- --------------- -------------------------------web.1 up for 10h bundle exec thin start -p $PORT
  • 124. Heroku BETAProcess State Command------- --------------- -------------------------------web.1 up for 10h bundle exec thin start -p $PORT
  • 125. Heroku BETA$ git push heroku masterProcess State Command------- --------------- -------------------------------web.1 up for 10h bundle exec thin start -p $PORT
  • 126. Heroku BETA$ git push heroku masterProcess State Command------- --------------- -------------------------------web.1 up for 10h bundle exec thin start -p $PORTweb.1 starting for 5s bundle exec thin start -p $PORT
  • 127. Heroku BETAProcess State Command------- --------------- -------------------------------web.1 up for 10h bundle exec thin start -p $PORTweb.1 up for 1s bundle exec thin start -p $PORT
  • 128. Heroku BETAProcess State Command------- --------------- -------------------------------web.1 up for 5s bundle exec thin start -p $PORT
  • 129. Heroku• Migrations
  • 130. Heroku• Migrations $ git push pre-production master $ heroku run rake db:migrate --remote pre-production $ git push production master
  • 131. Agenda• Introduction• Development practices• Production environment setup• Future• Conclusion
  • 132. Future
  • 133. Future• Asset pipeline
  • 134. Future• Asset pipeline• Mysql tooling
  • 135. Future• Asset pipeline• Mysql tooling• Can Rails help making the development of hot compatible applications easier?
  • 136. Future• Asset pipeline• Mysql tooling• Can Rails help making the development of hot compatible applications easier?• Can databases?
  • 137. Future
  • 138. Future• Hot compatibility beyond zero downtime deploys
  • 139. Conclusion
  • 140. Conclusion• Zero downtime is a pain in the ass
  • 141. Conclusion• Zero downtime is a pain in the ass• Hot compatibility to the rescue
  • 142. Conclusion• Zero downtime is a pain in the ass• Hot compatibility to the rescue• Make sure your production env is ready
  • 143. Thank you!http://pedro.heroku.com @ped