Reliably Deploying Web Apps with Database Migrations
Reliably Deploying Web
Apps with Database
To deploy new code
with a database migration
that takes 10 seconds to run per site
in batches of five sites at a time
Then the 600th site
would be running new code with
unpatched database for 20 minutes!
If we have to migrate 600 sites
for 600th site
Building an Environment
Use virtualenv and pip?
pip install would also need to
compile C extensions
eg. lxml takes a lot of cpu time and memory
Makes creating new environments
Wheels are the new standard of python
distribution (PEP 427)
Unzip the wheel in site-packages and the
package is installed … like magic!
Build wheels on a different machine and
watch a new environment get deployed
in a few seconds
Cost of creating virtualenv is
No compiler required on production server
PyPI availability is not a factor during
Benefits of Build Step
The new code will require us to start
a new wsgi process + workers
Use Jinja and generate
process list per virtualenv
Protip: For adding/removing a group of
processes, use reread and update instead of
reload. It prevents non-related processes
Stop executing scheduled jobs when in
Preferably, use a fresh broker for every build
Track pending jobs per site
(we prepend sitename to queue name for this)
The Final Picture
● Delay enqueuing scheduled tasks.
● Continue if no pending jobs, else
retry after 60 seconds.
● Run migrations.
● Move to build x+1.
(No HTTP requests)