Epic South Disasters
Or: Why You Need to Actually Pay Attention to Your DBMS
http://todaysmeet.com/epic-south-disasters
Ch...
What is South?
South fills a need in the Django ecosystem.
The Django ORM
A tool to write SQL for you
“You can still write SQL if needed.”
You don’t have to write SQL any more?
In other words, it's your responsibility.
Problem: Django does not change a table once
it’s created with syncdb.
So "if needed" is really "when needed."
Most web apps that change often are going to need schema
migrations.
Django 1.5 documentation
The recommended way to migrate schemas:
“If you have made changes to a model and wish
to alter th...
In practice...
BEGIN;
CREATE TABLE "foo_foo_bars" (
"id" serial NOT NULL PRIMARY KEY,
- "foo_id" integer NOT NULL,
"bar_id" integer NOT N...
IT’S EASY,
OH SO EASY
South Solves Some Problems for us.
Problems South solves:
1. Automates writing schema migrations for us.
2. In Python (no SQL).
3. Broadly “Don’t Repeat Your...
Levels of abstraction
● great because they take us away from the
messy details
● risky because they take us away from the
...
How to Use South (very quick 101)
Our Model
+++ b/minifier/models.py
class MinifiedURL(models.Model):
url = models.CharField(
max_length=100)
datetime = mod...
Initial Migration
$ ./manage.py schemamigration minifier --
initial
Creating migrations directory at '/...
/minifier/migra...
Initial Migration
$ ls -l minifier/migrations/
total 8
-rw-r--r-- 1 chris staff 1188 Aug 30 11:40 0001_initial.py
-rw-r--r...
Initial Migration
$ ./manage.py syncdb
Syncing...
Creating tables ...
Creating table south_migrationhistory
Synced:
> djan...
Initial Migration
$ ./manage.py migrate
Running migrations for minifier:
- Migrating forwards to 0001_initial.
> minifier:...
Ok, let’s add a field.
Adding a Field
class MinifiedURL(models.Model):
+ submitter = models.ForeignKey(
+ 'auth.user', null=True)
url = models.Ch...
$ ./manage.py schemamigration minifier --auto
+ Added field submitter on minifier.MinifiedURL
Created 0002_auto__add_field...
class Migration(SchemaMigration):
def forwards(self, orm):
# Adding field 'MinifiedURL.submitter'
db.add_column(u'minifier...
Adding a Field
$ ./manage.py migrate minifier 0002
- Soft matched migration 0002 to
0002_auto__add_field_minifiedurl_submi...
$ ./manage.py dbshell
psql-# d+ minifier_minifiedurl;
Table "public.minifier_minifiedurl"
Column | Type
--------------+---...
More details: follow the South Tutorial
http://south.readthedocs.org/en/latest/tutorial/
● Many people approach a new tool with a
broad set of expectations as to what they
think it will do for them.
● This may h...
Disaster situations
Don't panic.
Our Model
class MinifiedURL(models.Model):
submitter = models.ForeignKey(
'auth.user', null=True)
url = models.CharField(
...
Our Model
class MinifiedURL(models.Model):
submitter = models.ForeignKey(
'auth.user', null=True)
url = models.CharField(
...
$ vim minifier/models.py
$ ./manage.py schemamigration minifier --auto
$ git commit -am "Rename field."
$ git push
$ ./dep...
Fast iteration!
+++ b/minifier/migrations/0003_auto_del_field.
py
# Deleting field 'MinifiedURL.datetime'
db.delete_column...
Lesson #1
Always read migrations that are generated with --auto.
So how do we do this?
class MinifiedURL(models.Model):
submitter = models.ForeignKey(
'auth.user', null=True)
url = models...
Data migration - basic example
1. schemamigration - Create the new field.
2. datamigration - Copy the data to the new
fiel...
Data migration - basic example
class MinifiedURL(models.Model):
submitter = models.ForeignKey(
'auth.user', null=True)
url...
Data migration - basic example
$ ./manage.py schemamigration minifier --auto
Created 0003_auto__add_field_minifiedurl_crea...
# Note: Don't use "from
# appname.models import ModelName".
# Use orm.ModelName to refer to
# models in this application.....
Data migration - basic example
class Migration(DataMigration):
def forwards(self, orm):
+ for minified_url in orm.Minified...
Data migration - basic example
class MinifiedURL(models.Model):
submitter = models.ForeignKey(
'auth.user', null=True)
url...
$ ./manage.py migrate --list
minifier
(*) 0001_initial
(*) 0002_auto__add_field_minifiedurl_submitt
( ) 0003_auto__add_fie...
$ ./manage.py migrate
- Migrating forwards to 0005_auto__del_
field_minifiedurl_datetime.
> minifier:0003_auto__add_field_...
$ ./manage.py migrate
- Migrating forwards to 0005_auto__del_
field_minifiedurl_datetime.
> minifier:0003_auto__add_field_...
South’s frozen ORM is pretty nifty.
It will expose the model at an historical point-in-time.
Danger.
Many parts of the Django ORM still function.
Our Model
class MinifiedURL(models.Model):
submitter = models.ForeignKey(
'auth.user')
created = models.DateTimeField(
aut...
Our Model
class MinifiedURL(models.Model):
submitter = models.ForeignKey(
'auth.user')
created = models.DateTimeField(
aut...
class Migration(DataMigration):
def forwards(self, orm):
+ model = orm.MinifiedURL
+
+ for minified_url in model.objects.a...
$ git commit -am "Migrate 'domain' from 'url'
field."
$ git push
$ ./deploy-to-production.sh
Done!
Fast iteration!
Before and After
pk updated (before)
---|-----------------|
566|2013-03-01 09:01 |
567|2012-01-22 17:34 |
568|2012-12-31 1...
Before and After
pk updated (before) updated (after)
---|-----------------|-----------------|
566|2013-03-01 09:01 |2013-0...
Oh no! Why did we lose datetime
information?
Our Model
class MinifiedURL(models.Model):
submitter = models.ForeignKey(
'auth.user')
created = models.DateTimeField(
aut...
The South ORM wraps over the Django
ORM, which applies rules such as
auto_now=True and
auto_now_add=True.
Especially nasty because no exception
raised or warning given even with this
kind of data loss.
Lesson #2
Always confirm your migrations do what you expect --
and nothing more.
Workaround
+ opts = model._meta
+ field = opts.get_field_by_name('updated')[0]
+ old_auto_now = field.auto_now
+ field.aut...
Before and After
pk updated (before) updated (after)
---|-----------------|-----------------|
566|2013-03-01 09:01 |2013-0...
Oh no! Why did all our users suddenly
get emailed?
@receiver(post_save)
def email_user_on_save(sender, **kwargs):
"""
Not sure why I'm doing this here,
but it seems like a g...
The South ORM wraps over the Django
ORM, so it sends post_save signals.
However, the metaclass magic usually
takes care of avoiding problems.
@receiver(post_save)
def email_user_on_save(sender, **kwargs):
"""
Not sure why I'm doing this here,
but it seems like a g...
@receiver(post_save)
def email_user_on_save(sender, **kwargs):
"""
Not sure why I'm doing this here,
but it seems like a g...
ipdb> print repr(sender)
<class 'minifier.models.MinifiedURL'>
ipdb> print repr(MinifiedURL)
<class 'minifier.models.Minif...
Lesson #3
Always check data migrations for unintended consequences.
class MinifiedURL(models.Model):
created = models.DateTimeField(
auto_now_add=True)
updated = models.DateTimeField(
auto_n...
class MinifiedURL(models.Model):
created = models.DateTimeField(
auto_now_add=True)
updated = models.DateTimeField(
auto_n...
$ ./manage.py schemamigration minifier 
--auto
~ Changed field url on minifier.MinifiedURL
Created 0010_auto__chg_field_mi...
Seems fine...
$ ./manage.py migrate
Running migrations for minifier:
- Migrating forwards to
0010_auto__chg_field_minified...
Production vs. Development
Beware of differences in configuration.
From a Django blog
7. Local vs. Production Environments
Django comes with sqlite, a simple flat-file database that
doesn't...
$ git commit -am "Add some breathing
space to url fields."
$ git push
$ ./deploy-to-production.sh
Done!
Fast iteration!
Migration Failed
Running migrations for minifier:
- Migrating forwards to
0010_auto__chg_field_minifiedurl.
> minifier:001...
class MinifiedURL(models.Model):
created = models.DateTimeField(
auto_now_add=True)
updated = models.DateTimeField(
auto_n...
Always pay attention to the limitations of your DBMS.
Lesson #4
Schema-altering commands (DDL commands) cause a phantom
auto-commit.
Major limitation of MySQL
With InnoDB, when a client executes a
DDL change, the server executes an
implicit commit even if the normal auto-
commit b...
DDL transaction on Postgres
psql=# DROP TABLE IF EXISTS foo;
NOTICE: table "foo" does not exist
psql=# BEGIN;
psql=# CREAT...
No DDL transaction on MySQL
mysql> drop table if exists foo;
mysql> begin;
mysql> create table foo (bar int)
type=InnoDB;
...
South uses DDL transactions if they are
available.
Pay attention to your DBMS
FATAL ERROR - The following SQL query failed:
ALTER TABLE `minifier_minifiedurl` ADD CONSTRAINT...
1. Always read migrations that are generated
with --auto.
2. Always confirm your migrations do what
you expect.
3. Always ...
Encouragement
● Tools are not the problem. Tools are
why we are in this business.
● Knowledge is power. Know what
South is...
● David Cho
● Hadi Arbabi
● Mike Harris
Special Thanks
http://www.scrollmotion.com/careers
Epic South Disasters
Epic South Disasters
Epic South Disasters
Epic South Disasters
Epic South Disasters
Epic South Disasters
Upcoming SlideShare
Loading in...5
×

Epic South Disasters

1,098

Published on

Slides from my talk at DjangoCon 2013.

Published in: Technology, Business
1 Comment
4 Likes
Statistics
Notes
No Downloads
Views
Total Views
1,098
On Slideshare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
9
Comments
1
Likes
4
Embeds 0
No embeds

No notes for slide

Epic South Disasters

  1. 1. Epic South Disasters Or: Why You Need to Actually Pay Attention to Your DBMS http://todaysmeet.com/epic-south-disasters Christopher Adams Engineering Lead, Scrollmotion @adamsc64 http://todaysmeet.com/epic-south-disasters https://github.com/adamsc64/epic-south-disasters
  2. 2. What is South? South fills a need in the Django ecosystem.
  3. 3. The Django ORM A tool to write SQL for you
  4. 4. “You can still write SQL if needed.” You don’t have to write SQL any more?
  5. 5. In other words, it's your responsibility. Problem: Django does not change a table once it’s created with syncdb.
  6. 6. So "if needed" is really "when needed." Most web apps that change often are going to need schema migrations.
  7. 7. Django 1.5 documentation The recommended way to migrate schemas: “If you have made changes to a model and wish to alter the database tables to match, use the sql command to display the new SQL structure and compare that to your existing table schema to work out the changes.” https://docs.djangoproject.com/en/1.5/ref/django-admin/#syncdb
  8. 8. In practice...
  9. 9. BEGIN; CREATE TABLE "foo_foo_bars" ( "id" serial NOT NULL PRIMARY KEY, - "foo_id" integer NOT NULL, "bar_id" integer NOT NULL REFERENCES "baz_bar" ("id") DEFERRABLE INITIALLY DEFERRED, + "foo_id" integer NOT NULL, UNIQUE ("foo_id", "bar_id") ) ; CREATE TABLE "foo_foo_zazzs" ( "id" serial NOT NULL PRIMARY KEY, "foo_id" integer NOT NULL, + "user_id" integer REFERENCES "auth_user" ("id") DEFERRABLE INITIALLY DEFERRED, +) -; - "bang_id" integer NOT NULL REFERENCES "baz_bang" ("id") DEFERRABLE INITIALLY DEFERRED, - UNIQUE ("foo_id", "bang_id") -) -; -CREATE TABLE "foo_foo" ( - "user_id" integer REFERENCES "auth_user" ("id") DEFERRABLE INITIALLY DEFERRED, + "properties" text NOT NULL, - "id" serial NOT NULL PRIMARY KEY, "name" varchar(100) NOT NULL, "description" text NOT NULL, + "user_id" integer REFERENCES "auth_user" ("id") DEFERRABLE INITIALLY DEFERRED, "created_by_id" integer REFERENCES "auth_user" ("id") DEFERRABLE INITIALLY DEFERRED, "updated_at" timestamp with time zone NOT NULL, "bing" text NOT NULL, + "properties" text NOT NULL, - "lft" integer CHECK ("lft" >= 0) NOT NULL, - "rght" integer CHECK ("rght" >= 0) NOT NULL, - "tree_id" integer CHECK ("tree_id" >= 0) NOT NULL, - "level" integer CHECK ("level" >= 0) NOT NULL ) ; ALTER TABLE "foo_foo_bars" ADD CONSTRAINT "foo_id_refs_id_1401a163" FOREIGN KEY ("foo_id") REFERENCES "foo_foo" ("id") DEFERRABLE INITIALLY DEFERRED; ALTER TABLE "foo_foo_zazzs" ADD CONSTRAINT "foo_id_refs_id_ca2b0e5" FOREIGN KEY ("foo_id") REFERENCES "foo_foo" ("id") DEFERRABLE INITIALLY DEFERRED; +ALTER TABLE "foo_foo" ADD CONSTRAINT "parent_id_refs_id_1eb29019" FOREIGN KEY ("parent_id") REFERENCES "foo_foo" ("id") DEFERRABLE INITIALLY DEFERRED; SQL diffs are no fun
  10. 10. IT’S EASY, OH SO EASY
  11. 11. South Solves Some Problems for us.
  12. 12. Problems South solves: 1. Automates writing schema migrations for us. 2. In Python (no SQL). 3. Broadly “Don’t Repeat Yourself” (DRY). 4. Migrations are applied in order. 5. Version control migration code. 6. Shared migrations, dev and production. 7. Fast iteration.
  13. 13. Levels of abstraction ● great because they take us away from the messy details ● risky because they take us away from the messy details ● can obscure what’s going on
  14. 14. How to Use South (very quick 101)
  15. 15. Our Model +++ b/minifier/models.py class MinifiedURL(models.Model): url = models.CharField( max_length=100) datetime = models.DateTimeField( auto_now_add=True)
  16. 16. Initial Migration $ ./manage.py schemamigration minifier -- initial Creating migrations directory at '/... /minifier/migrations'... Creating __init__.py in '/... /minifier/migrations'... + Added model minifier.MinifiedURL Created 0001_initial.py. You can now apply this migration with: ./manage.py migrate minifier
  17. 17. Initial Migration $ ls -l minifier/migrations/ total 8 -rw-r--r-- 1 chris staff 1188 Aug 30 11:40 0001_initial.py -rw-r--r-- 1 chris staff 0 Aug 30 11:40 __init__.py
  18. 18. Initial Migration $ ./manage.py syncdb Syncing... Creating tables ... Creating table south_migrationhistory Synced: > django.contrib.auth > south Not synced (use migrations): - minifier (use ./manage.py migrate to migrate these)
  19. 19. Initial Migration $ ./manage.py migrate Running migrations for minifier: - Migrating forwards to 0001_initial. > minifier:0001_initial - Loading initial data for minifier. Installed 0 object(s) from 0 fixture(s)
  20. 20. Ok, let’s add a field.
  21. 21. Adding a Field class MinifiedURL(models.Model): + submitter = models.ForeignKey( + 'auth.user', null=True) url = models.CharField( max_length=100) datetime = models.DateTimeField( auto_now_add=True)
  22. 22. $ ./manage.py schemamigration minifier --auto + Added field submitter on minifier.MinifiedURL Created 0002_auto__add_field_minifiedurl_submitt er.py. You can now apply this migration with: ./manage. py migrate minifier Adding a Field
  23. 23. class Migration(SchemaMigration): def forwards(self, orm): # Adding field 'MinifiedURL.submitter' db.add_column(u'minifier_minifiedurl', 'submitter', ... ) def backwards(self, orm): # Deleting field 'MinifiedURL.submitter' db.delete_column(u'minifier_minifiedurl', 'submitter_id' ) Adding a Field
  24. 24. Adding a Field $ ./manage.py migrate minifier 0002 - Soft matched migration 0002 to 0002_auto__add_field_minifiedurl_submitter. Running migrations for minifier: - Migrating forwards to 0002_auto__add_field_minifiedurl_submitter. > minifier: 0002_auto__add_field_minifiedurl_submitter - Loading initial data for minifier. Installed 0 object(s) from 0 fixture(s)
  25. 25. $ ./manage.py dbshell psql-# d+ minifier_minifiedurl; Table "public.minifier_minifiedurl" Column | Type --------------+------------------------- id | integer url | character varying(100) submitter_id | integer datetime | timestamp with time zone It worked!
  26. 26. More details: follow the South Tutorial http://south.readthedocs.org/en/latest/tutorial/
  27. 27. ● Many people approach a new tool with a broad set of expectations as to what they think it will do for them. ● This may have little correlation with what the project actually has implemented. Expectations
  28. 28. Disaster situations Don't panic.
  29. 29. Our Model class MinifiedURL(models.Model): submitter = models.ForeignKey( 'auth.user', null=True) url = models.CharField( max_length=100) datetime = models.DateTimeField( auto_now_add=True)
  30. 30. Our Model class MinifiedURL(models.Model): submitter = models.ForeignKey( 'auth.user', null=True) url = models.CharField( max_length=100) + created = models.DateTimeField( auto_now_add=True)
  31. 31. $ vim minifier/models.py $ ./manage.py schemamigration minifier --auto $ git commit -am "Rename field." $ git push $ ./deploy-to-production.sh Done! Fast iteration!
  32. 32. Fast iteration! +++ b/minifier/migrations/0003_auto_del_field. py # Deleting field 'MinifiedURL.datetime' db.delete_column(u'minifier_minifiedurl', 'datetime') # Adding field 'MinifiedURL.created' db.add_column(u'minifier_minifiedurl', 'created', ... )
  33. 33. Lesson #1 Always read migrations that are generated with --auto.
  34. 34. So how do we do this? class MinifiedURL(models.Model): submitter = models.ForeignKey( 'auth.user', null=True) url = models.CharField( max_length=100) + created = models.DateTimeField( auto_now_add=True)
  35. 35. Data migration - basic example 1. schemamigration - Create the new field. 2. datamigration - Copy the data to the new field from the old field. 3. schemamigration - Delete the old field.
  36. 36. Data migration - basic example class MinifiedURL(models.Model): submitter = models.ForeignKey( 'auth.user', null=True) url = models.CharField( max_length=100) datetime = models.DateTimeField( auto_now_add=True) + created = models.DateTimeField( auto_now_add=True)
  37. 37. Data migration - basic example $ ./manage.py schemamigration minifier --auto Created 0003_auto__add_field_minifiedurl_crea ted.py. $ ./manage.py datamigration minifier datetime_to_created Created 0004_datetime_to_created.py. $ vim minifier/migrations/0004_datetime_to_ created.py
  38. 38. # Note: Don't use "from # appname.models import ModelName". # Use orm.ModelName to refer to # models in this application... Data migration - basic example
  39. 39. Data migration - basic example class Migration(DataMigration): def forwards(self, orm): + for minified_url in orm.MinifiedURL.objects.all(): + minified_url.created = minified_url.datetime + minified_url.save() def backwards(self, orm): + for minified_url in orm.MinifiedURL.objects.all(): + minified_url.datetime = minified_url.created + minified_url.save()
  40. 40. Data migration - basic example class MinifiedURL(models.Model): submitter = models.ForeignKey( 'auth.user', null=True) url = models.CharField( max_length=100) - datetime = models.DateTimeField( auto_now_add=True) created = models.DateTimeField( auto_now_add=True)
  41. 41. $ ./manage.py migrate --list minifier (*) 0001_initial (*) 0002_auto__add_field_minifiedurl_submitt ( ) 0003_auto__add_field_minifiedurl_created ( ) 0004_datetime_to_created ( ) 0005_auto__del_field_minifiedurl_datetim Data migration - basic example
  42. 42. $ ./manage.py migrate - Migrating forwards to 0005_auto__del_ field_minifiedurl_datetime. > minifier:0003_auto__add_field_minifiedurl_ created > minifier:0004_datetime_to_created > minifier:0005_auto__del_field_ minifiedurl_datetime Data migration - basic example
  43. 43. $ ./manage.py migrate - Migrating forwards to 0005_auto__del_ field_minifiedurl_datetime. > minifier:0003_auto__add_field_minifiedurl_ created > minifier:0004_datetime_to_created > minifier:0005_auto__del_field_ minifiedurl_datetime Data migration - basic example
  44. 44. South’s frozen ORM is pretty nifty. It will expose the model at an historical point-in-time.
  45. 45. Danger. Many parts of the Django ORM still function.
  46. 46. Our Model class MinifiedURL(models.Model): submitter = models.ForeignKey( 'auth.user') created = models.DateTimeField( auto_now_add=True) updated = models.DateTimeField( auto_now=True) url = models.CharField( max_length=100)
  47. 47. Our Model class MinifiedURL(models.Model): submitter = models.ForeignKey( 'auth.user') created = models.DateTimeField( auto_now_add=True) updated = models.DateTimeField( auto_now=True) url = models.CharField( max_length=100) + domain = models.CharField( + max_length=30)
  48. 48. class Migration(DataMigration): def forwards(self, orm): + model = orm.MinifiedURL + + for minified_url in model.objects.all(): + minified_url.domain = ( + minified_url.url.split('/')[2] + ) + minified_url.save() Data migration
  49. 49. $ git commit -am "Migrate 'domain' from 'url' field." $ git push $ ./deploy-to-production.sh Done! Fast iteration!
  50. 50. Before and After pk updated (before) ---|-----------------| 566|2013-03-01 09:01 | 567|2012-01-22 17:34 | 568|2012-12-31 19:11 | 569|2013-04-10 10:02 | ...
  51. 51. Before and After pk updated (before) updated (after) ---|-----------------|-----------------| 566|2013-03-01 09:01 |2013-09-04 14:01 | 567|2012-01-22 17:34 |2013-09-04 14:01 | 568|2012-12-31 19:11 |2013-09-04 14:01 | 569|2013-04-10 10:02 |2013-09-04 14:01 | ... ...
  52. 52. Oh no! Why did we lose datetime information?
  53. 53. Our Model class MinifiedURL(models.Model): submitter = models.ForeignKey( 'auth.user') created = models.DateTimeField( auto_now_add=True) updated = models.DateTimeField( auto_now=True) url = models.CharField( max_length=100)
  54. 54. The South ORM wraps over the Django ORM, which applies rules such as auto_now=True and auto_now_add=True.
  55. 55. Especially nasty because no exception raised or warning given even with this kind of data loss.
  56. 56. Lesson #2 Always confirm your migrations do what you expect -- and nothing more.
  57. 57. Workaround + opts = model._meta + field = opts.get_field_by_name('updated')[0] + old_auto_now = field.auto_now + field.auto_now = False for minified_url in model.objects.all(): minified_url.domain = ( minified_url.url.split('/')[2] ) minified_url.save() + field.auto_now = old_auto_now Peter Bengtsson, http://www.peterbe.com/plog/migration-south-auto_now_add
  58. 58. Before and After pk updated (before) updated (after) ---|-----------------|-----------------| 566|2013-03-01 09:01 |2013-03-01 09:01 | 567|2012-01-22 17:34 |2012-01-22 17:34 | 568|2012-12-31 19:11 |2012-12-31 19:11 | 569|2013-04-10 10:02 |2013-04-10 10:02 | ... ...
  59. 59. Oh no! Why did all our users suddenly get emailed?
  60. 60. @receiver(post_save) def email_user_on_save(sender, **kwargs): """ Not sure why I'm doing this here, but it seems like a good place! REFACTOR LATER TBD FYI!! """ if sender.__name__ == "MinifiedURL": email(kwargs['instance'].submitter, "Congratulations on changing " "your url!", ) Whoops, forgot about this.
  61. 61. The South ORM wraps over the Django ORM, so it sends post_save signals.
  62. 62. However, the metaclass magic usually takes care of avoiding problems.
  63. 63. @receiver(post_save) def email_user_on_save(sender, **kwargs): """ Not sure why I'm doing this here, but it seems like a good place! REFACTOR LATER TBD FYI!! """ if sender.__name__ == "MinifiedURL": email(kwargs['instance'].submitter, "Congratulations on changing " "your url!", ) So this...
  64. 64. @receiver(post_save) def email_user_on_save(sender, **kwargs): """ Not sure why I'm doing this here, but it seems like a good place! REFACTOR LATER TBD FYI!! """ - if sender.__name__ == "MinifiedURL": + if sender == MinifiedURL: email(kwargs['instance'].submitter, "Congratulations on changing " "your url!", ) ...should be this.
  65. 65. ipdb> print repr(sender) <class 'minifier.models.MinifiedURL'> ipdb> print repr(MinifiedURL) <class 'minifier.models.MinifiedURL'> ipdb> print MinifiedURL == sender False ipdb> print id(MinifiedURL) 140455961329984 ipdb> print id(sender) 140455961864000 Metaclass magic
  66. 66. Lesson #3 Always check data migrations for unintended consequences.
  67. 67. class MinifiedURL(models.Model): created = models.DateTimeField( auto_now_add=True) updated = models.DateTimeField( auto_now=True) url = models.CharField( max_length=100, db_index=True) Our Model
  68. 68. class MinifiedURL(models.Model): created = models.DateTimeField( auto_now_add=True) updated = models.DateTimeField( auto_now=True) - url = models.CharField( - max_length=100, db_index=True) + url = models.CharField( + max_length=1000, db_index=True) Our Model
  69. 69. $ ./manage.py schemamigration minifier --auto ~ Changed field url on minifier.MinifiedURL Created 0010_auto__chg_field_minifiedurl_ url. py. You can now apply this migration with: . /manage.py migrate minifier Create the schema migration.
  70. 70. Seems fine... $ ./manage.py migrate Running migrations for minifier: - Migrating forwards to 0010_auto__chg_field_minifiedurl_url. > minifier: 0010_auto__chg_field_minifiedurl_url - Loading initial data for minifier. Installed 0 object(s) from 0 fixture(s) “Works fine on development?” “Ship it!”
  71. 71. Production vs. Development Beware of differences in configuration.
  72. 72. From a Django blog 7. Local vs. Production Environments Django comes with sqlite, a simple flat-file database that doesn't need any configuration. This makes prototyping fast and easy right out of the box. However, once you've moved your project into a production environment, odds are you'll have to use a more robust database like Postgresql or MySQL. This means that you're going to have two separate environments: production and development. http://net.tutsplus.com/tutorials/other/10-django-troublespots-for-beginners/
  73. 73. $ git commit -am "Add some breathing space to url fields." $ git push $ ./deploy-to-production.sh Done! Fast iteration!
  74. 74. Migration Failed Running migrations for minifier: - Migrating forwards to 0010_auto__chg_field_minifiedurl. > minifier:0010_auto__chg_field_minifiedurl ! Error found during real run of migration! Aborting. Error in migration: minifier: 0010_auto__chg_field_minifiedurl Warning: Specified key was too long; max key length is 250 bytes
  75. 75. class MinifiedURL(models.Model): created = models.DateTimeField( auto_now_add=True) updated = models.DateTimeField( auto_now=True) url = models.CharField( max_length=1000, db_index=True) Our Model
  76. 76. Always pay attention to the limitations of your DBMS. Lesson #4
  77. 77. Schema-altering commands (DDL commands) cause a phantom auto-commit. Major limitation of MySQL
  78. 78. With InnoDB, when a client executes a DDL change, the server executes an implicit commit even if the normal auto- commit behavior is turned off.
  79. 79. DDL transaction on Postgres psql=# DROP TABLE IF EXISTS foo; NOTICE: table "foo" does not exist psql=# BEGIN; psql=# CREATE TABLE foo (bar int); psql=# INSERT INTO foo VALUES (1); psql=# ROLLBACK; # rolls back two cmds psql=# SELECT * FROM foo; ERROR: relation "foo" does not exist
  80. 80. No DDL transaction on MySQL mysql> drop table if exists foo; mysql> begin; mysql> create table foo (bar int) type=InnoDB; mysql> insert into foo values (1); mysql> rollback; # Table 'foo' exists! mysql> select * from foo; 0 rows in set (0.00 sec)
  81. 81. South uses DDL transactions if they are available.
  82. 82. Pay attention to your DBMS FATAL ERROR - The following SQL query failed: ALTER TABLE `minifier_minifiedurl` ADD CONSTRAINT `minifier_minifiedurl_url_263b28b6c6b349a8_uniq` UNIQUE (`name`) The error was: (1062, "Duplicate entry 'http://cnn.com' for key 'minifier_minifiedurl_url_263b28b6c6b349a8_uniq'") ! Error found during real run of migration! Aborting. ! Since you have a database that does not support running ! schema-altering statements in transactions, we have had ! to leave it in an interim state between migrations. ! You *might* be able to recover with: = ALTER TABLE `minifier_minifiedurl` DROP COLUMN `url` CASCADE; [] - no dry run output for alter_column() due to dynamic DDL, sorry ! The South developers regret this has happened, and would ! like to gently persuade you to consider a slightly ! easier-to-deal-with DBMS (one that supports DDL transactions) ! NOTE: The error which caused the migration to fail is further up.
  83. 83. 1. Always read migrations that are generated with --auto. 2. Always confirm your migrations do what you expect. 3. Always check data migrations for unintended consequences. 4. Always pay attention to the limitations of your DBMS. Lessons Recap
  84. 84. Encouragement ● Tools are not the problem. Tools are why we are in this business. ● Knowledge is power. Know what South is doing. ● Know what Django is doing for that matter.
  85. 85. ● David Cho ● Hadi Arbabi ● Mike Harris Special Thanks
  86. 86. http://www.scrollmotion.com/careers
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×