Going beyond Django ORM limitations with Postgres

8,393
-1

Published on

Published in: Technology
1 Comment
20 Likes
Statistics
Notes
  • That unclickable, uncopyable url in slide 3 is:
    https://speakerdeck.com/craigkerstiens/postgres-demystified
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
8,393
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
82
Comments
1
Likes
20
Embeds 0
No embeds

No notes for slide

Going beyond Django ORM limitations with Postgres

  1. 1. G!"#$ b%&!# ()% D*+#$! ORM,"-"(+("!#. w"() P/($r%. @0r+"$1%r.("%#.
  2. 2. PSA: Macs Postgres.app
  3. 3. Why Postgres “Its the emacs of databases” http://www.craigkerstiens.com/2012/04/30/why-postgres/https://speakerdeck.com/craigkerstiens/postgres-demystified
  4. 4. TLDR Datatypes Fast Column Addition Conditional Indexes Listen/Notify Transactional DDL Table Inheritance Foreign Data Wrappers Per Transaction sync replicationConcurrent Index Creation Window functions Extensions NoSQL inside SQLCommon Table Expressions Momentum
  5. 5. Limitations?Django attempts to support as many featuresas possible on all database backends. However,not all database backends are alike, and we’vehad to make design decisions on whichfeatures to support and which assumptionswe can make safely.
  6. 6. D+(+(2%.
  7. 7. __________< Postgres > ---------- ^__^ (oo)_______ (__) )/ ||----w | || ||
  8. 8. DatatypesUUID date booleantimestamptzarray intervalinteger bigint smallint line XML enum char money serial bytea point float inet polygon numeric circle cidr varchar tsquery timetzpath time text macaddr timestamp box tsvector
  9. 9. DatatypesUUID date booleantimestamptzarray intervalinteger bigint smallint line XML enum char money serial bytea point float inet polygon numeric circle cidr varchar tsquery timetzpath time text macaddr timestamp box tsvector
  10. 10. DatatypesUUID date booleantimestamptzarray intervalinteger bigint smallint line XML enum char money serial bytea point float inet polygon numeric circle cidr varchar tsquery timetzpath time text macaddr timestamp box tsvector
  11. 11. DatatypesUUID date booleantimestamptzarray intervalinteger bigint smallint line XML enum char money serial bytea point float inet polygon numeric circle cidr varchar tsquery timetzpath time text macaddr timestamp box tsvector
  12. 12. _______< SQLite > --------- ^__^ (oo)_______ (__) )/ ||----w | || ||
  13. 13. Datatypes null integertext real blob
  14. 14. DatatypesUUID date booleantimestamptzarray intervalinteger bigint smallint line XML enum char money serial bytea point float inet polygon numeric circle cidr varchar tsquery timetzpath time text macaddr timestamp box tsvector
  15. 15. I#3%.
  16. 16. ________< Postgres > ---------- ^__^ (oo)_______ (__) )/ ||----w | || ||
  17. 17. IndexesMultiple Types B-Tree, Gin, Gist, KNN, SP-GistCREATE INDEX CONCURRENTLY
  18. 18. _____< MySQL > ------- ^__^ (oo)_______ (__) )/ ||----w | || ||
  19. 19. IndexesThey exist
  20. 20. Digging In
  21. 21. ArraysCREATE TABLE item ( id serial NOT NULL, name varchar (255), tags varchar(255) [], created_at timestamp);
  22. 22. ArraysCREATE TABLE item ( id serial NOT NULL, name varchar (255), tags varchar(255) [], created_at timestamp);
  23. 23. ArraysINSERT INTO itemVALUES (1, Django Pony,{“Programming”,”Animal”}, now());INSERT INTO itemVALUES (2, Ruby Gem,{“Programming”,”Jewelry”}, now());
  24. 24. ArraysINSERT INTO itemVALUES (1, Django Pony,{“Programming”,”Animal”}, now());INSERT INTO itemVALUES (2, Ruby Gem,{“Programming”,”Jewelry”}, now());
  25. 25. Djangopip install djorm-ext-pgarraypip install djorm-ext-expressionsmodels.py:from djorm_pgarray.fields import ArrayFieldfrom djorm_expressions.models import ExpressionManagerclass Item(models.Model): name = models.CharField(max_length=255) tags = ArrayField(dbtype="varchar(255)") created_at = models.DateTimeField(auto_now=True)
  26. 26. Djangopip install djorm-ext-pgarraypip install djorm-ext-expressionsmodels.py:from djorm_pgarray.fields import ArrayFieldfrom djorm_expressions.models import ExpressionManagerclass Item(models.Model): name = models.CharField(max_length=255) tags = ArrayField(dbtype="varchar(255)") created_at = models.DateTimeField(auto_now=True)
  27. 27. Djangoi = Item( name=Django Pony, tags=[Programming,Animal])i.save()qs = Item.objects.where( SqlExpression("tags", "@>", [programming]))
  28. 28. Djangoi = Item( name=Django Pony, tags=[Programming,Animal])i.save()qs = Item.objects.where( SqlExpression("tags", "@>", [programming]))
  29. 29. Extensions dblink hstore trigram uuid-ossp pgstattuple citext pgcrypto isn ltree fuzzystrmatch cube dict_int earthdistanceunaccent dict_xsyn btree_gist tablefunc pgrowlocks
  30. 30. NoSQL in your SQL
  31. 31. NoSQL in your SQLCREATE EXTENSION hstore;CREATE TABLE item ( id integer NOT NULL, name varchar(255), data hstore,);
  32. 32. NoSQL in your SQLCREATE EXTENSION hstore;CREATE TABLE item ( id integer NOT NULL, name varchar(255), data hstore,);
  33. 33. NoSQL in your SQLINSERT INTO itemsVALUES ( 1, Pony, rating => "4.0", color => “Pink”,);
  34. 34. NoSQL in your SQLINSERT INTO itemsVALUES ( 1, Pony, rating => "4.0", color => “Pink”,);
  35. 35. Djangopip install django-hstorefrom django.db import modelsfrom django_hstore import hstoreclass Item(models.Model): name = models.CharField(max_length=250) data = hstore.DictionaryField(db_index=True) objects = hstore.Manager() def __unicode__(self): return self.name
  36. 36. Djangopip install django-hstorefrom django.db import modelsfrom django_hstore import hstoreclass Item(models.Model): name = models.CharField(max_length=250) data = hstore.DictionaryField(db_index=True) objects = hstore.Manager() def __unicode__(self): return self.name
  37. 37. DjangoItem.objects.create( name=Django Pony, data={rating: 5})Item.objects.create( name=Pony, data={color: pink, rating: 4})
  38. 38. DjangoItem.objects.create( name=Django Pony, data={rating: 5})Item.objects.create( name=Pony, data={color: pink, rating: 4})
  39. 39. Djangocolored_ponies = Product.objects.filter(data__contains=color)print colored_ponies[0].data[color]favorite_ponies = Product.objects.filter(data__contains={rating: 5})print colored_ponies[0]
  40. 40. Djangocolored_ponies = Product.objects.filter(data__contains=color)print colored_ponies[0].data[color]favorite_ponies = Product.objects.filter(data__contains={rating: 5})print colored_ponies[0]
  41. 41. JSON
  42. 42. JSONw/ PLV8
  43. 43. Q4%4%"#$
  44. 44. Pub/Sub?
  45. 45. Postgres agreat Queue
  46. 46. Postgres a great QueueNot With Polling
  47. 47. Trunkpip install celery trunkpsql < sql/*.sqlcelery worker -A tasks --loglevel=infoipython -i tasks.py>>> add.delay(2, 2)
  48. 48. Trunkpip install celery trunkpsql < sql/*.sqlcelery worker -A tasks --loglevel=infoipython -i tasks.py>>> add.delay(2, 2)
  49. 49. Trunkfrom celery import Celerycelery = Celery(tasks)celery.config_from_object({ BROKER_URL: trunk.transport.Transport://localhost/trunk,})@celery.taskdef add(x, y): return x + y
  50. 50. Trunkfrom celery import Celerycelery = Celery(tasks)celery.config_from_object({ BROKER_URL: trunk.transport.Transport://localhost/trunk,})@celery.taskdef add(x, y): return x + y
  51. 51. T3( S%+r0)
  52. 52. Searching Text
  53. 53. Searching Text Lucene Elastic SearchSphinx Solr
  54. 54. Searching Text Lucene Elastic SearchSphinx Postgres Solr
  55. 55. Full Text SearchCREATE TABLE posts ( id serial, title varchar(255), content text, tags varchar(255)[], post_text tsvector);CREATE INDEX posttext_gin ONposts USING GIN(post_text);CREATE TRIGGER update_posttextBEFORE INSERT OR UPDATE ON postsFOR EACH ROW EXECUTE PROCEDUREtsvector_update_trigger( ‘PostText’,‘english’,title, content, tags);
  56. 56. Full Text SearchCREATE TABLE posts ( id serial, title varchar(255), content text, tags varchar(255)[], post_text tsvector);CREATE INDEX posttext_gin ONposts USING GIN(post_text);CREATE TRIGGER update_posttextBEFORE INSERT OR UPDATE ON postsFOR EACH ROW EXECUTE PROCEDUREtsvector_update_trigger( ‘PostText’,‘english’,title, content, tags);
  57. 57. Djangofrom djorm_pgfulltext.models import SearchManagerfrom djorm_pgfulltext.fields import VectorFieldfrom django.db import modelsclass Posts(models.Model): title = models.CharField(max_length=200) content = models.TextField() search_index = VectorField() objects = SearchManager( fields = (title, content), config = pg_catalog.english, search_field = search_index, auto_update_search_field = True )
  58. 58. Djangofrom djorm_pgfulltext.models import SearchManagerfrom djorm_pgfulltext.fields import VectorFieldfrom django.db import modelsclass Posts(models.Model): title = models.CharField(max_length=200) content = models.TextField() search_index = VectorField() objects = SearchManager( fields = (title, content), config = pg_catalog.english, search_field = search_index, auto_update_search_field = True )
  59. 59. DjangoPost.objects.search("documentation & about")Post.objects.search("about | documentation")
  60. 60. DjangoPost.objects.search("documentation & about")Post.objects.search("about | documentation")
  61. 61. I#3%.
  62. 62. IndexesB-TreeGeneralized Inverted Index (GIN)Generalized Search Tree (GIST)K Nearest Neighbors (KNN)Space Partitioned GIST (SP-GIST)
  63. 63. IndexesWhich do I use?
  64. 64. IndexesB-TreeGeneralized Inverted Index (GIN)Generalized Search Tree (GIST)K Nearest Neighbors (KNN)Space Partitioned GIST (SP-GIST)
  65. 65. Generalized Inverted Index (GIN) Use with multiple values in 1 column Array/hStore
  66. 66. Generalized Search Tree (GIST) Full text search Shapes PostGIS
  67. 67. IndexesB-TreeGeneralized Inverted Index (GIN)Generalized Search Tree (GIST)K Nearest Neighbors (KNN)Space Partitioned GIST (SP-GIST)
  68. 68. G%!Sp+("+,
  69. 69. GeoDjango http://geodjango.org/https://speakerdeck.com/pyconslides/location-location- location
  70. 70. O#% M!r%
  71. 71. Connections
  72. 72. Connectionsdjango-postgrespooldjorm-ext-pooldjango-db-pool
  73. 73. django-postgrespoolimport dj_database_urlimport django_postgrespoolDATABASE = { default: dj_database_url.config() }DATABASES[default][ENGINE] = django_postgrespoolSOUTH_DATABASE_ADAPTERS = { default: south.db.postgresql_psycopg2}
  74. 74. django-postgrespoolimport dj_database_urlimport django_postgrespoolDATABASE = { default: dj_database_url.config() }DATABASES[default][ENGINE] = django_postgrespoolSOUTH_DATABASE_ADAPTERS = { default: south.db.postgresql_psycopg2}
  75. 75. Limitations?Django attempts to support as many featuresas possible on all database backends. However,not all database backends are alike, and we’vehad to make design decisions on whichfeatures to support and which assumptionswe can make safely.
  76. 76. P/($r%.Its great
  77. 77. D*+#$! ORMIts not so bad
  78. 78. 5+#1.!
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×