Omnibus database machine

Aleck Landgraf
Aleck LandgrafVP Software Engineering and Co-founder at Building Energy Inc.
omnibus database machine 
How to NoSQL in Postgres with Django
so much database 
I’m Aleck 
mainly: django + python + js 
github.com/alecklandgraf 
@aleck_landgraf 
aleck@buildingenergy.com 
I’m Gavin 
github.com/gmcquillan 
@gmcquillan
our use case 
people want to load their data… 
… and play with it: 
- django app for managing building data 
- unstructured data 
- relational data 
- orgs/perms/projects 
- utility meter ts data
we have unstructured data 
- lack of fixed schema 
- can create new fields without DB migration 
- 2,000 fields in the standard ontology 
- business logic (MCM) to map raw data to an 
ontology if possible 
- other logic to keep track of keys 
- different for each user or organization
we have relational data 
- everything we load relates to everything else 
in the system 
- buildings → organizations → permissions 
- buildings → utilities → meters → ts data 
- building v12010 → building v22011
unstructured and relational data 
- django can connect to mongo 
- django can connect to postgres 
- complex 
- is there something better?
PostgreSQL 9.2+ JSON type 
- native type for JSON (9.4+ jsonb) 
- GIN index, soon GiST index 
- supports search and lookup 
- easy to convert python dict to json 
- json is the de facto standard for web APIs
why not mongo * 
- MongoDB doesn’t seem to be more 
performant than PostgreSQL. 
- And you still get all of PostgreSQL’s goodies. 
- Larger documents will probably continue to 
favor PostgreSQL. 
- As will larger tables. 
*from Christophe Pettus’ talk at OSCON ‘13
did someone say django app? 
- are you ready to get beta 
- django-pgjson (version 0.2.0, 2014-09-13) 
- works best with dev postgres 9.4 and Django 
1.7+ 
- in beta development 
- also a kickstarter project to help build some of 
this out
setting up your django model 
from django.db import models 
from django_pgjson.fields import JsonField 
class Something(models.Model): 
name = models.CharField(max_length=32) 
data = JsonField()
querying the JSON data 
>>> Something.object.bulk_create([ 
... Something(data={"name": "foo", "tags": ["sad", "romantic"]}), 
... Something(data={"name": "bar", "tags": ["sad", "intelligent"]}) 
... ]) 
>>> Something.objects.filter(data__at_name="foo").count() 
1 
>>> Something.objects.filter(data__jcontains={"tags": ["sad"]}).count() 
2
what about order_by and distinct? 
- unsupported until PSQL 9.4 with jsonb 
- we wrote our own inside a custom queryset 
if order_by not in known_columns: 
qs = list(qs) 
qs.sort( 
key=lambda x: getattr(x, field).get(order_by), 
reverse=order_by_rev 
)
JSON value order_by in SQL 
SELECT id, NULLIF(extra_data->>'Total GHG Emissions (MtCO2e)', 
'')::float AS ghg_emissions from seed_buildingsnapshot WHERE 
extra_data->>'Total GHG Emissions (MtCO2e)' != '' ORDER BY 
NULLIF(extra_data->>'Total GHG Emissions (MtCO2e)', '')::float 
DESC LIMIT 10; 
Django order_by doesn’t 
work on ‘non-model-fields’
order_by 
- can also push it to Postgres with a PL script 
- http://hyperthese.net/post/sorting-json-fields-in- 
postgresql/ 
- jsonb support in Postgres 9.4 for this, but not 
for json
is this fast?
who is using this? 
With JSONB and other enhancements in 9.4 "we 
now have full document storage and awesome 
performance with little effort," explained Craig 
Kerstiens, a developer at Salesforce-backed 
Heroku, in a personal blog post.
further thoughts 
- Postgresql full text search with JSON? 
- joins
links 
- https://github.com/djangonauts/django-pgjson 
- http://www.postgresql.org/docs/9.4/static/functions-json.html 
- http://thebuild.com/presentations/pg-as-nosql-pgday-fosdem-2013.pdf 
- http://www.postgresql.org/docs/9.4/static/datatype-json.html 
- http://lwn.net/Articles/599705/ 
- http://simko.home.cern.ch/simko/postgresql-mongodb-json-select-speed. 
html
1 of 18

Recommended

MongoDB for Analytics by
MongoDB for AnalyticsMongoDB for Analytics
MongoDB for AnalyticsMongoDB
5K views67 slides
MongoDB: Intro & Application for Big Data by
MongoDB: Intro & Application  for Big DataMongoDB: Intro & Application  for Big Data
MongoDB: Intro & Application for Big DataTakahiro Inoue
4.9K views50 slides
Social Analytics with MongoDB by
Social Analytics with MongoDBSocial Analytics with MongoDB
Social Analytics with MongoDBPatrick Stokes
5.5K views28 slides
Intro to mongodb mongouk jun2010 by
Intro to mongodb mongouk jun2010Intro to mongodb mongouk jun2010
Intro to mongodb mongouk jun2010Skills Matter
821 views37 slides
Introduction to MongoDB by
Introduction to MongoDBIntroduction to MongoDB
Introduction to MongoDBNosh Petigara
801 views24 slides
Rethinkdb by
RethinkdbRethinkdb
RethinkdbAbhi Dey
878 views9 slides

More Related Content

What's hot

MongoDB - Ekino PHP by
MongoDB - Ekino PHPMongoDB - Ekino PHP
MongoDB - Ekino PHPFlorent DENIS
11.1K views36 slides
Andrew Dunstan 9.3 JSON Presentation @ Postgres Open 2013 by
Andrew Dunstan 9.3 JSON Presentation @ Postgres Open 2013Andrew Dunstan 9.3 JSON Presentation @ Postgres Open 2013
Andrew Dunstan 9.3 JSON Presentation @ Postgres Open 2013PostgresOpen
2.3K views40 slides
MongoDB.local DC 2018: Ch-Ch-Ch-Ch-Changes: Taking Your MongoDB Stitch Applic... by
MongoDB.local DC 2018: Ch-Ch-Ch-Ch-Changes: Taking Your MongoDB Stitch Applic...MongoDB.local DC 2018: Ch-Ch-Ch-Ch-Changes: Taking Your MongoDB Stitch Applic...
MongoDB.local DC 2018: Ch-Ch-Ch-Ch-Changes: Taking Your MongoDB Stitch Applic...MongoDB
188 views28 slides
Introduction to MongoDB at IGDTUW by
Introduction to MongoDB at IGDTUWIntroduction to MongoDB at IGDTUW
Introduction to MongoDB at IGDTUWAnkur Raina
155 views47 slides
Latinoware by
LatinowareLatinoware
Latinowarekchodorow
560 views47 slides
Sydney Python Presentation (Feb 2010) - Tracking Large Metallic Objects / Goo... by
Sydney Python Presentation (Feb 2010) - Tracking Large Metallic Objects / Goo...Sydney Python Presentation (Feb 2010) - Tracking Large Metallic Objects / Goo...
Sydney Python Presentation (Feb 2010) - Tracking Large Metallic Objects / Goo...Kelvin Nicholson
325 views13 slides

What's hot(20)

MongoDB - Ekino PHP by Florent DENIS
MongoDB - Ekino PHPMongoDB - Ekino PHP
MongoDB - Ekino PHP
Florent DENIS11.1K views
Andrew Dunstan 9.3 JSON Presentation @ Postgres Open 2013 by PostgresOpen
Andrew Dunstan 9.3 JSON Presentation @ Postgres Open 2013Andrew Dunstan 9.3 JSON Presentation @ Postgres Open 2013
Andrew Dunstan 9.3 JSON Presentation @ Postgres Open 2013
PostgresOpen2.3K views
MongoDB.local DC 2018: Ch-Ch-Ch-Ch-Changes: Taking Your MongoDB Stitch Applic... by MongoDB
MongoDB.local DC 2018: Ch-Ch-Ch-Ch-Changes: Taking Your MongoDB Stitch Applic...MongoDB.local DC 2018: Ch-Ch-Ch-Ch-Changes: Taking Your MongoDB Stitch Applic...
MongoDB.local DC 2018: Ch-Ch-Ch-Ch-Changes: Taking Your MongoDB Stitch Applic...
MongoDB188 views
Introduction to MongoDB at IGDTUW by Ankur Raina
Introduction to MongoDB at IGDTUWIntroduction to MongoDB at IGDTUW
Introduction to MongoDB at IGDTUW
Ankur Raina155 views
Latinoware by kchodorow
LatinowareLatinoware
Latinoware
kchodorow560 views
Sydney Python Presentation (Feb 2010) - Tracking Large Metallic Objects / Goo... by Kelvin Nicholson
Sydney Python Presentation (Feb 2010) - Tracking Large Metallic Objects / Goo...Sydney Python Presentation (Feb 2010) - Tracking Large Metallic Objects / Goo...
Sydney Python Presentation (Feb 2010) - Tracking Large Metallic Objects / Goo...
Kelvin Nicholson325 views
Back to Basics 2017: Mí primera aplicación MongoDB by MongoDB
Back to Basics 2017: Mí primera aplicación MongoDBBack to Basics 2017: Mí primera aplicación MongoDB
Back to Basics 2017: Mí primera aplicación MongoDB
MongoDB3K views
MongoDB - A Document NoSQL Database by Ruben Inoto Soto
MongoDB - A Document NoSQL DatabaseMongoDB - A Document NoSQL Database
MongoDB - A Document NoSQL Database
Ruben Inoto Soto2.3K views
Ebook Pembuatan Aplikasi Rental film 2012 by yantoit2011
Ebook Pembuatan Aplikasi Rental film 2012Ebook Pembuatan Aplikasi Rental film 2012
Ebook Pembuatan Aplikasi Rental film 2012
yantoit20111.6K views
Back to Basics Spanish 4 Introduction to sharding by MongoDB
Back to Basics Spanish 4 Introduction to shardingBack to Basics Spanish 4 Introduction to sharding
Back to Basics Spanish 4 Introduction to sharding
MongoDB1.1K views
2011 mongo FR - scaling with mongodb by antoinegirbal
2011 mongo FR - scaling with mongodb2011 mongo FR - scaling with mongodb
2011 mongo FR - scaling with mongodb
antoinegirbal702 views
Mongo db updatedocumentusecases by zarigatongy
Mongo db updatedocumentusecasesMongo db updatedocumentusecases
Mongo db updatedocumentusecases
zarigatongy126 views
Connecting and using PostgreSQL database with psycopg2 [Python 2.7] by Dinesh Neupane
Connecting and using PostgreSQL database with psycopg2 [Python 2.7]Connecting and using PostgreSQL database with psycopg2 [Python 2.7]
Connecting and using PostgreSQL database with psycopg2 [Python 2.7]
Dinesh Neupane624 views
MongoDB Days Silicon Valley: Winning the Dreamforce Hackathon with MongoDB by MongoDB
MongoDB Days Silicon Valley: Winning the Dreamforce Hackathon with MongoDBMongoDB Days Silicon Valley: Winning the Dreamforce Hackathon with MongoDB
MongoDB Days Silicon Valley: Winning the Dreamforce Hackathon with MongoDB
MongoDB1.7K views
mongoDB Performance by Moshe Kaplan
mongoDB PerformancemongoDB Performance
mongoDB Performance
Moshe Kaplan3.9K views
Do something in 5 with gas 3-simple invoicing app by Bruce McPherson
Do something in 5 with gas 3-simple invoicing appDo something in 5 with gas 3-simple invoicing app
Do something in 5 with gas 3-simple invoicing app
Bruce McPherson5.1K views
Back to Basics Webinar 2: Your First MongoDB Application by MongoDB
Back to Basics Webinar 2: Your First MongoDB ApplicationBack to Basics Webinar 2: Your First MongoDB Application
Back to Basics Webinar 2: Your First MongoDB Application
MongoDB4.4K views

Viewers also liked

Hex Series by
Hex SeriesHex Series
Hex SeriesTECHNOMED INDIA
337 views12 slides
Website evaluation by
Website evaluationWebsite evaluation
Website evaluationMrsCragle
398 views10 slides
Innovators Jumpstart 2015- McInnes Cooper (Business Structure Basics) by
Innovators Jumpstart 2015- McInnes Cooper (Business Structure Basics)Innovators Jumpstart 2015- McInnes Cooper (Business Structure Basics)
Innovators Jumpstart 2015- McInnes Cooper (Business Structure Basics)PlanetHatch
287 views10 slides
CEC by
CECCEC
CECWSCCEC
1.4K views13 slides
La vote ory 2 (no animations) by
La vote ory 2 (no animations)La vote ory 2 (no animations)
La vote ory 2 (no animations)DrBamboozler
249 views8 slides
2014 EMBEDDED TITLES FOR BE, ME by
2014 EMBEDDED TITLES FOR BE, ME2014 EMBEDDED TITLES FOR BE, ME
2014 EMBEDDED TITLES FOR BE, MEBlueChip Technology
393 views5 slides

Viewers also liked(20)

Website evaluation by MrsCragle
Website evaluationWebsite evaluation
Website evaluation
MrsCragle398 views
Innovators Jumpstart 2015- McInnes Cooper (Business Structure Basics) by PlanetHatch
Innovators Jumpstart 2015- McInnes Cooper (Business Structure Basics)Innovators Jumpstart 2015- McInnes Cooper (Business Structure Basics)
Innovators Jumpstart 2015- McInnes Cooper (Business Structure Basics)
PlanetHatch287 views
CEC by WSCCEC
CECCEC
CEC
WSCCEC1.4K views
La vote ory 2 (no animations) by DrBamboozler
La vote ory 2 (no animations)La vote ory 2 (no animations)
La vote ory 2 (no animations)
DrBamboozler249 views
BionicMe by Bionicme
BionicMeBionicMe
BionicMe
Bionicme367 views
Baltijas ceļa pasākumi ārvalstīs augustā un septembrī by Valdības māja
Baltijas ceļa pasākumi ārvalstīs augustā un septembrīBaltijas ceļa pasākumi ārvalstīs augustā un septembrī
Baltijas ceļa pasākumi ārvalstīs augustā un septembrī
Valdības māja228 views
27.2015.qđ.t tg by Mèo Hoang
27.2015.qđ.t tg27.2015.qđ.t tg
27.2015.qđ.t tg
Mèo Hoang352 views
About profession by Chris Chang
About professionAbout profession
About profession
Chris Chang354 views
Advanta Healthcare Partners Capabilities by Advanta
Advanta Healthcare Partners Capabilities Advanta Healthcare Partners Capabilities
Advanta Healthcare Partners Capabilities
Advanta429 views
Kts c2-dai so boole by Wang Ruan
Kts c2-dai so booleKts c2-dai so boole
Kts c2-dai so boole
Wang Ruan420 views
39.2015.tt.bnnptnt by Mèo Hoang
39.2015.tt.bnnptnt39.2015.tt.bnnptnt
39.2015.tt.bnnptnt
Mèo Hoang351 views
Epam BI - Near Realtime Marketing Support System by Dmitry Tolpeko
Epam BI - Near Realtime Marketing Support SystemEpam BI - Near Realtime Marketing Support System
Epam BI - Near Realtime Marketing Support System
Dmitry Tolpeko742 views
Corbett gardens cat hearing presentation 2015 by WSCCEC
Corbett gardens cat hearing presentation 2015Corbett gardens cat hearing presentation 2015
Corbett gardens cat hearing presentation 2015
WSCCEC1.2K views
Innovators Jumpstart 2015-Booking for Small Businesses by PlanetHatch
Innovators Jumpstart 2015-Booking for Small Businesses Innovators Jumpstart 2015-Booking for Small Businesses
Innovators Jumpstart 2015-Booking for Small Businesses
PlanetHatch193 views
46/2015/NĐ-CP về quản lý chất lượng công trình xây dựng by Mèo Hoang
46/2015/NĐ-CP về quản lý chất lượng công trình xây dựng46/2015/NĐ-CP về quản lý chất lượng công trình xây dựng
46/2015/NĐ-CP về quản lý chất lượng công trình xây dựng
Mèo Hoang545 views
Innovators Jumpstart 2015-NBIF (The Perfect Pitch) by PlanetHatch
Innovators Jumpstart 2015-NBIF (The Perfect Pitch)Innovators Jumpstart 2015-NBIF (The Perfect Pitch)
Innovators Jumpstart 2015-NBIF (The Perfect Pitch)
PlanetHatch319 views
淺談測試Part1 by Chris Chang
淺談測試Part1淺談測試Part1
淺談測試Part1
Chris Chang268 views

Similar to Omnibus database machine

20160407_GTC2016_PgSQL_In_Place by
20160407_GTC2016_PgSQL_In_Place20160407_GTC2016_PgSQL_In_Place
20160407_GTC2016_PgSQL_In_PlaceKohei KaiGai
4.4K views36 slides
Full-Stack Data Science: How to be a One-person Data Team by
Full-Stack Data Science: How to be a One-person Data TeamFull-Stack Data Science: How to be a One-person Data Team
Full-Stack Data Science: How to be a One-person Data TeamGreg Goltsov
2.1K views53 slides
Mongo db dla administratora by
Mongo db dla administratoraMongo db dla administratora
Mongo db dla administratoraŁukasz Jagiełło
551 views24 slides
Webscale PostgreSQL - JSONB and Horizontal Scaling Strategies by
Webscale PostgreSQL - JSONB and Horizontal Scaling StrategiesWebscale PostgreSQL - JSONB and Horizontal Scaling Strategies
Webscale PostgreSQL - JSONB and Horizontal Scaling StrategiesJonathan Katz
18K views66 slides
NoSQL meets Microservices - Michael Hackstein by
NoSQL meets Microservices -  Michael HacksteinNoSQL meets Microservices -  Michael Hackstein
NoSQL meets Microservices - Michael Hacksteindistributed matters
612 views38 slides
Gur1009 by
Gur1009Gur1009
Gur1009Cdiscount
5.5K views18 slides

Similar to Omnibus database machine(20)

20160407_GTC2016_PgSQL_In_Place by Kohei KaiGai
20160407_GTC2016_PgSQL_In_Place20160407_GTC2016_PgSQL_In_Place
20160407_GTC2016_PgSQL_In_Place
Kohei KaiGai4.4K views
Full-Stack Data Science: How to be a One-person Data Team by Greg Goltsov
Full-Stack Data Science: How to be a One-person Data TeamFull-Stack Data Science: How to be a One-person Data Team
Full-Stack Data Science: How to be a One-person Data Team
Greg Goltsov2.1K views
Webscale PostgreSQL - JSONB and Horizontal Scaling Strategies by Jonathan Katz
Webscale PostgreSQL - JSONB and Horizontal Scaling StrategiesWebscale PostgreSQL - JSONB and Horizontal Scaling Strategies
Webscale PostgreSQL - JSONB and Horizontal Scaling Strategies
Jonathan Katz18K views
Gur1009 by Cdiscount
Gur1009Gur1009
Gur1009
Cdiscount5.5K views
Get expertise with mongo db by Amit Thakkar
Get expertise with mongo dbGet expertise with mongo db
Get expertise with mongo db
Amit Thakkar1K views
Giovanni Lanzani – SQL & NoSQL databases for data driven applications - NoSQL... by NoSQLmatters
Giovanni Lanzani – SQL & NoSQL databases for data driven applications - NoSQL...Giovanni Lanzani – SQL & NoSQL databases for data driven applications - NoSQL...
Giovanni Lanzani – SQL & NoSQL databases for data driven applications - NoSQL...
NoSQLmatters1.3K views
Data Types/Structures in DivConq by eTimeline, LLC
Data Types/Structures in DivConqData Types/Structures in DivConq
Data Types/Structures in DivConq
eTimeline, LLC644 views
Working with JSON Data in PostgreSQL vs. MongoDB by ScaleGrid.io
Working with JSON Data in PostgreSQL vs. MongoDBWorking with JSON Data in PostgreSQL vs. MongoDB
Working with JSON Data in PostgreSQL vs. MongoDB
ScaleGrid.io830 views
Lessons Learned While Scaling Elasticsearch at Vinted by Dainius Jocas
Lessons Learned While Scaling Elasticsearch at VintedLessons Learned While Scaling Elasticsearch at Vinted
Lessons Learned While Scaling Elasticsearch at Vinted
Dainius Jocas189 views
Practical JSON in MySQL 5.7 and Beyond by Ike Walker
Practical JSON in MySQL 5.7 and BeyondPractical JSON in MySQL 5.7 and Beyond
Practical JSON in MySQL 5.7 and Beyond
Ike Walker2.2K views
hadoop by longhao
hadoophadoop
hadoop
longhao1.4K views
MongoDB dla administratora by 3camp
MongoDB dla administratora MongoDB dla administratora
MongoDB dla administratora
3camp688 views
2016 foss4 g track: developing and implementing spatial etl processes with... by GIS in the Rockies
2016 foss4 g track:  developing and implementing  spatial etl processes  with...2016 foss4 g track:  developing and implementing  spatial etl processes  with...
2016 foss4 g track: developing and implementing spatial etl processes with...
GIS in the Rockies268 views
Introduction To MongoDB by ElieHannouch
Introduction To MongoDBIntroduction To MongoDB
Introduction To MongoDB
ElieHannouch134 views

Recently uploaded

DSD-INT 2023 SFINCS Modelling in the U.S. Pacific Northwest - Parker by
DSD-INT 2023 SFINCS Modelling in the U.S. Pacific Northwest - ParkerDSD-INT 2023 SFINCS Modelling in the U.S. Pacific Northwest - Parker
DSD-INT 2023 SFINCS Modelling in the U.S. Pacific Northwest - ParkerDeltares
9 views16 slides
ict act 1.pptx by
ict act 1.pptxict act 1.pptx
ict act 1.pptxsanjaniarun08
13 views17 slides
Software testing company in India.pptx by
Software testing company in India.pptxSoftware testing company in India.pptx
Software testing company in India.pptxSakshiPatel82
7 views9 slides
DSD-INT 2023 Thermobaricity in 3D DCSM-FM - taking pressure into account in t... by
DSD-INT 2023 Thermobaricity in 3D DCSM-FM - taking pressure into account in t...DSD-INT 2023 Thermobaricity in 3D DCSM-FM - taking pressure into account in t...
DSD-INT 2023 Thermobaricity in 3D DCSM-FM - taking pressure into account in t...Deltares
9 views26 slides
DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ... by
DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ...DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ...
DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ...Deltares
9 views32 slides
DevsRank by
DevsRankDevsRank
DevsRankdevsrank786
11 views1 slide

Recently uploaded(20)

DSD-INT 2023 SFINCS Modelling in the U.S. Pacific Northwest - Parker by Deltares
DSD-INT 2023 SFINCS Modelling in the U.S. Pacific Northwest - ParkerDSD-INT 2023 SFINCS Modelling in the U.S. Pacific Northwest - Parker
DSD-INT 2023 SFINCS Modelling in the U.S. Pacific Northwest - Parker
Deltares9 views
Software testing company in India.pptx by SakshiPatel82
Software testing company in India.pptxSoftware testing company in India.pptx
Software testing company in India.pptx
SakshiPatel827 views
DSD-INT 2023 Thermobaricity in 3D DCSM-FM - taking pressure into account in t... by Deltares
DSD-INT 2023 Thermobaricity in 3D DCSM-FM - taking pressure into account in t...DSD-INT 2023 Thermobaricity in 3D DCSM-FM - taking pressure into account in t...
DSD-INT 2023 Thermobaricity in 3D DCSM-FM - taking pressure into account in t...
Deltares9 views
DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ... by Deltares
DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ...DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ...
DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ...
Deltares9 views
MariaDB stored procedures and why they should be improved by Federico Razzoli
MariaDB stored procedures and why they should be improvedMariaDB stored procedures and why they should be improved
MariaDB stored procedures and why they should be improved
Mark Simpson - UKOUG23 - Refactoring Monolithic Oracle Database Applications ... by marksimpsongw
Mark Simpson - UKOUG23 - Refactoring Monolithic Oracle Database Applications ...Mark Simpson - UKOUG23 - Refactoring Monolithic Oracle Database Applications ...
Mark Simpson - UKOUG23 - Refactoring Monolithic Oracle Database Applications ...
marksimpsongw76 views
Unmasking the Dark Art of Vectored Exception Handling: Bypassing XDR and EDR ... by Donato Onofri
Unmasking the Dark Art of Vectored Exception Handling: Bypassing XDR and EDR ...Unmasking the Dark Art of Vectored Exception Handling: Bypassing XDR and EDR ...
Unmasking the Dark Art of Vectored Exception Handling: Bypassing XDR and EDR ...
Donato Onofri711 views
BushraDBR: An Automatic Approach to Retrieving Duplicate Bug Reports by Ra'Fat Al-Msie'deen
BushraDBR: An Automatic Approach to Retrieving Duplicate Bug ReportsBushraDBR: An Automatic Approach to Retrieving Duplicate Bug Reports
BushraDBR: An Automatic Approach to Retrieving Duplicate Bug Reports
A first look at MariaDB 11.x features and ideas on how to use them by Federico Razzoli
A first look at MariaDB 11.x features and ideas on how to use themA first look at MariaDB 11.x features and ideas on how to use them
A first look at MariaDB 11.x features and ideas on how to use them
Federico Razzoli45 views
Consulting for Data Monetization Maximizing the Profit Potential of Your Data... by Flexsin
Consulting for Data Monetization Maximizing the Profit Potential of Your Data...Consulting for Data Monetization Maximizing the Profit Potential of Your Data...
Consulting for Data Monetization Maximizing the Profit Potential of Your Data...
Flexsin 15 views
Tridens DevOps by Tridens
Tridens DevOpsTridens DevOps
Tridens DevOps
Tridens9 views
Upgrading Incident Management with Icinga - Icinga Camp Milan 2023 by Icinga
Upgrading Incident Management with Icinga - Icinga Camp Milan 2023Upgrading Incident Management with Icinga - Icinga Camp Milan 2023
Upgrading Incident Management with Icinga - Icinga Camp Milan 2023
Icinga38 views
2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx by animuscrm
2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx
2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx
animuscrm13 views
DSD-INT 2023 HydroMT model building and river-coast coupling in Python - Bove... by Deltares
DSD-INT 2023 HydroMT model building and river-coast coupling in Python - Bove...DSD-INT 2023 HydroMT model building and river-coast coupling in Python - Bove...
DSD-INT 2023 HydroMT model building and river-coast coupling in Python - Bove...
Deltares17 views
Software evolution understanding: Automatic extraction of software identifier... by Ra'Fat Al-Msie'deen
Software evolution understanding: Automatic extraction of software identifier...Software evolution understanding: Automatic extraction of software identifier...
Software evolution understanding: Automatic extraction of software identifier...
DSD-INT 2023 Delft3D FM Suite 2024.01 2D3D - New features + Improvements - Ge... by Deltares
DSD-INT 2023 Delft3D FM Suite 2024.01 2D3D - New features + Improvements - Ge...DSD-INT 2023 Delft3D FM Suite 2024.01 2D3D - New features + Improvements - Ge...
DSD-INT 2023 Delft3D FM Suite 2024.01 2D3D - New features + Improvements - Ge...
Deltares16 views

Omnibus database machine

  • 1. omnibus database machine How to NoSQL in Postgres with Django
  • 2. so much database I’m Aleck mainly: django + python + js github.com/alecklandgraf @aleck_landgraf aleck@buildingenergy.com I’m Gavin github.com/gmcquillan @gmcquillan
  • 3. our use case people want to load their data… … and play with it: - django app for managing building data - unstructured data - relational data - orgs/perms/projects - utility meter ts data
  • 4. we have unstructured data - lack of fixed schema - can create new fields without DB migration - 2,000 fields in the standard ontology - business logic (MCM) to map raw data to an ontology if possible - other logic to keep track of keys - different for each user or organization
  • 5. we have relational data - everything we load relates to everything else in the system - buildings → organizations → permissions - buildings → utilities → meters → ts data - building v12010 → building v22011
  • 6. unstructured and relational data - django can connect to mongo - django can connect to postgres - complex - is there something better?
  • 7. PostgreSQL 9.2+ JSON type - native type for JSON (9.4+ jsonb) - GIN index, soon GiST index - supports search and lookup - easy to convert python dict to json - json is the de facto standard for web APIs
  • 8. why not mongo * - MongoDB doesn’t seem to be more performant than PostgreSQL. - And you still get all of PostgreSQL’s goodies. - Larger documents will probably continue to favor PostgreSQL. - As will larger tables. *from Christophe Pettus’ talk at OSCON ‘13
  • 9. did someone say django app? - are you ready to get beta - django-pgjson (version 0.2.0, 2014-09-13) - works best with dev postgres 9.4 and Django 1.7+ - in beta development - also a kickstarter project to help build some of this out
  • 10. setting up your django model from django.db import models from django_pgjson.fields import JsonField class Something(models.Model): name = models.CharField(max_length=32) data = JsonField()
  • 11. querying the JSON data >>> Something.object.bulk_create([ ... Something(data={"name": "foo", "tags": ["sad", "romantic"]}), ... Something(data={"name": "bar", "tags": ["sad", "intelligent"]}) ... ]) >>> Something.objects.filter(data__at_name="foo").count() 1 >>> Something.objects.filter(data__jcontains={"tags": ["sad"]}).count() 2
  • 12. what about order_by and distinct? - unsupported until PSQL 9.4 with jsonb - we wrote our own inside a custom queryset if order_by not in known_columns: qs = list(qs) qs.sort( key=lambda x: getattr(x, field).get(order_by), reverse=order_by_rev )
  • 13. JSON value order_by in SQL SELECT id, NULLIF(extra_data->>'Total GHG Emissions (MtCO2e)', '')::float AS ghg_emissions from seed_buildingsnapshot WHERE extra_data->>'Total GHG Emissions (MtCO2e)' != '' ORDER BY NULLIF(extra_data->>'Total GHG Emissions (MtCO2e)', '')::float DESC LIMIT 10; Django order_by doesn’t work on ‘non-model-fields’
  • 14. order_by - can also push it to Postgres with a PL script - http://hyperthese.net/post/sorting-json-fields-in- postgresql/ - jsonb support in Postgres 9.4 for this, but not for json
  • 16. who is using this? With JSONB and other enhancements in 9.4 "we now have full document storage and awesome performance with little effort," explained Craig Kerstiens, a developer at Salesforce-backed Heroku, in a personal blog post.
  • 17. further thoughts - Postgresql full text search with JSON? - joins
  • 18. links - https://github.com/djangonauts/django-pgjson - http://www.postgresql.org/docs/9.4/static/functions-json.html - http://thebuild.com/presentations/pg-as-nosql-pgday-fosdem-2013.pdf - http://www.postgresql.org/docs/9.4/static/datatype-json.html - http://lwn.net/Articles/599705/ - http://simko.home.cern.ch/simko/postgresql-mongodb-json-select-speed. html