IMPORTANT: The primary content for this presentation is in the "Notes" tab below.
Django is undoubtedly one of the most powerful web frameworks in existence today, and one of those sources of its power is its ORM. But, despite great documentation, it can be confusing as hell. And I’ve come into numerous projects which were plagued by poor usages of the ORM, causing headaches and heartaches for the developers, and often greatly degraded performance. So here in this presentation, I’d like to pass on some of the things I’ve learned over the many years I’ve been using Django and its ORM.
DevOps @Scale (Greek Tragedy in 3 Acts) as it was presented at DevOps Detroit...Baruch Sadogursky
As in a good Greek Tragedy, scaling devops to big teams has 3 stages and usually end badly. In this play (it’s more than a talk!) we’ll present you with Pentagon Inc, and their way to scaling devops from a team of 3 engineers to a team of 100 (spoiler – it’s painful!)
DevOps @Scale (Greek Tragedy in 3 Acts) as it was presented at Oracle Code SF...Baruch Sadogursky
As in a good Greek Tragedy, scaling devops to big teams has 3 stages and usually end badly. In this play (it’s more than a talk!) we’ll present you with Pentagon Inc, and their way to scaling devops from a team of 3 engineers to a team of 100 (spoiler – it’s painful!)
Patterns and antipatterns in Docker image lifecycle as was presented at Oracl...Baruch Sadogursky
While Docker has enabled an unprecedented velocity of software production, it is all too easy to spin out of control. A promotion-based model is required to control and track the flow of Docker images as much as it is required for a traditional software development lifecycle. New tools often introduce new paradigms. We will examine the patterns and the antipatterns for Docker image management, and what impact the new tools have on the battle-proven paradigms of the software development lifecycle.
DevOps @Scale (Greek Tragedy in 3 Acts) as it was presented at DevOps Detroit...Baruch Sadogursky
As in a good Greek Tragedy, scaling devops to big teams has 3 stages and usually end badly. In this play (it’s more than a talk!) we’ll present you with Pentagon Inc, and their way to scaling devops from a team of 3 engineers to a team of 100 (spoiler – it’s painful!)
DevOps @Scale (Greek Tragedy in 3 Acts) as it was presented at Oracle Code SF...Baruch Sadogursky
As in a good Greek Tragedy, scaling devops to big teams has 3 stages and usually end badly. In this play (it’s more than a talk!) we’ll present you with Pentagon Inc, and their way to scaling devops from a team of 3 engineers to a team of 100 (spoiler – it’s painful!)
Patterns and antipatterns in Docker image lifecycle as was presented at Oracl...Baruch Sadogursky
While Docker has enabled an unprecedented velocity of software production, it is all too easy to spin out of control. A promotion-based model is required to control and track the flow of Docker images as much as it is required for a traditional software development lifecycle. New tools often introduce new paradigms. We will examine the patterns and the antipatterns for Docker image management, and what impact the new tools have on the battle-proven paradigms of the software development lifecycle.
5 best practices for (web/ software) development (2010)Erwin Elling
Some of the best practices we've acquired while developing for the web! A presentation for students Communication and Multimedia Design at Noordelijke Hogeschool Leeuwarden in their Discover Web2.0 lecture series.
Two Scoops of Django is a book series written by Daniel & Audrey. This book introduces Django developer with various tips, tricks patterns, code snippets & techniques. I have been reading these books, and each has valuable advice for people working with Django in production.
Just to highlight some of the key insights from this book:
Deployment - Platform as a Service, development with Docker
Settings and requirement files
Coding style
Model best practices
Continuous Integrations
Security best practices
and more
The talk will cover top 10 learnings from the book and from my perspective with examples.
The aim of this talk is the bring out best practices for people working with Django in production.
Gonchik Tsymzhitov will share stories of improving the performance of DC (+Server) installations.
His stories will start from simple examples to complex ones where it is necessary to trace HTTP requests to the DBMS server.
At the end of the report, you will receive an approximate range of steps to improve your installation.
Also, the speaker will be happy to discuss the Atlassian ecosystem and Observability tools (such as NewRelic, glowroot etc.,) as they allow you to practically proactively respond to possible incidents.
Modernisation of legacy PHP applications using Symfony2 - PHP Northeast Confe...Fabrice Bernhard
PHP and its community has evolved really fast in the last few years to allow for professional architectures and solutions. However, there are thousands of existing PHP applications which have not evolved in the meantime and are now crippled and unmaintainable because of that. These applications represent a real threat to the competitiveness of the business that relies on them.
The best approach in terms of business to solve this problem is progressive rewrite. Symfony2 and its modular architecture make it possible. This talk covers the main technical difficulties of the progressive approach when rewriting legacy PHP applications, and the corresponding solutions, some of which rely on the modularity of Symfony2.
A small presentation by me at the Barcamp Vienna in Sept. 2007 where I tried to give a short overview about some of the features I really love about Django :D
Из презентации вы узнаете:
про большинство утилит из арсенала Go, предназначенных для оптимизации производительности;
— как и когда их (утилиты) использовать, а также мы посмотрим как они устроены внутри;
— про применимость linux утилиты perf для оптимизации программ на Go.
Кроме того, устроим небольшой crash course, в рамках которого поэтапно соптимизируем несколько небольших программ на Go с использованием вышеперечисленных утилит.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
More Related Content
Similar to The Django ORM: Common patterns, gotchas, and neat tricks
5 best practices for (web/ software) development (2010)Erwin Elling
Some of the best practices we've acquired while developing for the web! A presentation for students Communication and Multimedia Design at Noordelijke Hogeschool Leeuwarden in their Discover Web2.0 lecture series.
Two Scoops of Django is a book series written by Daniel & Audrey. This book introduces Django developer with various tips, tricks patterns, code snippets & techniques. I have been reading these books, and each has valuable advice for people working with Django in production.
Just to highlight some of the key insights from this book:
Deployment - Platform as a Service, development with Docker
Settings and requirement files
Coding style
Model best practices
Continuous Integrations
Security best practices
and more
The talk will cover top 10 learnings from the book and from my perspective with examples.
The aim of this talk is the bring out best practices for people working with Django in production.
Gonchik Tsymzhitov will share stories of improving the performance of DC (+Server) installations.
His stories will start from simple examples to complex ones where it is necessary to trace HTTP requests to the DBMS server.
At the end of the report, you will receive an approximate range of steps to improve your installation.
Also, the speaker will be happy to discuss the Atlassian ecosystem and Observability tools (such as NewRelic, glowroot etc.,) as they allow you to practically proactively respond to possible incidents.
Modernisation of legacy PHP applications using Symfony2 - PHP Northeast Confe...Fabrice Bernhard
PHP and its community has evolved really fast in the last few years to allow for professional architectures and solutions. However, there are thousands of existing PHP applications which have not evolved in the meantime and are now crippled and unmaintainable because of that. These applications represent a real threat to the competitiveness of the business that relies on them.
The best approach in terms of business to solve this problem is progressive rewrite. Symfony2 and its modular architecture make it possible. This talk covers the main technical difficulties of the progressive approach when rewriting legacy PHP applications, and the corresponding solutions, some of which rely on the modularity of Symfony2.
A small presentation by me at the Barcamp Vienna in Sept. 2007 where I tried to give a short overview about some of the features I really love about Django :D
Из презентации вы узнаете:
про большинство утилит из арсенала Go, предназначенных для оптимизации производительности;
— как и когда их (утилиты) использовать, а также мы посмотрим как они устроены внутри;
— про применимость linux утилиты perf для оптимизации программ на Go.
Кроме того, устроим небольшой crash course, в рамках которого поэтапно соптимизируем несколько небольших программ на Go с использованием вышеперечисленных утилит.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
46. The perfect beer...
They want…
● A stout which was produced between 3 and 5 years ago, OR
● An IPA which was produced less than 1 month ago with at least 90 IBUs, OR
● A sour beer which took at least 3 years to ferment and condition, OR
● A Belgian style beer produced in Belgium, at a brewery at least 100 years old
With a rating of at least 4.5/5.0
46
Django is undoubtedly one of the most powerful web frameworks in existence today, and one of those sources of its power is its ORM. But, despite great documentation, it can be confusing as hell. And I’ve come into numerous projects which were plagued by poor usages of the ORM, causing headaches and heartaches for the developers, and often greatly degraded performance. So here in this presentation, I’d like to pass on some of the things I’ve learned over the many years I’ve been using Django and its ORM.
And in order to do that, I will be using beer! Well, at least, building an application about beer. Because I like beer.
So the first obvious question here, for those who might be new to Django: What the hell is an ORM? And that’s a very good question. The answer is, an Object-Relational Mapper! Totally self explanatory, right? No? Okay so the ORM is the way that Django allows you to communicate with a database. Really any database, in fact. Using Django’s ORM, you can communicate with a SQLite database, MySQL, PostgreSQL, Oracle, even a Microsoft SQL Server database, all without having to know how to connect to it, the intricacies of its libraries, or the differences in how it handles the SQL language. You don’t have to know or deal with any of that, because Django’s ORM knows and does all of that for you. The other thing which the ORM affords you is that you no longer have to learn another language. Talking to databases normally requires learning the inticate and arcane SQL language, but no more! The Django ORM understands it for you, so you don’t have to know a single thing about it.
I’m sorry to disappoint, but working with Django doesn’t automatically mean you get to consort with the likes of Cindy Crawford and… random handsome men.
What we want at Django models! In a database, each different type of data is stored in a table. In Django, those tables are represented in what we call “models”. One model equals one table in the database. And it all starts with this one simple import!
The exact details of how you draw those lines and split up those types of data is a topic for an entirely separate discussion, but hopefully you’ll get the general idea as we continue.
Now, let’s make our first model. As mentioned before, we have to import the models from Django. Then to start with, we’re going to create a table to store breweries. With those breweries, we are going to store a name, description, and location. Pretty basic stuff. But the astute among you may notice a couple of oddities here. The first thing I want to cover is the name of this model.
Breweries. Now, this is a perfectly legitimate name for a model and corresponding database table. However, it flies in the face of convention and will make for some slightly awkward code. Specifically, the issue is the plurality: “breweries”.
The more common convention is to use a singular form. Now, this isn’t a hard and fast rule, but more of a common convention, both in Django development and in database design in general. It’s really just a defacto pattern followed by the community at large, and generally considered a best practice.
Now, the other thing here is far less obvious, because it is something which is not here: a primary key for this model. In database design, you use a primary key to identify each record individually, and it’s typically an auto-incrementing number. Of course, we could just make the name field unique and identify breweries by that field… but what if two breweries come about with the same name? It has been known to happen!
So why, then, did I not add a primary key field to this model? Well mostly because I don’t have to, and because I’m lazy. This is one of the magical pieces of Django. If you don’t specifically add a field with an attribute of primary_key=True, it will add a field for you, behind the scenes, named “id”. You don’t even have to think about it. All you have to think about is the fields which actually matter to you: the ones describing your data. So join me in being lazy, and don’t bother adding primary key fields when you really don’t have to.
Now, as we continue, we’re going to need to add some more models. We want a list of styles… because there are a crazy number of beer styles, which we want to track. And of course, we want to store the list of beers themselves. Cool. Good. Moving right along.
Yeah, that was too easy.
This here Beer model has got a few problems. Specifically, those ForeignKeys.
Let’s start with this weird bit here, the “on_delete”. This might be a new thing to you, because you haven’t necessarily had to deal with it before. But coming up in Django 2.0, it’s actually going to be a required argument. So it’s good to know. So what’s the deal with it? Well, first I will need to describe a bit about the ForeignKey, for those who are not aware. What the ForeignKey does is points to a single instance of another model. What that means is, let’s we have one single instance of beer, let’s say it’s one called “Kilt Lifter”. That beer can be associated with one single style, so let’s say it’s associated with the style of “Scottish Ale”. And then that beer is produced by one particular brewery, so in this case, we would associate that beer with the brewery named “Four Peaks”. Now where the on_delete comes into play is if you try to delete one of those associated records. Let’s say, for whatever reason, somebody tries to delete the Four Peaks brewery from the database. Well, this beer happens to be linked to that brewery, and deleting that brewery could cause havoc in our system, so we need to tell that beer what to do. And in this case, “do nothing” sounds like a logical thing to do. Right? Not exactly. “DO_NOTHING” actually means that Django will not try to interfere in the deletion, which will cause problems when it gets to the database. You see, the database wants to maintain what’s known as “referential integrity”, meaning that this beer record can’t point to a brewery record which has already been deleted. So then what happens when you go in and delete the brewery? BOOM! Big error. Bad things. Moral of the story, don’t use “do nothing” unless you’re a very advanced user and have a contingency plan. So what should we do instead?
There are a number of options available to you, and I have provided a link here to Django’s wonderful documentation to explain those, but in this case, we are going to use two different options. For the style, we are going to use “protect”. What this means is that if I have my Kilt Lifter in the database, and it’s linked as a Scottish Ale, then I can’t delete the style of Scottish Ale. As long as at least one reference exists to that style, Django won’t allow it. Cool! So what about the “cascade” on brewery? That one means that if I delete the Four Peaks brewery, then all beers associated with that brewery will be deleted along with it. Maybe not the best choice for this application, but worth showing here. So now that those are fixed, let’s move on!
Actually, those foreign keys still have a bit of an issue.
Well what could be wrong now? It all looks good. Except, this is where more Django magic comes into play.
These field names are a little off. In the database, these fields contain the primary key of another record; its “id”, and these are in fact the field names we want in the database. But the thing is, thanks to Django’s ORM, you don’t deal directly with those foreign keys. Instead, you deal with instances of models. Remember when I said that you don’t have to worry about or deal with a model’s primary key? I really meant that. This is another example of that. So what should this model look like instead?
Remove the “_id” and we’re good to go! And in fact, there’s even more magic going on behind the scenes here. In the database, that “style” and “brewery” will be stored in fields named, respectively, “style_id”, and “brewery_id”. And an instance of this model will even still have attributes named “style_id” and “brewery_id”. But you don’t ever have to or want to deal with those. Because these “style” and “brewery” attributes will give you direct access to those other models, for no extra work! What I mean by that is if I have an instance of the Beer model, I can reference “beer.style.name”, and it will give me that value, instead of having to look up the style by the ID which is stored in the beer’s record. Fancy stuff! As an additional bonus, this makes things such as creating new instance ever so slightly easier.
This is an example of how you would create a new brewery, style, and beer. It all looks very much above board, and especially to somebody with a background in databases, it makes absolute sense.
But as I just mentioned, you never actually have to deal with those “_id” fields, or referring to “.id” on an instance. Django does all of that automagically for you!
So all you have to do is remove all of those “id” mentions, and Django handles everything for you! It’s a very small thing, but it’s really just one less thing which you have to worry about. Django will handle all the actual details of these relationships for you.
Now for the last of our models, we need to list out venues. You have to know where you can get this wonderful beer, right? Of course! This is a simple enough model, but with two new things. You may notice a slight difference with how this model was built, with all of these big bold “type” things up top. What this is doing is building a list of possible selections for the “venue_type” field. The first 3 values are what’s stored in the database, then the following are what would be presented to a user on a webpage. So why did I build it like this, instead of having a separate model like I did with beer styles? Well, for a couple of reasons. The main one is that I only want to support a very small list of venue types, and they are unlikely to change frequently, so it doesn’t make sense to incur more database overhead for them. The second is that I might need the application to behave differently based on the venue type: I might offer different features, display different information, etc, depending on what type of venue it is. So if the code depends on the values, you ALWAYS want a strict list of choices, such as I’ve done here. And then the flip side of that is that beer styles are something which are constantly changing, and different styles are unlikely to require different application behavior. Those two things combined make style a great candidate for a lookup table.
One other thing I’ve done differently here is using this ManyToManyField. This allows for a different kind of relationship than a regular old ForeignKey. This means that one venue can have many beers, and one beer can be at many venues. To manage this, Django actually creates another database table in the background to manage the relationship. But that’s not something you have to worry about. It’s all done magically.
Then finally, I also added a “related_name” attribute to that ManyToManyField. This is something which you should get in the habit of using on all relationship fields, be they ForeignKeys or ManyToManyFields. What this does is sets up a friendly name for what Django calls a reverse relation. That means that, with this one field, I can have an instance of a venue and call “venue.beers.all”, using the name of the field here, or I can have an instance of a beer and call “beer.venues.all”, using that related_name on the field here.
Alright! So you’ve built your models, time has passed, you’ve worked on building up your application, you have some data and some views doing, and then…. It slows down. And you come to realize that you’re querying the database, a LOT. I mean, a lot.
This is a relatively small amount of data here. It’s 4 venues, with a total of 16 beers available between them, and this page is calling out to the database 43 different times. That doesn’t seem very optimal. Imagine if you had 100 venues, each with 20 beers on taps. Something tells me that’s just simply not going to perform well.
And for those who are wondering, that handy little toolbar on the right handle side is django-debug-toolbar. It’s one of the best things you can add to your toolbelt while building a Django application, even if only for detecting query problems like this one. I’ve included a link here to the docs so that you can get started with it!
So now that we’ve identified an issue, we need to figure out A) Why is it happening? And B) How do we fix it?
The first place we have to look is at our template. This is what’s generating the page we’re looking at. And at first glance, it looks pretty innocuous. All we’re doing is displaying names, links, and a count. What could be wrong with that?
Well the problem happens to be with how Django handles querysets, and when it pulls data out of the database. Actually, to be clear, this isn’t a problem with Django; Django handles this very elegantly and very efficiently. The problem is with this code just hoping that Django will be a little more magical than it is.
So, what are the problems here, and how do we solve them?
Let’s start from the top! This seemingly harmless statement right here is actually generating an extra SQL query for every venue in this listing! But wait, you might be thinking, it’s using the .count() method on the queryset, which the Django docs say is the best way to count the objects in a queryset! And this is true. You should definitely prefer using .count() over, for instance, using the built-in len() method to check the length of a queryset. It will be far more efficient in most cases. The problem here is that we’re doing it inside of a loop. And every time this method is called, it’s executing a SQL query to count how many results would be returned. And this now brings us to our first optimization, and first advanced usage of the ORM.
For this next part, we’re going to dive into the view where we generate this page. As you can see, right now, it’s very simple. Nothing more than two lines telling Django, in essence, that we want to get a list of venue instances. But this isn’t quite enough for all of the data we’re trying to load. We need to tell Django to be a little smarter about how it retrieves the data. And since we’re starting with that count, we’re going to first look at annotations!
So now, instead of our simple 2-liner telling Django what model to use, we have an explicit queryset! Now this syntax might seem a little weird and arcane. And it is, if you aren’t familiar with a whole lot of related concepts. But what it boils down to is this: The .annotate() method on a queryset is used for adding fields to every instance that queryset returns. In most cases, you are trying to add calculations of some sort to every field. And that’s exactly what we’re doing here. This code says: For every Venue in the database, add a field named “beer_count”, which is a count of the beers available at that venue. And again, I have provided a link to the documentation so that you can get more information on the topic later.
Now that the view has been updated, we just need to update the template to use that new field, as shown here, and then ….
We’ve gone from 43 queries, down to 39! Okay, so it’s not a huge change, but it’s a start. And we can do better!
The next thing we’re going to look at is the beers themselves. If you remember from when we constructed the models, the beers are a ManyToManyField, meaning that they’re just a relation to another model. What this means is that, every time we try to get the beers associated with a venue, we’re executing another SQL query to traverse that relationship. And since this, also, is happening inside a loop, that is another query per venue. So how can we prevent that from happening? For this, we are going to look into prefetching.
So what is prefetching, and how does it work? Well, the docs to which I have linked will again be your best source of knowledge. But to sum them up, prefetching is how you preload and cache related objects. Now keep in mind, this is only useful and usable for ManyToManyFields, or the reverse relation of a foreign key. So in our models, a venues beers are a perfect use case, as you see here. But a beer’s style, for example, would not. And we’ll get to that one later. But for now, just know that we are caching all of the beers related to all of the venues on this page. So what does this extra bit gain us?
3 fewer queries! Wait, 3? But didn’t that last optimization eliminate 4, one per venue? Why yes. Yes it did. But prefetch_related is a little different. Instead of consolidating into a single query, this tells Django to execute one additional query, wherein it performs its caching of all the related results. So this is good progress, we’ve gotten rid of 7 queries. But we can do better. Before we do, though, we’re going to sidetrack to a different view for a moment.
Here we have the beer list. And, as you can see, this much simpler list has a similar issue. For displaying 7 beers, it’s issuing 17 queries. And that just seems ridiculous, right? Of course it does. So let’s tweak this one really quick.
Here we have our template, with absolutely nothing we haven’t seen before. I actually copied this from the venues list. So what are we looking at here?
We’re looking at these related objects. And remember what happens when you look at related objects in a loop? Django executes more SQL queries, that’s what! We had better do something about that then. Now these two happen to be ForeignKeys, unlike what we’ve worked with before, meaning that each beer has only one brewery and one style. This is an important fact in constructing our optimization.
Here we see a very simple view, just like the venue list when we started. Except this one is for listing beers. And again, we need to change this view from using a simple model, to a slightly more explicit queryset.
The difference this time is that we’re using the “select_related” method of the queryset. This is because these are ForeignKeys, meaning that we will be point to a single related object. What this does is consolidates this all into a single query, meaning that looking for these related objects inside of our loop will no longer execute extra SQL queries! So what’s the result of this?
Only 3 queries, from just one small change! Excellent! Clearly, this select_related() thing is pretty powerful. So now that we’ve got that little bit of information, let’s apply that to our venue list! Maybe we can cut those queries down some more!
What we’re trying to accomplish here is the same as we did with the beer list. We want to issue a select_related() for our beers. But wait, we’re doing a prefetch_related() just to get the beers. So how do we select_related() on that? This is getting confusing! Well, yes, it is getting a little confusing, and I will admit that this next part may take a little effort to fully digest.
So I’ll provide another documentation link here so that you can read up more in Django’s ever wonderful documentation! What I’ve introduced here is a Prefetch object, which allows us to control the query used to retrieve and cache those related objects. And if that queryset looks familiar, it should. That’s exactly the same query we just wrote up for the beer list in order to retrieve all of the related objects in one query! And yes, I realize I’m glazing over this bit rather fast, but the documentation truly can do a better job of explaining exactly how this works. What I am to show you here is what can be achieved, given the right tools. And here’s what I mean.
4 queries, down from 43. All from just a bit of prefetching in a few short lines of code. And the nicer thing about that query count is that it’s not going to increase, no matter how many more venues, beers, breweries, or styles are added to the system!
Alright, now thanks to the tools you’ve learned so far, you’ve got a great functional and efficient site.
Then the hipsters and beer snobs arrive. And they turn their noses up at your list of beer styles. They don’t want to just classify beers as an “IPA”. They want their English IPAs, West Coast, East Coast, Dry Hopped, Wet Hopped, Single, Double, etc etc. And you realize that you need a way to categorize sub-styles.
So you create a SubStyle model. Because that’s clearly the right thing to do. Right? Yes? You have Styles, and then you have SubStyles.
Well, actually no. This leads to a big mess. Not only is it another unnecessary table, but you then have data linking problems. Does a beer now get linked to a style, a substyle, both? No, this isn’t what you want. There has to be a better way to do this.
And there is! You can create a self-referential, or recursive, ForeignKey. This allows a model to create a relationship with itself. In this way, you could create an entire tree of styles, with infinite depth and breadth. And the hipsters and beer snobs of the world will rejoice. But there’s a problem. Now that you’ve made all of them happy, and they’re using their site more… they’re starting to complain again. They want you to help them find the perfect beer, and the criteria for that are not simple.
The criteria for the “perfect beer” can be pretty specific, and pretty crazy. But you have to cater to your crowd. So you want to build a page to display some of the most optimal beers they can find. But how the hell do you do THIS in an efficient manner, without issuing all kinds of queries to the database and then a bunch of server-side processing?
This is where we introduce the power of Q!
Oops no sorry. Wrong Q. But similarly powerful!
Holy cow, is that a wall of text. And I’m not expecting you to be able to digest this entire thing. It’s a lot. It even confused me a bit when I was writing it. Damn those hipsters. I want to point a few things out here though.
The first thing is all of these joining operators. If you’ve done any work with bit math, you may recognize these. What these do is let you control how to parts of a query are combined: with either an “OR” using the pipe operator, or an “AND” using the ampersand operator. And this is actually the only way you can use an “OR” query, is by using Q objects like this.
Another important thing is the way we are constructing these. We don’t have to just build a single queryset filter statement for this. We can actually split it out into many statements, and combine those statements, to help keep our code more readable and understandable. And this is very important. Always remember that code be written only once, but read many times. This also makes the individual pieces of the query reusable, should you want to build multiple different queries out of the same building blocks.
And of course, because no self respecting beer snob wants to drink something from a particular big-name brewery who has been buying up all the little ones, we need the negation operator, for a “NOT” query. This allows us to exclude a set of records from our results.
And finally, one thing you may have noticed is that I’m not actually doing a filter here; I’m not actually operating on a queryset. I’m simply building up a query. This is due to the way Django does lazy evaluation of queries. It will never actually generate the SQL and communicate with the database until it really needs to, at the very last moment.
And in fact even when I do a filter on my model, it still doesn’t execute a query! Django really will be as lazy as it can, and that’s a very good thing. This can drastically cut down on the number of database queries your code executes, as long as you’re careful. The only time the query will actually go out and talk to the database is when you try to use the results, by doing something like printing them, or turning them into a list. There are few more ways that can happen, but I would again refer you to the documentation for more information on that.
And now, since I’ve taken up so much of your time, I will open up to questions from the floor!