Behind the curtain - How Django handles a request

Behind the curtain
How Django handles a request
Daniel Hepper

About this tutorial
• not a top-down Django tutorial

• for people who want to get to the bottom of things

What we'll cover
• what is Django?

• dissecting the tiniest Django web app

• the same app, but more complicated

About me
Daniel Hepper

• degree in computer science

• freelancing for 7 years

• ﬁrst contact with Django
around 0.95

• NOT a core developer

• varying amounts beard

The web framework for
perfectionists with deadlines
Django makes it easier to build better Web apps
more quickly and with less code.

Web apps
• web app

• Django project

• django-admin.py startproject

• Django app

• module, part of an web app

• might be re-usable: re-usable app

• python manage.py startapp

Web Framework
• you take things from a library, 
you put something in a framework

• "Don't call us, we call you"

• But you have to tell us your number 
you have to provide an entrypoint

Web Framework
• turn HTTP requests into HTTP response

Browser Cloud
HTTP Request
HTTP Response

HTTP Client
HTTP Request
HTTP Response
HTTP Server

HTTP-Request & Reponse
GET / HTTP/1.1
Host: 127.0.0.1:8000
User-Agent: curl/7.54.0
Accept: */*
HTTP/1.0 200 OK
Date: Sat, 26 May 2018 07:27:52 GMT
Server: WSGIServer/0.2 CPython/3.6.4
Content-Type: text/html; charset=utf-8
Hello world

HTTP Client
HTTP
HTTP Server Django
?

HTTP Client
HTTP
HTTP Server Django
WSGI

Browser
HTTP
Gunicorn Django
WSGI

Browser
HTTP
Nginx Django
WSGI
Gunicorn
HTTP

Browser
HTTP
Nginx Django
WSGI
uwsgi
uwsgi

WSGI
• Deﬁned in PEP-333 / PEP-3333

• "A [...] standard interface between web servers and
Python web applications or frameworks, to promote web
application portability across a variety of web servers."

• "WSGI is a tool for framework and server developers"

WSGI
• servers

• applications

• middleware

WSGI Application
• Callable that accepts two arguments

• a function, method, class, instance with a __call__ method

• result = application(environ, start_response)
• environ: dictionary object, containing CGI-style environment variables

• start_response: callable, accepting two required positional
arguments, third optional

• def start_response(status, response_headers,
exc_info=None):
• When called by the server, the application object must return an iterable
yielding zero or more bytestrings

Hello WSGI
HELLO_WORLD = b"Hello world!n"
def simple_app(environ, start_response):
"""Simplest possible application object"""
status = '200 OK'
response_headers = [('Content-type', 'text/plain')]
start_response(status, response_headers)
return [HELLO_WORLD]

The web framework for
perfectionists with deadlines
Django makes it easier to build better Web apps
more quickly and with less code.
WSGI

(venv)$ pip install django==2.0.5
(venv)$ django-admin.py startproject default
(venv)$ tree default
default
├── manage.py
└── default
├── __init__.py
├── settings.py
├── urls.py
└── wsgi.py
(venv)$ cat default/default/settings.py

"""
Django settings for mysite project...
"""
import os
# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/2.0/howto/deployment/checklist/
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = '-^oi_8%de()$d3qqa_4fgwds#&nlfn2(j-@_hu(j-m7-+-@q@p'
# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True
ALLOWED_HOSTS = []
# Application definition
INSTALLED_APPS = [
'django.contrib.admin',
...
'django.contrib.staticfiles',
]
MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
...
'django.middleware.clickjacking.XFrameOptionsMiddleware',
]
ROOT_URLCONF = 'default.urls'
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [],
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'django.template.context_processors.debug',
"Django is too complicated!"

Web development is
complicated
Django oﬀers reasonable defaults

to make your life easier

All you need is...
• a view

• an URLConf

• settings

• a WSGI application object

https://github.com/consideratecode/django-behind-the-curtain/

views.py
from django.http import HttpResponse
def hello(request):
return HttpResponse('Hello world')

urls.py
from django.urls import path
from . import views
urlpatterns = [
path('', views)
]

settings.py
SECRET_KEY = 'not-so-secret-123%^&'
ROOT_URLCONF = 'simple.urls'

wsgi.py
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault(
"DJANGO_SETTINGS_MODULE",
"simple.settings"
)
application = get_wsgi_application()

Running a
Django web application
• job of a WSGI server

• two phases

• import the application object

• call the application object for each request

A simple WSGI server
from wsgiref.simple_server import make_server
from simple.wsgi import application
httpd = make_server('127.0.0.1', 8000, application)
print("Serving HTTP on port 8000...")
httpd.handle_request()

(venv)$ tree simple
.
├── server.py
└── simple
├── __init__.py
├── __pycache__
├── settings.py
├── urls.py
├── views.py
└── wsgi.py

import os
from django.core.wsgi import
get_wsgi_application
os.environ.setdefault(
"DJANGO_SETTINGS_MODULE",
"simple.settings"
)
application = get_wsgi_application()
wsgi.py

import django
from django.core.handlers.wsgi import WSGIHandler
def get_wsgi_application():
"""
The public interface to Django's WSGI support.
Return a WSGI callable.
Avoids making django.core.handlers.WSGIHandler a public
API, in case the internal WSGI implementation changes or
moves in the future.
"""
django.setup(set_prefix=False)
return WSGIHandler()
django/core/wsgi.py

from django.utils.version import get_version
VERSION = (2, 0, 0, 'final', 0)
__version__ = get_version(VERSION)
def setup(set_prefix=True):
"""
Configure the settings (this happens as a side effect of accessing the
first setting), configure logging and populate the app registry.
Set the thread-local urlresolvers script prefix if `set_prefix` is True.
"""
from django.apps import apps
from django.conf import settings
from django.urls import set_script_prefix
from django.utils.log import configure_logging
configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
if set_prefix:
set_script_prefix(
'/'
if settings.FORCE_SCRIPT_NAME is None
else settings.FORCE_SCRIPT_NAME
)
apps.populate(settings.INSTALLED_APPS)
django/__init__.py

[...]
class LazySettings(LazyObject):
"""
A lazy proxy for either global Django settings or a custom settings
object. The user can manually configure settings prior to using
them. Otherwise,Django uses the settings module pointed to by
DJANGO_SETTINGS_MODULE.
"""
def __getattr__(self, name):
"""
Return the value of a setting and cache it in self.__dict__.
"""
if self._wrapped is empty:
self._setup(name)
val = getattr(self._wrapped, name)
self.__dict__[name] = val
return val
[...]
settings = LazySettings()
django/conf/__init__.py

[...]
class LazySettings(LazyObject):
"""
A lazy proxy for either global Django settings or a custom settings
object. The user can manually configure settings prior to using
them. Otherwise,Django uses the settings module pointed to by
DJANGO_SETTINGS_MODULE.
"""
def _setup(self, name=None):
"""
Load the settings module pointed to by the environment
variable. This is used the first time settings are needed, if
the user hasn't configured settings manually.
"""
settings_module = os.environ.get(ENVIRONMENT_VARIABLE)
if not settings_module:
desc = ("setting %s" % name) if name else "settings"
raise ImproperlyConfigured(...)
self._wrapped = Settings(settings_module)
[...]

[...]
class Settings:
def __init__(self, settings_module):
for setting in dir(global_settings):
if setting.isupper():
setattr(self, setting,
getattr(global_settings, setting))
[...]
mod = importlib.import_module(self.SETTINGS_MODULE)
self._explicit_settings = set()
for setting in dir(mod):
if setting.isupper():
setting_value = getattr(mod, setting)
[...]
setattr(self, setting, setting_value)
[...]
[...]

class WSGIHandler(base.BaseHandler):
request_class = WSGIRequest
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.load_middleware()
def __call__(self, environ, start_response):
[...]
django/core/handlers/wsgi.py

class BaseHandler:
[...]
def load_middleware(self):
"""
Populate middleware lists from settings.MIDDLEWARE.
"""
self._request_middleware = []
self._view_middleware = []
self._template_response_middleware = []
self._response_middleware = []
self._exception_middleware = []
handler = convert_exception_to_response(
self._get_response)
for middleware_path in reversed(settings.MIDDLEWARE):
[...]
self._middleware_chain = handler
django/core/handlers/base.py

[...]
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.load_middleware()
[...]

[...]
set_script_prefix(get_script_name(environ))
signals.request_started.send(sender=self.__class__,
environ=environ)
request = self.request_class(environ)
response = self.get_response(request)
[...]

class WSGIRequest(HttpRequest):
def __init__(self, environ):
[...]

class HttpRequest:
"""A basic HTTP request."""
# The encoding used in GET/POST dicts. None means use default
# setting.
_encoding = None
_upload_handlers = []
def __init__(self):
# WARNING: The `WSGIRequest` subclass doesn't call
# `super`.
# Any variable assignment made here should also happen in
# `WSGIRequest.__init__()`.
[...]
django/core/http/request.py

script_name = get_script_name(environ)
path_info = get_path_info(environ)
if not path_info:
# Sometimes PATH_INFO exists, but is empty (e.g.
# accessing the SCRIPT_NAME URL without a
# slash). We really need to operate as if they'd
# requested '/'. Not amazingly nice to force
# the path like this, but should be harmless.
path_info = '/'
[...]

path vs.
script_name & path_info
• http://mydjangoproject.com/

• http://mydjangoproject.com/polls/

• http://mydjangoprojects.com/project1/

• http://mydjangoprojects.com/project2/polls/

• http://mydjangoprojects.com/project3

if not path_info:
path_info = '/'
[...]
def get_script_name(environ):
"""
Return the equivalent of the HTTP request's SCRIPT_NAME
environment variable. If Apache mod_rewrite is used, return
what would have been the script name prior to any rewriting
(so it's the script name as seen from the client's
perspective), unless the FORCE_SCRIPT_NAME setting is
set (to anything).
"""
[...]

if not path_info:
path_info = '/'
[...]
def get_path_info(environ):
"""Return the HTTP request's PATH_INFO as a string."""
path_info = get_bytes_from_wsgi(environ, 'PATH_INFO', '/')
return repercent_broken_unicode(path_info).decode()

[...]
self.environ = environ
self.path_info = path_info
# be careful to only replace the first slash in the path
# because of http://test/something and
# http://test//something being different as
# stated in http://www.ietf.org/rfc/rfc2396.txt
self.path = '%s/%s' % (script_name.rstrip('/'),
path_info.replace('/', '', 1))
[...]

[...]
self.META = environ
self.META['PATH_INFO'] = path_info
self.META['SCRIPT_NAME'] = script_name
self.method = environ['REQUEST_METHOD'].upper()
[...]

[...]
self.content_type, self.content_params =
cgi.parse_header(environ.get('CONTENT_TYPE', ''))
if 'charset' in self.content_params:
try:
codecs.lookup(self.content_params['charset'])
except LookupError:
pass
else:
self.encoding = self.content_params['charset']
[...]

[...]
self._post_parse_error = False
try:
content_length = int(environ.get('CONTENT_LENGTH'))
except (ValueError, TypeError):
content_length = 0
self._stream = LimitedStream(self.environ['wsgi.input'],
content_length)
self._read_started = False
self.resolver_match = None

[...]
self._post_parse_error = False
try:
content_length = int(environ.get('CONTENT_LENGTH'))
except (ValueError, TypeError):
content_length = 0
self._stream = LimitedStream(self.environ['wsgi.input'],
content_length)
self._read_started = False
self.resolver_match = None
class LimitedStream:
"""
Wrap another stream to disallow reading it past a number of
bytes.
"""
[...]

class BaseHandler:
[...]
def get_response(self, request):
"""
Return an HttpResponse object for the given
HttpRequest.
"""
# Setup default url resolver for this thread
set_urlconf(settings.ROOT_URLCONF)
response = self._middleware_chain(request)

# Overridden URLconfs for each thread are stored here.
_urlconfs = local()
[...]
def set_urlconf(urlconf_name):
"""
Set the URLconf for the current thread (overriding the
default one in settings). If urlconf_name is None, revert
back to the default.
"""
if urlconf_name:
_urlconfs.value = urlconf_name
else:
if hasattr(_urlconfs, "value"):
del _urlconfs.value
django/urls/base.py

class BaseHandler:
"""
"""
handler = convert_exception_to_response(
self._get_response
)
[...]
"""
HttpRequest.
"""

class BaseHandler:
[...]
def _get_response(self, request):
"""
Resolve and call the view,
then apply view, exception, and template_response
middleware.
This method is everything that happens
inside the request/response middleware.
"""
[...]

class BaseHandler:
[...]
"""
Resolve and call the view,
then apply view, exception, and template_response
middleware.
This method is everything that happens
inside the request/response middleware.
"""
response = None
if hasattr(request, 'urlconf'):
urlconf = request.urlconf
set_urlconf(urlconf)
resolver = get_resolver(urlconf)
else:
resolver = get_resolver()
resolver_match = resolver.resolve(request.path_info)
callback, callback_args, callback_kwargs = resolver_match
request.resolver_match = resolver_match

class BaseHandler:
[...]
[...]
[...]
# Apply view middleware
for middleware_method in self._view_middleware:
response = middleware_method(request, callback, ...)
if response:
break
[...]

class BaseHandler:
[...]
[...]
[...]
[...]
if response is None:
wrapped_callback = self.make_view_atomic(callback)
try:
response = wrapped_callback(
request, *callback_args, **callback_kwargs
)
except Exception as e:
response = self.process_exception_by_middleware(
e, request
)

class BaseHandler:
[...]
[...]
[...]
[...]
response = wrapped_callback(
request, *callback_args, **callback_kwargs
)
# Complain if the view returned None (a common error).
[...]
# If the response supports deferred rendering, apply
# template response middleware and then render the response
[...]
return response

class BaseHandler:
[...]
[...]
response._closable_objects.append(request)
[...]
return response

class HttpResponseBase:
[...]
# The WSGI server must call this method upon completion of
# the request.
# See http://blog.dscpl.com.au/2012/10/obligations-for-calling-close
def close(self):
for closable in self._closable_objects:
try:
closable.close()
except Exception:
pass
self.closed = True
signals.request_finished.send(sender=self._handler_class)
django/http/response.py

class BaseHandler:
[...]
[...]
# If the exception handler returns a TemplateResponse
# that has not been rendered, force it to be rendered.
if not getattr(response, 'is_rendered', True) and
callable(getattr(response, 'render', None)):
response = response.render()
[...]
return response

class BaseHandler:
[...]
[...]
# If the exception handler returns a TemplateResponse
# that has not been rendered, force it to be rendered.
if not getattr(response, 'is_rendered', True) and
callable(getattr(response, 'render', None)):
if response.status_code == 404:
logger.warning(
'Not Found: %s', request.path,
extra={'status_code': 404, 'request': request})
return response

[...]
[...]
response._handler_class = self.__class__
status = '%d %s' % (response.status_code, response.reason_phrase
response_headers = list(response.items())
for c in response.cookies.values():
response_headers.append(('Set-Cookie', c.output(header='')))
start_response(status, response_headers)
if (getattr(response, 'file_to_stream', None) is not None
and environ.get('wsgi.file_wrapper')):
response = environ['wsgi.file_wrapper'](response.file_to_str
return response

Default
MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

Function based middleware
def simple_middleware(get_response):
# One-time configuration and initialization.
def middleware(request):
# Code to be executed for each request before
# the view (and later middleware) are called.
response = get_response(request)
# Code to be executed for each request/response after
# the view is called.
return response
return middleware

Class based middleware
class SimpleMiddleware:
def __init__(self, get_response):
self.get_response = get_response
# One-time configuration and initialization.
def __call__(self, request):
# Code to be executed for each request before
# the view (and later middleware) are called.
# Code to be executed for each request/response after
# the view is called.
return response

Additional hooks
• process_view(request, view_func, view_args, view_kwargs)
• process_exception(request, exception)
• process_template_response(request, response)

Pre-Django 1.10-style
middleware
• process_request()
• process_response()
• django.utils.deprecation.MiddlewareMixin

class Middleware1:
def __init__(self, get_response):
print('> Middleware1.__init__')
self.get_response = get_response
def __call__(self, request):
print('> Middleware1.__call__')
print('< Middleware1.__call__')
return response
def process_view(self, request, view_func, view_args,
view_kwargs):
print('> Middleware1.process_view')
return None
def process_exception(self, request, exception):
print('> Middleware1.process_exception')
return None
def process_template_response(self, request, response):
print('> Middleware1.process_template_response')
return response
middleware/middleware.py

SECRET_KEY = 'not-so-secret-123%^&'
ROOT_URLCONF = 'middleware.urls'
MIDDLEWARE = [
'middleware.middleware.Middleware1',
]
middleware/settings.py

$ python server.py
> Middleware3.__init__
Serving HTTP on port 8000...

class BaseHandler:
[...]
"""
"""
self._request_middleware = []
self._view_middleware = []
self._template_response_middleware = []
self._response_middleware = []
self._exception_middleware = []
handler = convert_exception_to_response(self._get_response)
[...]

class BaseHandler:
[...]
"""
"""
[...]
middleware = import_string(middleware_path)
try:
mw_instance = middleware(handler)
except MiddlewareNotUsed as exc:
if settings.DEBUG:
[...]
continue
[...]
handler = convert_exception_to_response(mw_instance)

settings.MIDDLEWARE = [
]
handler = convert_exception_to_response(Middleware3(handler))

'middleware1',
'middleware2',
'middleware3',
]
handler = middleware1(middleware2(middleware3(self._get_response)))))))

class BaseHandler:
[...]
[...]
[...]
if hasattr(mw_instance, 'process_view'):
self._view_middleware.insert(
0, mw_instance.process_view)
if hasattr(mw_instance, 'process_template_response'):
self._template_response_middleware.append(
mw_instance.process_template_response)
if hasattr(mw_instance, 'process_exception'):
self._exception_middleware.append(
mw_instance.process_exception)

'middleware1',
'middleware2',
'middleware3',
]
self._view_middleware = [
middleware1_instance.process_view,
]
self._template_response_middleware = [
middleware3_instance.process_template_response,
]
self._exception_middleware = [
middleware3_instance.process_exception,
]

$ python server.py
Serving HTTP on port 8000...
> Middleware1.__call__
> Middleware1.process_view
< Middleware3.__call__

class BaseHandler:
"""
"""
[...]
"""
HttpRequest.
"""
middleware1(middleware2(middleware3(self._get_response)))))))

Middleware1
Middleware2
__call__
Middleware3
__call__
BaseHandler
_get_response

class BaseHandler:
[...]
"""
Resolve and call the view, then apply view, exception, and
template_response middleware. This method is everything
that happen inside the request/response middleware.
"""
response = None
[...]
return response

class BaseHandler:
[...]
"""
"""
response = None
if hasattr(request, 'urlconf'):
urlconf = request.urlconf
set_urlconf(urlconf)
resolver = get_resolver(urlconf)
else:
resolver = get_resolver()
[...]
return response

[...]
@functools.lru_cache(maxsize=None)
def get_resolver(urlconf=None):
if urlconf is None:
from django.conf import settings
urlconf = settings.ROOT_URLCONF
return URLResolver(RegexPattern(r'^/'), urlconf)
[...]
django/urls/resolvers/base.py

class BaseHandler:
[...]
"""
"""
[...]
resolver_match = resolver.resolve(request.path_info)
request.resolver_match = resolver_match
[...]
return response

class BaseHandler:
[...]
"""
"""
[...]
[...]
for middleware_method in self._view_middleware:
response = middleware_method(request,
callback,
callback_args,
callback_kwargs)
if response:
break
[...]
return response

Middleware1
Middleware2
__call__
Middleware3
__call__
BaseHandler
_get_response
process_view
process_view
process_view

class BaseHandler:
[...]
"""
"""
[...]
wrapped_callback = self.make_view_atomic(callback)
try:
response = wrapped_callback(request,
*callback_args,
**callback_kwargs)
response = self.process_exception_by_middleware(
e, request)
[...]
return response

class BaseHandler:
[...]
"""
"""
[...]
# Complain if the view returned None (a common error).
if isinstance(callback, types.FunctionType): # FBV
view_name = callback.__name__
else: # CBV
view_name = callback.__class__.__name__ + '.__call__'
raise ValueError(
"The view %s.%s didn't return an HttpResponse object. It
"returned None instead." % (callback.__module__, view_na
)
[...]
return response

class BaseHandler:
[...]
"""
"""
[...]
# If the response supports deferred rendering, apply template
# response middleware and then render the response
elif hasattr(response, 'render') and callable(response.render):
for middleware_method in self._template_response_middleware:
response = middleware_method(request, response)
# Complain if the template response middleware
# returned None (a common error).
raise ValueError(...)
[...]
return response

class BaseHandler:
[...]
"""
"""
[...]
# If the response supports deferred rendering, apply template
# response middleware and then render the response
elif hasattr(response, 'render') and callable(response.render):
[...]
try:
response = self.process_exception_by_middleware(e, reque
return response

class BaseHandler:
[...]
def process_exception_by_middleware(self, exception, request):
"""
Pass the exception to the exception middleware.
If no middleware return a response for this exception,
raise it.
"""
for middleware_method in self._exception_middleware:
response = middleware_method(request, exception)
if response:
return response
raise

Middleware1
Middleware2
__call__
Middleware3
__call__
BaseHandler
_get_response
process_exception
process_exception
process_exception

class BaseHandler:
[...]
"""
HttpRequest.
"""
[...]
return response

class BaseHandler:
[...]
[...]
[...]
return response

TIL
• Django is made by humans, not wizards

• Just code, no magic

Room for contribution
• Unused attributes _response_middleware and
_request_middleware in BaseHandler

• Wrong or misleading comments

• ExceptionMiddleware does not exist

Thanks
Daniel Hepper

@danielhepper

daniel@consideratecode.com

consideratecode.com

Image credits

• https://pixabay.com/en/interior-house-home-
curtain-2570933/

• https://unsplash.com/photos/MEW1f-yu2KI

• https://pixabay.com/en/rocket-launch-smoke-rocket-take-
oﬀ-67723/

• https://pixabay.com/en/gears-cogs-machine-
machinery-1236578/

• https://pixabay.com/en/balance-background-harmony-
stacked-3356546/

Behind the curtain - How Django handles a request

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Behind the curtain - How Django handles a request

Similar to Behind the curtain - How Django handles a request (20)

Recently uploaded

Recently uploaded (20)

Behind the curtain - How Django handles a request