Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Secrets of a WSGI master


Published on

The WSGI (Web Server Gateway Interface) specification for hosting Python web applications was created in 2003. Measured in Internet time, it is ancient. The oldest main stream implementation of the WSGI specification is mod_wsgi, for the Apache HTTPD server and it is 10 years old.

WSGI is starting to be regarded as not up to the job, with technologies such as HTTP/2, web sockets and async dispatching being the way forward. Reality is that WSGI will be around for quite some time yet and for the majority of use cases is more than adequate.

The real problem is not that we need to move to these new technologies, but that we aren't using the current WSGI servers to their best advantage. Moving to a new set of technologies will not necessarily make things better and will only create a new set of problems you have to solve.

As one of the oldest WSGI server implementations, Apache and mod_wsgi may be regarded as boring and not cool, but it is still the most stable option for hosting WSGI applications available. It also hasn't been sitting still, with a considerable amount of development work being done on mod_wsgi in the last few years to make it even more robust and easier to use in a development environment as well as production, including in containerised environments.

In this talk you will learn about many features of mod_wsgi which you probably didn't even know existed, features which can help towards ensuring your Python web application deployment performs to its best, is secure, and has a low maintenance burden.

Published in: Internet
  • Hey guys! Who wants to chat with me? More photos with me here 👉
    Are you sure you want to  Yes  No
    Your message goes here

Secrets of a WSGI master

  1. 1. Secrets of
 a WSGI master. Graham Dumpleton @GrahamDumpleton
  2. 2. What is WSGI? Web Browser Web Browser Web Browser Web Server HTTP HTTP HTTP File System (Static Files) Python Web Application WSGI WSGI == Web Server Gateway Interface (PEP 3333)
  3. 3. WSGI is a specification for an Application Programming Interface WSGI is NOT a wire protocol WSGI is NOT an implementation of any anything
  4. 4. def application(environ, start_response): status = '200 OK' output = b'Hello World!' response_headers = [ ('Content-Type', 'text/plain'), ('Content-Length', str(len(output)))] start_response(status, response_headers) return [output]
  5. 5. Friends don’t let friends use raw WSGI
  6. 6. You still need a way to host a WSGI application The development servers builtin to a framework are not good enough
  7. 7. Installing mod_wsgi the easy way pip install mod_wsgi
  8. 8. Run mod_wsgi from the command line mod_wsgi-express start-server
  9. 9. No Apache configuration required $ mod_wsgi-express start-server Server URL : http://localhost:8000/ Server Root : /tmp/mod_wsgi-localhost:8000:502 Server Conf : /tmp/mod_wsgi-localhost:8000:502/httpd.conf Error Log File : /tmp/mod_wsgi-localhost:8000:502/error_log (warn) Request Capacity : 5 (1 process * 5 threads) Request Timeout : 60 (seconds) Startup Timeout : 15 (seconds) Queue Backlog : 100 (connections) Queue Timeout : 45 (seconds) Server Capacity : 20 (event/worker), 20 (prefork) Server Backlog : 500 (connections) Locale Setting : en_AU.UTF-8
  10. 10. Django framework integration INSTALLED_APPS = [ 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'mod_wsgi.server', ] Add mod_wsgi.server
  11. 11. Run with Django management command python runmodwsgi
  12. 12. Automatic code reloading python runmodwsgi --reload-on-changes
  13. 13. Interactive debugger python runmodwsgi --enable-debugger
  14. 14. Production configuration #1 python runmodwsgi --server-root /etc/wsgi-port-80 --user www-data --group www-data --port 80 --setup-only
  15. 15. Production configuration #2 mod_wsgi-express start-server --server-root /etc/wsgi-port-80 --user www-data --group www-data --port 80 --url-alias /static static --setup-only
  16. 16. Running in production /etc/wsgi-port-80/apachectl start /etc/wsgi-port-80/apachectl restart /etc/wsgi-port-80/apachectl stop
  17. 17. Build a container image FROM python:3 RUN apt-get update && apt-get install -y --no-install-recommends apache2 apache2-dev locales && apt-get clean && rm -r /var/lib/apt/lists/* RUN echo 'en_US.UTF-8 UTF-8' >> /etc/locale.gen && locale-gen ENV LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 RUN pip install --no-cache-dir mod_wsgi WORKDIR /opt/app-root COPY . /opt/app-root EXPOSE 80 CMD [ "mod_wsgi-express", "start-server", "--port", "80", "--user", "www-data", "--group", "www-data", "--log-to-terminal", "" ] Install Apache Fix Unicode Problems Run mod_wsgi-express
  18. 18. Building the container $ docker build -t mypyapp . Sending build context to Docker daemon 3.584kB Step 1/9 : FROM python:3 ---> 968120d8cbe8 Step 2/9 : WORKDIR /opt/app-root ---> Using cache ---> 003096a40d39 Step 3/9 : .......
  19. 19. Starting the container $ docker run --rm -p 80:80 mypyapp
  20. 20. Friends don’t let friends run containers as root
  21. 21. Friends don’t let friends use Python without a Python virtual environment
  22. 22. A better container image #1 FROM python:3 RUN apt-get update && apt-get install -y --no-install-recommends apache2 apache2-dev locales && apt-get clean && rm -r /var/lib/apt/lists/* RUN adduser --disabled-password --gecos "Warp Drive" --uid 1001 --gid 0 --home /opt/app-root warpdrive && chmod g+w /etc/passwd RUN echo 'en_US.UTF-8 UTF-8' >> /etc/locale.gen && locale-gen ENV LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 PATH=/opt/app-root/bin:$PATH HOME=/opt/app-root Create non root user
  23. 23. A better container image #2 RUN pip install --no-cache-dir virtualenv && virtualenv /opt/app-root && . /opt/app-root/bin/activate && pip install --no-cache-dir warpdrive && warpdrive fixup /opt/app-root WORKDIR /opt/app-root COPY . /opt/app-root/src RUN warpdrive fixup /opt/app-root/src USER 1001 RUN warpdrive build && warpdrive fixup /opt/app-root EXPOSE 8080 CMD [ "warpdrive", "start" ] Create a Python
 virtual environment Use warpdrive, it's magicUse non root user Non privileged port
  24. 24. Running as non root $ docker run mypyapp warpdrive exec id uid=1001(warpdrive) gid=0(root) groups=0(root) $ docker run -u 100001 mypyapp warpdrive exec id uid=100001(warpdrive) gid=0(root) groups=0(root) Default to assigned non root user Can be run as arbitrary high user ID
  25. 25. Same tools for development $ warpdrive project mypyapp (warpdrive+mypyapp) $ warpdrive build (warpdrive+mypyapp) $ warpdrive start
  26. 26. Generate image with no Dockerfile $ warpdrive image mypyapp $ docker run --rm -p 80:8080 mypyapp
  27. 27. Source-to-Image $ s2i build getwarped/warp0-debian8-python35 mypyapp
  28. 28. OpenShift $ oc new-app --image-stream getwarped/warp0-debian8-python35 --code --name mypyapp
  29. 29. Manual Apache configuration $ mod_wsgi-express module-config LoadModule wsgi_module "/.../venv/.../mod_wsgi/server/" WSGIPythonHome "/.../venv"
  30. 30. Friends don’t let friends use embedded mode of mod_wsgi
  31. 31. Embedded mode
  32. 32. Daemon mode
  33. 33. Manual daemon mode configuration WSGIRestrictEmbedded On WSGIDaemonProcess mypyapp python-home=/.../env WSGIScriptAlias / /.../src/ process-group=mypyapp application-group=%{GLOBAL} <Directory /.../src> <Files> Require all granted </Files> </Directory>
  34. 34. The missing options • display-name='%{GROUP}' • lang='en_US.UTF-8' • locale='en_US.UTF-8' • startup-timeout=15 • connect-timeout=15 • socket-timeout=60 • queue-timeout=45 • request-timeout=60 • inactivity-timeout=0 • restart-interval=0 • maximum-requests=0 • shutdown-timeout=5 • deadlock-timeout=60 • graceful-timeout=15 • eviction-timeout=0
  35. 35. Failed application loading • startup-timeout=15
 Defines the maximum number of seconds allowed to pass waiting to see if a WSGI script file can be loaded successfully by a daemon process. When the timeout is passed, the process will be restarted.
  36. 36. Connection timeouts • connect-timeout=15
 Defines the maximum amount of time for an Apache child process to wait trying to get a successful connection to the mod_wsgi daemon processes. This defaults to 15 seconds. • socket-timeout=60
 Defines the timeout on individual reads/writes on the socket connection between the Apache child processes and the mod_wsgi daemon processes. If not specified, the number of seconds specified by the Apache Timeout directive will be used instead. See also response-socket- timeout if need to control this only for writing back content of the response. • queue-timeout=45
 Defines the timeout on how long to wait for a mod_wsgi daemon process to accept a request for processing. Not enabled by default.
  37. 37. Time triggered restart • request-timeout=60
 Defines the maximum number of seconds that a request is allowed to run before the daemon process is restarted. • inactivity-timeout=0
 Defines the maximum number of seconds allowed to pass before the daemon process is shutdown and restarted when the daemon process has entered an idle state. To restart on stuck requests use request-timeout instead.
  38. 38. Request Monitoring import mod_wsgi def event_handler(name, **kwargs): if name == 'request_started': ... elif name == 'request_finished': environ = kwargs['request_environ'] response_time = kwargs.get('response_time') cpu_user_time = kwargs.get('cpu_user_time') cpu_system_time = kwargs.get('cpu_system_time') ... elif name == 'request_exception': ... mod_wsgi.subscribe_events(event_handler)
  39. 39. Resources • mod_wsgi - • warpdrive - • Source-to-Image - • OpenShift -
  40. 40. Friends don't let friends use Windows for running Python web applications
  41. 41. Friends don't let friends use the mod_wsgi which comes packaged with the operating system
  42. 42. Friends don't let friends use those other WSGI servers
  43. 43. Friends don't let friends make things too complicated, simple is good
  44. 44. @GrahamDumpleton