GSoC 2022 proposal

1. Django :
Django benchmarking
GSoC 2022 Proposal
Table Of Contents:
1. Abstract
1.1. Django benchmarking
1.2. Load testing
1.3. Goals
2. Implementation Details
2.1. Integration of Airspeed velocity
2.2. Running benchmarks regularly
2.3. Load Testing
2.4. Storing Results
3. Schedule And Milestones
4. About Me

2. 1.Abstract:
1.1 Django Benchmarking:
Django has a benchmarking suite that helps in tracking the effects of
each commit and release, it helps in the identification of regressions
introduced by the changes made to the code. By identifying these
regressions and fixing them it can be made sure that Django does
not get slower due to the changes made.
Airspeed velocity is a package used for benchmarking, it is currently
being integrated into the Django benchmarking suite, and most of
the benchmarks have been migrated, by migrating a few missing
benchmarks and running these benchmarks regularly against each
release or on a schedule identification of potential regressions
becomes much easier.
1.2 Load Testing:
Since asynchronous support is currently being added to Django,
being able to see the effects using different setups and different
ASGI and WSGI servers will be very helpful in its progress.
By utilizing Locust a test harness can be created and used to load
test different ASGI and WSGI servers. Their performance and other
attributes can be measured. This will help in adding asynchronous
support to Django.

3. 1.3 Goals:
1. Integrate airspeed velocity with djangobench
2. Running the benchmarks regularly by using Github actions with
a self-hosted runner
3. Perform load tests on a Django project by configuring it with
different ASGI and WSGI servers and comparing the results.
4. Storing the benchmark and the load testing results in a
database
5. Updating the documentation
2.Implementation Details
2.1 Integration of Airspeed Velocity:
David Smith worked on integrating ASV into djangobench in the
django-asv repository and most of the benchmarks have already
been added to the repository but the following benchmarks are yet
to be added
1. Model deletion benchmark
2. Multi-Valued dictionary
3. Query set filter chaining
4. Some query related benchmarks
so I will add them to the repository after modifying its structure as
currently, the benchmarks belonging to a certain aspect of Django
are grouped in a single file, this does not provide the required
granularity and also makes it hard to locate individual benchmarks,
so I will move the benchmarks into separate directories in the
following manner

4. benchmarks
├── model_benchmarks
│ ├── model_create
│ │ └── benchmark.py
│ └── model_save_existing
│ └── benchmark.py
│ └── …….
└── query_benchmarks
├── query_date
└── query_delete
└── …….
├── ……
├── other_benchmarks
ASV provides support to run benchmarks on different branches and
different python versions, I will select the branches and python
versions to run the benchmarks against after discussion with the
mentor.
2.2 Running benchmarks regularly:
The benchmarks can be run regularly using the following
methods:
1. Repository dispatch event
Github's repository dispatch event allows the workflow of the
target repository to be triggered whenever an event that
triggers a workflow run occurs in the given repository.
To run the benchmarks when pull requests are opened in the
Django repository, the Github action Trigger Workflow And Wait

5. can be used. This action calls the benchmark workflow in the
djangobench repository.
The action can be used in the workflow file in Django as follows-
- uses: convictional/trigger-workflow-and-wait@v1.3.0
with:
owner: Django
repo: djangobench
github_token: ${{ secrets.ACCESS_TOKEN }}
workflow_file_name: benchmark.yml
This method requires the creation of a personal access token
with repo and workflow access and may raise some security
concerns but it helps in the identification of the effects of each
pull request.
2. Scheduling workflow
The benchmarks can be run regularly by triggering the
benchmark workflow on a schedule.
The schedule can be specified in the workflow file as follows
- on:
schedule:
- cron: ‘< schedule specified in a cron syntax>’
I will implement both of these methods and choose one or both of
them.

6. David Smith pointed out in a comment that the test results he got
when he ran the benchmarks with Github actions were very noisy, so
I will look into other options such as a docker-compose to run the
benchmarks.
2.3 Load testing a Django test project:
An existing test project will be used to perform load testing and
determine the performance of different ASGI and WSGI servers.
Docker-compose provides a way to set up multiple different
containers with Django projects running on different ASGI and WSGI
servers. Since docker-compose creates a network by default
communication between these containers can be handled easily.
A container with a locust file that defines a set of tasks can be used
to benchmark the servers and the results of the load test can be
written to an external database.
The tasks can either be GET or POST requests to the server. Tasks
can be defined in the locust file as follows -
from locust import HttpUser, task
class Test(HttpUser):
@task
def task1(self):
self.client.get("<url>")
self.client.post("<url>", data)

7. @task
def task2(self):
self.client.get("<url>")
self.client.post("<url>", data)
Load testing the servers through other means has some cons which
make it difficult to choose them:
- In the case of Github actions the Django server would have to
be deployed to some host before testing, and deploying every
server every time load testing is to be done seems like a hassle,
also Github actions kept reporting that the CPU usage was too
high while running locust with just a single task.
- On a self-hosted runner, we run into the problem of
daemonizing the Django server process so that locust tasks can
be performed, daemonizing some of the servers is easy but
others are pretty time-consuming.
2.4 Storing the results:
2.4.1 Storing ASV results:
ASV Compare:
ASV results for a particular branch or commit can be obtained
directly with the asv-show <commit/branch> command. The

8. benchmark results of two different branches can be compared by
using the asv compare <branch1> <branch2> command.
But, since the results are not provided in a particular format, such as
JSON or CSV, some amount of processing is required to obtain raw
data that can be inserted into a database. A python script can be
used to process the result of asv compare and insert it into the
database.
Sample Output of comparison between the main and
stable/4.0.x branch -

9. Github pages:
ASV also provides an option to push the results to the gh-pages
branch and the results can be viewed on a website. To do this a
separate repository with the name django-bench-results.github.com
can be used, and upon every run of the benchmark the results can
be pushed to the repository and the results can be viewed on a
website.
The result would look like this:
When the project is completed the benchmark results can be viewed
using the URL:
https://django-benchmark-results.github.io

10. 2.4.2 Storing Locust results:
Locust generates a CSV file result_stats.csv, which contains the
required results.
Here is a sample output of a locust run -
result _stats.csv
The data in the following columns will be written to the database:
1. Type of request
2. Request Count
3. Failure Count
4. Median Response Time
5. Average Response Time
6. Max Response Time
7. Average Content Size
8. Requests per second
9. Failures per second
2.5 Update Documentation:
The following changes will be made to the existing readme.md file
● Modification of the instructions to install djangobench
● Addition of sample results of benchmarking and load testing
● Modification of the instructions to write a new benchmark

11. 3.Schedule And Milestones
Before I start to work on the project I will complete the following
tasks
- The template_render benchmark is currently failing, I will work
on fixing it
- Add some of the benchmarks mentioned in the TODO to
djangobench
- Experiment with different docker images and python versions
to run Django ASGI and WSGI servers
I will work on these tasks during the application review period.
As I will be having offline classes during the project period, I will be
able to work for about 4-6 hours during weekdays and about 6-8
hours on weekends.
During the project, I will blog every week about my progress on the
medium and I will update my progress with the Django community.
Community Bonding (May 20 -June 12)
● Discuss the plan with mentors and senior developers and make any
changes if necessary
● Figure out alternate approaches and compare them with the current
plan and make improvements
● Discuss hardware and software specifications of the virtual machine.

12. First Milestone -Integrating Airspeed velocity (June 13 -June 30)
● Modify the directory structure of django-asv
● Add model deletion benchmark
● Add multi-valued dict benchmark
● Add Query set filter chaining benchmark
● Add query-related benchmarks.
● Run all the benchmarks and obtain the results.
● Modify asv.conf.json to include older releases and some python
versions
● Move the django-asv repository to djangobench
Second Milestone -Setting up a VM and running the benchmarks -(July
1 -July 18)
● Create a personal access token
● Add run_benchmarks.yaml file to Django repository that triggers the
benchmarks run when a pull request is made to it
● Add a benchmark.yaml file to djangobench that defines how the
workflow should be run when a pull request is made.
● Setup a self-hosted runner to run the actions
● Run the actions.
● Obtain the results.
● Add a schedule to run the benchmarks in benchmark.yaml

13. Third Milestone -Creating a database and inserting the results -(July 18
-July 25)
● Obtain the output of several runs of the action.
● Write a python script to parse the output of asv compare command
● Create a database to store the results from the asv compare
command
● Write the results into the database
● Create a repository with the name django_bench_results.github.io
and add action to push ASV results from djangobench to the
repository
Fourth Milestone -Create a test harness using Locust -(July 25 -July 30)
● Add locustfile.py
● Define the tasks to be performed during load testing
● Load test locally with different servers and compare results
● Vary the locust parameters to get different results and choose the
ideal parameters
Fifth Milestone -Perform Load testing -(July 31 -August 25)
● Create a new folder dockerfiles to store all the docker files required
to perform load testing
● Select a virtual machine and set it up to perform load testing
● Add docker-compose.yaml to the dockerfiles folder

14. ● Create locust.dockerfile which checks out the repo and performs
load testing using the tasks defined in locustfile.py and add it to
dockerfiles
● Create gunicorn.dockerfile which creates a docker container that
runs django using a gunicorn server and add this container as a
service to the docker-compose.yaml file
● Check the load testing results by running the docker-compose file
● Make any changes if necessary
● Create uWSGI.dockerfile which runs django using a uWSGI server
and add it to dockerfiles
● Create mod_wsgi.dockerfile which runs django using an apache
server and configure the apache server by modifying the http.confd
file
● Load-test to find the performance difference between different
WSGI servers.
● Create daphne.dockerfile, hypercorn.dockerfile, and
uvicorn.dockerfile which run django using daphne, hypercorn,
and uvicorn servers, and add it to the dockerfiles folder
● Create services in docker-compose.yaml to create containers
with the given dockerfiles
● Load-test using the locust container to find the performance
difference between different ASGI and WSGI servers.
● Add load testing to benchmark.yml or create a new workflow
Sixth Milestone -Add load testing results to the database -(August 25 -
August 30)
● Create a database to store the load test results
● Create a user with write access

15. ● Add user credentials to docker-compose.yml to write results into the
database
● Create save_db.py which saves the results from result_stats.csv into
the database
● Save the results into the database
Seventh Milestone -Update Documentation -(August 31 -September 4)
● Update benchmark setup instructions
● Update instructions on running the benchmark
● Upload sample images of ASV output
● Upload sample images of Load testing output
● Add the https://django_bench_results.github.io link so that people
can view ASV results on the website
● Update instructions on writing new benchmarks
Final Week
● Wrap up any remaining work
● Prepare a final report and summarise my work
Stretch Goals
● Visualize asv compare results of different branches and identify
regressions
● Use asv find on a range of commits to find commits that produce a
large regression

16. Other Commitments
● College Examinations:
○ I will be having three internal assessments and one
end-semester examination.
○ During internal assessments, I can only work for
about an hour a day but these assessments usually
take only 3 days.
○ During end semester exams I can work for about 5-6
hours as there will be study holidays between two
exams.
○ The schedules of both IA’s and exams have not been
released yet, I will inform the mentors as soon I get it.
Code Contributions
https://github.com/django/djangobench/pull/42
https://github.com/django/djangobench/pull/43
About me:
Name: Deepak Dinesh
Github ID: deepakdinesh1123

17. Email: deepakdinesh13@protonmail.com
LinkedIn: Deepak
College: Adichunchanagiri Institute Of Technology
Program: B.E in Computer Science And Engineering
Year: 3rd Year
Expected Graduation Year: 2023
Time Zone: Indian Standard Time,UTC+5:30
Country of residence: India
Languages: Kannada,English,Hindi