This document summarizes techniques for optimizing performance when working with large datasets in Django. It discusses using select_related, prefetch_related, values, and values_list to retrieve data in a lean way without entire model objects. It also covers string aggregation, setdefault, and get for serializing to-many relationships efficiently. The goal is to retrieve and serialize data from the database with as few queries as possible for better performance with large amounts of data.

1. SOLVING PERF ISSUES IN . ORMDJANGO

2. SIAN LERK LAU
linkedin.com/in/sianlerk
@kiawin

3. A little bit about Jewel…
We’re a Singapore-based company with offices in KL and HK too.
We’re hiring!
We’re always looking for talented developers, data scientists, data engineers, devops engineers and more!
Check out our Careers page at
https://www.jewelpaymentech.com/careers.html
Image
Jewel Paymentech
© Copyright 2018 Jewel Paymentech Pte .Ltd. – Reproduction and distribution of this presentation without written permission is
prohibited.
VISION

4. JOIN THE TEAM
Python Developers
React Developers
Data Scientists
Data Engineers
and many more…
Scan the QR code to
visit our Careers page:

5. (UN)ORTHODOX
THE JOURNEY OF AN OPTIMIZATION

6. Jafnee Jasmee Willy CahyadiSwee Meng

7. BACKGROUND
(OUR STORY)

8. M V P

9. M V P

10. TOOLS
(TO GET US STARTED)

11. browser inspection tool

13. code inspection tool

14. code profiling tool

15. jazzband/django-silk

20. ORM
(THE DJANGO WAY)

21. just works

22. just works

23. django documentation

24. D.10K
(THE CASE STUDY)

25. #the10kreality

26. REALITY
HURTS
(if not your eyes)

27. setting baseline

28. #the10kdream

29. process of elimination

30. lazy load
1

31. # 10k Records found
Form.objects.count()
# How many queries made?
for form in Form.objects.all():
print(form.author.name)

32. # 10k Records found
Form.objects.count()
# How many queries made?
for form in Form.objects.all():
print(form.author.name)
# 1+10k queries made!
Big-O Notation

33. # 10k Records found
Form.objects.count()
# How many queries made?
for form in Form.objects.all():
print(form.author.name)
# 1+10k queries made!
Big-O Notation
O(n+1)

34. one-to-one
one-to-many
many-to-many

35. one-to-one
one-to-many
.select_related()
# One query made
for form in Form.objects.select_related('author'):
print(form.author.name)
O(1)

36. many-to-many .prefetch_related()
# Two queries made, join done at python level
for form in Form.objects.prefetch_related('categories'):
for category in form.categories.all():
print(category.name)
O(2)

37. model object
2

38. #expensive

39. skip model object
slim data retrieval

40. # 10k Records found
Form.objects.count()
# List of 10k model objects
for form in Form.objects.select_related('author'):
print(book.author.name)
fat + slow

41. skip model object
slim data retrieval
.values()
# List of 10k dictionaries
form_qs = Form.objects.values(
'application_id',
'author__name',
'categories__name',
)
for form in form_qs:
print(
f"{form['application_id']}, "
f"{form['author__name']}, "
f"{form['categories__name']}"
)
slim + quick

42. skip model object
slim data retrieval
.values_list()
# List of 10k tuples
form_qs = Form.objects.values_list(
'application_id',
'author__name',
'categories__name',
)
for application_id, author, categories in form_qs:
print(f'{application_id}, {author}, {categories}')
slim + quick

43. skip model object
slim data retrieval
.values_list()
# List of 10k tuples
form_qs = Form.objects.values_list(
'application_id',
flat=True,
)
for application_id in form_qs:
print(f'{application_id}')
slim + quick

44. prove to me

45. from django.db import connection
# prints 0
print(len(connection.queries))
# How many queries made?
forms = list(Form.objects.values_list('id',
'categories__name'))
# prints x?
print(len(connection.queries))

46. print(connection.queries[-1])
# actual query performed, with time taken
{'sql': "QUERY = 'SELECT ...', 'time': '0.009'}

47. DRF
(serializer)
3

48. #slow

49. manual serialization

50. return a dict

51. #eww

52. three minutes saved

53. #okay...

54. database
agnostic
support
4

55. to-many relationship

56. string aggregation

57. SELECT
form.id AS form,
...
STRING_AGG(category.name, ',') AS categories,
...
GROUP BY form
...
string aggregation

58. SELECT
form.id AS form,
...
STRING_AGG(category.name, ',') AS categories,
...
GROUP BY form
...
string aggregation

59. string aggregation

60. #...

61. setdefault
get

62. form_qs = Form.objects.values('id', 'categories__name')
serialized_forms = {}
for id, category in form_qs:
serialized_form = serialized_forms.setdefault(
id,
{},
)
categories = serialized_form.get('categories', [])
categories.append(category)
setdefault x get

63. #jobdone

64. disclaimer

65. THANK YOU
linkedin.com/in/sianlerk
@kiawin