1. @louisemeta
Postgres index types
(And where to find them)
Louise Grandjonc
DjangoCon 2019

2. @louisemeta
About me
Software engineer at Citus/Microsoft
Previously lead python developer
Postgres enthusiast
PostgresWomen co-founder
@louisemeta on twitter
www.louisemeta.com
louise.grandjonc@microsoft.com
@louisemeta !2

3. @louisemeta
What we’re going to talk about
1. What are indexes for?
2. Creating indexes
3. B-Tree
4. GIN
5. GiST
6. Brin
@louisemeta !3

4. @louisemeta
First things first: the crocodiles
• 250k crocodiles
• 100k birds
• 400K appointments
@louisemeta !4

5. @louisemeta
What are indexes for?

6. @louisemeta
Constraints
Some constraints transform into indexes.
- PRIMARY KEY
- UNIQUE
- EXCLUDE USING
"crocodile_pkey" PRIMARY KEY, btree (id)
"crocodile_email_uq" UNIQUE CONSTRAINT, btree (email)
Indexes:
"appointment_pkey" PRIMARY KEY, btree (id)
"appointment_crocodile_id_schedule_excl" EXCLUDE USING gist
(crocodile_id WITH =, schedule WITH &&)
In the crocodile table
In the appointment table
@louisemeta !6

7. @louisemeta
Query optimization
Often the main reason why we create indexes
Why do indexes make queries faster
In an index, tuples (value, pointer) are stored.
Instead of reading the entire table for a value, you just go to the index (kind of like in an
encyclopedia)
@louisemeta !7

8. @louisemeta
Creating indexes
@louisemeta

9. @louisemeta
Creating an index
@louisemeta !9
Let’s say we would like to do queries like:
Crocodile.objects.filter(number_of_teeth=10)
SELECT * FROM crocodile WHERE number_of_teeth = 10;
Time: 31ms
class Crocodile(models.Model):
…
number_of_teeth = models.IntegerField(db_index=True)
CREATE INDEX (optional index name) ON crocodile (number_of_teeth);
SELECT * FROM crocodile WHERE number_of_teeth = 10;
Time: 6ms
Raw SQL
New timing

10. @louisemeta
Creating a unique index
@louisemeta !10
We want to make sure that you can’t create two account with
the same email:
class Crocodile(models.Model):
…
email = models.EmailField(max_length=255, unique=True)
CREATE UNIQUE INDEX ON crocodile (email);
Raw SQL
Crocodile.objects.create(
email='louise@croco.com',
first_name='Louise',
last_name='grandjonc',
birthday='1991-12-21',
number_of_teeth=32)
DETAIL: Key (email)=(louise@croco.com)
already exists.
Insert result if duplicated row

11. @louisemeta
Creating a partial index
95% of the appointments in our database have the field done=True.
Appointment.objects.filter(emergency_level__gt=8, done=False)
CREATE INDEX ON appointment (emergency_level);
@louisemeta !11
class Appointment(models.Model):
…
class Meta:
indexes = [models.Index(fields=['emergency_level'],
name='apptmt_emergency_level_idx',
condition=Q(done=False))]
In Django >= 2.2

12. @louisemeta
Creating a partial index
@louisemeta !12
CREATE INDEX ON crocodile (emergency_level) WHERE done is False;
Raw SQL
Size on the index: 352 kB
Time of the query: 3.639 ms
Size on the index:13MB
Time of the query: 29.106 ms
Old index without condition New index with condition

13. @louisemeta
Creating a partial unique index
@louisemeta !13
Want to add a UNIQUE index but have duplicates due to history or
soft delete?
class Crocodile(models.Model):
...
class Meta:
constraints = [
models.UniqueConstraint(
fields=['email'],
name='croco_email_uniq’,
condition=Q(created_at__gte='2019-09-01'))]
In Django >= 2.2

14. @louisemeta
Creating a partial unique index
@louisemeta !14
CREATE UNIQUE INDEX ON crocodile (email) WHERE created_at >
‘2019-01-01’;
Size of previous index: 6648 kB
Size of the new index: 112 kB
Why use a partial unique index?
- Save disk space with smaller index
- Faster inserts because the index tree is shorter to explore (especially
when you have a huge volume of old data)
Raw SQL

15. @louisemeta
Creating a multi-column index
@louisemeta !15
We have a job running regularly to list a bird’s emergencies, it runs
the following query:
class Appointment(models.Model):
…
class Meta:
indexes = [models.Index(fields=['plover_bird', ‘emergency_level'],
name='apptmt_plover_emergency_idx')]
Appointment.objects.filter(emergency_level__gte=9,
plover_bird=plover_bird)
CREATE INDEX ON appointment (plover_bird_id, emergency_level);
Time: 41.560 ms
Raw SQL
Time after: 0.606 ms

16. @louisemeta
Ordering the columns in a multi-column index
@louisemeta !16
Two things to consider:
- Re-using the index: the rightmost columns can be re-used for other queries.
The first column will be ordered, so the index can be used by this query:
Appointment.objects.filter(plover_bird=plover_bird)
30 8
55 10
96 7
31 5
31 10
55 10
56 10
57 3
31 6 RowID
31 10 RowID
31 10 RowID
32 1 RowID
32 6 RowID
…
55 10 RowID
plover_bird_id
em
ergency_level

17. @louisemeta
Ordering the columns in a multi-column index
@louisemeta !17
- The most filtering columns should come first
croco_talk=# SELECT COUNT(*) FROM appointment WHERE emergency_level
>= 9;
count
-------
75982
(1 row)
croco_talk=# SELECT COUNT(*) FROM appointment WHERE plover_bird_id =
22551;
count
-------
5
(1 row)
In this case, with plover_bird_id as the first column, it will first filter out and
the second filter will be applied on only 5 rows.

18. @louisemeta
B-Trees
@louisemeta

19. @louisemeta
B-Trees internal data structure
@louisemeta !19
Root
High Key: None
begin 16 31
Parent
High Key: 16
begin 12 14
Parent
High Key: 31
16 20
Parent
High Key: None
31 33
Leave
High Key: 12
Value: 1
Pointer:
croco 10
Value: 1
Pointer:
croco 12
Value : 2
Pointer:
croco 23
… Value 10
Pointer:
croco 1
Value: 11
Pointer:
croco 2
Leave
High Key: 14
Value:
12
Pointer:
croco 17
Value:
13
Pointer:
croco 3
Value :
13
Pointer:
croco 4
… Value 13
Pointer:
croco 27
…
Leave
High Key: None
Value: 33
Pointer:
croco 5
Value: 33
Pointer:
croco 6
Value : 33
Pointer:
croco 123
… Value: 38
Pointer:
croco 26
- A BTree in a balanced tree
- All the leaves are at equal distance from the root.
- A parent node can have multiple children minimizing the tree’s depth

20. @louisemeta
B-Trees internal data structure - 2
Pages
The root, the parents, and the leaves are all pages with the same structure.
Pages have:
- A block number (pointer)
- A high key (defines the highest value found in a page)
- Items
@louisemeta !20

21. @louisemeta
B-Trees internal data structure - 4
Pages high key
- Any item in the page will have a value lower or equal to the high key
And in page 575, there is no high key as it’s the
rightmost page.
In page 3, I will find crocodiles with 16 or less teeth
In page 289, with 31 and less
@louisemeta !21

22. @louisemeta
B-Trees internal data structure - 5
Items
An item contains:
- A value (of the indexed row in the leaves, of the first row in the parents)
- Pointer (to the row in the leaves, to the child page in the parents)
@louisemeta !22

23. @louisemeta
To sum it up
@louisemeta !23
Root
High Key: None
begin 16 31
Parent
High Key: 16
begin 12 14
Parent
High Key: 31
16 20
Parent
High Key: None
31 33
Leave
High Key: 12
Value: 1
Pointer:
croco 10
Value: 1
Pointer:
croco 12
Value : 2
Pointer:
croco 23
… Value 10
Pointer:
croco 1
Value: 11
Pointer:
croco 2
Leave
High Key: 14
Value:
12
Pointer:
croco 17
Value:
13
Pointer:
croco 3
Value :
13
Pointer:
croco 4
… Value 13
Pointer:
croco 27
…
Leave
High Key: None
Value: 33
Pointer:
croco 5
Value: 33
Pointer:
croco 6
Value : 33
Pointer:
croco 123
… Value: 38
Pointer:
croco 26
- A Btree is a balanced tree
- The values indexed are the values of the rows
- Data is stored in pages
- Pages have a high key defining the biggest value in the page
- Pages have items pointing to an other page or the row.

24. @louisemeta
What are BTree good for?
@louisemeta !24
Root
High Key: None
begin 16 31
Parent
High Key: 16
begin 12 14
Parent
High Key: 31
16 20
Parent
High Key: None
31 33
Leave
High Key: 12
Value: 1
Pointer:
croco 10
Value: 1
Pointer:
croco 12
Value : 2
Pointer:
croco 23
… Value 10
Pointer:
croco 1
Value: 11
Pointer:
croco 2
Leave
High Key: 14
Value:
12
Pointer:
croco 17
Value:
13
Pointer:
croco 3
Value :
13
Pointer:
croco 4
… Value 13
Pointer:
croco 27
…
Leave
High Key: None
Value: 33
Pointer:
croco 5
Value: 33
Pointer:
croco 6
Value : 33
Pointer:
croco 123
… Value: 38
Pointer:
croco 26
BTrees are good for the following operations: =, >, <, >=, <=
Why?
Because the value indexed is the value of the column(s) so we can
easily perform binary search in the BTree

25. @louisemeta
GIN

26. @louisemeta
GIN
- Used to index arrays, jsonb, and tsvector (for fulltext search) columns.
- Efficient for <@, &&, @@@ operators
New column healed_teeth:
croco=# SELECT email, number_of_teeth, healed_teeth FROM crocodile WHERE id =1;
-[ RECORD 1 ]---+--------------------------------------------------------
email | louise.grandjonc1@croco.com
number_of_teeth | 58
healed_teeth | {16,11,55,27,22,41,38,2,5,40,52,57,28,50,10,15,1,12,46}
!26

27. @louisemeta
Creating a GIN index
Here is how to create the GIN index for this column
CREATE INDEX ON crocodile USING GIN(healed_teeth);
!27
from django.contrib.postgres.indexes import GinIndex
class Crocodile(models.Model):
...
class Meta:
indexes = [GinIndex(fields=['healed_teeth'])]
Raw SQL

28. @louisemeta
GIN
How is it different from a BTree?
- In a GIN index, the array is split and each value is an entry
- The values are unique
- As the value is unique, in the leaves, we keep a list of pointers to the rows
!28
Root
Value: <begin>
Value: 10
Value: 20
…
Parent
Value: 1
Value: 4
Value: 6
…
Parent
Value: 10
Value: 15
Value: 17
…
Parent
Value: 20
Value: 24
Value: 26
…
Leaf
Pointers: {(269, 49),
(296, 51), (296, 54),
(296, 57), …}
Pointers: { (306, 33),
(306, 35), (306,36), …}
…
Leaf
Pointer to posting tree
…
Page
Page
Page
Root
Posting tree

29. @louisemeta
GIN
How is it different from a BTree?
Bitmap Heap Scan on crocodile
(cost=516.59..6613.42 rows=54786 width=29)
(actual time=15.960..38.197 rows=73275 loops=1)
Recheck Cond: ('{1,2}'::integer[] <@ healed_teeth)
Heap Blocks: exact=4218
-> Bitmap Index Scan on crocodile_healed_teeth_idx
(cost=0.00..502.90 rows=54786 width=0)
(actual time=15.302..15.302 rows=73275 loops=1)
Index Cond: ('{1,2}'::integer[] <@ healed_teeth)
Planning time: 0.124 ms
Execution time: 41.018 ms
(7 rows)
Seq Scan on crocodile (cost=…)
Filter: ('{1,2}'::integer[] <@ healed_teeth)
Rows Removed by Filter: 250728
Planning time: 0.157 ms
Execution time: 161.716 ms
(5 rows)
!29

30. @louisemeta
To sum it up
@louisemeta !30
- A GIN index is a balanced tree
- Each value in the tree is unique
- The row value is split and each value is an entry
- Efficient for <@, &&, @@@ operators

31. @louisemeta
GIST

32. @louisemeta
GiST - keys
Differences with a BTree index
- Data isn’t ordered
- The key ranges can overlap
Which means that a same value can be inserted in different pages
!32
To be more readable, the following example here is for a Integer Range type :)
Root
Page block number: 0
Page level: 0
Value: [3, 5]
Value: [0, 2]
Value: [4, 8]
Value: [7, 9]
Parent
Page block number: 4699
Page level: 1
Parent
Page block number: 1610
Page level: 1
Parent
Page block number: 813
Page level: 1
Parent
Page block number: 6249
Page level: 1

33. @louisemeta
Creating a GiST index
!33
Here is how to create the GiST index for this column
CREATE INDEX ON appointment USING GIST(schedule);
from django.contrib.postgres.indexes import GistIndex
class Appointment(models.Model):
...
class Meta:
indexes = [GistIndex(fields=[‘schedule'])]
Raw SQL

34. @louisemeta
Why use GiST
- Useful for overlapping (geometries, array, range etc.)
- Especially useful when using postgis
- Nearest neighbor
- Can be used for full text search (tsvector, tsquery)
!34

35. @louisemeta
GiST or GIN for fulltext search
movies=# CREATE INDEX ON film USING GIN(fulltext) with (fastupdate=off);
CREATE INDEX
Time: 8.083 ms
movies=# INSERT INTO film (title, description, language_id) VALUES ('Nightmare at the
dentist', 'A crocodile calls his dentist on halloween and ends up toothless and very
sad, warning: not for kids, or teeth-sensitive crocodiles', 1);
INSERT 0 1
Time: 3.057 ms
movies=# INSERT INTO film (title, description, language_id) VALUES ('Nightmare at the
dentist', 'The terrible adventure of a crocodile who never goes to the dentist', 1);
INSERT 0 1
Time: 1.323 ms
- Maintaining a GIN index is slower than GiST
!35

36. @louisemeta
GiST or GIN for fulltext search
- Lookups are faster with GIN
movies=# SELECT COUNT(*) FROM film WHERE fulltext @@ to_tsquery('crocodile');
count
-------
106
(1 row)
Time: 1.275 ms
movies=# SELECT COUNT(*) FROM film WHERE fulltext @@ to_tsquery('crocodile');
count
-------
106
(1 row)
Time: 0.467 ms
!36

37. @louisemeta
GiST or GIN for fulltext search
- GIN indexes are larger than GiST
movies=# di+ film_fulltext_idx
List of relations
Schema | Name | Type | Owner | Table | Size | Description
--------+-------------------+-------+----------+-------+-------+-------------
public | film_fulltext_idx | index | postgres | film | 88 kB |
(1 row)
movies=# di+ film_fulltext_gin_idx
List of relations
Schema | Name | Type | Owner | Table | Size | Description
--------+-----------------------+-------+----------+-------+--------+-------------
public | film_fulltext_gin_idx | index | postgres | film | 112 kB |
(1 row)
!37

38. @louisemeta
BRIN

39. @louisemeta
BRIN
Internal data structure
- Block Range Index
- Not a balanced tree
- Not even a tree
- Block range: group of pages physically adjacent
- For each block range: the range of values is stored
- BRIN indexes are very small
- Fast scanning on large tables
!39

40. @louisemeta
BRIN
Internal data structure
SELECT * FROM brin_page_items(get_raw_page('appointment_created_at_idx', 2), 'appointment_created_at_idx');
itemoffset | blknum | attnum | allnulls | hasnulls | placeholder | value
------------+--------+--------+----------+----------+-------------+---------------------------------------------------
1 | 0 | 1 | f | f | f | {2008-03-01 00:00:00-08 .. 2009-07-07 07:30:00-07}
2 | 128 | 1 | f | f | f | {2009-07-07 08:00:00-07 .. 2010-11-12 15:30:00-08}
3 | 256 | 1 | f | f | f | {2010-11-12 16:00:00-08 .. 2012-03-19 23:30:00-07}
4 | 384 | 1 | f | f | f | {2012-03-20 00:00:00-07 .. 2013-07-26 07:30:00-07}
5 | 512 | 1 | f | f | f | {2013-07-26 08:00:00-07 .. 2014-12-01 15:30:00-08}
SELECT id, created_at FROM appointment WHERE ctid='(0, 1)'::tid;
id | created_at
--------+------------------------
101375 | 2008-03-01 00:00:00-08
(1 row)
!40

41. @louisemeta
BRIN
Internal data structure
SELECT * FROM brin_page_items(get_raw_page('crocodile_birthday_idx', 2),
'crocodile_birthday_idx');
itemoffset | blknum | attnum | allnulls | hasnulls | placeholder | value
------------+--------+--------+----------+----------+-------------+----------------------------
1 | 0 | 1 | f | f | f | {1948-09-05 .. 2018-09-04}
2 | 128 | 1 | f | f | f | {1948-09-07 .. 2018-09-03}
3 | 256 | 1 | f | f | f | {1948-09-05 .. 2018-09-03}
4 | 384 | 1 | f | f | f | {1948-09-05 .. 2018-09-04}
5 | 512 | 1 | f | f | f | {1948-09-05 .. 2018-09-02}
6 | 640 | 1 | f | f | f | {1948-09-09 .. 2018-09-04}
…
(14 rows)
In this case, the values in birthday has no correlation with the physical
location, the index would not speed up the search as all pages would have
to be visited.
BRIN is interesting for data where the value is correlated with the
physical location.
!41

42. @louisemeta
BRIN
Warning on DELETE and INSERT
SELECT * FROM brin_page_items(get_raw_page('appointment_created_at_idx', 2), 'appointment_created_at_idx');
itemoffset | blknum | attnum | allnulls | hasnulls | placeholder | value
------------+--------+--------+----------+----------+-------------+---------------------------------------------------
1 | 0 | 1 | f | f | f | {2008-03-01 00:00:00-08 .. 2018-07-01 07:30:00-07}
2 | 128 | 1 | f | f | f | {2009-07-07 08:00:00-07 .. 2018-07-01 23:30:00-07}
3 | 256 | 1 | f | f | f | {2010-11-12 16:00:00-08 .. 2012-03-19 23:30:00-07}
4 | 384 | 1 | f | f | f | {2012-03-20 00:00:00-07 .. 2018-07-06 23:30:00-07}
DELETE FROM appointment WHERE created_at >= '2009-07-07' AND created_at < ‘2009-07-08';
DELETE FROM appointment WHERE created_at >= '2012-03-20' AND created_at < ‘2012-03-25';
Deleted and then vacuum on the appointment table
New rows are inserted in the free space after VACUUM
BRIN index has some ranges with big data ranges.
Search will visit a lot of pages.
!42

43. @louisemeta
Creating a BRIN index
!43
Here is how to create the BRIN index for this column
CREATE INDEX ON crocodile USING BRIN(created_at);
from django.contrib.postgres.indexes import BrinIndeex
class Crocodile(models.Model):
...
class Meta:
indexes = [BrinIndex(fields=['created_at'])]
Raw SQL

44. @louisemeta
Conclusion
- B-Tree
- Great for <, >, =, >=, <=
- GIN
- Fulltext search, jsonb, arrays
- Inserts can be slow because of unicity of the
keys
- GiST
- Great for overlapping
- Using key class functions
- Can be implemented for any data type
- BRIN
- Great for huge table with correlation
between value and physical location
- <, >, =, >=, <=
!44

45. @louisemeta
Questions
Thanks for your attention
Go read the articles www.louisemeta.com
Now only the ones on BTrees and GIN are
published, but I’ll announce the rest on
twitter @louisemeta
Crocodiles by https://www.instagram.com/zimmoriarty/?hl=en
!45