SlideShare a Scribd company logo
1 of 46
© 2013 by Markus Winand
iStockPhoto/Mitshu
Pagination done
the Right Way
@MarkusWinand
@SQLPerfTips
© 2013 by Markus Winand
Takeaways
‣OFFSET kills performance
‣Paging can be done without OFFSET
‣It’s faster and has fewer side effects
© 2013 by Markus Winand
Note
In this presentation
index means B-tree index.
A Trivial Example
A query to fetch the 10 most recent news:
se!e"# *
$%om news
whe%e #op&" ' 1234
o!de! b" da#e des$, %d des$
&%m%# 10;
$!ea#e %ndex .. on news(#op%$);
Using o%de% b( to get the most recent first and
!&m&# to fetch only the first 10.
Alternative SQL-2008 syntax (since PostgreSQL 8.4)
$e#"h $&%s# 10 %ows on!(
Worst Case: No Index for o%de% b(
L&m&# (a$#ua& !ows'10)
-> So%# (a$#ua& !ows'10)
So%# Me#hod: #op-N heapso!# Memo!": 18(B
-> B&#map Heap S"an (!ows'10000)
Re"he") *ond: (#op&" ' 1234)
-> B&#map Index S"an (!ows'10000)
Index *ond: (#op&" ' 1234)
Worst Case: No Index for order by
The limiting factor is the number of rows that match
the whe%e clause
(“Base-Set Size”).
The database might use
an index to satisfy the
whe%e clause, but must
still fetch all matching
rows to “sort” them.
Another Benchmark: Fetch Next Page
Fetching the next page is easy using the o$$se#
keyword:
se!e"# *
$%om news
whe%e #op&" ' 1234
o%de% b( da#e des", &d des"
o))se# 10
!&m&# 10;
Worst Case: No Index for order by
L&m&# (a"#ua! %ows'10)
-> So%# (a$#ua& !ows'20)
So%# Me#hod: #op-N heapso!# Memo!": 19(B
-> B&#map Heap S"an (a"#ua! %ows'10000)
Re"he") *ond: (#op&" ' 1234)
-> B&#map Index S"an (a"#ua! %ows'10000)
Index *ond: (#op&" ' 1234)
Worst Case: No Index for order by
L&m&# (a"#ua! %ows'10)
-> So%# (a$#ua& !ows'30)
So%# Me#hod: #op-N heapso!# Memo!": 20(B
-> B&#map Heap S"an (a"#ua! %ows'10000)
Re"he") *ond: (#op&" ' 1234)
-> B&#map Index S"an (a"#ua! %ows'10000)
Index *ond: (#op&" ' 1234)
Worst Case: No Index for order by
L&m&# (a"#ua! %ows'10)
-> So%# (a$#ua& !ows'40)
So%# Me#hod: #op-N heapso!# Memo!": 22(B
-> B&#map Heap S"an (a"#ua! %ows'10000)
Re"he") *ond: (#op&" ' 1234)
-> B&#map Index S"an (a"#ua! %ows'10000)
Index *ond: (#op&" ' 1234)
Worst Case: No Index for order by
L&m&# (a"#ua! %ows'10)
-> So%# (a$#ua& !ows'10000)
So%# Me#hod: ex#e!na& me!*e D%s(: 1200(B
-> B&#map Heap S"an (a"#ua! %ows'10000)
Re"he") *ond: (#op&" ' 1234)
-> B&#map Index S"an (a"#ua! %ows'10000)
Index *ond: (#op&" ' 1234)
Worst Case: No Index for order by
Sorting might become the limiting factor when
browsing farther back.
Fetching the last page
can take considerably
longer than fetching
the first page.
Improvement 1: Indexed order by
se!e"# *
$%om news
whe%e #op&" ' 1234
o%de% b( da#e des", &d des"
o$$se# 10
!&m&# 10;
$!ea#e %ndex .. on news (#op%$, da#e, %d);
A single index to support the whe%e and o%de% b(
clauses.
Improvement 1: Indexed order by
L&m&# (a$#ua& !ows'10)
-> Index S"an Ba")wa%d (a$#ua& !ows'10)
Index *ond: (#op&" ' 0)
Improvement 1: Indexed order by
L&m&# (a$#ua& !ows'10)
-> Index S"an Ba")wa%d (a$#ua& !ows'20)
Index *ond: (#op&" ' 0)
Improvement 1: Indexed order by
L&m&# (a$#ua& !ows'10)
-> Index S"an Ba")wa%d (a$#ua& !ows'30)
Index *ond: (#op&" ' 0)
Improvement 1: Indexed order by
L&m&# (a$#ua& !ows'10)
-> Index S"an Ba")wa%d (a$#ua& !ows'40)
Index *ond: (#op&" ' 0)
Improvement 1: Indexed order by
L&m&# (a"#ua! %ows'10)
-> So%# (a$#ua& !ows'10000)
So%# Me#hod: ex#e!na& me!*e D%s(: 1200(B
-> B&#map Heap S"an (a"#ua! %ows'10000)
Re"he") *ond: (#op&" ' 1234)
-> B&#map Index S"an (a"#ua! %ows'10000)
Index *ond: (#op&" ' 1234)
Improvement 1: Indexed order by
Fetching the first page is not affected by the
Base-Set size!
Fetching the next page
is also faster.
However, PostgreSQL
might take a Bitmap
Index Scan when
browsing to the end.
© 2013 by Markus Winand
We can do better!
© 2013 by Markus Winand
Don’t touch what you don’t need
Improvement 2: The Seek Method
Instead of o$$se#, use a whe%e filter to remove the
rows from previous pages.
se!e"# *
$%om news
whe%e #op&" ' 1234
and (da#e, %d) < (p!e+_da#e, p!e+_%d)
o%de% b( da#e des", &d des"
!&m&# 10;
Only select the rows “before” (=earlier date) the
last!row from the previous page.
A definite sort order is really required!
© 2013 by Markus Winand
Side Note: Row Values
Besides scalar values, SQL also defines
“row values” or “composite values.”
‣ In the SQL standard since ages (SQL-92)
‣ All comparison operators are well defined
‣ E.g.: (x, y) > (a, b) is true iff
(x > a or (x=a and y>b))
‣ In other words, when (x,y) sorts after (a,b)
‣ Great PostgreSQL support since 8.0!
© 2013 by Markus Winand
Row Values: How it works
© 2013 by Markus Winand
Row Values: How it works
© 2013 by Markus Winand
Row Values: How it works
© 2013 by Markus Winand
Row Values: Emulating them
‣ There is only a single database that has sufficient
Row-Values support for this trick (PostgreSQL).
‣ With all other database, you must emulate it
according to this scheme:
!"#$#%&'()*+',)%-.%/
%%%%012%3
%%%%%%%%%%%3&'()*+',)%-%/4
%%%%%%%%%5$
%%%%%%%%%%%3&'()*+',)%.%/%012%&'()*6+%-%/4
%%%%%%%%4
See: http://use-the-index-luke.com/sql/partial-results/fetch-next-page#sb-equivalent-logic
Seek-Method without Optimal Index
se!e"# *
$%om news
whe%e #op&" ' 1234
and (da#e, &d) < (p%e+_da#e, p%e+_&d)
o%de% b( da#e des", &d des"
!&m&# 10;
$!ea#e %ndex .. on news (#op%$);
Seek Method w/o Index for order by
L&m&# (a$#ua& !ows'10)
-> So%# (a$#ua& !ows'10)
So%# Me#hod: #op-N heapso!# Memo!": 18(B
-> B&#map Heap S"an (!ows'10000)
Re"he") *ond: (#op&" ' 1234)
-> B&#map Index S"an (!ows'10000)
Index *ond: (#op&" ' 1234)
Seek Method w/o Index for order by
L&m&# (a"#ua! %ows'10)
-> So%# (a$#ua& !ows'10)
So%# Me#hod: #op-N heapso!# Memo!": 18(B
-> B&#map Heap S"an (a$#ua& !ows'9990)
Rows Remo+ed b" F%&#e!: 10 (new &n 9.2)
-> B&#map Index S"an (a$#ua& !ows'10000)
Index *ond: (#op&" ' 1234)
Seek Method w/o Index for order by
L&m&# (a"#ua! %ows'10)
-> So%# (a$#ua& !ows'10)
So%# Me#hod: #op-N heapso!# Memo!": 18(B
-> B&#map Heap S"an (a$#ua& !ows'9980)
Rows Remo+ed b" F%&#e!: 20 (new &n 9.2)
-> B&#map Index S"an (a$#ua& !ows'10000)
Index *ond: (#op&" ' 1234)
Seek Method w/o Index for order by
L&m&# (a"#ua! %ows'10)
-> So%# (a$#ua& !ows'10)
So%# Me#hod: #op-N heapso!# Memo!": 18(B
-> B&#map Heap S"an (a$#ua& !ows'9970)
Rows Remo+ed b" F%&#e!: 30 (new &n 9.2)
-> B&#map Index S"an (a$#ua& !ows'10000)
Index *ond: (#op&" ' 1234)
Seek Method w/o Index for order by
L&m&# (a"#ua! %ows'10)
-> So%# (a$#ua& !ows'10)
So%# Me#hod: #op-N heapso!# Memo!": 18(B
-> B&#map Heap S"an (a$#ua& !ows'10)
Rows Remo+ed b" F%&#e!: 9990 (new &n 9.2)
-> B&#map Index S"an (a$#ua& !ows'10000)
Index *ond: (#op&" ' 1234)
Seek Method w/o Index for order by
Always needs to retrieve the full base set, but the
top-n sort buffer needs
to hold only 10 rows.
The response time remains
the same even when
browsing to the last page.
And the memory footprint
is very low!
Seek-Method with Optimal Index
se!e"# *
$%om news
whe%e #op&" ' 1234
and (da#e, &d) < (p%e+_da#e, p%e+_&d)
o%de% b( da#e des", &d des"
!&m&# 10;
$!ea#e %ndex .. on news (#op%$, da#e, %d);
Seek Method with index for order by
L&m&# (a$#ua& !ows'10)
-> Index S"an Ba")wa%d (a$#ua& !ows'10)
Index *ond: (#op&" ' 1234)
Seek Method with index for order by
L&m&# (a$#ua& !ows'10)
-> Index S"an Ba")wa%d (a$#ua& !ows'10)
Index *ond: ((#op&" ' 1234)
AND (ROW(d#, &d) < ROW(‘...’, 23456)))
Seek Method with index for order by
L&m&# (a$#ua& !ows'10)
-> Index S"an Ba")wa%d (a$#ua& !ows'10)
Index *ond: ((#op&" ' 1234)
AND (ROW(d#, &d) < ROW(‘...’, 34567)))
Seek Method with index for order by
L&m&# (a$#ua& !ows'10)
-> Index S"an Ba")wa%d (a$#ua& !ows'10)
Index *ond: ((#op&" ' 1234)
AND (ROW(d#, &d) < ROW(‘...’, 45678)))
Seek Method with index for order by
L&m&# (a$#ua& !ows'10)
-> Index S"an Ba")wa%d (a$#ua& !ows'10)
Index *ond: ((#op&" ' 1234)
AND (ROW(d#, &d) < ROW(‘...’, 56789)))
Seek Method with index for order by
Successively browsing back doesn’t slow down.
Neither the size of the
base set(*) nor the fetched
page number affects the
response time.
(*) the index tree depth still affects the response time.
ComparisonOffsetSeek
W/O index for o%de% b( With index for o%de% b(
© 2013 by Markus Winand
Too good to be true?
The Seek Method has serious limitations
‣You cannot directly navigate to arbitrary pages
‣because you need the values from the previous page
‣Bi-directional navigation is possible but tedious
‣you need to revers the o%de% b( direction and RV comparison
‣Works best with full row values support
‣Workaround is possible, but ugly and less performant
‣Framework support?
‣jOOQ 3.3 introduced native support for the Seek-Method
‣order_query for Ruby by @glebm: http://github.com/glebm/order_query
© 2013 by Markus Winand
A Perfect Match for Infinite Scrolling
The “Infinite Scrolling” UI doesn’t need to ...
‣navigate to arbitrary pages
‣there are no buttons
‣Browse backwards
‣previous pages are
still in the browser
‣Stable results
‣New rows don’t affect the result
No need for de-duplication in JS
Also a Perfect Match for PostgreSQL
o%de% b(
support matrix
row values
support matrix
Popular
Popular
Advanced
Advanced
© 2013 by Markus Winand
About Markus Winand
Tuning developers for
high SQL performance
Training & co (one-man show):
winand.at
Geeky blog:
use-the-index-luke.com
Author of:
SQL Performance Explained

More Related Content

Similar to Pagination Done the Right Way

Un petit guide de la domination du monde
Un petit guide de la domination du mondeUn petit guide de la domination du monde
Un petit guide de la domination du mondeGeoffrey Dorne
 
E-Primer Your Business Online
E-Primer Your Business OnlineE-Primer Your Business Online
E-Primer Your Business Onlineguestfc9d8a
 
Al Fazl International Weekly26 June 2015
Al Fazl International  Weekly26 June 2015Al Fazl International  Weekly26 June 2015
Al Fazl International Weekly26 June 2015muzaffertahir9
 
The Lean Startup - simplified
The Lean Startup - simplifiedThe Lean Startup - simplified
The Lean Startup - simplifiedStefano Bernardi
 
Interaction design
Interaction designInteraction design
Interaction designfeifei2011
 
treinamento-sdh-alcatel_compress.pdf
treinamento-sdh-alcatel_compress.pdftreinamento-sdh-alcatel_compress.pdf
treinamento-sdh-alcatel_compress.pdfMaster22
 
LAMP_TRAINING_SESSION_6
LAMP_TRAINING_SESSION_6LAMP_TRAINING_SESSION_6
LAMP_TRAINING_SESSION_6umapst
 
What the !@#$ is UX? A fun and concise introduction
What the !@#$ is UX? A fun and concise introductionWhat the !@#$ is UX? A fun and concise introduction
What the !@#$ is UX? A fun and concise introductionSean Buch
 
Understanding variations of entanglement and complexity: A way to influence e...
Understanding variations of entanglement and complexity: A way to influence e...Understanding variations of entanglement and complexity: A way to influence e...
Understanding variations of entanglement and complexity: A way to influence e...RSD7 Symposium
 
Roberto Isaias - Derechos humanos
Roberto Isaias - Derechos humanosRoberto Isaias - Derechos humanos
Roberto Isaias - Derechos humanosRoberto Isaias
 
William Isaías-Derechos humanos
William Isaías-Derechos humanosWilliam Isaías-Derechos humanos
William Isaías-Derechos humanosWilliam Isaias
 

Similar to Pagination Done the Right Way (20)

Un petit guide de la domination du monde
Un petit guide de la domination du mondeUn petit guide de la domination du monde
Un petit guide de la domination du monde
 
OSGi - beyond the myth
OSGi -  beyond the mythOSGi -  beyond the myth
OSGi - beyond the myth
 
E-Primer Your Business Online
E-Primer Your Business OnlineE-Primer Your Business Online
E-Primer Your Business Online
 
Al Fazl International Weekly26 June 2015
Al Fazl International  Weekly26 June 2015Al Fazl International  Weekly26 June 2015
Al Fazl International Weekly26 June 2015
 
FCIP SASS Talk
FCIP SASS TalkFCIP SASS Talk
FCIP SASS Talk
 
The Lean Startup - simplified
The Lean Startup - simplifiedThe Lean Startup - simplified
The Lean Startup - simplified
 
Interaction design
Interaction designInteraction design
Interaction design
 
Curriculo de primer año
Curriculo de primer añoCurriculo de primer año
Curriculo de primer año
 
treinamento-sdh-alcatel_compress.pdf
treinamento-sdh-alcatel_compress.pdftreinamento-sdh-alcatel_compress.pdf
treinamento-sdh-alcatel_compress.pdf
 
Composicion danza2
Composicion danza2Composicion danza2
Composicion danza2
 
Mips1
Mips1Mips1
Mips1
 
All about Apache ACE
All about Apache ACEAll about Apache ACE
All about Apache ACE
 
LAMP_TRAINING_SESSION_6
LAMP_TRAINING_SESSION_6LAMP_TRAINING_SESSION_6
LAMP_TRAINING_SESSION_6
 
Device deployment
Device deploymentDevice deployment
Device deployment
 
What the !@#$ is UX? A fun and concise introduction
What the !@#$ is UX? A fun and concise introductionWhat the !@#$ is UX? A fun and concise introduction
What the !@#$ is UX? A fun and concise introduction
 
referente
referentereferente
referente
 
The Project Trap
The Project TrapThe Project Trap
The Project Trap
 
Understanding variations of entanglement and complexity: A way to influence e...
Understanding variations of entanglement and complexity: A way to influence e...Understanding variations of entanglement and complexity: A way to influence e...
Understanding variations of entanglement and complexity: A way to influence e...
 
Roberto Isaias - Derechos humanos
Roberto Isaias - Derechos humanosRoberto Isaias - Derechos humanos
Roberto Isaias - Derechos humanos
 
William Isaías-Derechos humanos
William Isaías-Derechos humanosWilliam Isaías-Derechos humanos
William Isaías-Derechos humanos
 

More from Markus Winand

Standard SQL features where PostgreSQL beats its competitors
Standard SQL features where PostgreSQL beats its competitorsStandard SQL features where PostgreSQL beats its competitors
Standard SQL features where PostgreSQL beats its competitorsMarkus Winand
 
Four* Major Database Releases of 2017 in Review
Four* Major Database Releases of 2017 in ReviewFour* Major Database Releases of 2017 in Review
Four* Major Database Releases of 2017 in ReviewMarkus Winand
 
Row Pattern Matching in SQL:2016
Row Pattern Matching in SQL:2016Row Pattern Matching in SQL:2016
Row Pattern Matching in SQL:2016Markus Winand
 
SQL Transactions - What they are good for and how they work
SQL Transactions - What they are good for and how they workSQL Transactions - What they are good for and how they work
SQL Transactions - What they are good for and how they workMarkus Winand
 
Volkskrankheit "Stiefmuetterliche Indizierung"
Volkskrankheit "Stiefmuetterliche Indizierung"Volkskrankheit "Stiefmuetterliche Indizierung"
Volkskrankheit "Stiefmuetterliche Indizierung"Markus Winand
 
SQL Performance - Vienna System Architects Meetup 20131202
SQL Performance - Vienna System Architects Meetup 20131202SQL Performance - Vienna System Architects Meetup 20131202
SQL Performance - Vienna System Architects Meetup 20131202Markus Winand
 

More from Markus Winand (6)

Standard SQL features where PostgreSQL beats its competitors
Standard SQL features where PostgreSQL beats its competitorsStandard SQL features where PostgreSQL beats its competitors
Standard SQL features where PostgreSQL beats its competitors
 
Four* Major Database Releases of 2017 in Review
Four* Major Database Releases of 2017 in ReviewFour* Major Database Releases of 2017 in Review
Four* Major Database Releases of 2017 in Review
 
Row Pattern Matching in SQL:2016
Row Pattern Matching in SQL:2016Row Pattern Matching in SQL:2016
Row Pattern Matching in SQL:2016
 
SQL Transactions - What they are good for and how they work
SQL Transactions - What they are good for and how they workSQL Transactions - What they are good for and how they work
SQL Transactions - What they are good for and how they work
 
Volkskrankheit "Stiefmuetterliche Indizierung"
Volkskrankheit "Stiefmuetterliche Indizierung"Volkskrankheit "Stiefmuetterliche Indizierung"
Volkskrankheit "Stiefmuetterliche Indizierung"
 
SQL Performance - Vienna System Architects Meetup 20131202
SQL Performance - Vienna System Architects Meetup 20131202SQL Performance - Vienna System Architects Meetup 20131202
SQL Performance - Vienna System Architects Meetup 20131202
 

Recently uploaded

Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfIngrid Airi González
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentPim van der Noll
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Strongerpanagenda
 
Accelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with PlatformlessAccelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with PlatformlessWSO2
 
All These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFAll These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFMichael Gough
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen
 
QCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesQCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesBernd Ruecker
 
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkReact Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkPixlogix Infotech
 
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...amber724300
 
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Alkin Tezuysal
 
Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Kaya Weers
 
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical InfrastructureVarsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructureitnewsafrica
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI AgeCprime
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integrationmarketing932765
 
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesAssure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesThousandEyes
 
Kuma Meshes Part I - The basics - A tutorial
Kuma Meshes Part I - The basics - A tutorialKuma Meshes Part I - The basics - A tutorial
Kuma Meshes Part I - The basics - A tutorialJoão Esperancinha
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfNeo4j
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Farhan Tariq
 
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security ObservabilityGlenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security Observabilityitnewsafrica
 

Recently uploaded (20)

Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdf
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
 
Accelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with PlatformlessAccelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with Platformless
 
All These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFAll These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDF
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
 
QCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesQCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architectures
 
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkReact Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App Framework
 
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
 
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
 
Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)
 
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical InfrastructureVarsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI Age
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
 
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesAssure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
 
Kuma Meshes Part I - The basics - A tutorial
Kuma Meshes Part I - The basics - A tutorialKuma Meshes Part I - The basics - A tutorial
Kuma Meshes Part I - The basics - A tutorial
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdf
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...
 
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security ObservabilityGlenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
 

Pagination Done the Right Way

  • 1. © 2013 by Markus Winand iStockPhoto/Mitshu Pagination done the Right Way @MarkusWinand @SQLPerfTips
  • 2. © 2013 by Markus Winand Takeaways ‣OFFSET kills performance ‣Paging can be done without OFFSET ‣It’s faster and has fewer side effects
  • 3. © 2013 by Markus Winand Note In this presentation index means B-tree index.
  • 4. A Trivial Example A query to fetch the 10 most recent news: se!e"# * $%om news whe%e #op&" ' 1234 o!de! b" da#e des$, %d des$ &%m%# 10; $!ea#e %ndex .. on news(#op%$); Using o%de% b( to get the most recent first and !&m&# to fetch only the first 10. Alternative SQL-2008 syntax (since PostgreSQL 8.4) $e#"h $&%s# 10 %ows on!(
  • 5. Worst Case: No Index for o%de% b( L&m&# (a$#ua& !ows'10) -> So%# (a$#ua& !ows'10) So%# Me#hod: #op-N heapso!# Memo!": 18(B -> B&#map Heap S"an (!ows'10000) Re"he") *ond: (#op&" ' 1234) -> B&#map Index S"an (!ows'10000) Index *ond: (#op&" ' 1234)
  • 6. Worst Case: No Index for order by The limiting factor is the number of rows that match the whe%e clause (“Base-Set Size”). The database might use an index to satisfy the whe%e clause, but must still fetch all matching rows to “sort” them.
  • 7. Another Benchmark: Fetch Next Page Fetching the next page is easy using the o$$se# keyword: se!e"# * $%om news whe%e #op&" ' 1234 o%de% b( da#e des", &d des" o))se# 10 !&m&# 10;
  • 8. Worst Case: No Index for order by L&m&# (a"#ua! %ows'10) -> So%# (a$#ua& !ows'20) So%# Me#hod: #op-N heapso!# Memo!": 19(B -> B&#map Heap S"an (a"#ua! %ows'10000) Re"he") *ond: (#op&" ' 1234) -> B&#map Index S"an (a"#ua! %ows'10000) Index *ond: (#op&" ' 1234)
  • 9. Worst Case: No Index for order by L&m&# (a"#ua! %ows'10) -> So%# (a$#ua& !ows'30) So%# Me#hod: #op-N heapso!# Memo!": 20(B -> B&#map Heap S"an (a"#ua! %ows'10000) Re"he") *ond: (#op&" ' 1234) -> B&#map Index S"an (a"#ua! %ows'10000) Index *ond: (#op&" ' 1234)
  • 10. Worst Case: No Index for order by L&m&# (a"#ua! %ows'10) -> So%# (a$#ua& !ows'40) So%# Me#hod: #op-N heapso!# Memo!": 22(B -> B&#map Heap S"an (a"#ua! %ows'10000) Re"he") *ond: (#op&" ' 1234) -> B&#map Index S"an (a"#ua! %ows'10000) Index *ond: (#op&" ' 1234)
  • 11. Worst Case: No Index for order by L&m&# (a"#ua! %ows'10) -> So%# (a$#ua& !ows'10000) So%# Me#hod: ex#e!na& me!*e D%s(: 1200(B -> B&#map Heap S"an (a"#ua! %ows'10000) Re"he") *ond: (#op&" ' 1234) -> B&#map Index S"an (a"#ua! %ows'10000) Index *ond: (#op&" ' 1234)
  • 12. Worst Case: No Index for order by Sorting might become the limiting factor when browsing farther back. Fetching the last page can take considerably longer than fetching the first page.
  • 13. Improvement 1: Indexed order by se!e"# * $%om news whe%e #op&" ' 1234 o%de% b( da#e des", &d des" o$$se# 10 !&m&# 10; $!ea#e %ndex .. on news (#op%$, da#e, %d); A single index to support the whe%e and o%de% b( clauses.
  • 14. Improvement 1: Indexed order by L&m&# (a$#ua& !ows'10) -> Index S"an Ba")wa%d (a$#ua& !ows'10) Index *ond: (#op&" ' 0)
  • 15. Improvement 1: Indexed order by L&m&# (a$#ua& !ows'10) -> Index S"an Ba")wa%d (a$#ua& !ows'20) Index *ond: (#op&" ' 0)
  • 16. Improvement 1: Indexed order by L&m&# (a$#ua& !ows'10) -> Index S"an Ba")wa%d (a$#ua& !ows'30) Index *ond: (#op&" ' 0)
  • 17. Improvement 1: Indexed order by L&m&# (a$#ua& !ows'10) -> Index S"an Ba")wa%d (a$#ua& !ows'40) Index *ond: (#op&" ' 0)
  • 18. Improvement 1: Indexed order by L&m&# (a"#ua! %ows'10) -> So%# (a$#ua& !ows'10000) So%# Me#hod: ex#e!na& me!*e D%s(: 1200(B -> B&#map Heap S"an (a"#ua! %ows'10000) Re"he") *ond: (#op&" ' 1234) -> B&#map Index S"an (a"#ua! %ows'10000) Index *ond: (#op&" ' 1234)
  • 19. Improvement 1: Indexed order by Fetching the first page is not affected by the Base-Set size! Fetching the next page is also faster. However, PostgreSQL might take a Bitmap Index Scan when browsing to the end.
  • 20. © 2013 by Markus Winand We can do better!
  • 21. © 2013 by Markus Winand Don’t touch what you don’t need
  • 22. Improvement 2: The Seek Method Instead of o$$se#, use a whe%e filter to remove the rows from previous pages. se!e"# * $%om news whe%e #op&" ' 1234 and (da#e, %d) < (p!e+_da#e, p!e+_%d) o%de% b( da#e des", &d des" !&m&# 10; Only select the rows “before” (=earlier date) the last!row from the previous page. A definite sort order is really required!
  • 23. © 2013 by Markus Winand Side Note: Row Values Besides scalar values, SQL also defines “row values” or “composite values.” ‣ In the SQL standard since ages (SQL-92) ‣ All comparison operators are well defined ‣ E.g.: (x, y) > (a, b) is true iff (x > a or (x=a and y>b)) ‣ In other words, when (x,y) sorts after (a,b) ‣ Great PostgreSQL support since 8.0!
  • 24. © 2013 by Markus Winand Row Values: How it works
  • 25. © 2013 by Markus Winand Row Values: How it works
  • 26. © 2013 by Markus Winand Row Values: How it works
  • 27. © 2013 by Markus Winand Row Values: Emulating them ‣ There is only a single database that has sufficient Row-Values support for this trick (PostgreSQL). ‣ With all other database, you must emulate it according to this scheme: !"#$#%&'()*+',)%-.%/ %%%%012%3 %%%%%%%%%%%3&'()*+',)%-%/4 %%%%%%%%%5$ %%%%%%%%%%%3&'()*+',)%.%/%012%&'()*6+%-%/4 %%%%%%%%4 See: http://use-the-index-luke.com/sql/partial-results/fetch-next-page#sb-equivalent-logic
  • 28. Seek-Method without Optimal Index se!e"# * $%om news whe%e #op&" ' 1234 and (da#e, &d) < (p%e+_da#e, p%e+_&d) o%de% b( da#e des", &d des" !&m&# 10; $!ea#e %ndex .. on news (#op%$);
  • 29. Seek Method w/o Index for order by L&m&# (a$#ua& !ows'10) -> So%# (a$#ua& !ows'10) So%# Me#hod: #op-N heapso!# Memo!": 18(B -> B&#map Heap S"an (!ows'10000) Re"he") *ond: (#op&" ' 1234) -> B&#map Index S"an (!ows'10000) Index *ond: (#op&" ' 1234)
  • 30. Seek Method w/o Index for order by L&m&# (a"#ua! %ows'10) -> So%# (a$#ua& !ows'10) So%# Me#hod: #op-N heapso!# Memo!": 18(B -> B&#map Heap S"an (a$#ua& !ows'9990) Rows Remo+ed b" F%&#e!: 10 (new &n 9.2) -> B&#map Index S"an (a$#ua& !ows'10000) Index *ond: (#op&" ' 1234)
  • 31. Seek Method w/o Index for order by L&m&# (a"#ua! %ows'10) -> So%# (a$#ua& !ows'10) So%# Me#hod: #op-N heapso!# Memo!": 18(B -> B&#map Heap S"an (a$#ua& !ows'9980) Rows Remo+ed b" F%&#e!: 20 (new &n 9.2) -> B&#map Index S"an (a$#ua& !ows'10000) Index *ond: (#op&" ' 1234)
  • 32. Seek Method w/o Index for order by L&m&# (a"#ua! %ows'10) -> So%# (a$#ua& !ows'10) So%# Me#hod: #op-N heapso!# Memo!": 18(B -> B&#map Heap S"an (a$#ua& !ows'9970) Rows Remo+ed b" F%&#e!: 30 (new &n 9.2) -> B&#map Index S"an (a$#ua& !ows'10000) Index *ond: (#op&" ' 1234)
  • 33. Seek Method w/o Index for order by L&m&# (a"#ua! %ows'10) -> So%# (a$#ua& !ows'10) So%# Me#hod: #op-N heapso!# Memo!": 18(B -> B&#map Heap S"an (a$#ua& !ows'10) Rows Remo+ed b" F%&#e!: 9990 (new &n 9.2) -> B&#map Index S"an (a$#ua& !ows'10000) Index *ond: (#op&" ' 1234)
  • 34. Seek Method w/o Index for order by Always needs to retrieve the full base set, but the top-n sort buffer needs to hold only 10 rows. The response time remains the same even when browsing to the last page. And the memory footprint is very low!
  • 35. Seek-Method with Optimal Index se!e"# * $%om news whe%e #op&" ' 1234 and (da#e, &d) < (p%e+_da#e, p%e+_&d) o%de% b( da#e des", &d des" !&m&# 10; $!ea#e %ndex .. on news (#op%$, da#e, %d);
  • 36. Seek Method with index for order by L&m&# (a$#ua& !ows'10) -> Index S"an Ba")wa%d (a$#ua& !ows'10) Index *ond: (#op&" ' 1234)
  • 37. Seek Method with index for order by L&m&# (a$#ua& !ows'10) -> Index S"an Ba")wa%d (a$#ua& !ows'10) Index *ond: ((#op&" ' 1234) AND (ROW(d#, &d) < ROW(‘...’, 23456)))
  • 38. Seek Method with index for order by L&m&# (a$#ua& !ows'10) -> Index S"an Ba")wa%d (a$#ua& !ows'10) Index *ond: ((#op&" ' 1234) AND (ROW(d#, &d) < ROW(‘...’, 34567)))
  • 39. Seek Method with index for order by L&m&# (a$#ua& !ows'10) -> Index S"an Ba")wa%d (a$#ua& !ows'10) Index *ond: ((#op&" ' 1234) AND (ROW(d#, &d) < ROW(‘...’, 45678)))
  • 40. Seek Method with index for order by L&m&# (a$#ua& !ows'10) -> Index S"an Ba")wa%d (a$#ua& !ows'10) Index *ond: ((#op&" ' 1234) AND (ROW(d#, &d) < ROW(‘...’, 56789)))
  • 41. Seek Method with index for order by Successively browsing back doesn’t slow down. Neither the size of the base set(*) nor the fetched page number affects the response time. (*) the index tree depth still affects the response time.
  • 42. ComparisonOffsetSeek W/O index for o%de% b( With index for o%de% b(
  • 43. © 2013 by Markus Winand Too good to be true? The Seek Method has serious limitations ‣You cannot directly navigate to arbitrary pages ‣because you need the values from the previous page ‣Bi-directional navigation is possible but tedious ‣you need to revers the o%de% b( direction and RV comparison ‣Works best with full row values support ‣Workaround is possible, but ugly and less performant ‣Framework support? ‣jOOQ 3.3 introduced native support for the Seek-Method ‣order_query for Ruby by @glebm: http://github.com/glebm/order_query
  • 44. © 2013 by Markus Winand A Perfect Match for Infinite Scrolling The “Infinite Scrolling” UI doesn’t need to ... ‣navigate to arbitrary pages ‣there are no buttons ‣Browse backwards ‣previous pages are still in the browser ‣Stable results ‣New rows don’t affect the result No need for de-duplication in JS
  • 45. Also a Perfect Match for PostgreSQL o%de% b( support matrix row values support matrix Popular Popular Advanced Advanced
  • 46. © 2013 by Markus Winand About Markus Winand Tuning developers for high SQL performance Training & co (one-man show): winand.at Geeky blog: use-the-index-luke.com Author of: SQL Performance Explained