The document discusses a robust access path selection method that avoids relying on cardinality estimation, highlighting the shortcomings of classical approaches which depend on outdated statistics and naive cost models. It proposes a new scanning mechanism that adapts at runtime, allowing for more effective selection without needing extensive statistics. The document details various policies for morphing scan methods and incorporates these innovations into a PostgreSQL implementation to enhance performance in diverse data situations.

1. Robust Access Path Selection without
Cardinality Estimation
Presented by Kenan Yao

2. PingCAP.com
Problem to Solve
● access path selection
● classical approach
○ predicate push down
○ ranger extracts access conditions
○ selectivity estimation / statistics
○ cost model
○ independent / uniform assumptions
Transaction

3. PingCAP.com
Drawbacks
● drawbacks of classical approach
○ heavily depends on statistics
○ outdated statistics / expensive to maintain
○ naive cost model
○ assumptions not hold
○ wrong choices / not robust

4. PingCAP.com
Proposal
● a proposal to dodge those drawbacks
○ new scan operator in executor
○ observe and behave accordingly at runtime
○ relieve planner from access path selection
○ robust by eschewing statistics / cardinality estimation

5. PingCAP.com
Modeling the Problem
● page caches not considered
● double read
● sensitive / not robust
Transaction

6. PingCAP.com
Alternative Approaches
● optimizer level: not robust enough
● runtime reoptimization
○ start from index scan
○ switch to table scan if scanned rows exceeds
tipping point
○ binary switch
○ bound worst case
○ risk area
○ not robust
Transaction

7. PingCAP.com
Smooth Scan
● rough idea
○ executor operator / runtime
○ start from index scan
○ morph between index scan and table scan
○ morph continuously and adaptively
○ trade off CPU and memory for I/O reduction

8. PingCAP.com
Storage Model
● PostgreSQL style: heap table / B+-tree index
● access path types
○ table scan / index scan
○ bitmap scan
■ reduce random access / cache miss
■ execution model: pipeline breaker
■ order property lost
Transaction

9. PingCAP.com
Smooth Scan Details
● targeted behavior
○ near-optimal
○ bullet proof of the estimation errors
○ no performance cliff / robust
Transaction

10. PingCAP.com
Morphing Mechanism
● start from simple index scan
○ monitor retrieved row count
● start morphing if selectivity exceeds threshold
○ probe entire heap page
○ fetch and probe adjacent heap pages if selectivity keeps
increasing
■ start from extra one page
■ morphing region size increases exponentially

11. PingCAP.com
Correctness
● no tuple missed
○ driven by index scan
● no duplicate tuple
○ bookkeeping
○ CPU / memory cost

12. PingCAP.com
Morphing Policy
● greedy policy
○ fast convergence for high selectivity
○ unnecessary overhead for low selectivity
● selectivity driven policy
○ monitor local and global selectivity
○ selectivity computed in page level
○ keep or increase morphing region size
● elastic policy
○ skewed data distribution
○ double morphing region size for dense region
○ halve morphing region size for sparse region

13. PingCAP.com
Threshold
● optimizer driven
○ retrieved row count exceeds optimizer’s estimate
● eager approach
○ start morphing from first tuple
● SLA driven

14. PingCAP.com
Implementation
● integrated with PostgreSQL
○ page ID cache
○ tuple ID cache
○ result cache to respect order property
■ hash-based data structure
■ store additional tuples found
■ check result cache before index probe
■ pipeline breaker to some extent
■ spill to disk when short of memory

15. PingCAP.com
Evaluation
● TPC-H and synthetic datasets
● clear database and OS caches

16. Any Questions ?
Thank You!