PelotonDB is a self-driving database that uses a hybrid storage layout to support both OLTP and OLAP workloads. It uses logical tiles to decouple storage management from query execution. Tuples can be stored in either a narrow storage model, dense storage model, or flexible storage model depending on how "hot" the data is. The system continuously monitors queries and reorganizes the physical data layout in the background to optimize for the workload over time using k-means clustering. It employs MVCC for concurrency control and minimizes overhead during data reorganization by only modifying versioning metadata. An evaluation using the ADAPT benchmark shows PelotonDB can adapt the storage layout to improve performance.

1. PelotonDB
A self-driving database for hybrid workloads

2. 2

3. Agenda
• Overview of Peloton
• Hybrid Storage Layout
• Concurrency Control
• Layout Reorganization
• Evaluation
3

4. Overview of Peloton
4

5. How to be Self-driving?
• Understand the workload – OLTP or OLAP
• Forecast resource utilization trends
• Identify potential actions that tune and optimize the database
5

6. Architecture
6

7. Hybrid Storage Layout
7

8. Problems
• Many existing systems: OLTP + OLAP
• Takes minutes or even hours to propagate changes
• Administrative overhead
• Developer needs to write query for multiple systems
Transactional
Database
(e.g. MySQL)
Analytical
Database
(e.g. HiStore)
ETL
8

9. HTAP
• Classic solution – 2 separated engines
• OLTP engine with row-oriented data
• OLAP engine with column-oriented data
• Use a synchronization method (e.g. 2PC) to combine the results
• Well, this looks better but still too complex
• Limited types of queries
• Performance overhead
9

10. Hybrid Storage in Peloton
• A unified architecture for 'hot' and 'cold' tuples, based on a logical
abstraction over these different layouts
• A novel online reorganization technique that continuously enhances
the physical design
10

11. Storage Models
• NSM is good for OLTP
• DSM is good for OLAP
• FSM: adaptive as data get cooler
• NSM/DSM is special case of FSM
11

12. 12

13. Physical Tiles
• A tuple can be stored in
different layouts over time.
• Stores new tuples in NSM
• Reorganizes as they become
colder
13

14. Logical Tiles
Composed of
• Underlying physical tiles
• Mapping of attributes
• Offsets of values
14

15. Logical Tile Algebra
• Decoupling storage managing and execution engine
• Intermediate results can be represented as passthrough logical tile
15

16. Bridge Operator
Pipeline Breaker
Metadata Operator
Mutator
16

17. Benefits of Logical Tiles
• Layout Transparency
• Vectorized Processing
• Flexible Materialization
• Caching Behavior
17

18. Concurrency Control
18

19. MVCC
• HTAP workloads are comprised of short-duration transactions
alongside long-running analytical queries.
• Every transaction holds
• A unique transaction Id
• A unique commit timestamp (assigned on committing)
• Timestamp of last committed transaction
19

20. Versioning Metadata
• For each tuple
• TxnId: The transaction id that currently holds a latch
• BeginCTS: The commit timestamp from which it becomes visible
• EndCTS: The commit timestamp after which it ceases to be visible
• PreV: Reference to the previous version
20

21. Operations under MVCC
• Insert / Delete / Update
• TableScan / IndexScan
21

22. Layout Reorganization
22

23. Approach
• Track the recent query workloads
• Periodically compute a workload-optimized storage layout in the
background
23

24. Query Monitoring
• Collects information about the attributes in queries
24

25. Find Optimized Partition
• Naive algorithm takes . Infeasible!
• Heuristic approach
1. Clustering similar queries by k-means
• distance(q, p) = #attributes appears only in one side / #attributes
• Prioritizes each query based on its plan cost to avoid partial to TP queries
• Prioritizes the older samples with a weight w
2. Generate a layout in greedy way
• Iterates over the clusters in the weight-descending order
• For each cluster, groups the attributes accessed by that its representative
query together into a tile
25

26. K-means
26

27. Data Layout Reorganization
• Copies over the data to the new layout then atomically swaps in
• Concurrent DML operation only modifies the versioning metadata
• Old data is reclaimed if not referenced by any logical tile
27

28. Evaluation
28

29. ADAPT benchmark
Simulating enterprise workloads
29
# Attributes Size of tuple # Tuples PK
Narrow table 50 200B
10m a0
Wide table 500 2KB

30. 30
Performance Impact of Storage Models

31. 31Workload-Aware Adaptation

32. Thanks!
Q&A
32