CockroachDB: Architecture of a Geo-Distributed SQL Database
•
8 likes•1,022 views
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2nZwuQF. Peter Mattis talks about how Cockroach Labs addressed the complexity of distributed databases with CockroachDB. He gives a tour of CockroachDB’s internals, covering the usage of Raft for consensus, the challenges of data distribution, distributed transactions, distributed SQL execution, and distributed SQL optimizations. Filmed at qconnewyork.com. Peter Mattis is the co-founder of Cockroach Labs where he works on a bit of everything, from low-level optimization of code to refining the overall design. He has worked on distributed systems, designing and implementing the original Gmail back-end search and storage system at Google and designing and implementing Colossus, the successor to Google's original distributed file system.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to CockroachDB: Architecture of a Geo-Distributed SQL Database
Similar to CockroachDB: Architecture of a Geo-Distributed SQL Database (20)
More from C4Media
More from C4Media (20)
Recently uploaded
Recently uploaded (20)
CockroachDB: Architecture of a Geo-Distributed SQL Database
- 1. Architecture of a Geo-Distributed SQL Database CockroachDB Peter Mattis (@petermattis), Co-founder & CTO
- 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ cockroachdb-distributed-sql/
- 3. Presented at QCon New York www.qconnewyork.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
- 4. CockroachDB: Geo-distributed SQL Database Make Data Easy • Distributed ○ Horizontally scalable to grow with your application • Geo-distributed ○ Handle datacenter failures ○ Place data near usage ○ Push computation near data • SQL ○ Lingua-franca for rich data storage ○ Schemas, indexes, and transactions make app development easier
- 5. AGENDA ● Introduction ● Ranges and Replicas ● Transactions ● SQL Data in a KV World ● SQL Execution ● SQL Optimization
- 6. Distributed, Replicated, Transactional KV* • Keys and values are strings ○ Lexicographically ordered by key • Multi-version concurrency control (MVCC) ○ Values are never updated “in place”, newer versions shadow older versions ○ Tombstones are used to delete values ○ Provides snapshot to each transaction • Monolithic key-space * Not exposed for external usage
- 7. Monolithic Key Space DOGS carl dagne figment jack pinetop sooshi stella zee muddy peetey lula lady Monolithic logical key space ● Ordered lexicographically by key
- 8. Ranges DOGS carl dagne figment jack pinetop sooshi stella zee muddy peetey lula lady carl dagne figment jack muddy peetey lula lady pinetop sooshi stella zee Key space divided into contiguous ~64MB ranges Ranges are small enough to be moved/split quickly Ranges are large enough to amortize indexing overhead
- 9. Range Indexing DOGS carl dagne figment jack pinetop sooshi stella zee muddy peetey lula lady Index structure used to locate ranges (very much like a B-tree) 1 2 3 carl - jack lady - peetey pinetop - zee carl dagne figment jack muddy peetey lula lady pinetop sooshi stella zee
- 10. Ordered Range Scans DOGS carl dagne figment jack pinetop sooshi zee peetey lula lady Ordered keys enable efficient range scans dogs >= “muddy” AND <= “stella” 1 2 3 carl - jack lady - peetey pinetop - zee carl dagne figment jack peetey lula lady pinetop sooshi zee muddy stella stella muddy
- 11. Transactional Updates DOGS carl dagne figment jack pinetop sooshi zee peetey lula lady Transactions used to insert records into ranges 1 2 3 carl - jack lady - peetey pinetop - zee stella muddy INSERT[sunny] INSERT[sunny] Space available in range? - YES carl dagne figment jack muddy peetey lula lady pinetop sooshi stella zee pinetop sooshi stella zee muddy peetey lula lady ✓?
- 12. Transactional Updates DOGS carl dagne figment jack pinetop sooshi zee peetey lula lady 1 2 3 carl - jack lady - peetey pinetop - zee stella muddy INSERT[sunny] carl dagne figment jack muddy peetey lula lady pinetop sooshi stella sunny zee pinetop sooshi stella zee muddy peetey lula lady ✓ Transactions used to insert records into ranges INSERT[sunny]
- 13. Range Splits DOGS carl dagne figment jack pinetop sooshi zee peetey lula lady 1 2 3 carl - jack lady - peetey pinetop - zee stella muddy INSERT[rudy] carl dagne figment jack muddy peetey lula lady pinetop sooshi stella sunny zee pinetop sooshi stella zee muddy peetey lula lady BUT… what happens when a range is full? ✓? INSERT[rudy] Space available in range? - NO
- 14. Range Splits DOGS carl dagne figment jack pinetop sooshi zee peetey lula lady stella muddy INSERT[rudy] carl dagne figment jack muddy peetey lula lady pinetop rudy sooshi pinetop sooshi stella zee muddy peetey lula lady Ranges are automatically split, a new range index is created & order maintained ✓ INSERT[rudy] split range and insert stella sunny zee 1 2 3 carl - jack lady - peetey pinetop - sooshi 4 stella - zee
- 15. Raft and Replication Ranges (~64MB) are the unit of replication Each range is a Raft group (Raft is a consensus replication protocol) Default to 3 replicas, though this is configurable • Important system ranges default to 5 replicas • Note: 2 replicas doesn’t make sense in consensus replication Raft group
- 16. Raft and Replication Raft provides “atomic replication” of commands Commands are proposed by the leaseholder replica and distributed to the follower replicas, but only accepted when a quorum of replicas have acknowledged receipt * Leaseholder == Raft leader Raft group LEASEHOLDER
- 17. node1 node2 node4 node3 Range Leases muddy peetey lula lady pinetop sooshi stella zee carl dagne figment jack Reads with consensus Reads must talk to a quorum of replicas READ[carl]
- 18. node1 node2 node4 node3 Range Leases muddy peetey lula lady pinetop sooshi stella zee carl dagne figment jack Reads without consensus One replica is chosen as the leaseholder READ[carl] leaseholder
- 19. node1 node2 node4 node3 Range Leases muddy peetey lula lady pinetop sooshi stella zee carl dagne figment jack Reads without consensus One replica is chosen as the leaseholder ● Coordinates writes (proposal, key locking) ● Performs reads READ[carl] leaseholder
- 20. node1 node2 node4 node3 Replica Placement muddy peetey lula lady pinetop sooshi stella zee ● Space ● Diversity ● Load ● Latency carl dagne figment jack Each Range is a Raft state machine A Range has 1 or more Replicas
- 21. node1 node2 node4 node3 Replica Placement: Diversity muddy peetey lula lady carl dagne figment jack Diversity optimizes placement of replicas across “failure domains” ● Disk ● Single machine ● Rack ● Datacenter ● Region pinetop sooshi stella zee
- 22. node1 node2 node6 node4 node5 Replica Placement: Load muddy peetey lula lady pinetop sooshi stella zee carl dagne figment jack Load Balances placement using heuristics that considers real-time usage metrics of the data itself This range is high load as it is accessed more than others While we show this for ranges within a single table, this is also applicable across all ranges across ALL tables, which is the more typical situation node1node3
- 23. Replica Placement: Latency & Geo-partitioning muddy peetey lula lady carl dagne figment jack pinetop sooshi stella zee USE/muddy USE/stella USE/figment USE/dagne USW/jack USW/lady USW/peetey USW/pinetop EU/carl EU/lula EU/sooshi EU/zee We apply a constraint that indicates regional placement so we can ensure low latency access or jurisdictional control of data
- 24. Rebalancing Replicas node1 node5 node4 node2 node3 NEW Scale: Add a node If we add a node to the cluster, CockroachDB automatically redistributed replicas to even load across the cluster Uses the replica placement heuristics from previous slides to decide which node to add to and which to remove from
- 25. Rebalancing Replicas node1 node5 node4 node2 node3 NEW Scale: Add a node If we add a node to the cluster, CockroachDB automatically redistributed replicas to even load across the cluster Uses the replica placement heuristics from previous slides Movement is decomposed into adding a replica followed by removing a replica
- 26. Rebalancing Replicas node1 node5 node4 node2 node3 NEW Scale: Add a node If we add a node to the cluster, CockroachDB automatically redistributed replicas to even load across the cluster Uses the replica placement heuristics from previous slides Movement is decomposed into adding a replica followed by removing a replica
- 27. Rebalancing Replicas node1 node5 node4 node2 node3 Loss of a node Permanent Failure If a node goes down, the Raft group realizes a replica is missing and replaces it with a new replica on an active node Uses the replica placement heuristics from previous slides
- 28. Rebalancing Replicas node1 node5 node4 node2 node3 Loss of a node Permanent Failure If a node goes down, the Raft group realizes a replica is missing and replaces it with a new replica on an active node Uses the replica placement heuristics from previous slides The failed replica is removed from the Raft group and a new replica created. The leaseholder sends a snapshot of the Range’s state to bring the new replica up to date.
- 29. Rebalancing Replicas node1 node5 node4 node2 Loss of a node Temporary Failure If a node goes down for a moment, the leaseholder can “catch up” any replica that is behind The leaseholder can send commands to be replayed OR it can send a snapshot of the current Range data. We apply heuristics to decide which is most efficient for a given failure. node3
- 30. AGENDA ● Introduction ● Ranges and Replicas ● Transactions ● SQL Data in a KV World ● SQL Execution ● SQL Optimization
- 31. Transactions Atomicity, Consistency, Isolation, Durability Serializable Isolation • As if the transactions are run in a serial order • Gold standard isolation level • Make Data Easy - weaker isolation levels are too great a burden Transactions can span arbitrary ranges Conversational • The full set of operations is not required up front
- 32. Transactions Raft provides atomic writes to individual ranges Bootstrap transaction atomicity using Raft atomic writes Transaction record atomically flipped from PENDING to COMMIT
- 34. Distributed Transactions node1 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady node2 node3 carl peetey dagne lady lula muddy peetey lady node4 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady carl dagne figment jack pinetop sooshi stella zee pinetop sooshi stella zee BEGIN TXN1 WRITE[sunny] GATEWAY INSERT INTO dogs VALUES (sunny, ozzie) transactions TXN1: PENDING pinetop sooshi stella zee
- 35. Distributed Transactions node1 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady node2 node3 carl peetey dagne lady lula muddy peetey lady node4 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady carl dagne figment jack pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee BEGIN TXN1 WRITE[sunny] GATEWAY INSERT INTO dogs VALUES (sunny, ozzie) transactions TXN1: PENDING pinetop sooshi stella sunny zee
- 36. Distributed Transactions node1 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady node2 node3 carl peetey dagne lady lula muddy peetey lady node4 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady carl dagne figment jack pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee BEGIN TXN1 WRITE[sunny] GATEWAY INSERT INTO dogs VALUES (sunny, ozzie) transactions TXN1: PENDING pinetop sooshi stella sunny zee ACK pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee
- 37. Distributed Transactions node1 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady node2 node3 carl peetey dagne lady lula muddy peetey lady node4 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady carl dagne figment jack pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee BEGIN TXN1 WRITE[sunny] GATEWAY INSERT INTO dogs VALUES (sunny, ozzie) transactions TXN1: PENDING pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee
- 38. Distributed Transactions node1 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady node2 node3 carl peetey dagne lady lula muddy peetey lady node4 carl dagne figment jack carl dagne figment jack pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee BEGIN TXN1 WRITE[sunny] WRITE[ozzie] GATEWAY INSERT INTO dogs VALUES (sunny, ozzie) transactions TXN1: PENDING pinetop sooshi stella sunny zee carl peetey dagne lady lula muddy ozzie peetey lady ACK pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee
- 39. Distributed Transactions node1 carl dagne figment jack carl peetey dagne lady lula muddy ozzie peetey lady node2 node3 carl peetey dagne lady lula muddy ozzie peetey lady node4 carl dagne figment jack carl dagne figment jack BEGIN TXN1 WRITE[sunny] WRITE[ozzie] GATEWAY INSERT INTO dogs VALUES (sunny, ozzie) transactions TXN1: PENDING carl peetey dagne lady lula muddy ozzie peetey lady pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee
- 40. Distributed Transactions node1 carl dagne figment jack node2 node3 node4 carl dagne figment jack carl dagne figment jack BEGIN TXN1 WRITE[sunny] WRITE[ozzie] GATEWAY INSERT INTO dogs VALUES (sunny, ozzie) transactions TXN1: PENDING ACK pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee carl peetey dagne lady lula muddy ozzie peetey lady carl peetey dagne lady lula muddy ozzie peetey lady carl peetey dagne lady lula muddy ozzie peetey lady
- 41. Distributed Transactions node1 carl dagne figment jack node2 node3 node4 carl dagne figment jack carl dagne figment jack BEGIN TXN1 WRITE[sunny] WRITE[ozzie] GATEWAY INSERT INTO dogs VALUES (sunny, ozzie) transactions TXN1: PENDING ACK pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee carl peetey dagne lady lula muddy ozzie peetey lady carl peetey dagne lady lula muddy ozzie peetey lady carl peetey dagne lady lula muddy ozzie peetey lady
- 42. Distributed Transactions node1 carl dagne figment jack carl peetey dagne lady lula muddy ozzie peetey lady node2 node3 carl peetey dagne lady lula muddy ozzie peetey lady node4 carl dagne figment jack carl dagne figment jack pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee BEGIN TXN1 WRITE[sunny] WRITE[ozzie] COMMIT GATEWAY INSERT INTO dogs VALUES (sunny, ozzie) transactions TXN1: COMMIT pinetop sooshi stella sunny zee carl peetey dagne lady lula muddy ozzie peetey lady
- 43. Distributed Transactions node1 carl dagne figment jack carl peetey dagne lady lula muddy ozzie peetey lady node2 node3 carl peetey dagne lady lula muddy ozzie peetey lady node4 carl dagne figment jack carl dagne figment jack pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee BEGIN TXN1 WRITE[sunny] WRITE[ozzie] COMMIT GATEWAY INSERT INTO dogs VALUES (sunny, ozzie) pinetop sooshi stella sunny zee carl peetey dagne lady lula muddy ozzie peetey lady ACK
- 46. Transactions: Pipelining Serial Pipelined txn:sunny (pending) sunny ozzie sunny ozzie BEGIN WRITE[sunny] WRITE[ozzie]
- 47. Transactions: Pipelining Serial Pipelined txn:sunny (pending) sunny ozzie txn:sunny (commit)[keys: sunny, ozzie] txn:sunny (staged)[keys: sunny, ozzie] sunny ozzie BEGIN WRITE[sunny] WRITE[ozzie] COMMIT
- 48. Transactions: Pipelining Serial Pipelined txn:sunny (pending) sunny ozzie txn:sunny (commit)[keys: sunny, ozzie] BEGIN WRITE[sunny] WRITE[ozzie] COMMIT Committed once all operations complete We replaced the centralized commit marker with a distributed one t sunny ozzie txn:sunny (staged)[keys: sunny, ozzie] * “Proved” with TLA+
- 49. AGENDA ● Introduction ● Ranges and Replicas ● Transactions ● SQL Data in a KV World ● SQL Execution ● SQL Optimization
- 50. SQL Structured Query Language Declarative, not imperative • These are the results I want vs perform these operations in this sequence Relational data model • Typed: INT, FLOAT, STRING, ... • Schemas: tables, rows, columns, foreign keys
- 51. SQL: Tabular Data in a KV World SQL data has columns and types?!? How do we store typed and columnar data in a distributed, replicated, transactional key-value store? • The SQL data model needs to be mapped to KV data • Reminder: keys and values are lexicographically sorted
- 52. CREATE TABLE inventory ( id INT PRIMARY KEY, name STRING, price FLOAT ) SQL Data Mapping: Inventory Table ID Name Price 1 Bat 1.11 2 Ball 2.22 3 Glove 3.33 Key Value /1 “Bat”,1.11 /2 “Ball”,2.22 /3 “Glove”,3.33
- 53. CREATE TABLE inventory ( id INT PRIMARY KEY, name STRING, price FLOAT ) SQL Data Mapping: Inventory Table ID Name Price 1 Bat 1.11 2 Ball 2.22 3 Glove 3.33 Key Value /<Table>/<Index>/1 “Bat”,1.11 /<Table>/<Index>/2 “Ball”,2.22 /<Table>/<Index>/3 “Glove”,3.33
- 54. CREATE TABLE inventory ( id INT PRIMARY KEY, name STRING, price FLOAT ) SQL Data Mapping: Inventory Table ID Name Price 1 Bat 1.11 2 Ball 2.22 3 Glove 3.33 Key Value /inventory/primary/1 “Bat”,1.11 /inventory/primary/2 “Ball”,2.22 /inventory/primary/3 “Glove”,3.33
- 55. CREATE TABLE inventory ( id INT PRIMARY KEY, name STRING, price FLOAT, INDEX name_idx (name) ) SQL Data Mapping: Inventory Table ID Name Price 1 Bat 1.11 2 Ball 2.22 3 Glove 3.33 Key Value /inventory/name_idx/”Bat”/1 ∅ /inventory/name_idx/”Ball”/2 ∅ /inventory/name_idx/”Glove”/3 ∅
- 56. CREATE TABLE inventory ( id INT PRIMARY KEY, name STRING, price FLOAT, INDEX name_idx (name) ) SQL Data Mapping: Inventory Table ID Name Price 1 Bat 1.11 2 Ball 2.22 3 Glove 3.33 4 Bat 4.44 Key Value /inventory/name_idx/”Bat”/1 ∅ /inventory/name_idx/”Ball”/2 ∅ /inventory/name_idx/”Glove”/3 ∅
- 57. CREATE TABLE inventory ( id INT PRIMARY KEY, name STRING, price FLOAT, INDEX name_idx (name) ) SQL Data Mapping: Inventory Table ID Name Price 1 Bat 1.11 2 Ball 2.22 3 Glove 3.33 4 Bat 4.44 Key Value /inventory/name_idx/”Bat”/1 ∅ /inventory/name_idx/”Ball”/2 ∅ /inventory/name_idx/”Glove”/3 ∅ /inventory/name_idx/”Bat”/4 ∅
- 58. AGENDA ● Introduction ● Ranges and Replicas ● Transactions ● SQL Data in a KV World ● SQL Execution ● SQL Optimization
- 59. SQL Execution Relational operators • Projection (SELECT <columns>) • Selection (WHERE <filter>) • Aggregation (GROUP BY <columns>) • Join (JOIN), union (UNION), intersect (INTERSECT) • Scan (FROM <table>) • Sort (ORDER BY) ○ Technically, not a relational operator
- 60. SQL Execution • Relational expressions have input expressions and scalar expressions ○ For example, a “filter” expression has 1 input expression and a scalar expression that filters the rows from the child ○ The scan expression has zero inputs • Query plan is a tree of relational expressions • SQL execution takes a query plan and runs the operations to completion
- 61. SQL Execution: Example SELECT name FROM inventory WHERE name >= “b” AND name < “c”
- 62. SQL Execution: Scan SELECT name FROM inventory WHERE name >= “b” AND name < “c” Scan inventory
- 63. SQL Execution: Filter SELECT name FROM inventory WHERE name >= “b” AND name < “c” Scan inventory Filter name >= “b” AND name < “c”
- 64. SQL Execution: Project SELECT name FROM inventory WHERE name >= “b” AND name < “c” Scan inventory Filter name >= “b” AND name < “c” Project name
- 65. SQL Execution: Project SELECT name FROM inventory WHERE name >= “b” AND name < “c” Scan inventory Filter name >= “b” AND name < “c” Project name Results
- 66. SQL Execution: Index Scans SELECT name FROM inventory WHERE name >= “b” AND name < “c” Scan inventory@name [“b” - “c”) The filter gets pushed into the scan
- 67. SQL Execution: Index Scans SELECT name FROM inventory WHERE name >= “b” AND name < “c” Scan inventory@name [“b” - “c”) Project name Results
- 68. SQL Execution: Correctness Correct SQL execution involves lots of bookkeeping • User defined tables, and indexes • Queries refer to table and column names • Execution uses table and column IDs • NULL handling
- 69. SQL Execution: Performance Performant SQL execution • Tight, well written code • Operator specialization ○ hash group by, stream group by ○ hash join, merge join, lookup join, zig-zag join • Distributed execution
- 70. SQL Execution: Group By SELECT COUNT(*), country FROM customers GROUP BY country Name Country Bob United States Hans Germany Jacques France Marie France Susan United States
- 71. SQL Execution: Hash Group By SELECT COUNT(*), country FROM customers GROUP BY country Name Country Bob United States Hans Germany Jacques France Marie France Susan United States
- 72. SQL Execution: Hash Group By SELECT COUNT(*), country FROM customers GROUP BY country Name Country Bob United States Hans Germany Jacques France Marie France Susan United States United States 1
- 73. SQL Execution: Hash Group By SELECT COUNT(*), country FROM customers GROUP BY country Name Country Bob United States Hans Germany Jacques France Marie France Susan United States United States 1 Germany 1
- 74. SQL Execution: Hash Group By SELECT COUNT(*), country FROM customers GROUP BY country Name Country Bob United States Hans Germany Jacques France Marie France Susan United States United States 1 Germany 1 France 1
- 75. SQL Execution: Hash Group By SELECT COUNT(*), country FROM customers GROUP BY country Name Country Bob United States Hans Germany Jacques France Marie France Susan United States United States 1 Germany 1 France 2
- 76. SQL Execution: Hash Group By SELECT COUNT(*), country FROM customers GROUP BY country Name Country Bob United States Hans Germany Jacques France Marie France Susan United States United States 2 Germany 1 France 2
- 77. SQL Execution: Group By Revisited SELECT COUNT(*), country FROM customers GROUP BY country Name Country Bob United States Hans Germany Jacques France Marie France Susan United States
- 78. SQL Execution: Sort on Grouping Column(s) SELECT COUNT(*), country FROM customers GROUP BY country Name Country Jacques France Marie France Hans Germany Bob United States Susan United States
- 79. SQL Execution: Streaming Group By SELECT COUNT(*), country FROM customers GROUP BY country Name Country Jacques France Marie France Hans Germany Bob United States Susan United States France 1
- 80. SQL Execution: Streaming Group By SELECT COUNT(*), country FROM customers GROUP BY country Name Country Jacques France Marie France Hans Germany Bob United States Susan United States France 2
- 81. SQL Execution: Streaming Group By SELECT COUNT(*), country FROM customers GROUP BY country Name Country Jacques France Marie France Hans Germany Bob United States Susan United States France 2 Germany 1
- 82. SQL Execution: Streaming Group By SELECT COUNT(*), country FROM customers GROUP BY country Name Country Jacques France Marie France Hans Germany Bob United States Susan United States France 2 Germany 1 United States 1
- 83. SQL Execution: Streaming Group By SELECT COUNT(*), country FROM customers GROUP BY country Name Country Jacques France Marie France Hans Germany Bob United States Susan United States France 2 Germany 1 United States 2
- 84. Distributed SQL Execution Network latencies and throughput are important considerations in geo-distributed setups Push fragments of computation as close to the data as possible
- 85. Distributed SQL Execution: Streaming Group By SELECT COUNT(*), country FROM customers GROUP BY country Scan customers Scan customers Scan customers scan scan scan
- 86. Distributed SQL Execution: Streaming Group By SELECT COUNT(*), country FROM customers GROUP BY country Scan customers Scan customers Scan customers Group-By “country” Group-By “country” Group-By “country” group-by group-by group-by
- 87. Distributed SQL Execution: Streaming Group By SELECT COUNT(*), country FROM customers GROUP BY country Scan customers Scan customers Scan customers Group-By “country” Group-By “country” Group-By “country” Group-By “country” group-by
- 88. AGENDA ● Introduction ● Ranges and Replicas ● Transactions ● SQL Data in a KV World ● SQL Execution ● SQL Optimization
- 89. SQL Optimization An optimizer explores many plans that are logically equivalent to a given query and chooses the best one Parse ExecuteSearch Memo Prep AST Plan Fold Constants Check Types Resolve Names Report Semantic Errors Compute properties Retrieve and attach stats Cost-independent transformations Cost-based transformationsParse SQL
- 90. SQL Optimization: Cost-Independent Transformations • Some transformations always make sense ○ Constant folding ○ Filter push-down ○ Decorrelating subqueries* ○ ... • These transformations are cost-independent ○ If the transformation can be applied to the query, it is applied • Domain Specific Language for transformations ○ Compiled down to code which efficiently matches query fragments in the memo ○ ~200 transformations currently defined * Actually cost-based, but we’re treating it as cost-independent right now
- 91. SQL Optimization: Filter Push-Down SELECT * FROM a JOIN b WHERE x > 10 Scan a@primary Filter x > 10 Results Scan b@primary Join Initial plan
- 92. SQL Optimization: Filter Push-Down SELECT * FROM a JOIN b WHERE x > 10 Scan a@primary Filter x > 10 Results Scan b@primary Join Filter x > 10 After filter push-down
- 93. SQL Optimization: Cost-Based Transformations • Some transformations are not universally good ○ Index selection ○ Join reordering ○ ... • These transformations are cost-based ○ When should the transformation be applied? ○ Need to try both paths and maintain both the original and transformed query ○ State explosion: thousands of possible query plans ■ Memo data structure maintains a forest of query plans ○ Estimate cost of each query, select query with lowest cost • Costing ○ Based on table statistics and estimating cardinality of inputs to relational expressions
- 94. SQL Optimization: Cost-based Index Selection The index to use for a query is affected by multiple factors • Filters and join conditions • Required ordering (ORDER BY) • Implicit ordering (GROUP BY) • Covering vs non-covering (i.e. is an index-join required) • Locality
- 95. SQL Optimization: Cost-based Index Selection SELECT * FROM a WHERE x > 10 ORDER BY y Required orderings affect index selection Sorting is expensive if there are a lot of rows Sorting can be the better option if there are few rows
- 96. SQL Optimization: Cost-based Index Selection Required orderings affect index selection Sorting is expensive if there are a lot of rows Sorting can be the better option if there are few rows Scan a@primary Filter x > 10 Sort y SELECT * FROM a WHERE x > 10 ORDER BY y
- 97. SQL Optimization: Cost-based Index Selection Required orderings affect index selection Sorting is expensive if there are a lot of rows Sorting can be the better option if there are few rows Scan a@primary Scan a@x [10 - ) Filter x > 10 Sort y Sort y SELECT * FROM a WHERE x > 10 ORDER BY y
- 98. SQL Optimization: Cost-based Index Selection Required orderings affect index selection Sorting is expensive if there are a lot of rows Sorting can be the better option if there are few rows Scan a@primary Scan a@x [10 - ) Filter x > 10 Scan a@y Sort y Sort y Filter x > 10 SELECT * FROM a WHERE x > 10 ORDER BY y
- 99. SQL Optimization: Cost-based Index Selection Required orderings affect index selection Sorting is expensive if there are a lot of rows Sorting can be the better option if there are few rows Scan a@primary Scan a@x [10 - ) Filter x > 10 Scan a@y Sort y Sort y Filter x > 10 SELECT * FROM a WHERE x > 10 ORDER BY y 10 100,000 10 10 Lowest Cost
- 100. SQL Optimization: Cost-based Index Selection Required orderings affect index selection Sorting is expensive if there are a lot of rows Sorting can be the better option if there are few rows Scan a@primary Scan a@x [10 - ) Filter x > 10 Scan a@y Sort y Sort y Filter x > 10 SELECT * FROM a WHERE x > 10 ORDER BY y 50,000 100,000 50,000 50,000 Lowest Cost
- 101. Locality-Aware SQL Optimization Network latencies and throughput are important considerations in geo-distributed setups Duplicate read-mostly data in each locality Plan queries to use data from the same locality
- 102. Locality-Aware SQL Optimization Three copies of the postal_codes table data Use replication constraints to pin the copies to different geographic regions (US-East, US-West, EU) CREATE TABLE postal_codes ( id INT PRIMARY KEY, code STRING, INDEX idx_eu (id) STORING (code), INDEX idx_usw (id) STORING (code) )
- 103. Locality-Aware SQL Optimization Optimizer includes locality in cost model Automatically selects index from same locality: primary, idx_eu, or idx_usw CREATE TABLE postal_codes ( id INT PRIMARY KEY, code STRING, INDEX idx_eu (id) STORING (code), INDEX idx_usw (id) STORING (code) ) SELECT * FROM postal_codes
- 104. Conclusion ● Distributed, replicated, transactional key-value store ● Monolithic key space ● Raft replication of ranges (~64MB) ● Replica placement signals: space, diversity, load, latency ● Pipelined transaction operations ● Mapping SQL data to KV storage ● Distributed SQL execution ● Distributed SQL optimization
- 107. A Simple Transaction: One Range node1 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady node1 node1 carl peetey dagne lady lula muddy peetey lady node1 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady carl dagne figment jack pinetop sooshi stella zee pinetop sooshi stella zee pinetop sooshi stella sunny zee BEGIN WRITE[sunny] COMMIT GATEWAY INSERT INTO DOGS (sunny); NOTE: a gateway can be ANY CockroachDB instance. It can find the leaseholder for any range and execute a transaction
- 108. A Simple Transaction: One Range node1 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady node1 node1 carl peetey dagne lady lula muddy peetey lady node1 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady carl dagne figment jack pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee BEGIN WRITE[sunny] COMMIT GATEWAY INSERT INTO DOGS (sunny); ACK
- 109. A Simple Transaction: One Range node1 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady node1 node1 carl peetey dagne lady lula muddy peetey lady node1 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady carl dagne figment jack pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee BEGIN WRITE[sunny] COMMIT GATEWAY INSERT INTO DOGS (sunny); ACK
- 110. Ranges CockroachDB implements order-preserving data distribution • Automates sharding of key/value data into “ranges” • Supports efficient range scans • Requires an indexing structure Foundational capability that enables efficient distribution of data across nodes within a CockroachDB cluster * This approach is also used by Bigtable (tablets), HBase (regions) & Spanner (ranges)
- 111. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ cockroachdb-distributed-sql/