SQLFire at Strata 2012

SQLFireJags Ramnarayan – Chief Architect, SQLFireCarter Shanklin – Product Manager, SQLFire

Sponsor Sessions Suck• – – –

Speed MattersUsers demand fast applications and fast websites. The database is the hardest thing to scale.

SQLFire: Speed, Scale, SQL Speed Scale SQL• In-memory for maximum • Horizontally scalable. • Familiar SQL interface. speed and minimum • Add or remove nodes at • SQL 92 compliant. latency. any time for more • JDBC and ADO.NET capacity or availability. interfaces.

How does SQLFire get scale and speed?• –• –• –

Diverging needs for online and analytics

SQLFire: What does it really look like?

SQLFire Tables Are Replicated By Default.1 CREATE TABLE sales SQLFire Node 12 (product_id int, store_id int, Replica3 price float); sales456 SQLFire Node 27 Replica8 Best for small and9 frequently accessed data.10

Partitioned Tables Are Split Among Members.1 CREATE TABLE sales SQLFire Node 12 (product_id int, store_id int, Replica3 price float) sales Partition 14 PARTITION BY5 COLUMN (product_id);6 SQLFire Node 27 Replica8 Best for large Partition 29 data sets.10

Types Of Partitioning In SQLFire. Type Purpose Example Built-in hashing algorithmHash Partitioning splits data at random across PARTITION BY COLUMN (customer_id); (Default) available servers. Manually divide data across PARTITION BY LIST (home_state) List servers based on discrete (VALUES (‘CA’, ‘WA’), criteria. VALUES (‘TX’, ‘OK’)); Manually divide data across PARTITION BY RANGE (date) Range servers based on continuous (VALUES BETWEEN ‘2008-01-01’ AND ‘2008-12-31’, criteria. VALUES BETWEEN ‘2009-01-01’ AND ‘2009-12-30’); Fully dynamic division of data Expression based on function execution. PARTITION BY (MONTH(date)); Can use UDFs.

How does it scale for queries? 1M Partitioned Table 1000PK queries per second 790k (1kb Rows) 800 604k 600 420k 400 200k 200 # Clients = 2*N N= 2 4 6 8 10 Number Of Servers

How does it scale for updates? 1.3M Partitioned Table 1000Updates Per Second (3 columns) 950k 800 750k 600 490k 400 220k 85% < 1ms 200 latency # Clients = 2*N N= 2 4 6 8 10 Number Of Servers

Redundancy Increases Availability.1 CREATE TABLE sales SQLFire Node 12 (product_id int, store_id int, Replica3 price float) sales Partition 14 PARTITION BY Partition 2*5 COLUMN (product_id);6 REDUNDANCY 1; SQLFire Node 27 Replica8 All data is available Partition 29 if Node 1 fails. Partition 1*10

Partitioning and redundancy Replication issynchronous but done Replication can be in parallel “rack aware” Single owner Redundancy = 2 for any row at point (but tunable) in time

SQLFire: Derp-Proof Database••• Was that cord supposed to be in the wall?

Linearly scaling joins•• –

Partition Aware DB Design–

Collocate Data For Fast Joins.1 CREATE TABLE sales Related data placed SQLFire Node 12 (product_id int, store_id int, on the same node. Replica3 price float) Customer 14 PARTITION BY C1 Customer 1 Sales5 COLUMN (product_id);6 COLOCATE WITH customers; SQLFire Node 27 C2 Replica8 SQLFire can join Customer 29 tables without Customer 2 Sales10 network hops.

Collocate Data For Fast Joins. Related data placed SQLFire Node 1 on the same node. Replica Customer 1 C1 Customer 1 Sales SQLFire Node 2 C2 Replica SQLFire can join Customer 2 tables without Customer 2 Sales network hops.

Collocate Data For Fast Joins. Related data placed SQLFire Node 1 on the same node. Replica Customer 1 C1 Customer 1 Sales Parallel scatter-gather SQLFire Node 2 C2 Replica Customer 2 In parallel, each node does hash join, aggregation locally Customer 2 Sales

Dynamic Data Colocation Dynamic entity Based on foreigngroup formation key relationships Single master for Redundancy = 2 any entity group

Data-Aware Stored Procs••••• Like Map/Reduce But Different

Scaling Stored Procedures1 CALL maxSales(arguments) SQLFire uses data- maxSales on2 aware routing to local data ON TABLE sales route processing to3 WHERE (Location in (‘CA’,’WA’,’OR) the data.4 WITH RESULT PROCESSOR5 maxSalesReducer maxSalesReducer678 Result Processors9 give map/reduce maxSales on functionality. local data10

Scalability: Consistency Assumes:Most x-actions small in space and timeWrite-write conflicts rare

Scalability: High performance persistence• Memory Memory Tables Tables• LOG Compressor LOG Compressor• – OS Buffers OS Buffers Record1 Record1 Record1 Record2 Record2 Append only Record1 Record2 Record2 Append only Record3 Record3 Operation logs Record3 Record3 Operation logs

Demos!

Demo: Distributed Procedures••••

Demo: Caching••••

:sigh:Download: Just Google it Try SQLFire Today! Free for developer use to 3 nodes. Forum: Got questions? Get answers. Twitter: I need more followers to get a promotion.

Demo Details

Scaling Stored Procs (1) Ubuntu (database)Insert Timeseries

Scaling Stored Procs (2) Ubuntu (database)Insert Timeseries Compute Autocorrelations Complete

Scaling Stored Procs (3) Ubuntu Ubuntu Ubuntu (database) (database) (database) Insert Timeseries Rebalance Rebalance Compute Autocorrelations Compute Autocorrelations Compute Autocorrelations All usingstandard SQL APIs Complete Complete Complete

Caching Analytics (1) Continuous Batch Processing

Caching Analytics (2) Ubuntu (database) Low latency In-memory caching JDBC row loader Continuous Batch Processing

Caching Analytics (3) Ubuntu (database) Low latency In-memory caching Scalable + Tunable Cache Policies Continuous Batch Processing

Caching Policies• LRU Count – Overflow to disk or destroy.• Time To Live – Counter ticks as soon as the row is loaded.• Idle Time – Destroy rows when they are not accessed for a while.• Specified in CREATE TABLE syntax.

http://strataconf.com	228
http://jp02.dev.medplus.com	24
http://lanyrd.com	15
https://twitter.com	6
http://us-w1.rockmelt.com	1
http://localhost	1
http://patricks-imac.corp.medplus.com	1

SQLFire at Strata 2012

by Carter Shanklin

Accessibility

Categories

Upload Details

Usage Rights

7 Embeds 276

Statistics