SortaSQL is a proposal to add seamless horizontal scalability to SQL databases by using the filesystem to store and retrieve data. The SQL database would store metadata and handle queries, while an embedded key-value store manages record storage on files in the local or distributed filesystem. This allows queries to scale across many servers by letting the filesystem handle replication, performance and locking of distributed data files. The architecture involves an application communicating with PostgreSQL over SQL, which uses a SortaSQL plugin to retrieve rows from Kyoto Cabinet key-value files on the POSIX filesystem. Case studies at CloudFlare show how a 400GB per day dataset can be efficiently stored and queried at scale using this approach.

1. SortaSQL
Ian Pye <ian@cloudflare.com>

2. Motivation

3. Everyone likes SQL
• Tables
• Joins
• Online Transaction Processing
• Transactions!
• Arbitrary Queries

4. Scaling?
• What happens to joins when your data
doesn’t fit in memory?
• I only need get and set for my data
• Sharding is too hard/unreliable
• A “monopolistically competitive market”?

5. Scaling
Seamless Horizontal Scalability from 1 to N

6. Proposal:
Let the Filesystem do the hard work
• RDBMS presents a full SQL interface to
applications, automatically accessing files to get
data as needed
• RDBMS stores metadata allowing it to find the
right data files
• Embedded key/value store handles the record
level storage, locking, caching, etc.
• FS (local or distributed) stores data and is
responsible for replication, performance, locking,
etc.

7. Major Wins
• Scales continuously from 1-100 servers (FS
permitting)
• Hot/cold storage hierarchy
• Allows ad-hoc queries via mature SQL
• Everyone already has built in bindings

8. Architecture
• Application Talks SQL to PostgreSQL
• PostgreSQL stores metadata
• Performs post processing on rows
retrieved from KC files
• KC files live on a POSIX filesystem

9. Architecture
Application
SQL
PostgreSQL
SortaSQL Plugin
libKC libKC libKC libKC
Kyoto Kyoto Kyoto Kyoto
Cabinet Cabinet Cabinet Cabinet
Filesystem Filesystem

10. Big Table
“A Bigtable is a sparse, distributed, persistent
multidimensional sorted map.”

11. Multi-Dimensional
• Storing values as protocol
buffers allow for arbitrarily
complex maps
• Logic so that when maps
get too big, they are
promoted to top level KC
stores

12. Persistent and Sorted
• Any Key/Value store which allows for
binary values accessed via a B+Tree of keys
will do
• We use Kyoto Cabinet (successor to Tokyo
Cabinet)

13. Sparse
• Values can be arbitrarily different.
• NULLs are free (or cheap)
• Protocol Buffers again to the rescue.

14. Distributed
• All about the filesystem here

15. Sharding Made Easy
• Fine grained metadata allowed for efficient
storage hierarchy

16. SortaSQL: Summary
• BigTable like structure
• Accessed via SQL (PHP bindings come for free!)
• Offload the hard part to the Filesystem Folks

17. Case Study: CloudFlare
• 400 GB data/day (Medium Data?)
• Facebook = 25 TB data/day
• USPS = 25.6 GB text data delivered/day
• Mix of Flash and Magnetic storage
• Mirrored
• Fixed user queries
• Random BizDev queries

18. Data Scheme

19. Metadata
Partitioned by owner, period and data type

20. Disk Layout
• 2 80GB SSDs (small
and blazing)
• 2.5T RAID5 (big and
slow)

21. Disk Layout

22. First Steps
(access some records)

23. Silly SQL Tricks

24. Window Functions are
Your Friends

25. NoSQL to MySQL with
Memcached
• Replace Language not
Storage Engine
• Speak Memcached not
SQL

26. In Context
• https://github.com/cloudflare/SortaSQL
• http://dev.mysql.com/tech-resources/articles/
nosql-to-mysql-with-memcached.html
• http://queue.acm.org/detail.cfm?id=1961297