A Storage Engine for
Amazon S3
How to store a hundred billion BLOBs
without buying a single disk.

Who is this guy?

http://fallenpegasus.com/code

Storage Engine

Protocol Translator

Traditional
Storage Engines
use the local disk

Network Storage Engines

Federated
ODBC
HTTP
MemCacheD
and...

S3

What is S3?

Petabytes?!

This can't be free

Why do I want to use it?

“An empty disk costs the same as
a full one.”

Can I move my
existing database over?

Translating S3 to MySQL
AWS Account → CREATE SERVER
•
S3 Bucket → Table
•
S3 Item → Row
•
S3 Item Key → VARCHAR PRIMARY KEY
•
S3 Item Contents → BLOB or VARCHAR
•

CREATE SERVER 'MyAWSAcct'
FOREIGN DATA WRAPPER 'AWS'
OPTIONS
(USER 'aws id string',
PASSWORD 'aws secret string');

CREATE TABLE 'bierce' (
'word' VARCHAR(255) NOT NULL PRIMARY KEY,
'defn' BLOB)
CHARSET=utf-8
ENGINE=AWSS3
CONNECTION='awss3 DevilDictionary $server MyAWSAcct';

SELECT defn FROM bierce WHERE word='WIT';

SELECT defn FROM bierce WHERE word='WIT';
WIT, n. The salt with which the American humorist spoils his
intellectual cookery by leaving it out.

INSERT INTO bierce (word, defn) VALUES
('AUTHOR',
'One noted for confusing bitterness with humor.');

DELETE FROM bierce WHERE word='AUTHOR';

So, what's it good for?

“If you build it,
they will come.”

Saving EC2 work
•
SQL CMS for S3 virtual hosting
•
Huge list of persistent primary keys
•
Big slow BLOBs, to join against fast local tables.
•
“The Image Server Problem”
•
... and more
•

Transactions?

What do you mean,
“No temporal guarantees”?!

No WHERE clause
means
“MAKE MONEY FAST”
for Amazon

The Future
(watch me wave my hands)

Code Improvement
•
S3 & HTTP Metadata
•
Multiple Data Columns
•
Information Schema
•
Security & Authentication
•
Transfer & Storage Compression
•
Streaming BLOB
•
More Storage Engines
•

Code Improvement

S3 & HTTP
Metadata

Multiple Data Columns

Sharing a solution with
HTTP and MemCacheD
Engines

Information Schema

Security &
Authentication

Transfer & Storage
Compression

(new slide)
Streaming BLOBs
like PBXT

More storage engines

More storage engines
“A Storage Engine for AWS SQS”
•
There are more AWS services rolling out
•
Replication distribution via S3
•
Cluster persistent storage via S3
•
... and the EC2 holy grail:
•

“A Generic Schema
Storage Engine for S3”

Back to the present...

“Where do I get it?”

“Where do I get it?”
MySQL 5.1
•
GPL
•
http://fallenpegasus.com/code
•

“You never write,
you never call...”

Suggestions
•
Use Cases
•
Bug Reports
•
Patches
•
Feature Requests
•
Support Requests
•
Gigs & Proposals
•
mark@fallenpegasus.com

Questions?