Amazon S3 storage engine plugin for MySQL

Slide 1: A Storage Engine for Amazon S3 How to store a hundred billion BLOBs without buying a single disk.

Slide 2: Who is this guy?

Slide 3: http://fallenpegasus.com/code

Slide 4: Storage Engine

Slide 5: Protocol Translator

Slide 6: Traditional Storage Engines use the local disk

Slide 7: Network Storage Engines

Slide 8: Federated ODBC HTTP MemCacheD and...

Slide 9: S3

Slide 10: What is S3?

Slide 11: Petabytes?!

Slide 12: This can't be free

Slide 13: Why do I want to use it?

Slide 14: “An empty disk costs the same as a full one.”

Slide 15: Can I move my existing database over?

Slide 16: Translating S3 to MySQL AWS Account → CREATE SERVER • S3 Bucket → Table • S3 Item → Row • S3 Item Key → VARCHAR PRIMARY KEY • S3 Item Contents → BLOB or VARCHAR •

Slide 17: CREATE SERVER 'MyAWSAcct' FOREIGN DATA WRAPPER 'AWS' OPTIONS (USER 'aws id string', PASSWORD 'aws secret string');

Slide 18: CREATE TABLE 'bierce' ( 'word' VARCHAR(255) NOT NULL PRIMARY KEY, 'defn' BLOB) CHARSET=utf-8 ENGINE=AWSS3 CONNECTION='awss3 DevilDictionary $server MyAWSAcct';

Slide 19: SELECT defn FROM bierce WHERE word='WIT';

Slide 20: SELECT defn FROM bierce WHERE word='WIT'; WIT, n. The salt with which the American humorist spoils his intellectual cookery by leaving it out.

Slide 21: INSERT INTO bierce (word, defn) VALUES ('AUTHOR', 'One noted for confusing bitterness with humor.');

Slide 22: DELETE FROM bierce WHERE word='AUTHOR';

Slide 23: So, what's it good for?

Slide 24: “If you build it, they will come.”

Slide 25: Saving EC2 work • SQL CMS for S3 virtual hosting • Huge list of persistent primary keys • Big slow BLOBs, to join against fast local tables. • “The Image Server Problem” • ... and more •

Slide 26: Transactions?

Slide 27: What do you mean, “No temporal guarantees”?!

Slide 28: No WHERE clause means “MAKE MONEY FAST” for Amazon

Slide 29: The Future (watch me wave my hands)

Slide 30: Code Improvement • S3 & HTTP Metadata • Multiple Data Columns • Information Schema • Security & Authentication • Transfer & Storage Compression • Streaming BLOB • More Storage Engines •

Slide 31: Code Improvement

Slide 32: S3 & HTTP Metadata

Slide 33: Multiple Data Columns

Slide 34: Sharing a solution with HTTP and MemCacheD Engines

Slide 35: Information Schema

Slide 36: Security & Authentication

Slide 37: Transfer & Storage Compression

Slide 38: (new slide) Streaming BLOBs like PBXT

Slide 39: More storage engines

Slide 40: More storage engines “A Storage Engine for AWS SQS” • There are more AWS services rolling out • Replication distribution via S3 • Cluster persistent storage via S3 • ... and the EC2 holy grail: •

Slide 41: “A Generic Schema Storage Engine for S3”

Slide 42: Back to the present...

Slide 43: “Where do I get it?”

Slide 44: “Where do I get it?” MySQL 5.1 • GPL • http://fallenpegasus.com/code •

Slide 45: “You never write, you never call...”

Slide 46: Suggestions • Use Cases • Bug Reports • Patches • Feature Requests • Support Requests • Gigs & Proposals • mark@fallenpegasus.com

Slide 47: Questions?