Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Amazon S3 storage engine plugin for MySQL

6,174 views

Published on

Published in: Technology, Business

Amazon S3 storage engine plugin for MySQL

  1. 1. A Storage Engine for Amazon S3 How to store a hundred billion BLOBs without buying a single disk.
  2. 2. Who is this guy?
  3. 3. http://fallenpegasus.com/code
  4. 4. Storage Engine
  5. 5. Protocol Translator
  6. 6. Traditional Storage Engines use the local disk
  7. 7. Network Storage Engines
  8. 8. Federated ODBC HTTP MemCacheD and...
  9. 9. S3
  10. 10. What is S3?
  11. 11. Petabytes?!
  12. 12. This can't be free
  13. 13. Why do I want to use it?
  14. 14. “An empty disk costs the same as a full one.”
  15. 15. Can I move my existing database over?
  16. 16. Translating S3 to MySQL AWS Account → CREATE SERVER • S3 Bucket → Table • S3 Item → Row • S3 Item Key → VARCHAR PRIMARY KEY • S3 Item Contents → BLOB or VARCHAR •
  17. 17. CREATE SERVER 'MyAWSAcct' FOREIGN DATA WRAPPER 'AWS' OPTIONS (USER 'aws id string', PASSWORD 'aws secret string');
  18. 18. CREATE TABLE 'bierce' ( 'word' VARCHAR(255) NOT NULL PRIMARY KEY, 'defn' BLOB) CHARSET=utf-8 ENGINE=AWSS3 CONNECTION='awss3 DevilDictionary $server MyAWSAcct';
  19. 19. SELECT defn FROM bierce WHERE word='WIT';
  20. 20. SELECT defn FROM bierce WHERE word='WIT'; WIT, n. The salt with which the American humorist spoils his intellectual cookery by leaving it out.
  21. 21. INSERT INTO bierce (word, defn) VALUES ('AUTHOR', 'One noted for confusing bitterness with humor.');
  22. 22. DELETE FROM bierce WHERE word='AUTHOR';
  23. 23. So, what's it good for?
  24. 24. “If you build it, they will come.”
  25. 25. Saving EC2 work • SQL CMS for S3 virtual hosting • Huge list of persistent primary keys • Big slow BLOBs, to join against fast local tables. • “The Image Server Problem” • ... and more •
  26. 26. Transactions?
  27. 27. What do you mean, “No temporal guarantees”?!
  28. 28. No WHERE clause means “MAKE MONEY FAST” for Amazon
  29. 29. The Future (watch me wave my hands)
  30. 30. Code Improvement • S3 & HTTP Metadata • Multiple Data Columns • Information Schema • Security & Authentication • Transfer & Storage Compression • Streaming BLOB • More Storage Engines •
  31. 31. Code Improvement
  32. 32. S3 & HTTP Metadata
  33. 33. Multiple Data Columns
  34. 34. Sharing a solution with HTTP and MemCacheD Engines
  35. 35. Information Schema
  36. 36. Security & Authentication
  37. 37. Transfer & Storage Compression
  38. 38. (new slide) Streaming BLOBs like PBXT
  39. 39. More storage engines
  40. 40. More storage engines “A Storage Engine for AWS SQS” • There are more AWS services rolling out • Replication distribution via S3 • Cluster persistent storage via S3 • ... and the EC2 holy grail: •
  41. 41. “A Generic Schema Storage Engine for S3”
  42. 42. Back to the present...
  43. 43. “Where do I get it?”
  44. 44. “Where do I get it?” MySQL 5.1 • GPL • http://fallenpegasus.com/code •
  45. 45. “You never write, you never call...”
  46. 46. Suggestions • Use Cases • Bug Reports • Patches • Feature Requests • Support Requests • Gigs & Proposals • mark@fallenpegasus.com
  47. 47. Questions?

×