Amazon S3 storage engine plugin for MySQL

5,958 views

Published on

Published in: Technology, Business
9 Comments
5 Likes
Statistics
Notes
No Downloads
Views
Total views
5,958
On SlideShare
0
From Embeds
0
Number of Embeds
34
Actions
Shares
0
Downloads
60
Comments
9
Likes
5
Embeds 0
No embeds

No notes for slide

Amazon S3 storage engine plugin for MySQL

  1. 1. A Storage Engine for Amazon S3 How to store a hundred billion BLOBs without buying a single disk.
  2. 2. Who is this guy?
  3. 3. http://fallenpegasus.com/code
  4. 4. Storage Engine
  5. 5. Protocol Translator
  6. 6. Traditional Storage Engines use the local disk
  7. 7. Network Storage Engines
  8. 8. Federated ODBC HTTP MemCacheD and...
  9. 9. S3
  10. 10. What is S3?
  11. 11. Petabytes?!
  12. 12. This can't be free
  13. 13. Why do I want to use it?
  14. 14. “An empty disk costs the same as a full one.”
  15. 15. Can I move my existing database over?
  16. 16. Translating S3 to MySQL AWS Account → CREATE SERVER • S3 Bucket → Table • S3 Item → Row • S3 Item Key → VARCHAR PRIMARY KEY • S3 Item Contents → BLOB or VARCHAR •
  17. 17. CREATE SERVER 'MyAWSAcct' FOREIGN DATA WRAPPER 'AWS' OPTIONS (USER 'aws id string', PASSWORD 'aws secret string');
  18. 18. CREATE TABLE 'bierce' ( 'word' VARCHAR(255) NOT NULL PRIMARY KEY, 'defn' BLOB) CHARSET=utf-8 ENGINE=AWSS3 CONNECTION='awss3 DevilDictionary $server MyAWSAcct';
  19. 19. SELECT defn FROM bierce WHERE word='WIT';
  20. 20. SELECT defn FROM bierce WHERE word='WIT'; WIT, n. The salt with which the American humorist spoils his intellectual cookery by leaving it out.
  21. 21. INSERT INTO bierce (word, defn) VALUES ('AUTHOR', 'One noted for confusing bitterness with humor.');
  22. 22. DELETE FROM bierce WHERE word='AUTHOR';
  23. 23. So, what's it good for?
  24. 24. “If you build it, they will come.”
  25. 25. Saving EC2 work • SQL CMS for S3 virtual hosting • Huge list of persistent primary keys • Big slow BLOBs, to join against fast local tables. • “The Image Server Problem” • ... and more •
  26. 26. Transactions?
  27. 27. What do you mean, “No temporal guarantees”?!
  28. 28. No WHERE clause means “MAKE MONEY FAST” for Amazon
  29. 29. The Future (watch me wave my hands)
  30. 30. Code Improvement • S3 & HTTP Metadata • Multiple Data Columns • Information Schema • Security & Authentication • Transfer & Storage Compression • Streaming BLOB • More Storage Engines •
  31. 31. Code Improvement
  32. 32. S3 & HTTP Metadata
  33. 33. Multiple Data Columns
  34. 34. Sharing a solution with HTTP and MemCacheD Engines
  35. 35. Information Schema
  36. 36. Security & Authentication
  37. 37. Transfer & Storage Compression
  38. 38. (new slide) Streaming BLOBs like PBXT
  39. 39. More storage engines
  40. 40. More storage engines “A Storage Engine for AWS SQS” • There are more AWS services rolling out • Replication distribution via S3 • Cluster persistent storage via S3 • ... and the EC2 holy grail: •
  41. 41. “A Generic Schema Storage Engine for S3”
  42. 42. Back to the present...
  43. 43. “Where do I get it?”
  44. 44. “Where do I get it?” MySQL 5.1 • GPL • http://fallenpegasus.com/code •
  45. 45. “You never write, you never call...”
  46. 46. Suggestions • Use Cases • Bug Reports • Patches • Feature Requests • Support Requests • Gigs & Proposals • mark@fallenpegasus.com
  47. 47. Questions?

×