Resumable File Upload API using GridFS and TUS

2,740 views

Published on

TUS is a resumable file upload protocol and with MongoDB GridFS, we build an API for uploading files through a REST API and show how to scale this API horizontally using MongoDB as the storage for these files.

Singapore MongoDB User Group March Meetup

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,740
On SlideShare
0
From Embeds
0
Number of Embeds
468
Actions
Shares
0
Downloads
5
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Resumable File Upload API using GridFS and TUS

  1. 1. MONGO DB FOR CAT LOVERS
  2. 2. About me • Developer / Entrepreneur • Co-Founder / CTO at PicoCandy Singapore and MogiMe Inc, San Francisco :)
  3. 3. What do you do as an entrepreneur?
  4. 4. You think of ideas
  5. 5. You see funny cats pics at work Sometimes you get bored and google for funny cats at work
  6. 6. And when you suddenly thought of this awesome idea, you can’t sleep
  7. 7. What if !!??!! User Generated Funny Cat Pics and Videos! Shot using their mobile phone camera! Using the Catstagram app!
  8. 8. MVP Development 101 • Wrote ruby server code • Wrote iOS code • Wrote ruby server code • Wrote iOS code
  9. 9. Problem How to do file upload on mobile?
  10. 10. Why File Upload for mobile sucks*
  11. 11. Why File Upload for mobile sucks*
  12. 12. Why File Upload for mobile sucks* SUCKY SG MOBILE NETWORK
  13. 13. How to make file upload RESUMABLE for mobile?
  14. 14. Then you google like every developer should*
  15. 15. TUS.IO a resumable file upload protocol on top of HTTP
  16. 16. What is the TUS protocol? • Simple, open and free • HTTP-based ( POST, HEAD, PATCH ) • Split file into chunks • Send smaller pieces, server keeps track of chunks / offset • Web and mobile clients library
  17. 17. What is the TUS protocol? POST "/files" HTTP/1.1 Host: tus.example.org Content-Length: 0 Final-Length: 100 RESPONSE: HTTP/1.1 201 Created Location: http://tus.example.org/files/1
  18. 18. What is the TUS protocol? PATCH "/files/1" HTTP/1.1 Host: tus.example.org Content-Length: 70 Offset: 0 [file data] RESPONSE: HTTP/1.1 200 Ok
  19. 19. What is the TUS protocol? HEAD "/files/1" HTTP/1.1 Host: tus.example.org RESPONSE: HTTP/1.1 200 Ok Offset: 70
  20. 20. What is the TUS protocol? PATCH "/files/1" HTTP/1.1 Host: tus.example.org Content-Length: 30 Offset: 70 [remaining file data] RESPONSE: HTTP/1.1 200 Ok
  21. 21. Awesome! What about the servers and the clients? • tusd - reference server implementation using Go • tus-ios-client, native objective-C client library • tus-jquery, javascript client library • rubytus, ruby gem https://github.com/picocandy/rubytus
  22. 22. Where are the chunks? • tusd - reference server implementation using Go • tus-ios-client, native objective-C client library • tus-jquery, javascript client library • rubytus, ruby gem https://github.com/picocandy/rubytus
  23. 23. Where are the chunks? Take 2.
  24. 24. What is GridFS? • GridFS is a specification for storing and retrieving files that exceed the BSON-document size limit of 16MB. • Just like the other collections • By default, GridFS uses two collections with names prefixed by fs bucket: • fs.files and fs.chunks • Use a different bucket name? Yes! Create multiple buckets in a single database? Oh yes!
  25. 25. What is GridFS? { "_id" : <ObjectId>, "length" : <num>, "chunkSize" : <num> "uploadDate" : <timestamp> "md5" : <hash> "filename" : <string>, "contentType" : <string>, "aliases" : <string array>, "metadata" : <dataObject>, } The files Collection - fs.files
  26. 26. What is GridFS? { "_id" : <ObjectId>, "files_id" : <ObjectId>, "n" : <num>, "data" : <binary> } The chunks Collection - fs.chunks
  27. 27. Mongo Ruby Driver require 'mongo' include Mongo @db = MongoClient.new('localhost').db('picotusd') @grid = Grid.new(@db) Saving Fileimage = File.open(“cat.jpg") id_for_cat = @grid.put(image, :filename => “cat.jpg") Retrieving File the_first_cat_image = @grid.get(id_for_cat)
  28. 28. Demo / Code Code
  29. 29. Questions?

×