Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

File Upload 2015

3,801 views

Published on

for Singapore Ruby Meetup

Published in: Software
  • Be the first to comment

File Upload 2015

  1. 1. File Upload 2015 @choonkeat
 choonkeat.com jollygoodcode.com
  2. 2. How’s that predefined styles doing for you? has_attached_file :asset, styles: { thumb: "100x100#" }
  3. 3. How’s that predefined styles doing for you? has_attached_file :asset, styles: { thumb: "100x100#", medium: "320x>" }
  4. 4. How’s that predefined styles doing for you? has_attached_file :asset, styles: { thumb: "100x100#", medium: "320x>", large: "1024x>" }
  5. 5. How’s that predefined styles doing for you? has_attached_file :asset, styles: { thumb: "100x100#", medium: "320x>",
 big: "640x>", large: "1024x>" } Names starting to lose meaning…
  6. 6. How’s that predefined styles doing for you? has_attached_file :asset, styles: { thumb: "100x100#", thumb2x: "200x200#", medium: "320x>", medium2x: "640x>",
 big: "640x>", big2x: "1280x>", large: "1024x>", large2x: "2048x>" }
  7. 7. How’s that predefined styles doing for you? has_attached_file :asset, styles: { thumb: "100x100#", thumb2x: "200x200#", medium: "320x>", medium2x: "640x>",
 big: "640x>", big2x: "1280x>", large: "1024x>", large2x: "2048x>" } Reprocess all the production files, each time, we make changes. 404 while rake runs or do at midnight?
  8. 8. How’s that transformation juggling doing for you?
  9. 9. How’s that transformation juggling doing for you? class MyUploader < CarrierWave::Uploader::Base version :thumb do process resize_to_fill: [280, 280] end version :small_thumb, from_version: :thumb do process resize_to_fill: [20, 20] end end
  10. 10. How’s that transformation juggling doing for you? class MyUploader < CarrierWave::Uploader::Base version :thumb do process resize_to_fill: [280, 280] end version :small_thumb, from_version: :thumb do process resize_to_fill: [20, 20] end end Did your users wait in the foreground or background?
  11. 11. How’s that file path config doing for you?
  12. 12. How’s that file path config doing for you? class Avatar < ActiveRecord::Base has_attached_file :image,
 url: '/system/:class/:attachment/:id/:hash-:style.:extension', hash_secret: Rails.application.secrets.paperclip end
  13. 13. How’s that file path config doing for you? class Avatar < ActiveRecord::Base has_attached_file :image,
 url: '/system/:class/:attachment/:id/:hash-:style.:extension', hash_secret: Rails.application.secrets.paperclip end You sure this is the format? Or will they need to change?
  14. 14. How’s that file path config doing for you? class Avatar < ActiveRecord::Base has_attached_file :image,
 url: '/system/:class/:attachment/:id/:hash-:style.:extension', hash_secret: Rails.application.secrets.paperclip end
  15. 15. How’s that file path config doing for you? class Avatar < ActiveRecord::Base has_attached_file :image,
 url: '/system/:class/:attachment/:id/:hash-:style.:extension', hash_secret: Rails.application.secrets.paperclip end
  16. 16. How’s that file path config doing for you? CarrierWave.configure do |config| config.permissions = 0666 config.directory_permissions = 0777 config.storage = :file end
  17. 17. How’s that file path config doing for you? CarrierWave.configure do |config| config.permissions = 0666 config.directory_permissions = 0777 config.storage = :file end Did you configure your MySQL data file in your app too?
  18. 18. How’s that file path config doing for you? class Avatar < ActiveRecord::Base self.table = { name: "avatars", data: "/var/lib/mysql/data/avatars.MYD", index: "/var/lib/mysql/data/avatars.MYI" } end Did you configure your MySQL data file in your app too?
  19. 19. How’s that form validation error dance doing for you?
  20. 20. How’s that form validation error dance doing for you? • User chooses a file • Submit & wait for file to upload ⌛… • Validation error: “Username is already taken!” • Re-render form
  21. 21. How’s that form validation error dance doing for you? • User chooses a file • Submit & wait for file to upload ⌛… • Validation error: “Username is already taken!” • Re-render form Where dat file go?
  22. 22. How’s that form validation error dance doing for you? http://stackoverflow.com/questions/5198602/not-losing-paperclip-attachment-when-model-cannot-be-saved-due-to-validation-err Closed: Won’t Fix
  23. 23. How’s that form validation error dance doing for you? http://stackoverflow.com/questions/5198602/not-losing-paperclip-attachment-when-model-cannot-be-saved-due-to-validation-err
  24. 24. How’s that form validation error dance doing for you? http://stackoverflow.com/questions/5198602/not-losing-paperclip-attachment-when-model-cannot-be-saved-due-to-validation-err Answer: use CarrierWave
  25. 25. How’s that form validation error dance doing for you?
  26. 26. How’s that form validation error dance doing for you? “Is there a standard approach? This seems like a very common use case.”
  27. 27. How’s the schema pollution doing for you?
  28. 28. How’s the schema pollution doing for you? class StoreMetadata < ActiveRecord::Migration def change add_column :users, :profile_image_filename, :string add_column :users, :profile_image_size, :integer add_column :users, :profile_image_content_type, :string end end
  29. 29. How’s the schema pollution doing for you? class StoreMetadata < ActiveRecord::Migration def change add_column :users, :profile_image_filename, :string add_column :users, :profile_image_size, :integer add_column :users, :profile_image_content_type, :string end end
  30. 30. How’s that multiple files doing for you?
  31. 31. How’s that multiple files doing for you? class Post < ActiveRecord::Base has_many :images, dependent: :destroy end
  32. 32. How’s that multiple files doing for you? class Post < ActiveRecord::Base has_many :images, dependent: :destroy end class Image < ActiveRecord::Base belongs_to :post attachment :file end
  33. 33. How’s that multiple files doing for you? class Post < ActiveRecord::Base has_many :images, dependent: :destroy end class Image < ActiveRecord::Base belongs_to :post attachment :file end
  34. 34. How’s that multiple files doing for you? class Post < ActiveRecord::Base has_many :images, dependent: :destroy end class Image < ActiveRecord::Base belongs_to :post attachment :file end Is this what you want or just what you’re accustomed to?
  35. 35. How’s Amazon Lambda doing for you? • User chooses a file • Submit & wait for file to upload ⌛… • Success! • Render page with thumbnail… How many thumbnails - 404?
  36. 36. How’s Amazon Lambda doing for you? • User chooses a file • Submit & wait for file to upload ⌛… • Success! • Render page with thumbnail… Direct upload to AWS? Cancel form submit - delete files & thumbnails? Deep integration & assumption
  37. 37. How’s Dragonfly doing for you? http://markevans.github.io/dragonfly/
  38. 38. How’s refile doing for you? https://github.com/refile/refile
  39. 39. http://thecooperreview.com/10-tricks-appear-smart-meetings/ 10 Tricks to Appear Smart During Meetings
  40. 40. Take a step back
  41. 41. Take a step back • We want to store a bunch of attributes in a model • e.g. Title, Body, Tags, Photo
 
 
 
 
 
 
 

  42. 42. Take a step back {photo} {title} {body}
  43. 43. Take a step back <img src={photo}> <h1>{title}</h1> {body}
  44. 44. Take a step back • Why should photo be a disproportionately complicated attribute in my Article model? • stored file path • conversion • background job • aws config • clean up on delete
  45. 45. Take a step back • Why should photo be a disproportionately complicated attribute in my Article model? • stored file path • conversion • background job • validation error dance • aws config
  46. 46. Take a step back • Frankly photo_url is best; least intrusive
 
 
 
 
 
 
 
 
 

  47. 47. Take a step back • Frankly photo_url is best; least intrusive • Problems • Remote url 404? (not exclusive to your app) • Asking users to give us a URL is a hard sell • Need to render other sizes • Filter by meta data
  48. 48. Take a step back • Frankly photo_url is best; least intrusive • Problems • Remote url 404? (not exclusive to your app) • Asking users to give us a URL is a hard sell • Need to render other sizes • Filter by meta data
  49. 49. Take a step back • Frankly photo_url is best; least intrusive • Problems • Remote url 404? (not exclusive to your app) • Asking users to give us a URL is a hard sell • Need to render other sizes • Filter by meta data
  50. 50. Take a step back • Frankly photo_url is best; least intrusive • Solutions • Exclusive server for your app • Upload to that server • On-the-fly resize based on URL • Store url with meta data: photo_json instead?
  51. 51. Just add server
  52. 52. • PostgreSQL, MySQL • Redis • Memcached • SMTP server (Mail)
 
 
 
 You are already Generic server to do specialised work Not specific to your business logic
  53. 53. • Not a new pattern • Mostly commercial services
 
 
 
 
 • Maybe it has to be Free & Open Source to become a default pattern Image server
  54. 54. Want • Move the “concern” out of my app • photo is a regular attribute • configure my app & forget it exist
 
 
 

  55. 55. Want • Move the “concern” out of my app • photo is a regular attribute • configure my app & forget it exist
 
 
 
 What would my app look like in a better world?
  56. 56. My app: Bare minimum create_table "users" do |t| t.string "name" t.json "avatar" t.json "photos" end
  57. 57. My app: Bare minimum create_table "users" do |t| t.string "name" t.json "avatar" t.json "photos" # multiple files in a column end
  58. 58. My app: Bare minimum Image serverRails appBrowser { avatar: #<File..> }
  59. 59. My app: Bare minimum Image serverRails appBrowser { avatar: #<File..> } {“path”:“x.jpg”, “geometry”:“200x600”} #<File..>
  60. 60. My app: Bare minimum Image serverRails appBrowser {“path”:“x.jpg”, “geometry”:“200x600”} #<File..> user.avatar={“path”: “x.jpg”…} user.save <img src=“x.jpg”> { avatar: #<File..> }
  61. 61. My app: Bare minimum Image serverRails appBrowser {“path”:“x.jpg”, “geometry”:“200x600”} <img src=“x.jpg”> #<File..> GET x.jpg #<File..> user.avatar={“path”: “x.jpg”…} user.save { avatar: #<File..> }
  62. 62. My app: Bare minimum • Browser render <file> field; regular form submit • Receive binary param, uploads to attache server and stores the json response instead • Your app render <img src> requesting for image in certain size, e.g. http://example/200x/ file.png
 
 

  63. 63. Image serverRails appBrowser #<File..> {“path”:“x.jpg”, “geometry”:“200x600”} Progressive Enhancement
  64. 64. Image serverRails appBrowser #<File..> {“path”:“x.jpg”, “geometry”:“200x600”} Progressive Enhancement { avatar: {“path”:“x.jpg”, … }
  65. 65. { avatar: {“path”:“x.jpg”, … } Image serverRails appBrowser #<File..> {“path”:“x.jpg”, “geometry”:“200x600”} user.avatar={“path”: “x.jpg”…} user.save <img src=“x.jpg”> Progressive Enhancement
  66. 66. • Browser render <file> field; regular form submit • Receive binary param, uploads to attache server and stores the json response instead • Your app render <img src> requesting for image in certain size, e.g. http://example/200x/ file.png
 
 
 Progressive Enhancement
  67. 67. • Browser render <file> field; regular form submit • Receive binary param, uploads to attache server and stores the json response instead • Your app render <img src> requesting for image in certain size, e.g. http://example/200x/ file.png
 
 
 • JS upload directly to attache server; “Direct upload” in AWS parlance • No binary in params; receive and store json attribute Progressive Enhancement • When after_update & after_destroy removes obsolete file from attache via delete API
  68. 68. • Just use Ruby; just use your framework • Pre-render multiple sizes • fetch the urls, server will generate and cache • Validation • validating a regular json attribute How do I…
  69. 69. • Move the “concern” out of my app • photo is a regular attribute • configure my app & forget it exist
 
 
 
 Want (cont’d)
  70. 70. Want (cont’d) • Move the “concern” out of my app • photo is a regular attribute • configure my app & forget it exist • Separate, standalone server • Minimal / zero ongoing administration
  71. 71. Want (cont’d) • Move the “concern” out of my app • photo is a regular attribute • configure my app & forget it exist • Separate, standalone server • Minimal / zero ongoing administration How does this server work?
  72. 72. Attache File Server • HTTP server with simple APIs • upload • download • delete • Rack app + ImageMagick • Go? Node? C++? PHP? • GraphicsMagick? MyResizer.bash?
  73. 73. • Uploaded files are stored locally • Resize local file on-the-fly • configurable pool size to limit concurrent resizing • Sync upload to cloud storage • 2 hop problem vs complexity • Fixed local storage, purge LRU (zero maintenance) • Spin up new fresh servers anytime… 
 Attache File Server
  74. 74. • When requested file does not exist locally • fetch from cloud storage & write locally • resume operations
 
 
 
 
 
 Attache File Server
  75. 75. • Remove obsolete file is “best effort” • If photo delete failed, do you Error 500 or stop the world?
 
 
 
 • OCDs can schedule rake task remove dangling files? Attache File Server
  76. 76. • Caching in production • Browser → CDN → Varnish → Disk → Cloud
 
 
 
 
 
 
 Attache File Server
  77. 77. Demo https://attache-demo.herokuapp.com/
  78. 78. https://github.com/choonkeat/attache_api ATTACHE_URL=http://localhost:9292 ATTACHE_SECRET_KEY=topsecret rake Compatibility Check
  79. 79. tus.io • Open protocol for resumable uploads built on HTTP • Perfect for mobile apps • Rack middleware implemented in choonkeat/ attache#10
  80. 80. Responsive Images with Client Hints http://blog.imgix.com/2015/10/13/next-generation-responsive-images-with-client.html
  81. 81. SmartCrop https://github.com/jwagner/smartcrop.js/
  82. 82. Dragonfly & refile
  83. 83. Dragonfly & refile • tldr: we can fuss over implementation, but it is mostly about architecture
  84. 84. Dragonfly & refile • Rack middleware in your Rails app by default • performing on-the-fly image resize 😱 • Rack standalone end point • Dragonfly.app - upload, download, delete • Refile::App - upload, download, delete • Downloads are unthrottled
  85. 85. Dragonfly & refile • BEFORE: Rails integrate with AWS • AFTER: Rails integrate with AWS + Rack app • Maintain identical AWS config in both apps • Rails app couldn’t shed the “concern” • Multiple images still require multiple models
  86. 86. refile class Post < ActiveRecord::Base has_many :images, dependent: :destroy accepts_attachments_for :images, attachment: :file end class Image < ActiveRecord::Base belongs_to :post attachment :file end
  87. 87. refile class Post < ActiveRecord::Base has_many :images, dependent: :destroy accepts_attachments_for :images, attachment: :file end class Image < ActiveRecord::Base belongs_to :post attachment :file end “Note it must be possible to persist images given only the associated post and a file. There must not be any other validations or constraints which prevent images from being saved” i.e. Must be pure dummy wrapper; no validations here
  88. 88. refile download https://github.com/refile/refile/blob/master/lib/refile/app.rb get "/:token/:backend/:processor/:id/:filename" do halt 404 unless download_allowed? stream_file processor.call(file) end
  89. 89. refile download https://github.com/refile/refile/blob/master/lib/refile/app.rb get "/:token/:backend/:processor/:id/:filename" do halt 404 unless download_allowed? stream_file processor.call(file) end How many ImageMagick can you run in parallel? has_many :images?
  90. 90. refile upload https://github.com/refile/refile/blob/master/lib/refile/app.rb post "/:backend" do halt 404 unless upload_allowed? tempfile = request.params.fetch("file").fetch(:tempfile) file = backend.upload(tempfile) content_type :json { id: file.id }.to_json end def file file = backend.get(params[:id]) unless file.exists? log_error("Could not find attachment by id: #{params[:id]}") halt 404 end file.download end
  91. 91. post "/:backend" do halt 404 unless upload_allowed? tempfile = request.params.fetch("file").fetch(:tempfile) file = backend.upload(tempfile) content_type :json { id: file.id }.to_json end def file file = backend.get(params[:id]) unless file.exists? log_error("Could not find attachment by id: #{params[:id]}") halt 404 end file.download end refile upload https://github.com/refile/refile/blob/master/lib/refile/app.rb 2 hops problem when uploading and downloading • 3mb file becomes 6mb each way • #create and #show becomes 12mb process • has_many :images?
  92. 92. refile • CarrierWave-styled named processors (e.g. :fill, :thumb) vs passing through syntax to underlying ImageMagick • personally prefer leveraging off existing knowledge • instead of manually configured syntax sugar • “2 hop problem” however provide higher consistency when running multiple refile servers • upload-here-download-there problem • (considering to perform 2-hop upload instead of async)
  93. 93. refile • Can upload to S3 direct and/or upload to refile • Redundancy interesting, but prefer less concern in Rails app • Concept of cache and store to manage “dangling file problem” is worth considering • Download urls are signed-only (this practice should be adopted) • can partly counter motivation to abuse “dangling file problem” (aka free file hosting, whee!)
  94. 94. Questions? Attache server
 https://github.com/choonkeat/attache Gem for Rails
 https://github.com/choonkeat/attache_rails Demo
 https://attache-demo.herokuapp.com/

×