File Upload 2015
@choonkeat
choonkeat.com
jollygoodcode.com

How’s that predefined styles
doing for you?
has_attached_file :asset, styles: {
thumb: "100x100#"
}

How’s that predefined styles
doing for you?
has_attached_file :asset, styles: {
thumb: "100x100#",
medium: "320x>"
}

How’s that predefined styles
doing for you?
has_attached_file :asset, styles: {
thumb: "100x100#",
medium: "320x>",
large: "1024x>"
}

How’s that predefined styles
doing for you?
has_attached_file :asset, styles: {
thumb: "100x100#",
medium: "320x>",
big: "640x>",
large: "1024x>"
}
Names starting to lose
meaning…

How’s that predefined styles
doing for you?
has_attached_file :asset, styles: {
thumb: "100x100#", thumb2x: "200x200#",
medium: "320x>", medium2x: "640x>",
big: "640x>", big2x: "1280x>",
large: "1024x>", large2x: "2048x>"
}

How’s that predefined styles
doing for you?
has_attached_file :asset, styles: {
thumb: "100x100#", thumb2x: "200x200#",
medium: "320x>", medium2x: "640x>",
big: "640x>", big2x: "1280x>",
large: "1024x>", large2x: "2048x>"
}
Reprocess all the production files, each
time, we make changes.
404 while rake runs or do at midnight?

How’s that transformation
juggling doing for you?

How’s that transformation
juggling doing for you?
class MyUploader < CarrierWave::Uploader::Base
version :thumb do
process resize_to_fill: [280, 280]
end
version :small_thumb, from_version: :thumb do
process resize_to_fill: [20, 20]
end
end

How’s that transformation
juggling doing for you?
class MyUploader < CarrierWave::Uploader::Base
version :thumb do
process resize_to_fill: [280, 280]
end
version :small_thumb, from_version: :thumb do
process resize_to_fill: [20, 20]
end
end
Did your users wait in the foreground
or background?

How’s that file path config
doing for you?

How’s that file path config
doing for you?
class Avatar < ActiveRecord::Base
has_attached_file :image,
url: '/system/:class/:attachment/:id/:hash-:style.:extension',
hash_secret: Rails.application.secrets.paperclip
end

How’s that file path config
doing for you?
class Avatar < ActiveRecord::Base
has_attached_file :image,
url: '/system/:class/:attachment/:id/:hash-:style.:extension',
hash_secret: Rails.application.secrets.paperclip
end
You sure this is the format?
Or will they need to change?

How’s that file path config
doing for you?
class Avatar < ActiveRecord::Base
has_attached_file :image,
url: '/system/:class/:attachment/:id/:hash-:style.:extension',
hash_secret: Rails.application.secrets.paperclip
end

How’s that file path config
doing for you?
class Avatar < ActiveRecord::Base
has_attached_file :image,
url: '/system/:class/:attachment/:id/:hash-:style.:extension',
hash_secret: Rails.application.secrets.paperclip
end

How’s that file path config
doing for you?
CarrierWave.configure do |config|
config.permissions = 0666
config.directory_permissions = 0777
config.storage = :file
end

How’s that file path config
doing for you?
CarrierWave.configure do |config|
config.permissions = 0666
config.directory_permissions = 0777
config.storage = :file
end
Did you configure your MySQL
data file in your app too?

How’s that file path config
doing for you?
class Avatar < ActiveRecord::Base
self.table = {
name: "avatars",
data: "/var/lib/mysql/data/avatars.MYD",
index: "/var/lib/mysql/data/avatars.MYI"
}
end
Did you configure your MySQL
data file in your app too?

How’s that form validation
error dance doing for you?

How’s that form validation
error dance doing for you?
• User chooses a file
• Submit & wait for file to upload ⌛…
• Validation error: “Username is already taken!”
• Re-render form

How’s that form validation
error dance doing for you?
• User chooses a file
• Submit & wait for file to upload ⌛…
• Validation error: “Username is already taken!”
• Re-render form
Where dat file go?

How’s that form validation
error dance doing for you?
http://stackoverflow.com/questions/5198602/not-losing-paperclip-attachment-when-model-cannot-be-saved-due-to-validation-err
Closed: Won’t Fix

How’s that form validation
error dance doing for you?
http://stackoverflow.com/questions/5198602/not-losing-paperclip-attachment-when-model-cannot-be-saved-due-to-validation-err

How’s that form validation
error dance doing for you?
http://stackoverflow.com/questions/5198602/not-losing-paperclip-attachment-when-model-cannot-be-saved-due-to-validation-err
Answer: use CarrierWave

How’s that form validation
error dance doing for you?

How’s that form validation
error dance doing for you?
“Is there a standard
approach? This seems like
a very common use case.”

How’s the schema pollution
doing for you?

How’s the schema pollution
doing for you?
class StoreMetadata < ActiveRecord::Migration
def change
add_column :users, :profile_image_filename, :string
add_column :users, :profile_image_size, :integer
add_column :users, :profile_image_content_type, :string
end
end

How’s the schema pollution
doing for you?
class StoreMetadata < ActiveRecord::Migration
def change
add_column :users, :profile_image_filename, :string
add_column :users, :profile_image_size, :integer
add_column :users, :profile_image_content_type, :string
end
end

How’s that multiple files
doing for you?

How’s that multiple files
doing for you?
class Post < ActiveRecord::Base
has_many :images, dependent: :destroy
end

How’s that multiple files
doing for you?
class Post < ActiveRecord::Base
has_many :images, dependent: :destroy
end
class Image < ActiveRecord::Base
belongs_to :post
attachment :file
end

How’s that multiple files
doing for you?
class Post < ActiveRecord::Base
has_many :images, dependent: :destroy
end
class Image < ActiveRecord::Base
belongs_to :post
attachment :file
end

How’s that multiple files
doing for you?
class Post < ActiveRecord::Base
has_many :images, dependent: :destroy
end
class Image < ActiveRecord::Base
belongs_to :post
attachment :file
end
Is this what you want or just
what you’re accustomed to?

How’s Amazon Lambda
doing for you?
• User chooses a file
• Submit & wait for file to upload ⌛…
• Success!
• Render page with thumbnail…
How many thumbnails -
404?

How’s Amazon Lambda
doing for you?
• User chooses a file
• Submit & wait for file to upload ⌛…
• Success!
• Render page with thumbnail…
Direct upload to AWS?
Cancel form submit -
delete files & thumbnails?
Deep integration &
assumption

How’s Dragonfly
doing for you?
http://markevans.github.io/dragonfly/

How’s refile
doing for you?
https://github.com/refile/refile

http://thecooperreview.com/10-tricks-appear-smart-meetings/
10 Tricks to Appear Smart
During Meetings

Take a step back

Take a step back
• We want to store a bunch of attributes in a model
• e.g. Title, Body, Tags, Photo

Take a step back
{photo}
{title}
{body}

Take a step back
<img src={photo}>
<h1>{title}</h1>
{body}

Take a step back
• Why should photo be a disproportionately
complicated attribute in my Article model?
• stored file path
• conversion
• background job
• aws config
• clean up on delete

Take a step back
• Why should photo be a disproportionately
complicated attribute in my Article model?
• stored file path
• conversion
• background job
• validation error dance
• aws config

Take a step back
• Frankly photo_url is best; least intrusive

Take a step back
• Frankly photo_url is best; least intrusive
• Problems
• Remote url 404? (not exclusive to your app)
• Asking users to give us a URL is a hard sell
• Need to render other sizes
• Filter by meta data

Take a step back
• Frankly photo_url is best; least intrusive
• Problems
• Remote url 404? (not exclusive to your app)
• Asking users to give us a URL is a hard sell
• Need to render other sizes
• Filter by meta data

Take a step back
• Frankly photo_url is best; least intrusive
• Problems
• Remote url 404? (not exclusive to your app)
• Asking users to give us a URL is a hard sell
• Need to render other sizes
• Filter by meta data

Take a step back
• Frankly photo_url is best; least intrusive
• Solutions
• Exclusive server for your app
• Upload to that server
• On-the-fly resize based on URL
• Store url with meta data: photo_json instead?

Just add server

• PostgreSQL, MySQL
• Redis
• Memcached
• SMTP server (Mail)
You are already
Generic server to do
specialised work
Not specific to your
business logic

• Not a new pattern
• Mostly commercial services
• Maybe it has to be Free & Open Source to become
a default pattern
Image server

Want
• Move the “concern” out of my app
• photo is a regular attribute
• configure my app & forget it exist

Want
• Move the “concern” out of my app
• photo is a regular attribute
• configure my app & forget it exist
What would my app look
like in a better world?

My app: Bare minimum
create_table "users" do |t|
t.string "name"
t.json "avatar"
t.json "photos"
end

My app: Bare minimum
create_table "users" do |t|
t.string "name"
t.json "avatar"
t.json "photos" # multiple files in a column
end

My app: Bare minimum
Image serverRails appBrowser
{ avatar: #<File..> }

My app: Bare minimum
Image serverRails appBrowser
{ avatar: #<File..> }
{“path”:“x.jpg”,
“geometry”:“200x600”}
#<File..>

My app: Bare minimum
Image serverRails appBrowser
{“path”:“x.jpg”,
“geometry”:“200x600”}
#<File..>
user.avatar={“path”: “x.jpg”…}
user.save
<img src=“x.jpg”>
{ avatar: #<File..> }

My app: Bare minimum
Image serverRails appBrowser
{“path”:“x.jpg”,
“geometry”:“200x600”}
<img src=“x.jpg”>
#<File..>
GET x.jpg
#<File..>
user.avatar={“path”: “x.jpg”…}
user.save
{ avatar: #<File..> }

My app: Bare minimum
• Browser render <file> field; regular form submit
• Receive binary param, uploads to attache server
and stores the json response instead
• Your app render <img src> requesting for image in
certain size, e.g. http://example/200x/
file.png

Image serverRails appBrowser
#<File..>
{“path”:“x.jpg”,
“geometry”:“200x600”}
Progressive Enhancement

Image serverRails appBrowser
#<File..>
{“path”:“x.jpg”,
“geometry”:“200x600”}
Progressive Enhancement
{ avatar: {“path”:“x.jpg”, … }

{ avatar: {“path”:“x.jpg”, … }
Image serverRails appBrowser
#<File..>
{“path”:“x.jpg”,
“geometry”:“200x600”}
user.avatar={“path”: “x.jpg”…}
user.save
<img src=“x.jpg”>
Progressive Enhancement

• Browser render <file> field; regular form submit
• Receive binary param, uploads to attache server
and stores the json response instead
• Your app render <img src> requesting for image in
certain size, e.g. http://example/200x/
file.png
Progressive Enhancement

• Browser render <file> field; regular form submit
• Receive binary param, uploads to attache server
and stores the json response instead
• Your app render <img src> requesting for image in
certain size, e.g. http://example/200x/
file.png
• JS upload directly to attache server; “Direct upload” in
AWS parlance
• No binary in params; receive and store json attribute
Progressive Enhancement
• When after_update & after_destroy
removes obsolete file from attache via delete API

• Just use Ruby; just use your framework
• Pre-render multiple sizes
• fetch the urls, server will generate and cache
• Validation
• validating a regular json attribute
How do I…

• Move the “concern” out of my app
• photo is a regular attribute
• configure my app & forget it exist
Want (cont’d)

Want (cont’d)
• Move the “concern” out of my app
• photo is a regular attribute
• configure my app & forget it exist
• Separate, standalone server
• Minimal / zero ongoing administration

Want (cont’d)
• Move the “concern” out of my app
• photo is a regular attribute
• configure my app & forget it exist
• Separate, standalone server
• Minimal / zero ongoing administration
How does this
server work?

Attache File Server
• HTTP server with simple APIs
• upload
• download
• delete
• Rack app + ImageMagick
• Go? Node? C++? PHP?
• GraphicsMagick? MyResizer.bash?

• Uploaded files are stored locally
• Resize local file on-the-fly
• configurable pool size to limit concurrent resizing
• Sync upload to cloud storage
• 2 hop problem vs complexity
• Fixed local storage, purge LRU (zero maintenance)
• Spin up new fresh servers anytime…
Attache File Server

• When requested file does not exist locally
• fetch from cloud storage & write locally
• resume operations
Attache File Server

• Remove obsolete file is “best effort”
• If photo delete failed, do you Error 500 or stop the
world?
• OCDs can schedule rake task remove dangling files?
Attache File Server

• Caching in production
• Browser → CDN → Varnish → Disk → Cloud
Attache File Server

Demo
https://attache-demo.herokuapp.com/

https://github.com/choonkeat/attache_api
ATTACHE_URL=http://localhost:9292
ATTACHE_SECRET_KEY=topsecret
rake
Compatibility Check

tus.io
• Open protocol for resumable uploads built on HTTP
• Perfect for mobile apps
• Rack middleware implemented in choonkeat/
attache#10

Responsive Images with
Client Hints
http://blog.imgix.com/2015/10/13/next-generation-responsive-images-with-client.html

SmartCrop
https://github.com/jwagner/smartcrop.js/

Dragonfly & refile

Dragonfly & refile
• tldr: we can fuss over implementation, but it is
mostly about architecture

Dragonfly & refile
• Rack middleware in your Rails app by default
• performing on-the-fly image resize 😱
• Rack standalone end point
• Dragonfly.app - upload, download, delete
• Refile::App - upload, download, delete
• Downloads are unthrottled

Dragonfly & refile
• BEFORE: Rails integrate with AWS
• AFTER: Rails integrate with AWS + Rack app
• Maintain identical AWS config in both apps
• Rails app couldn’t shed the “concern”
• Multiple images still require multiple models

refile
class Post < ActiveRecord::Base
has_many :images, dependent: :destroy
accepts_attachments_for :images, attachment: :file
end
class Image < ActiveRecord::Base
belongs_to :post
attachment :file
end

refile
class Post < ActiveRecord::Base
has_many :images, dependent: :destroy
accepts_attachments_for :images, attachment: :file
end
class Image < ActiveRecord::Base
belongs_to :post
attachment :file
end
“Note it must be possible to persist images
given only the associated post and a file. There
must not be any other validations or constraints
which prevent images from being saved”
i.e. Must be pure dummy wrapper; no
validations here

refile download
https://github.com/refile/refile/blob/master/lib/refile/app.rb
get "/:token/:backend/:processor/:id/:filename" do
halt 404 unless download_allowed?
stream_file processor.call(file)
end

refile download
https://github.com/refile/refile/blob/master/lib/refile/app.rb
get "/:token/:backend/:processor/:id/:filename" do
halt 404 unless download_allowed?
stream_file processor.call(file)
end
How many ImageMagick can
you run in parallel?
has_many :images?

refile upload
https://github.com/refile/refile/blob/master/lib/refile/app.rb
post "/:backend" do
halt 404 unless upload_allowed?
tempfile = request.params.fetch("file").fetch(:tempfile)
file = backend.upload(tempfile)
content_type :json
{ id: file.id }.to_json
end
def file
file = backend.get(params[:id])
unless file.exists?
log_error("Could not find attachment by id: #{params[:id]}")
halt 404
end
file.download
end

post "/:backend" do
halt 404 unless upload_allowed?
tempfile = request.params.fetch("file").fetch(:tempfile)
file = backend.upload(tempfile)
content_type :json
{ id: file.id }.to_json
end
def file
file = backend.get(params[:id])
unless file.exists?
log_error("Could not find attachment by id: #{params[:id]}")
halt 404
end
file.download
end
refile upload
https://github.com/refile/refile/blob/master/lib/refile/app.rb
2 hops problem when
uploading and downloading
• 3mb file becomes 6mb each
way
• #create and #show
becomes 12mb process
• has_many :images?

refile
• CarrierWave-styled named processors (e.g. :fill, :thumb) vs
passing through syntax to underlying ImageMagick
• personally prefer leveraging off existing knowledge
• instead of manually configured syntax sugar
• “2 hop problem” however provide higher consistency
when running multiple refile servers
• upload-here-download-there problem
• (considering to perform 2-hop upload instead of async)

refile
• Can upload to S3 direct and/or upload to refile
• Redundancy interesting, but prefer less concern in Rails
app
• Concept of cache and store to manage “dangling file
problem” is worth considering
• Download urls are signed-only (this practice should be
adopted)
• can partly counter motivation to abuse “dangling file
problem” (aka free file hosting, whee!)

Questions?
Attache server
https://github.com/choonkeat/attache
Gem for Rails
https://github.com/choonkeat/attache_rails
Demo
https://attache-demo.herokuapp.com/