SlideShare a Scribd company logo
1 of 46
Download to read offline
📦 Active Storage - Modern File Storage? 📦
by Michael Yagudaev
About Michael
• Co-Founder @ Nano 3 Labs & PictureThat

• Full-stack Developer Rails/React

• Been working with Rails since 2012

• Web dev since 2001

• Passionate about Product Design & AR

• Fun Fact: offline snowboarding 🏂
Outline
• Battle Stories: file upload is hard…
• What is Modern File Storage?
• What is Active Record?
• Uploading a File
• Transforming a File
• Downloading/Previewing a File
• Alternatives to Active Storage
• Conclusion
Battle Stories
Story 1: Bail on Rails
• Back in 2012 (Rails 2.3) — small startup, terrible uploader. Users upload
entire hi-res albums (100s of photos). Blocking all other users from
doing anything on the platform! :(
• Sales staff hides in hope of not running into angry users
• Can you build us a better file uploader?
• Me: yes! Let’s use Node.js + Direct Upload to S3
• Me 3 months later: FML… node… ;(
• We created a robust uploader
• auto-retry failed uploads, priority resize images users requested,
pre-create resize jobs for all images sizes, custom uploader ui
Story 2: Rails re-discovered
• 0idle - our own startup helping event organizers find venues
• 2013: implemented multi-file upload
• Carrierwave + Carrierwave-Direct S3
• Only took a few days
• ❤ Rails
• Life is good :)
• Custom upload UI
Story 3: DDoS Day
• Summer 2018 - implemented ActiveStorage for PictureThat. Loved
the variants feature
• Fall 2018 -
• DDoS attack on one of our projects
• Targeted slow pdf conversation end-points
• Made us look into slow running requests
• Second slowest was asset upload
• A DevOps Engineer I was working with outlined “Modern File
Storage” to me… it got me thinking, is Active Storage Modern?
What is Modern File Storage?
• Modular
• Supports cloud providers
• Fast
• Scalable
• Cheap
• Easy to setup
• API Based
What is ActiveStorage?
• Extracted from BaseCamp
• Supports
• Cloud Providers (S3, digital ocean, GCP, Azure)
• Variants
• Mirroring
• Previews
• Accessible MetaData* (sorta we will see soon)
• Upload: documents, images and videos (yes!)
File Life Cycle
File Life Cycle
1. select file(s)
2. upload file
3. analyze file
4. transform file
5. preview / download
$ bundle open activestorage
File Life Cycle - AS Mapping
1. select file(s) - rails form / bring your own ui
2. upload file - activestorage/app/controllers/
3. analyze file - activestorage/jobs/…/analyze_job.rb
4. transform file - activestorage/lib/…/
transformers.rb
5. preview / download - activestorage/models/…/
variant.rb (or) activestorage/models/…/preview
1. File Selection
1. Select File(s) - Traditional
• Easy with rails forms
• Add <%= form.file_field :files, multiple: true,%>
• For S3 Direct Upload (work with Digital Ocean, GCP,
etc)
• <%= form.file_field :files, multiple: true, direct_upload:
true,%>
• Use direct upload, no long requests
Rails Form File Field Be Like…
1. Select File(s) - BYOU
• Use modern Javascript
framework like React
• Try uppy.io
• Can work as Multi-part
XHR Upload
• But a two-step API based
upload is better
• API should support upload
from iOS and Android apps
2. Uploading
2. Uploading
• Considerations
• size of files (resumable?)
• # files
• types of files
• file selection method in step #1
• Offload away from Rails, it wasn’t meant for it
• Maintain short connection timeouts. Let S3 handle upload
Multipart form file upload
• Sends metadata & file content in one request
• Chunks file and sends using a boundary, looks like
• Content-Type: multipart/form-data; boundary=——
WebKitFormBoundarydmmGebTloNbZGViV
• Old and meant for forms, not APIs
• Can be used with XHR request
2-Step API Upload
• Instead use 2 requests
1. Upload Meta data (file name, size, type, etc)
2. Get Location back from step #1 for direct upload
• Active Storage Direct does this!
• POST http://localhost:3000/rails/active_storage/direct_uploads
• { blob: { filename: …, content_type: “image/jpeg”, byte_size:
123, checksum: “aoEPYm6I0kn9sFqekqy5yg==" }}
• RESPONSE: { direct_upload:{url:”<s3_signed_url>”, headers: {…}} }
AS Schema
• Run: $ rails active_storage:install
• active_storage_blobs => Meta data about file
• byte_size
• content_type
• checksum (md5 as base64)
• metadata (height, width, etc)
• active_storage_attachments => Relationship between your models and blobs
• blob_id
• record_id
• record_type (polymorphic association)
Custom API
• Simple all you need is
• Then just
• POST /api/v1/files { blob: { filename, content_type,
byte_size, checksum } }
class	Api::V1::FilesController	<	ActiveStorage::DirectUploadsController	
		skip_forgery_protection	
end
3. Analyzing
3. Analyze
• File has already been uploaded
• File is in temp folder or cloud storage
• If in temp
• Easy, run appropriate tool like`identify` on images
• Compare meta to client received ones
• Save meta data back
3. Analyze
• If on cloud storage (s3), either:
• Download to tmp file, and run analysis locally (AS)
• Run cloud analysis and connect to DB + update or update
through a webhook (no help from AS here)
• Ideally run background job / cloud function
• Might want to run things like a virus scan
• ActiveStorage has an AnalyzeJob, use it
• What happens if user requests resource right away?
4. Transformations
4. Transformation
• Resize images, extract display frame from video
• Either pre-process or lazy-evaluate (see step 5)
• Active Storage uses background jobs
• Alternatively, cloud function + cloud storage
• But…
Active Storage on S3…
ATM: No way to customize paths for human readable names :(
no file extensions?!
5. Downloading / Previewing
5. Download / Preview
• Viewing the original is simple
• <%= image_tag @user.avatar %>
• To get S3 url: @user.avatar.service_url
• Return it in your API
Displaying Transformed
Version
• upload.variable? (i.e. images)
• png, jpeg, etc (not SVG)
• upload.previewable? (i.e. can get an image from them)
• pdf, mpeg (video), etc
Variants
• only applies to images, can change size
• <%= image_tag @user.avatar.variant(resize:
“400x400”) %>
• 💡 dynamically generate an image size based on
device
• to link to full-size image simply do:
• <%= link_to image_tag(@user.avatar.variant(resize:
“400x400”)), @user.avatar %>
PictureThat AR
App
PictureThat Backend
PictureThat Backend
Variants
• ActiveStorage can transform(resize, etc) documents on
the fly
• in process
• bad performance
• works for small images only (due to connection
timeouts)
• in background process
• dispatches a resize job
File Previews
• As easy as this!
• <%= image_tag file.preview(resize: '400x400') %>
• Works for documents: pdf, videos!
• Can build your own custom previewers
Behind the Scenes
• hides your files behind a rails route
• /rails/active_storage/representations/a84…3b/file_name.jpg
• great for
• switching between environments or mirrors
• dynamically generating sizes
• no more size having to call resize on all images in the database
• lazily evaluated
• bad for
• performance, another call to rails as opposed to going to nginx or directly to S3
Advanced Transformation
• none of these are supported by ActiveStorage
• on-the-fly resize with s3
• AWS ServerLess-Thumbor -> https://github.com/
awslabs/serverless-image-handler
• http://thumbor-server/filters:watermark(http://
my.site.com/img.png,10p,-20p,50)/some/image.jpg
Alternative: Shrine
• Good
• Been around for longer
• Retains file extension on storage
• Customizable with Plugins
• Supports Cloud Processing using Plugins
• Works with uppy.io out of the box
• Bad
• More work to setup
Alternative: No Rails…
• Firebase Storage => easy to setup from an JS frontend
• Storage as a Service:
• https://transloadit.com/
• https://www.filestack.com/
• https://cloudinary.com
Conclusion
Conclusion
• ActiveStorage shows potentials and works for simple
cases (Admin interfaces)
• For all other, start with Shrine
• Don’t over-optimize. But keep your options open
References
• https://edgeguides.rubyonrails.org/active_storage_overview.html
• https://gorails.com/episodes/file-uploading-with-activestorage-rails-5-2
• https://gorails.com/episodes/how-to-create-an-active-storage-previewer
• https://philsturgeon.uk/api/2016/01/04/http-rest-api-file-uploads/
• https://medium.com/typecode/a-strategy-for-handling-multiple-file-uploads-using-javascript-
eb00a77e15f
• https://www.netlify.com/blog/2016/11/17/serverless-file-uploads/
• https://bibwild.wordpress.com/2018/10/03/some-notes-on-whats-going-on-in-activestorage/
• https://github.com/rails/rails/issues/31419
• https://aws.amazon.com/blogs/compute/resize-images-on-the-fly-with-amazon-s3-aws-
lambda-and-amazon-api-gateway/
• https://github.com/texpert/shrine-lambda
Questions?
• twitter: @yagudaev, @nano3labs
• come talk shop, tell me about your Rails project
• Our friends @spacelist are hiring remote F/T senior
ruby/rails dev

More Related Content

What's hot

The WordPress University
The WordPress UniversityThe WordPress University
The WordPress University
Stephanie Leary
 
5 Common Mistakes You are Making on your Website
 5 Common Mistakes You are Making on your Website 5 Common Mistakes You are Making on your Website
5 Common Mistakes You are Making on your Website
Acquia
 
The things we found in your website
The things we found in your websiteThe things we found in your website
The things we found in your website
hernanibf
 
My site is slow
My site is slowMy site is slow
My site is slow
hernanibf
 
Challenges of Simple Documents: When Basic isn't so Basic - Cassandra Targett...
Challenges of Simple Documents: When Basic isn't so Basic - Cassandra Targett...Challenges of Simple Documents: When Basic isn't so Basic - Cassandra Targett...
Challenges of Simple Documents: When Basic isn't so Basic - Cassandra Targett...
Lucidworks
 

What's hot (20)

The WordPress University
The WordPress UniversityThe WordPress University
The WordPress University
 
Building intranet applications with ASP.NET AJAX and jQuery
Building intranet applications with ASP.NET AJAX and jQueryBuilding intranet applications with ASP.NET AJAX and jQuery
Building intranet applications with ASP.NET AJAX and jQuery
 
Riding the Edge with Ember.js
Riding the Edge with Ember.jsRiding the Edge with Ember.js
Riding the Edge with Ember.js
 
5 Common Mistakes You are Making on your Website
 5 Common Mistakes You are Making on your Website 5 Common Mistakes You are Making on your Website
5 Common Mistakes You are Making on your Website
 
Building intranet applications with ASP.NET AJAX and jQuery
Building intranet applications with ASP.NET AJAX and jQueryBuilding intranet applications with ASP.NET AJAX and jQuery
Building intranet applications with ASP.NET AJAX and jQuery
 
The things we found in your website
The things we found in your websiteThe things we found in your website
The things we found in your website
 
A Day of REST
A Day of RESTA Day of REST
A Day of REST
 
IBM Connect 2016 - AD1548 - Building Responsive XPages Applications
IBM Connect 2016 - AD1548 - Building Responsive XPages ApplicationsIBM Connect 2016 - AD1548 - Building Responsive XPages Applications
IBM Connect 2016 - AD1548 - Building Responsive XPages Applications
 
Georgia Tech Drupal Users Group - Local Drupal Development
Georgia Tech Drupal Users Group - Local Drupal DevelopmentGeorgia Tech Drupal Users Group - Local Drupal Development
Georgia Tech Drupal Users Group - Local Drupal Development
 
Play Architecture, Implementation, Shiny Objects, and a Proposal
Play Architecture, Implementation, Shiny Objects, and a ProposalPlay Architecture, Implementation, Shiny Objects, and a Proposal
Play Architecture, Implementation, Shiny Objects, and a Proposal
 
My site is slow
My site is slowMy site is slow
My site is slow
 
Extreme Web Performance for Mobile Devices - Velocity NY
Extreme Web Performance for Mobile Devices - Velocity NYExtreme Web Performance for Mobile Devices - Velocity NY
Extreme Web Performance for Mobile Devices - Velocity NY
 
Introduction to Apache Roller
Introduction to Apache RollerIntroduction to Apache Roller
Introduction to Apache Roller
 
Flexible UI Components for a Multi-Framework World
Flexible UI Components for a Multi-Framework WorldFlexible UI Components for a Multi-Framework World
Flexible UI Components for a Multi-Framework World
 
EJB 3.2 - Java EE 7 - Java One Hyderabad 2012
EJB 3.2 - Java EE 7 - Java One Hyderabad 2012EJB 3.2 - Java EE 7 - Java One Hyderabad 2012
EJB 3.2 - Java EE 7 - Java One Hyderabad 2012
 
Comparing Hot JavaScript Frameworks: AngularJS, Ember.js and React.js - Sprin...
Comparing Hot JavaScript Frameworks: AngularJS, Ember.js and React.js - Sprin...Comparing Hot JavaScript Frameworks: AngularJS, Ember.js and React.js - Sprin...
Comparing Hot JavaScript Frameworks: AngularJS, Ember.js and React.js - Sprin...
 
XPages: No Experience Needed
XPages: No Experience NeededXPages: No Experience Needed
XPages: No Experience Needed
 
User-percieved performance
User-percieved performanceUser-percieved performance
User-percieved performance
 
Tech Talk on Cloud Computing
Tech Talk on Cloud ComputingTech Talk on Cloud Computing
Tech Talk on Cloud Computing
 
Challenges of Simple Documents: When Basic isn't so Basic - Cassandra Targett...
Challenges of Simple Documents: When Basic isn't so Basic - Cassandra Targett...Challenges of Simple Documents: When Basic isn't so Basic - Cassandra Targett...
Challenges of Simple Documents: When Basic isn't so Basic - Cassandra Targett...
 

Similar to  Active Storage - Modern File Storage? 

GeneralMobile Hybrid Development with WordPress
GeneralMobile Hybrid Development with WordPressGeneralMobile Hybrid Development with WordPress
GeneralMobile Hybrid Development with WordPress
GGDBologna
 
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
DATAVERSITY
 
Should you use HTML5 to build your product? The pros & cons of using current ...
Should you use HTML5 to build your product? The pros & cons of using current ...Should you use HTML5 to build your product? The pros & cons of using current ...
Should you use HTML5 to build your product? The pros & cons of using current ...
boxuno
 
Netflix oss season 2 episode 1 - meetup Lightning talks
Netflix oss   season 2 episode 1 - meetup Lightning talksNetflix oss   season 2 episode 1 - meetup Lightning talks
Netflix oss season 2 episode 1 - meetup Lightning talks
Ruslan Meshenberg
 

Similar to  Active Storage - Modern File Storage?  (20)

GraphTour - Albelli: Running Neo4j on a large scale image platform
GraphTour - Albelli: Running Neo4j on a large scale image platformGraphTour - Albelli: Running Neo4j on a large scale image platform
GraphTour - Albelli: Running Neo4j on a large scale image platform
 
Rails scaling
Rails scalingRails scaling
Rails scaling
 
12-Step Program for Scaling Web Applications on PostgreSQL
12-Step Program for Scaling Web Applications on PostgreSQL12-Step Program for Scaling Web Applications on PostgreSQL
12-Step Program for Scaling Web Applications on PostgreSQL
 
Training on iOS app development - Samesh Swongamikha & Neetin Sharma
Training on iOS app development - Samesh Swongamikha & Neetin SharmaTraining on iOS app development - Samesh Swongamikha & Neetin Sharma
Training on iOS app development - Samesh Swongamikha & Neetin Sharma
 
GeneralMobile Hybrid Development with WordPress
GeneralMobile Hybrid Development with WordPressGeneralMobile Hybrid Development with WordPress
GeneralMobile Hybrid Development with WordPress
 
Mobile Hybrid Development with WordPress
Mobile Hybrid Development with WordPressMobile Hybrid Development with WordPress
Mobile Hybrid Development with WordPress
 
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
A Case Study of NoSQL Adoption: What Drove Wordnik Non-Relational?
 
Static Site Generators - Developing Websites in Low-resource Condition
Static Site Generators - Developing Websites in Low-resource ConditionStatic Site Generators - Developing Websites in Low-resource Condition
Static Site Generators - Developing Websites in Low-resource Condition
 
Introduction to Android Development and Security
Introduction to Android Development and SecurityIntroduction to Android Development and Security
Introduction to Android Development and Security
 
Advanced Site Studio Class, June 18, 2012
Advanced Site Studio Class, June 18, 2012Advanced Site Studio Class, June 18, 2012
Advanced Site Studio Class, June 18, 2012
 
Not Just ORM: Powerful Hibernate ORM Features and Capabilities
Not Just ORM: Powerful Hibernate ORM Features and CapabilitiesNot Just ORM: Powerful Hibernate ORM Features and Capabilities
Not Just ORM: Powerful Hibernate ORM Features and Capabilities
 
Should you use HTML5 to build your product? The pros & cons of using current ...
Should you use HTML5 to build your product? The pros & cons of using current ...Should you use HTML5 to build your product? The pros & cons of using current ...
Should you use HTML5 to build your product? The pros & cons of using current ...
 
Tuenti Release Workflow v1.1
Tuenti Release Workflow v1.1Tuenti Release Workflow v1.1
Tuenti Release Workflow v1.1
 
Optimization of modern web applications
Optimization of modern web applicationsOptimization of modern web applications
Optimization of modern web applications
 
Alfresco Tech Talk Live (Episode 70): Customizing Alfresco Share 4.2
Alfresco Tech Talk Live (Episode 70): Customizing Alfresco Share 4.2Alfresco Tech Talk Live (Episode 70): Customizing Alfresco Share 4.2
Alfresco Tech Talk Live (Episode 70): Customizing Alfresco Share 4.2
 
Ship It ! with Ruby/ Rails Ecosystem
Ship It ! with Ruby/ Rails EcosystemShip It ! with Ruby/ Rails Ecosystem
Ship It ! with Ruby/ Rails Ecosystem
 
Django Overview
Django OverviewDjango Overview
Django Overview
 
Netflix oss season 2 episode 1 - meetup Lightning talks
Netflix oss   season 2 episode 1 - meetup Lightning talksNetflix oss   season 2 episode 1 - meetup Lightning talks
Netflix oss season 2 episode 1 - meetup Lightning talks
 
Untangling - fall2017 - week 9
Untangling - fall2017 - week 9Untangling - fall2017 - week 9
Untangling - fall2017 - week 9
 
Ruby and Distributed Storage Systems
Ruby and Distributed Storage SystemsRuby and Distributed Storage Systems
Ruby and Distributed Storage Systems
 

Recently uploaded

TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc
 
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
Muhammad Subhan
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
panagenda
 

Recently uploaded (20)

ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage Intacct
 
Event-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream ProcessingEvent-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream Processing
 
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
 
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
 
Generative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfGenerative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdf
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptx
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024
 
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
 
ADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptx
 
2024 May Patch Tuesday
2024 May Patch Tuesday2024 May Patch Tuesday
2024 May Patch Tuesday
 
Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptx
 
Design and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data ScienceDesign and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data Science
 
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptxCyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
 
How to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cfHow to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cf
 
Top 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development CompaniesTop 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development Companies
 

 Active Storage - Modern File Storage? 

  • 1. 📦 Active Storage - Modern File Storage? 📦 by Michael Yagudaev
  • 2. About Michael • Co-Founder @ Nano 3 Labs & PictureThat • Full-stack Developer Rails/React • Been working with Rails since 2012 • Web dev since 2001 • Passionate about Product Design & AR • Fun Fact: offline snowboarding 🏂
  • 3. Outline • Battle Stories: file upload is hard… • What is Modern File Storage? • What is Active Record? • Uploading a File • Transforming a File • Downloading/Previewing a File • Alternatives to Active Storage • Conclusion
  • 5. Story 1: Bail on Rails • Back in 2012 (Rails 2.3) — small startup, terrible uploader. Users upload entire hi-res albums (100s of photos). Blocking all other users from doing anything on the platform! :( • Sales staff hides in hope of not running into angry users • Can you build us a better file uploader? • Me: yes! Let’s use Node.js + Direct Upload to S3 • Me 3 months later: FML… node… ;( • We created a robust uploader • auto-retry failed uploads, priority resize images users requested, pre-create resize jobs for all images sizes, custom uploader ui
  • 6. Story 2: Rails re-discovered • 0idle - our own startup helping event organizers find venues • 2013: implemented multi-file upload • Carrierwave + Carrierwave-Direct S3 • Only took a few days • ❤ Rails • Life is good :) • Custom upload UI
  • 7. Story 3: DDoS Day • Summer 2018 - implemented ActiveStorage for PictureThat. Loved the variants feature • Fall 2018 - • DDoS attack on one of our projects • Targeted slow pdf conversation end-points • Made us look into slow running requests • Second slowest was asset upload • A DevOps Engineer I was working with outlined “Modern File Storage” to me… it got me thinking, is Active Storage Modern?
  • 8. What is Modern File Storage? • Modular • Supports cloud providers • Fast • Scalable • Cheap • Easy to setup • API Based
  • 9. What is ActiveStorage? • Extracted from BaseCamp • Supports • Cloud Providers (S3, digital ocean, GCP, Azure) • Variants • Mirroring • Previews • Accessible MetaData* (sorta we will see soon) • Upload: documents, images and videos (yes!)
  • 11. File Life Cycle 1. select file(s) 2. upload file 3. analyze file 4. transform file 5. preview / download
  • 12. $ bundle open activestorage
  • 13. File Life Cycle - AS Mapping 1. select file(s) - rails form / bring your own ui 2. upload file - activestorage/app/controllers/ 3. analyze file - activestorage/jobs/…/analyze_job.rb 4. transform file - activestorage/lib/…/ transformers.rb 5. preview / download - activestorage/models/…/ variant.rb (or) activestorage/models/…/preview
  • 15. 1. Select File(s) - Traditional • Easy with rails forms • Add <%= form.file_field :files, multiple: true,%> • For S3 Direct Upload (work with Digital Ocean, GCP, etc) • <%= form.file_field :files, multiple: true, direct_upload: true,%> • Use direct upload, no long requests
  • 16. Rails Form File Field Be Like…
  • 17. 1. Select File(s) - BYOU • Use modern Javascript framework like React • Try uppy.io • Can work as Multi-part XHR Upload • But a two-step API based upload is better • API should support upload from iOS and Android apps
  • 19. 2. Uploading • Considerations • size of files (resumable?) • # files • types of files • file selection method in step #1 • Offload away from Rails, it wasn’t meant for it • Maintain short connection timeouts. Let S3 handle upload
  • 20. Multipart form file upload • Sends metadata & file content in one request • Chunks file and sends using a boundary, looks like • Content-Type: multipart/form-data; boundary=—— WebKitFormBoundarydmmGebTloNbZGViV • Old and meant for forms, not APIs • Can be used with XHR request
  • 21. 2-Step API Upload • Instead use 2 requests 1. Upload Meta data (file name, size, type, etc) 2. Get Location back from step #1 for direct upload • Active Storage Direct does this! • POST http://localhost:3000/rails/active_storage/direct_uploads • { blob: { filename: …, content_type: “image/jpeg”, byte_size: 123, checksum: “aoEPYm6I0kn9sFqekqy5yg==" }} • RESPONSE: { direct_upload:{url:”<s3_signed_url>”, headers: {…}} }
  • 22. AS Schema • Run: $ rails active_storage:install • active_storage_blobs => Meta data about file • byte_size • content_type • checksum (md5 as base64) • metadata (height, width, etc) • active_storage_attachments => Relationship between your models and blobs • blob_id • record_id • record_type (polymorphic association)
  • 23. Custom API • Simple all you need is • Then just • POST /api/v1/files { blob: { filename, content_type, byte_size, checksum } } class Api::V1::FilesController < ActiveStorage::DirectUploadsController skip_forgery_protection end
  • 25. 3. Analyze • File has already been uploaded • File is in temp folder or cloud storage • If in temp • Easy, run appropriate tool like`identify` on images • Compare meta to client received ones • Save meta data back
  • 26. 3. Analyze • If on cloud storage (s3), either: • Download to tmp file, and run analysis locally (AS) • Run cloud analysis and connect to DB + update or update through a webhook (no help from AS here) • Ideally run background job / cloud function • Might want to run things like a virus scan • ActiveStorage has an AnalyzeJob, use it • What happens if user requests resource right away?
  • 28. 4. Transformation • Resize images, extract display frame from video • Either pre-process or lazy-evaluate (see step 5) • Active Storage uses background jobs • Alternatively, cloud function + cloud storage • But…
  • 29. Active Storage on S3… ATM: No way to customize paths for human readable names :( no file extensions?!
  • 30. 5. Downloading / Previewing
  • 31. 5. Download / Preview • Viewing the original is simple • <%= image_tag @user.avatar %> • To get S3 url: @user.avatar.service_url • Return it in your API
  • 32. Displaying Transformed Version • upload.variable? (i.e. images) • png, jpeg, etc (not SVG) • upload.previewable? (i.e. can get an image from them) • pdf, mpeg (video), etc
  • 33. Variants • only applies to images, can change size • <%= image_tag @user.avatar.variant(resize: “400x400”) %> • 💡 dynamically generate an image size based on device • to link to full-size image simply do: • <%= link_to image_tag(@user.avatar.variant(resize: “400x400”)), @user.avatar %>
  • 37. Variants • ActiveStorage can transform(resize, etc) documents on the fly • in process • bad performance • works for small images only (due to connection timeouts) • in background process • dispatches a resize job
  • 38. File Previews • As easy as this! • <%= image_tag file.preview(resize: '400x400') %> • Works for documents: pdf, videos! • Can build your own custom previewers
  • 39. Behind the Scenes • hides your files behind a rails route • /rails/active_storage/representations/a84…3b/file_name.jpg • great for • switching between environments or mirrors • dynamically generating sizes • no more size having to call resize on all images in the database • lazily evaluated • bad for • performance, another call to rails as opposed to going to nginx or directly to S3
  • 40. Advanced Transformation • none of these are supported by ActiveStorage • on-the-fly resize with s3 • AWS ServerLess-Thumbor -> https://github.com/ awslabs/serverless-image-handler • http://thumbor-server/filters:watermark(http:// my.site.com/img.png,10p,-20p,50)/some/image.jpg
  • 41. Alternative: Shrine • Good • Been around for longer • Retains file extension on storage • Customizable with Plugins • Supports Cloud Processing using Plugins • Works with uppy.io out of the box • Bad • More work to setup
  • 42. Alternative: No Rails… • Firebase Storage => easy to setup from an JS frontend • Storage as a Service: • https://transloadit.com/ • https://www.filestack.com/ • https://cloudinary.com
  • 44. Conclusion • ActiveStorage shows potentials and works for simple cases (Admin interfaces) • For all other, start with Shrine • Don’t over-optimize. But keep your options open
  • 45. References • https://edgeguides.rubyonrails.org/active_storage_overview.html • https://gorails.com/episodes/file-uploading-with-activestorage-rails-5-2 • https://gorails.com/episodes/how-to-create-an-active-storage-previewer • https://philsturgeon.uk/api/2016/01/04/http-rest-api-file-uploads/ • https://medium.com/typecode/a-strategy-for-handling-multiple-file-uploads-using-javascript- eb00a77e15f • https://www.netlify.com/blog/2016/11/17/serverless-file-uploads/ • https://bibwild.wordpress.com/2018/10/03/some-notes-on-whats-going-on-in-activestorage/ • https://github.com/rails/rails/issues/31419 • https://aws.amazon.com/blogs/compute/resize-images-on-the-fly-with-amazon-s3-aws- lambda-and-amazon-api-gateway/ • https://github.com/texpert/shrine-lambda
  • 46. Questions? • twitter: @yagudaev, @nano3labs • come talk shop, tell me about your Rails project • Our friends @spacelist are hiring remote F/T senior ruby/rails dev