Large Files                         Without the Trials                          Aaron VanDerlip and Sally Kleinfeldt      ...
Acknowledgments                    • Bioneers provides environmental education                         and social connecti...
Acknowledgments                    • Aaron VanDerlip - Project Manager                    • Kapil Thangavelu - DeveloperTh...
What is a Big File?                    • Anything that makes you wait...Thursday, June 3, 2010
Plone Problems with                               Big Files                    1.Uploading/Downloading                    ...
Uploading Big Files                    • Both the user and a Zope thread are                         waiting for the file t...
Thursday, June 3, 2010
Uploading Big Files                    • Browser encodes file in multipart mime                         format             ...
Downloading Big Files                    • ...the same thing happens in reverseThursday, June 3, 2010
Learning from Rails                    • Get file encoding/unencoding and read/                         write operations ou...
Learning from Rails                    • Uploads: Apache plus mod_porter                         http://therailsway.com/ta...
Mod Porter                    • Parses the multipart mime data                    • Writes the file to disk                ...
Mod PorterThursday, June 3, 2010
Apache Config for                                 Mod Porter                         LoadModule apreq_module /usr/lib/Apach...
X-Sendfile                    • HTTP header                    • Set an X-Sendfile header and the path of a                 ...
Apache Config for                                  X-Sendfile                         LoadModule xsendfile_module /usr/lib/A...
Using X-Sendfile                              from Python                         def download(self, response, file_path): ...
Blob Storage                    • Uploads                     • Blob.consumeFile moves file from                           ...
Upload ProcessThursday, June 3, 2010
What About Really                          Really Big Files?                    • Use FTP                    • Supports co...
UIThursday, June 3, 2010
Uploading with FTPThursday, June 3, 2010
ore.bigfile                    • Minimally intrusive, works with the grain of                         Plone                ...
ore.bigfile                                 Limitations                    • Upload directory is hardcoded                 ...
Versioning Big FilesThursday, June 3, 2010
Solution                    • Bypass CMFEditions - no file size limitation                    • Create a new version only w...
UIThursday, June 3, 2010
Conclusion                    • ore.bigfile solves the Big File problem for a                         particular use case, ...
http://svn.objectrealms.net/                  view/public/browser/ore.bigfile                          QuestionsThursday, J...
Upcoming SlideShare
Loading in...5
×

Large Files without the Trials

594

Published on

Sally Kleinfeldt and Aaron VanDerlip describe ore.bigfile, a minimalist solution to the problem of uploading, downloading, and versioning very large files in Plone.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
594
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Large Files without the Trials

  1. 1. Large Files Without the Trials Aaron VanDerlip and Sally Kleinfeldt Plone Symposium East 2010Thursday, June 3, 2010
  2. 2. Acknowledgments • Bioneers provides environmental education and social connectivity through conferences, radio and TV, books, and online materials • Engaged Jazkarta to build a file asset server based on Plone to help them organize, capture, and store multimedia and textual content with files as large as 5 GB.Thursday, June 3, 2010
  3. 3. Acknowledgments • Aaron VanDerlip - Project Manager • Kapil Thangavelu - DeveloperThursday, June 3, 2010
  4. 4. What is a Big File? • Anything that makes you wait...Thursday, June 3, 2010
  5. 5. Plone Problems with Big Files 1.Uploading/Downloading 2.VersioningThursday, June 3, 2010
  6. 6. Uploading Big Files • Both the user and a Zope thread are waiting for the file transferThursday, June 3, 2010
  7. 7. Thursday, June 3, 2010
  8. 8. Uploading Big Files • Browser encodes file in multipart mime format • Zope must undo this encoding • CPU and memory intensive, and SLOW • Zope thread is blocked during this processThursday, June 3, 2010
  9. 9. Downloading Big Files • ...the same thing happens in reverseThursday, June 3, 2010
  10. 10. Learning from Rails • Get file encoding/unencoding and read/ write operations out of Plone • Web servers are really good at this - Apache, Nginx, and Lighttpd • Our implementation uses Apache • Apache file streaming is fast and threads are cheapThursday, June 3, 2010
  11. 11. Learning from Rails • Uploads: Apache plus mod_porter http://therailsway.com/tags/porter • Downloads: Apache plus mod_xsendfile http://john.guen.in/past/2007/4/17/ send_files_faster_with_xsendfile/ • ...and of course ZODB Blob storageThursday, June 3, 2010
  12. 12. Mod Porter • Parses the multipart mime data • Writes the file to disk • Changes the Request to contain a pointer to the temp file on disk • All done efficiently in C code inside your Apache processThursday, June 3, 2010
  13. 13. Mod PorterThursday, June 3, 2010
  14. 14. Apache Config for Mod Porter LoadModule apreq_module /usr/lib/Apache2/modules/mod_apreq2.so LoadModule porter_module /usr/lib/Apache2/modules/mod_porter.so # Apache has a default read limit of 64MB, set it higher APREQ2_ReadLimit 2G ... Porter On # Files below this size will not be handled by mod-porter PorterMinSize 14M # Where the uploaded files are stored PorterDir /mnt/uploads-ApacheThursday, June 3, 2010
  15. 15. X-Sendfile • HTTP header • Set an X-Sendfile header and the path of a file on your response • Apache does the restThursday, June 3, 2010
  16. 16. Apache Config for X-Sendfile LoadModule xsendfile_module /usr/lib/Apache2/modules/mod_xsendfile.so ... EnableSendfile On XSendFile on # Config to send file resources directly from blob storage XSendFilePath /mnt/bioneers/var/blobstorageThursday, June 3, 2010
  17. 17. Using X-Sendfile from Python def download(self, response, file_path): response.setHeader("X-Sendfile", file_path)Thursday, June 3, 2010
  18. 18. Blob Storage • Uploads • Blob.consumeFile moves file from Apache’s temp area to blob storage (ZODB/blob.py) • Uses os.rename, file never enters Plone • Downloads • Served directly from blob storageThursday, June 3, 2010
  19. 19. Upload ProcessThursday, June 3, 2010
  20. 20. What About Really Really Big Files? • Use FTP • Supports continuation and batching • Handles files too large for browser limits • Content editors use FTP to transfer files to an upload directoryThursday, June 3, 2010
  21. 21. UIThursday, June 3, 2010
  22. 22. Uploading with FTPThursday, June 3, 2010
  23. 23. ore.bigfile • Minimally intrusive, works with the grain of Plone • Provides Big File content type • IFrontendFileServer interface defines two methods that provide web server support for upload and download • Apache and Nginx implementations providedThursday, June 3, 2010
  24. 24. ore.bigfile Limitations • Upload directory is hardcoded • Possibility of error on very large images which Mod Porter interceptsThursday, June 3, 2010
  25. 25. Versioning Big FilesThursday, June 3, 2010
  26. 26. Solution • Bypass CMFEditions - no file size limitation • Create a new version only when file changes (not metadata) • Allow old versions to be purged • Version information stored on Big File object using annotationsThursday, June 3, 2010
  27. 27. UIThursday, June 3, 2010
  28. 28. Conclusion • ore.bigfile solves the Big File problem for a particular use case, not feature complete • It does so by taking advantage of mature web server technology • The code is minimally intrusive • It provides a strategy for implementation we can learn from as we improve Plone’s Big File storyThursday, June 3, 2010
  29. 29. http://svn.objectrealms.net/ view/public/browser/ore.bigfile QuestionsThursday, June 3, 2010
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×