Successfully reported this slideshow.
Your SlideShare is downloading. ×

pixiv thumbnails

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 68 Ad

More Related Content

Similar to pixiv thumbnails (20)

Advertisement

pixiv thumbnails

  1. 1. pixiv thumbnails Tatsuhiko Kubo  bokko@pixiv.com
  2. 2. Introduction to pixiv ● Illust Communication Site ○ http://www.pixiv.net ● Users ○ 5 million ● Monthly page view ○ 3.3 billion ● Network Traffic ○ Over 6Gbps
  3. 3. About me ● Tatsuhiko Kubo(H.N:bokko) ● @cubicdaiya ● Infrastructure & Software Engineer@pixiv. Inc
  4. 4. My Works@pixiv. Inc Responsible for ● Middle-Ware Development ● Technical Operation & Administration ● Datastore & Caching Strategy ● Image Upload & Transformation ● Illust & User Recommendation ● Notification( called Popboard at pixiv) etc...
  5. 5. My works@private ■Software ● neoagent ○ A Yet Another Memcached Protocol Proxy Server ● dtl ○ diff template library with C++ ● ngx_small_light ○ Dynamic Transformation Module for Nginx ■Writing ● Software Design 2009 Sep ○ どのようにして差分を導き出すのか~diffの動作原理を知る~ ○ http://gihyo.jp/dev/column/01/prog/2011/diff_sd200906 And More -> http://cccis.jp
  6. 6. Agenda ● Following mechanism at pixiv ○ Image Upload ○ Thumbnail Generation
  7. 7. Scale of pixiv according to thumbnails ● pixiv has about 30,000,000 illusts and comics ● Each illust has about 12 ~ too many thumbnails ● 20,000 illusts and comics are uploaded every day ● Total volume is about 30TB
  8. 8. Image Upload
  9. 9. Image Upload Detail 1 ● Generate too many thumbnails ○ 12 ~ too many thumbnails ● Save a original image and thumbnails to storage ○ not NFS ○ with in-house WebDAV Client(ImageClient)
  10. 10. ImageClient WebDAV Client
  11. 11. ImageClient Powered by @?????
  12. 12. ImageClient Powered by @?????
  13. 13. ImageClient Powered by @kamipo
  14. 14. ImageClient ● Enable WebDAV operations to multiple servers transparently ○ put image ○ delete image ○ move image ○ make directory PUT ImageClient ○ move directory DELETE MOVE PUT DELETE ○ delete directory MKCOL MOVE MKCOL Image Image Storage ・・・・・ Storage
  15. 15. Image Upload Detail 2 ● A image is uploaded semi-asynchronously ○ Generating thumbnails takes a long time
  16. 16. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View
  17. 17. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View
  18. 18. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View create lock file
  19. 19. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View create lock file
  20. 20. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View create lock file poll until lock file is deleted
  21. 21. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View create lock file poll until lock file is deleted Server Side Action
  22. 22. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View create lock file poll until lock file is deleted Server Side Action prefork server
  23. 23. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View create lock file poll until lock file is deleted Server Side Action prefork prefork server upload worker upload worker ・ ・ ・ ・ ・ upload worker
  24. 24. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View create lock file poll until lock file is deleted Server Side Action prefork prefork server upload worker Genrate Thumbnails upload worker Genrate Thumbnails ・ ・ ・ ・ ・ upload worker Genrate Thumbnails
  25. 25. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View create lock file poll until lock file is deleted Server Side Action prefork prefork server upload worker Genrate Thumbnails delete lock file upload worker Genrate Thumbnails delete lock file ・ ・ ・ ・ ・ upload worker Genrate Thumbnails delete lock file
  26. 26. Inside Image Upload ● User Side Action ○ Apache ○ PHP ■ ImageClient ● Server Side Action ○ daemontools ○ Python ■ python-prefork, python-q4m, python-worker ● python-q4m and python-worker are in-house libraries ○ Q4M ■ Used As Upload Job Queue
  27. 27. Thumbnail Generation
  28. 28. Two types of thumbnails at pixiv ● Static Thumbnail ● Dynamic Thumbnail
  29. 29. Static Thumbnail
  30. 30. Static Thumbnail Generated on upload
  31. 31. Detail of generating thumbnails ● pixiv uses ImageMagick ○ ImageMagick is not fast ○ But quality of generated image is good ○ Quality is more important than speed for us ○ Of course, optimization is important, too
  32. 32. ImageMagick Optimization
  33. 33. libjpeg-turbo
  34. 34. libjpeg-turbo ● Optimized libjpeg for x86 and x86_64 ● Replace libjpeg simply by LD_LIBRARY_PATH ● ImageMagick uses libjpeg
  35. 35. benchmark of libjpeg and libjpeg-turbo processing JPEG-image with libjpeg and libjpeg-turbo ■libjpeg ■libjpeg-turbo
  36. 36. benchmark of libjpeg and libjpeg-turbo processing JPEG-image with libjpeg and libjpeg-turbo ■libjpeg ■libjpeg-turbo libjpeg-turbo is faster than libjpeg by 10% on x86_64
  37. 37. Disable OpenMP
  38. 38. Disable OpenMP ● Latest ImageMagick is OpenMP enabled at default ○ This is very slow in multi-process environment ● How to disable OpenMP ○ Re-compile with '--disable-openmp' ○ OMP_NUM_THREADS=1 ● pixiv takes the latter ○ Re-compiling and Re-packaging are complicated
  39. 39. OMP_NUM_THREADS=1 CPU Usage of Application Server
  40. 40. OMP_NUM_THREADS=1 CPU usage in peak dropped by 150~200% CPU Usage of Application Server
  41. 41. OMP_NUM_THREADS=1 Load Average of Application Server
  42. 42. OMP_NUM_THREADS=1 Load Average became dust Load Average of Application Server
  43. 43. Dynamic Thumbnail
  44. 44. Dynamic Thumbnail Generated on demand
  45. 45. Why is dynamic thumbnail needed? ● Static thumbnail consumes disk space ○ Dynamic thumbnail does not consume disk space ● Preparing new static thumbnails takes a long time ○ Dynamic thumbnail is ready in a second!
  46. 46. Other Companies' Cases ● Cookpad ○ mod_tofu ○ http://www.slideshare.net/mirakui/ss-8150494 ● livedoor(Now, NHN Japan) ○ mod_small_light ○ http://www.slideshare.net/livedoor/smalllight2
  47. 47. pixiv's Case
  48. 48. mod_small_light-1.1.1
  49. 49. mod_small_light-1.1.1 pixiv Edition5
  50. 50. mod_small_light-1.1.1 pixiv Edition5 heavily customized...
  51. 51. Heavily customized...
  52. 52. mod_small_light configuration
  53. 53. Resize image with mod_small_light GET /tank.jpg GET /small_light(dw=100,dh=100)/tank.jpg
  54. 54. Many Options q image quality(0~100) of output format(jpg,gif,png,tiff) e processing engine(imlib2,imagemagick) cc canvas color p pattern name with SmallLightPatternDefine etc Too many options
  55. 55. Pattern Name
  56. 56. Resize image with mod_small_light GET /small_light(p=small)/tank.jpg GET /tank.jpg or GET /small_light(dw=100,dh=100,e=imagemagick,jpeghint=y)/tank.jpg
  57. 57. Why not original mod_small_light? ● Some thumbnails are special ○ comic cover ○ various cropping algorithms depending on aspect ratio ● Default output format is JPEG ○ It is good for us that input and output formats are the same
  58. 58. Why not original mod_small_light? ● No support for CMYK ○ pixiv must support CMYK ● Needed support for strange aspect ratio ○ da=l -> long-edge ○ da=s -> short-edge ○ da=p -> pixiv-edge (only pixiv Edition)
  59. 59. Special Thumbnail 1 ● comic cover
  60. 60. Special Thumbnail 2 ● special cropping algorithm
  61. 61. Special Thumbnail 2 ● special cropping algorithm
  62. 62. Special Thumbnail 2 ● special cropping algorithm crop center of image if image is landscape
  63. 63. Special Thumbnail 3 ● special cropping algorithm
  64. 64. Special Thumbnail 3 ● special cropping algorithm
  65. 65. Special Thumbnail 3 ● special cropping algorithm crop top of image if image is portrait
  66. 66. Many Extend Options rmprof remove image profiles crop_square crop image squarly cover add manga cover samec conform dw & dh & cw & ch extendl extend long-edge etc twenty new options in all
  67. 67. Summary ● Optimization of image processing is very important ○ Generating thumbnail takes a long time ○ Let's tune and desynchronize ● pixiv has two types of thumbnails ○ Static Thumbnail ○ Dynamic Thumbnail ● Dynamic Thumbnail is diskspace-saving and flexible ○ Big image storage is expensive ○ Easy to correspond to shift of application ■ New thumbnail types for new designs
  68. 68. Thanks!

×