Downloaded 21 times
Uploaded byTatsuhiko Kubo
pixiv thumbnails
- Pixiv is an illustration communication site with over 5 million users and 3.3 billion monthly page views. - The presenter works on infrastructure and software engineering at Pixiv, where he is responsible for image upload, thumbnail generation, data storage, caching strategies and more. - Pixiv generates 12 or more thumbnails for each image uploaded to optimize loading and browsing on different devices. With over 30 million images, this amounts to over 30 terabytes of thumbnails.
pixiv thumbnails
- 1. pixiv thumbnails Tatsuhiko Kubo bokko@pixiv.com
- 2. Introduction to pixiv ●Illust Communication Site ○ http://www.pixiv.net ● Users ○ 5 million ● Monthly page view ○ 3.3 billion ● Network Traffic ○ Over 6Gbps
- 3. About me ● TatsuhikoKubo(H.N：bokko) ● @cubicdaiya ● Infrastructure & Software Engineer@pixiv. Inc
- 4. My Works@pixiv. Inc Responsiblefor ● Middle-Ware Development ● Technical Operation & Administration ● Datastore & Caching Strategy ● Image Upload & Transformation ● Illust & User Recommendation ● Notification( called Popboard at pixiv) etc...
- 5. My works@private ■Software ● neoagent ○ A Yet Another Memcached Protocol Proxy Server ● dtl ○ diff template library with C++ ● ngx_small_light ○ Dynamic Transformation Module for Nginx ■Writing ● Software Design 2009 Sep ○ どのようにして差分を導き出すのか～diffの動作原理を知る～ ○ http://gihyo.jp/dev/column/01/prog/2011/diff_sd200906 And More -> http://cccis.jp
- 6. Agenda ● Following mechanismat pixiv ○ Image Upload ○ Thumbnail Generation
- 7. Scale of pixivaccording to thumbnails ● pixiv has about 30,000,000 illusts and comics ● Each illust has about 12 ～ too many thumbnails ● 20,000 illusts and comics are uploaded every day ● Total volume is about 30TB
- 8.
- 9. Image Upload Detail1 ● Generate too many thumbnails ○ 12 ～ too many thumbnails ● Save a original image and thumbnails to storage ○ not NFS ○ with in-house WebDAV Client(ImageClient)
- 10.
- 11.
- 12.
- 13.
- 14. ImageClient ● Enable WebDAVoperations to multiple servers transparently ○ put image ○ delete image ○ move image ○ make directory PUT ImageClient ○ move directory DELETE MOVE PUT DELETE ○ delete directory MKCOL MOVE MKCOL Image Image Storage ・・・・・ Storage
- 15. Image Upload Detail2 ● A image is uploaded semi-asynchronously ○ Generating thumbnails takes a long time
- 16. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View
- 17. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View
- 18. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View create lock file
- 19. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View create lock file
- 20. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View create lock file poll until lock file is deleted
- 21. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View create lock file poll until lock file is deleted Server Side Action
- 22. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View create lock file poll until lock file is deleted Server Side Action prefork server
- 23. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View create lock file poll until lock file is deleted Server Side Action prefork prefork server upload worker upload worker ・ ・ ・ ・ ・ upload worker
- 24. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View create lock file poll until lock file is deleted Server Side Action prefork prefork server upload worker Genrate Thumbnails upload worker Genrate Thumbnails ・ ・ ・ ・ ・ upload worker Genrate Thumbnails
- 25. semin-asynchronous upload mechanism User Side Action File Selection View Input Information View Completed View create lock file poll until lock file is deleted Server Side Action prefork prefork server upload worker Genrate Thumbnails delete lock file upload worker Genrate Thumbnails delete lock file ・ ・ ・ ・ ・ upload worker Genrate Thumbnails delete lock file
- 26. Inside Image Upload ●User Side Action ○ Apache ○ PHP ■ ImageClient ● Server Side Action ○ daemontools ○ Python ■ python-prefork, python-q4m, python-worker ● python-q4m and python-worker are in-house libraries ○ Q4M ■ Used As Upload Job Queue
- 27.
- 28. Two types ofthumbnails at pixiv ● Static Thumbnail ● Dynamic Thumbnail
- 29.
- 30. Static Thumbnail Generatedon upload
- 31. Detail of generatingthumbnails ● pixiv uses ImageMagick ○ ImageMagick is not fast ○ But quality of generated image is good ○ Quality is more important than speed for us ○ Of course, optimization is important, too
- 32.
- 33.
- 34. libjpeg-turbo ● Optimized libjpegfor x86 and x86_64 ● Replace libjpeg simply by LD_LIBRARY_PATH ● ImageMagick uses libjpeg
- 35. benchmark of libjpegand libjpeg-turbo processing JPEG-image with libjpeg and libjpeg-turbo ■libjpeg ■libjpeg-turbo
- 36. benchmark of libjpegand libjpeg-turbo processing JPEG-image with libjpeg and libjpeg-turbo ■libjpeg ■libjpeg-turbo libjpeg-turbo is faster than libjpeg by 10% on x86_64
- 37.
- 38. Disable OpenMP ● LatestImageMagick is OpenMP enabled at default ○ This is very slow in multi-process environment ● How to disable OpenMP ○ Re-compile with '--disable-openmp' ○ OMP_NUM_THREADS=1 ● pixiv takes the latter ○ Re-compiling and Re-packaging are complicated
- 39. OMP_NUM_THREADS=1 CPU Usage ofApplication Server
- 40. OMP_NUM_THREADS=1 CPU usage in peak dropped by 150～200% CPU Usage of Application Server
- 41. OMP_NUM_THREADS=1 Load Average ofApplication Server
- 42. OMP_NUM_THREADS=1 Load Average became dust Load Average of Application Server
- 43.
- 44. Dynamic Thumbnail Generated on demand
- 45. Why is dynamicthumbnail needed? ● Static thumbnail consumes disk space ○ Dynamic thumbnail does not consume disk space ● Preparing new static thumbnails takes a long time ○ Dynamic thumbnail is ready in a second！
- 46. Other Companies' Cases ●Cookpad ○ mod_tofu ○ http://www.slideshare.net/mirakui/ss-8150494 ● livedoor(Now, NHN Japan) ○ mod_small_light ○ http://www.slideshare.net/livedoor/smalllight2
- 47.
- 48.
- 49. mod_small_light-1.1.1 pixiv Edition5
- 50. mod_small_light-1.1.1 pixiv Edition5 heavily customized...
- 51.
- 52.
- 53. Resize image withmod_small_light GET /tank.jpg GET /small_light(dw=100,dh=100)/tank.jpg
- 54. Many Options q image quality(0～100) of output format(jpg,gif,png,tiff) e processing engine(imlib2,imagemagick) cc canvas color p pattern name with SmallLightPatternDefine etc Too many options
- 55.
- 56. Resize image withmod_small_light GET /small_light(p=small)/tank.jpg GET /tank.jpg or GET /small_light(dw=100,dh=100,e=imagemagick,jpeghint=y)/tank.jpg
- 57. Why not originalmod_small_light? ● Some thumbnails are special ○ comic cover ○ various cropping algorithms depending on aspect ratio ● Default output format is JPEG ○ It is good for us that input and output formats are the same
- 58. Why not originalmod_small_light？ ● No support for CMYK ○ pixiv must support CMYK ● Needed support for strange aspect ratio ○ da=l -> long-edge ○ da=s -> short-edge ○ da=p -> pixiv-edge (only pixiv Edition)
- 59. Special Thumbnail 1 ●comic cover
- 60. Special Thumbnail 2 ●special cropping algorithm
- 61. Special Thumbnail 2 ●special cropping algorithm
- 62. Special Thumbnail 2 ●special cropping algorithm crop center of image if image is landscape
- 63. Special Thumbnail 3 ●special cropping algorithm
- 64. Special Thumbnail 3 ●special cropping algorithm
- 65. Special Thumbnail 3 ●special cropping algorithm crop top of image if image is portrait
- 66. Many Extend Options rmprof remove image profiles crop_square crop image squarly cover add manga cover samec conform dw & dh & cw & ch extendl extend long-edge etc twenty new options in all
- 67. Summary ● Optimization ofimage processing is very important ○ Generating thumbnail takes a long time ○ Let's tune and desynchronize ● pixiv has two types of thumbnails ○ Static Thumbnail ○ Dynamic Thumbnail ● Dynamic Thumbnail is diskspace-saving and flexible ○ Big image storage is expensive ○ Easy to correspond to shift of application ■ New thumbnail types for new designs
- 68.