0
Introducing high performance image gallery <ul><li>Remigijus Kiminas </li><ul><ul><ul><ul><ul><ul><ul><li>May 1 </li></ul>...
Who I am? <ul><li>Author of </li><ul><li>http://livehelperchat.com/
http://redmine.remdex.info  my projects :) </li></ul><li>Currently working </li><ul><li>http://www.coralsolutions.com/
Freelancing and building open-source software in free time </li></ul></ul>
Purpose of the presentation <ul><li>Present some architecture decisions witch were applied building image gallery </li></ul>
Issues with previous image gallery's I had <ul><li>A lot of users = a lot of problems </li><ul><li>No caching support
Unoptimized SQL query's
Resource hungry
No framework used (well, perhaps this is not a problem, but most of the time they just duplicate frameworks functionality,...
No Etag based caching, bandwidth saver... </li></ul></ul>
Requirements <ul><li>Optimized SQL queries
Fulltext search engine
Etag based caching
SQL querys caching
Fullpage caching
Low resource requirements </li></ul>
Adopted software <ul><li>APC – opcode cache for PHP
Sphinx – free open-source SQL full-text search engine (http://sphinxsearch.com/)
Memcached – free & open source, high-performance, distributed memory object caching system  (http://memcached.org/)
eZ Components – an enterprise-ready, general-purpose PHP library of components used independently or together for PHP appl...
JQuery – is a fast and concise JavaScript Library that simplifies HTML document traversing, event handling, animating, and...
Upcoming SlideShare
Loading in...5
×

Hppg

892

Published on

Presenting high performance photo gallery

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
892
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Hppg"

  1. 1. Introducing high performance image gallery <ul><li>Remigijus Kiminas </li><ul><ul><ul><ul><ul><ul><ul><li>May 1 </li></ul></ul></ul></ul></ul></ul></ul></ul>
  2. 2. Who I am? <ul><li>Author of </li><ul><li>http://livehelperchat.com/
  3. 3. http://redmine.remdex.info my projects :) </li></ul><li>Currently working </li><ul><li>http://www.coralsolutions.com/
  4. 4. Freelancing and building open-source software in free time </li></ul></ul>
  5. 5. Purpose of the presentation <ul><li>Present some architecture decisions witch were applied building image gallery </li></ul>
  6. 6. Issues with previous image gallery's I had <ul><li>A lot of users = a lot of problems </li><ul><li>No caching support
  7. 7. Unoptimized SQL query's
  8. 8. Resource hungry
  9. 9. No framework used (well, perhaps this is not a problem, but most of the time they just duplicate frameworks functionality, reinventing the wheel...)
  10. 10. No Etag based caching, bandwidth saver... </li></ul></ul>
  11. 11. Requirements <ul><li>Optimized SQL queries
  12. 12. Fulltext search engine
  13. 13. Etag based caching
  14. 14. SQL querys caching
  15. 15. Fullpage caching
  16. 16. Low resource requirements </li></ul>
  17. 17. Adopted software <ul><li>APC – opcode cache for PHP
  18. 18. Sphinx – free open-source SQL full-text search engine (http://sphinxsearch.com/)
  19. 19. Memcached – free & open source, high-performance, distributed memory object caching system (http://memcached.org/)
  20. 20. eZ Components – an enterprise-ready, general-purpose PHP library of components used independently or together for PHP application development. (http://ez.no/ezcomponents)
  21. 21. JQuery – is a fast and concise JavaScript Library that simplifies HTML document traversing, event handling, animating, and Ajax interactions for rapid web development. (http://jquery.com/)
  22. 22. Lighttpd – lightweight open-source web server. (http://www.lighttpd.net/)
  23. 23. Mysql – database engine (http://www.mysql.com) </li></ul>
  24. 24. Building process – core <ul><li>Gallery core is based on eZ Components. Used components: </li><ul><li>Authentication
  25. 25. Configuration
  26. 26. Database
  27. 27. Feed
  28. 28. ImageAnalysis
  29. 29. ImageConversion
  30. 30. PersistentObject
  31. 31. Translation
  32. 32. Cache
  33. 33. Url
  34. 34. UserInput </li></ul></ul>
  35. 35. Fulltext search implementation <ul><li>Why sphinx? </li><ul><li>Very very fast :) </li></ul><li>Used features of 9.9 </li><ul><li>SetSelect – this feature was introduced in 9.9 version and allowed to make fancy filtering.
  36. 36. Example in next slide </li></ul></ul>
  37. 37. Image full mode problem with previous and next image <ul><li>Search condition in literal. I need to find 2 previous images based on current image position including search keyword, sorting mode.
  38. 38. URL consists of </li><ul><li>Current image ID (16679)
  39. 39. Keyword (haposai)
  40. 40. Sort mode (popular) </li></ul><li>How do I find out what should I display in two first thumbnails (middle image is current our image)? </li></ul>
  41. 41. Solution <ul><li>Use SetSelect query $cl->SetSelect ( &quot;*, (hits > '.$Image->hits.' OR (hits = '.$Image->hits.' AND pid > '.$Image->pid.')) AS myfilter&quot; ); $cl->SetFilter ( &quot;myfilter&quot;, array(1) );
  42. 42. Things I do not know how to do till now. If sorting is based on relevance how to now previous two images. </li><ul><li>Any suggestions? </li></ul></ul>
  43. 43. Some search statistic <ul><li>Each day around 190 K querys. It were more if search result page were not be cached :) </li></ul>
  44. 44. Mysql performance tweaking <ul><li>Just optimise querys (EXPLAIN is you friend)
  45. 45. Not a single slow query
  46. 46. Some tips: </li><ul><li>With large lists use
  47. 47. SELECT * FROM `lh_gallery_images`
  48. 48. INNER JOIN ( SELECT pid FROM lh_gallery_images ORDER BY comtime DESC, pid DESC LIMIT 20 OFFSET 20 ) AS items
  49. 49. ON lh_gallery_images.pid = items.pid
  50. 50. This query is at least 5x times faster than normal select. Tested with (150 K records.)
  51. 51. See - http://www.mysqlperformanceblog.com </li></ul></ul>
  52. 52. Caching objects <ul><li>Version caching </li><ul><li>http://www.bestechvideos.com/2009/03/21/railslab-scaling-rails-episode-8-memcached
  53. 53. http://www.infoq.com/presentations/lutke-rockstar-memcaching
  54. 54. Version cache were used in </li><ul><li>Album pages
  55. 55. Last uploaded
  56. 56. Last hits
  57. 57. Popular images and so on. </li></ul><li>Then cache is cleared? </li><ul><li>It's not, only version number is increased, and automaticly cache self expire, because cache key does not exists. </li></ul></ul></ul>
  58. 58. Some code with version cache <ul><li>Cache Key calculation in Album
  59. 59. $cache = CSCacheAPC::getMem(); $cacheKey = md5('version_'.$cache->getCacheVersion('album_'.(int)$Params['user_parameters']['album_id']).$mode.'album_view_url'.(int)$Params['user_parameters']['album_id'].'_page_'.$Params['user_parameters_unordered']['page']); </li><ul><li>Includes: </li><ul><li>Album version
  60. 60. $mode – sorting mode (Ex. Popular)
  61. 61. Page </li></ul><li>this combination gives unique cache version for each page. </li></ul><li>Same logic applies to all listing pages </li></ul>
  62. 62. Some benchmarks [root@ks310613 ~]# ab -n 500 -c 10 http://animeonly.org/Fantasy/Mix-16a.html This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0 Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Copyright 2006 The Apache Software Foundation, http://www.apache.org/ Benchmarking animeonly.org (be patient) Completed 100 requests Completed 200 requests Completed 300 requests Completed 400 requests Finished 500 requests Server Software: lighttpd Server Hostname: animeonly.org Server Port: 80 Document Path: /Fantasy/Mix-16a.html Document Length: 26883 bytes Concurrency Level: 10 Time taken for tests: 0.545137 seconds Complete requests: 500 Failed requests: 0 Write errors: 0 Total transferred: 13593092 bytes HTML transferred: 13441500 bytes Requests per second: 917.20 [#/sec] (mean) Time per request: 10.903 [ms] (mean) Time per request: 1.090 [ms] (mean, across all concurrent requests) Transfer rate: 24349.84 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.0 0 0 Processing: 5 10 2.9 9 23 Waiting: 4 9 3.1 9 23 Total: 5 10 2.9 9 23 Percentage of the requests served within a certain time (ms) 50% 9 66% 12 75% 13 80% 13 90% 13 95% 13 98% 20 99% 22 100% 23 (longest request)
  63. 63. Etag base caching <ul><li>What is it? </li><ul><li>An ETag (entity tag) is part of HTTP, the protocol for the World Wide Web. It is a response header that may be returned by an HTTP/1.1 compliant web server and is used to determine change in content at a given URL (http://en.wikipedia.org/wiki/HTTP_ETag) </li></ul></ul>
  64. 64. How to use it? $ExpireTime = 3600; $currentKeyEtag = md5($cacheKey.'user_id_'.erLhcoreClassUser::instance()->getUserID());; header('Cache-Control: max-age=' . $ExpireTime); // must-revalidate header('Expires: '.gmdate('D, d M Y H:i:s', time()+$ExpireTime).' GMT'); header('ETag: ' . $currentKeyEtag); $iftag = isset($_SERVER['HTTP_IF_NONE_MATCH']) ? $_SERVER['HTTP_IF_NONE_MATCH'] == $currentKeyEtag : null; if ($iftag === true) { header (&quot;HTTP/1.0 304 Not Modified&quot;); header ('Content-Length: 0'); exit; } <ul><li>$cacheKey – from previous example cache key
  65. 65. User ID is needed if user is logged in.
  66. 66. Can be used for custom pages, that do not change
  67. 67. Then image is uploaded or deleted, we just increase cache version and Etag is expired automatic also. </li></ul>
  68. 68. Some MRTG screen shots 1 <ul><li>Hits per hour
  69. 69. Mysql queries </li></ul>
  70. 70. Some MRTG screen shots 2 <ul><li>Memcached status
  71. 71. Traffic stats </li></ul>
  72. 72. Conclusions <ul><li>Single server with sphinx, memcached, mysql, lighttpd handles per day around 120 K pageviews.
  73. 73. No performance issues at this time.
  74. 74. Gallery home page
  75. 75. http://redmine.remdex.info/projects/hppg </li></ul>
  76. 76. Thank you for your attention :) <ul><li>Questions etc: </li><ul><li>[email_address] </li></ul></ul>
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×