Loading…

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

Like this presentation? Why not share!

HPPG - high performance photo gallery

on

  • 2,432 views

New version of presentation a lot of changes since last time.

New version of presentation a lot of changes since last time.

Statistics

Views

Total Views
2,432
Views on SlideShare
2,431
Embed Views
1

Actions

Likes
0
Downloads
8
Comments
0

1 Embed 1

http://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as OpenOffice

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

HPPG - high performance photo gallery HPPG - high performance photo gallery Presentation Transcript

  • Introducing high performance photo gallery
    • Remigijus Kiminas
                  • 2010-11-29
                  • v5
  • Who I am?
    • Author of
      • http://livehelperchat.com/
      • http://redmine.remdex.info my projects :)
    • Currently working
      • http://www.coralsolutions.com/
      • Freelancing and building open-source software in free time
  • Purpose of the presentation 1
    • Present some architecture decisions witch were applied building image gallery
  • What's new since last presentation
    • Mobile devices get support
    • Image gallery can be used as shopping CMS
      • Credit's based buying
      • Checkout using paypal service
    • Uncached pages get speed improvement by finding bug in paginator.
    • Official ngnix support
  • What's new since last presentation 2
    • Extensions
    • Kernel modules override
    • Kernel classes override
    • CSS compile
    • Most popular images in 24 hours
    • Photo approvement functionality
    • Image filtering by resolution
  • What's new since last presentation 3
    • Thumbnails recreation script
    • 100% duplicates management accuracy
    • More configurable system aspects as:
      • Max upload photo size
      • Max archive size
      • Max file queue size
    • Animated gif support
  • What's new since last presentation 4
    • Animated gif support
    • Completely fixed AJAX navigation usability, no more confusing of available images to left or to right.
    • Front end design remake, thanks to http://pauliusc.lt
    • HTML output compression
    • HTML 5 frontend changes, saves bandwidth
  • What's new since last presentation 5
    • Some performance improvement regarding users permissions settings
    • More things moved to Memcached service
  • What's new since last presentation 5 V4
    • Sort by relevance was introduced
    • AddQuery usage implementation in search
    • Refactored search page. One query less now.
    • Paginator updates
    • Sphinx wildcard support
    • Images without original deletion script
    • SEO enchancement related to resolution and user current page
  • What's new since last presentation 5 V5
    • Refactored captcha, it's now AJAX/javacript based, performs well, plus saves one request on image preview window
    • Image preview full window cache!!! cached windows is as fast as cached pagination around 5ms
    • Image counter from log file, avoid insert on each image preview window
  • What's new since last presentation 5 V5
    • Last rated functionality
    • Cache status window
    • Recently top rated, in 24 hours
    • APC support as cache engine.
    • HTML5, SWF, FLV files support
    • Search suggest feature
  • What's new since last presentation 5 V5
    • Mysql query hint for album pagination, mysql planner choosed wrong indexes
    • Smart selects in image preview window
    • Full multilanguage support including translatable module URL!!! none of my known gallery/cms has this featyre. E.x gallery/search (engish) or gallerie/recherche (french)
    • Full InnoDB support. Performs well as MyISAM. Top process is PHP not Mysql :)
  • Future works
    • Pagination sharding with index filter shard table. It should boost large sets of pagination around 100% > and keep constant speed with millions of photos.
    • http://remdex.info/Optimising-mysql-limit-performance-99a.html
    • Backend redesign
  • Issues with previous image gallery's I had
    • A lot of users = a lot of problems
      • No caching support
      • Unoptimized SQL query's
      • Resource hungry
      • No framework used (well, perhaps this is not a problem, but most of the time they just duplicate frameworks functionality, reinventing the wheel...)
      • No Etag based caching, bandwidth saver...
  • Requirements
    • Optimized SQL queries
    • Fulltext search engine
    • Etag based caching
    • SQL querys caching
    • Fullpage caching
    • Low resource requirements
  • Adopted software
    • APC – opcode cache for PHP
    • Sphinx – free open-source SQL full-text search engine (http://sphinxsearch.com/)
    • Memcached – free & open source, high-performance, distributed memory object caching system (http://memcached.org/)
    • eZ Components – an enterprise-ready, general-purpose PHP library of components used independently or together for PHP application development. (http://ez.no/ezcomponents)
    • JQuery – is a fast and concise JavaScript Library that simplifies HTML document traversing, event handling, animating, and Ajax interactions for rapid web development. (http://jquery.com/)
    • Lighttpd – lightweight open-source web server. (http://www.lighttpd.net/)
    • Mysql – database engine (http://www.mysql.com)
  • Adopted software
    • Ngnix - A HTTP and mail proxy server licensed under a 2-clause BSD-like license. (http://nginx.org/)
    • Fully working ngnix config provided. For eshop requirements and standard
  • Building process – core
    • Gallery core is based on eZ Components. Used components:
      • Authentication
      • Configuration
      • Database
      • Feed
      • ImageAnalysis
      • ImageConversion
      • PersistentObject
      • Translation
      • Cache
      • Url
      • UserInput
  • Fulltext search implementation
    • Why sphinx?
      • Very very fast :)
    • Used features of 9.9
      • SetSelect – this feature was introduced in 9.9 version and allowed to make fancy filtering.
      • Example in next slide
  • Image full mode problem with previous and next image
    • Search condition in literal. I need to find 2 previous images based on current image position including search keyword, sorting mode.
    • URL consists of
      • Current image ID (16679)
      • Keyword (haposai)
      • Sort mode (popular)
    • How do I find out what should I display in two first thumbnails (middle image is current our image)?
  • Solution
    • Use SetSelect query $cl->SetSelect ( "*, (hits > '.$Image->hits.' OR (hits = '.$Image->hits.' AND pid > '.$Image->pid.')) AS myfilter" ); $cl->SetFilter ( "myfilter", array(1) );
    • Things I do not know how to do till now. If sorting is based on relevance how to now previous two images.
    • I know now. But:
      • SetSelect does not work with @weight attributes in it.
      • Had to use two query's. SetFilter() works with @weight
      • AddQuery comes in help here for perfromance. Mutch more relevance images now.
  • Some search statistic
    • Each day around 190 K querys. It were more if search result page were not be cached :)
  • Mysql performance tweaking
    • Just optimise querys (EXPLAIN is you friend)
    • Not a single slow query
    • Some tips:
      • With large data sets use
      • SELECT * FROM `lh_gallery_images`
      • INNER JOIN ( SELECT pid FROM lh_gallery_images ORDER BY comtime DESC, pid DESC LIMIT 20 OFFSET 20 ) AS items
      • ON lh_gallery_images.pid = items.pid
      • This query is at least 5x times faster than normal select. Tested with (150 K records.)
      • See - http://www.mysqlperformanceblog.com
  • Supported HTTP servers
    • Lighttpd
    • Apache
    • Ngnix
      • With ngnix managed to produce around 1200 Q/S on cached page. It's 30% more than with Lighttpd.
  • Caching objects
    • Version caching
      • http://www.bestechvideos.com/2009/03/21/railslab-scaling-rails-episode-8-memcached
      • http://www.infoq.com/presentations/lutke-rockstar-memcaching
      • Version cache were used in
        • Album pages
        • Last uploaded
        • Last hits
        • Popular images and so on.
        • The most popular images in 24 hours
      • Then cache is cleared?
        • It's not, only version number is increased, and automatic cache self expire, because cache key does not exists.
  • Some code with version cache
    • Cache Key calculation in Album
    • $cache = CSCacheAPC::getMem(); $cacheKey = md5('version_'.$cache->getCacheVersion('album_'.(int)$Params['user_parameters']['album_id']).$mode.'album_view_url'.(int)$Params['user_parameters']['album_id'].'_page_'.$Params['user_parameters_unordered']['page']);
      • Includes:
        • Album version
        • $mode – sorting mode (Ex. Popular)
        • Page
      • this combination gives unique cache version for each page.
    • Same logic applies to all listing pages
  • Some benchmarks [root@ks310613 ~]# ab -n 500 -c 10 http://animeonly.org/Fantasy/Mix-16a.html This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0 Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Copyright 2006 The Apache Software Foundation, http://www.apache.org/ Benchmarking animeonly.org (be patient) Completed 100 requests Completed 200 requests Completed 300 requests Completed 400 requests Finished 500 requests Server Software: lighttpd Server Hostname: animeonly.org Server Port: 80 Document Path: /Fantasy/Mix-16a.html Document Length: 26883 bytes Concurrency Level: 10 Time taken for tests: 0.545137 seconds Complete requests: 500 Failed requests: 0 Write errors: 0 Total transferred: 13593092 bytes HTML transferred: 13441500 bytes Requests per second: 917.20 [#/sec] (mean) Time per request: 10.903 [ms] (mean) Time per request: 1.090 [ms] (mean, across all concurrent requests) Transfer rate: 24349.84 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.0 0 0 Processing: 5 10 2.9 9 23 Waiting: 4 9 3.1 9 23 Total: 5 10 2.9 9 23 Percentage of the requests served within a certain time (ms) 50% 9 66% 12 75% 13 80% 13 90% 13 95% 13 98% 20 99% 22 100% 23 (longest request)
  • Etag base caching
    • What is it?
      • An ETag (entity tag) is part of HTTP, the protocol for the World Wide Web. It is a response header that may be returned by an HTTP/1.1 compliant web server and is used to determine change in content at a given URL (http://en.wikipedia.org/wiki/HTTP_ETag)
  • How to use it? $ExpireTime = 3600; $currentKeyEtag = md5($cacheKey.'user_id_'.erLhcoreClassUser::instance()->getUserID());; header('Cache-Control: max-age=' . $ExpireTime); // must-revalidate header('Expires: '.gmdate('D, d M Y H:i:s', time()+$ExpireTime).' GMT'); header('ETag: ' . $currentKeyEtag); $iftag = isset($_SERVER['HTTP_IF_NONE_MATCH']) ? $_SERVER['HTTP_IF_NONE_MATCH'] == $currentKeyEtag : null; if ($iftag === true) { header (&quot;HTTP/1.0 304 Not Modified&quot;); header ('Content-Length: 0'); exit; }
    • $cacheKey – from previous example cache key
    • User ID is needed if user is logged in.
    • Can be used for custom pages, that do not change
    • Then image is uploaded or deleted, we just increase cache version and Etag is expired automatic also.
  • Some MRTG screen shots 1
    • Hits per hour
    • Mysql queries
  • Some MRTG screen shots 2
    • Memcached status
    • Traffic stats
  • Conclusions
    • Single server with sphinx, memcached, mysql, nginx handles per day around 180 K pageviews daily.
    • No performance issues at this time.
    • Gallery home page
    • http://code.google.com/p/hppg/
  • Thank you for your attention :)
    • Questions etc:
      • [email_address]