Agenda● Introduction to Ranker● Performance Challenges In Ranker● Performance Tuning Strategies● Conclusion
Introduction To RankerRanker is a social site and platform that is in essence an operating system for Lists.Ranker makes it easy, fun, and social for users to Rank things – anything - via aNetflix-style drop-and-drag interface and a huge backend database. Everything in thesystem is an object, so that we can aggregate individual lists and answer the “wisdomof crowds” question “what is the best ___”.Ranker is fully distributable. So for example a travel blog can embed Ranker on theirsite that allows their users to easily rank their own favorite golf destinations. This givesthe blog a sticky interactive tool, as well as valuable content showcasing a continuallyupdated ranking of their community’s consensus picks for golf spots.The Ranker platform is flexible enough to be used for publishing, social networking,shopping, polling, even organization.
Performance Challenges In RankerThe Ranker application has to deal with two main performance issues:1. Data Volume - One of the biggest challenges while building Ranker, compared toother regular web applications is the volume of data that has to be managed. Rankerdeals with close to 10 million topics, most of which have been obtained from Freebase.Freebase uses a custom RDF store to persist and retrieve its data. However Rankerneeds to achieve the same performance levels using a relational database.2. Traffic - Ranker deals with a lot of social and entertaining content. This results intraffic spikes where its not uncommon to get a huge number of visitors in very shortspan of time. We have often seen around 40,000 visits to a single page within a spanof one hour, before subsiding back to a more reasonable traffic volume.
Performance Tuning StrategiesThe performance challenges explained in the previous slide arehandled using the following methods:● Caching● Database De-normalization● Hardware● Delayed Calculation/Aggregation● Event Based Post-Processing● Search Indexing
CachingRanker implements caching using the open source Ehcache framework. Caching isimplemented to various depths within the application. While some parts of theapplication only use caching to store backend objects that are time-consuming to load,other parts of the application cache the entire request by storing the generated HTML inthe cache.Caching in Ranker is also well integrated with the custom CMS that is used to configurevarious pages in the application. The CMS allows us to specify different cacheexpiration times for each block of each configured page in Ranker.
Database De-normalizationSince the Ranker traffic patterns indicate that a huge percentage of activity on the siteis for "reads" and a much smaller percentage for "writes", de-normalizing the databaseprovides huge performance benefits for the application.Database de-normalization often involves duplicating data across tables in order toavoid expensive joins in the SQL queries. Hence it involves a lot of overhead whileediting or deleting the de-normalized entities. A single user action might require theapplication to update multiple locations due to this technique. This is also very prone tocausing bugs in the system when the programmers are not aware of all the places thedata is duplicated in. Hence de-normalization has been used very cautiously and isused when none of the other approaches are applicable.
HardwareUsing better hardware is often a much simpler and cheaper option than investing a lotof time in improving the performance of some parts of the application. We have madesure that we have the most suitable hardware for the systems that are being built,based on the amount of memory and processing power needed.Ranker also uses hardware load balancers to distribute the load across multiple webservers. This makes a huge difference when there is a spike in traffic, as mentioned inthe “Performance Challenges In Ranker” section.Here is one situation where coding for better hardware made a huge difference to theperformance: One of the background processes in Ranker required to make around 9million queries to be able to complete its job. Later we realized that by loading all thedata into memory in one shot, we could reduce the number of queries to a fewthousand. However this would require us store around 3GB of data in memory. Hence itmade more sense to get systems with bigger memory capacity. This change resulted inthe performance increasing by about 20 times.
Delayed Calculation/AggregationRanker uses a large number of small batch programs that performcalculations using complex algorithms, on a regular interval. This allowsus to pre calculate scores for lists and items and hence avoid performingthe calculations every time data is retrieved or stored. The tricky part inusing this technique is to choose the right amount of pre-calculation ofdata. Too much pre-calculation will result in large number of results tostore, however too less of it can result in doing a lot of calculation whileloading the data.For example, this technique is used to calculate the most interesting listsin each domain in Ranker. The algorithm to identify the interesting listsuses the a lot of factors like number of views, number of votes, etc and isexecuted once a day. In this case, instead of determining the mostinteresting lists in each domain, we only determine a universal score foreach list. This score can be used to get the most interesting lists in anydomain.
Event Based Post-ProcessingThis is another form of Delayed Calculation. Some of the user actions inRanker will need the system to sometimes perform complex/time-consuming operations. For cases like these, Ranker uses an event basedpost-processing framework to perform these operations asynchronously.This will allow us to give the user a quick response time and also performtime-consuming operations within a few seconds after the action. The onlydisadvantage in using this approach is the difficulty it causes in reportingerrors and failures to the user.For example, when someone comments on a list, we need to notify the listauthor through an email. Even though the list author needs to be notifiedimmediately, having a delay of a few seconds is acceptable. Performingthis asynchronously will allow us to give a quick response to the userwithout having the user wait for the email to be sent.
Search IndexingRanker uses popular indexing tools like Lucene and Solr to index allsearchable data. Using a search index provides huge performance benefitswhile performing text based search in the application.Different strategies are used to add data into the index. Entities which arefrequently created / changed in the system are added to the index through anautomated SQL query, which runs every 5 minutes. Other entities, like thedata obtained from freebase is updated through a program that is triggeredmanually.Solr allows us to search across a number of fields and also do so usingdifferent weights for each type of field, without compromising on the speed ofthe search.
ConclusionMaking any changes in the application for improving the speed and performanceof the application always involve certain trade-offs. In Ranker, we have madesure that we only make changes once they are analyzed well and we are readyto handle all the side effects of the change. Changes often involve additionaleffort in maintenance and environment setup. Some of them even require us toacquire and maintain new servers, like in the case of search indexing andbackground processes.By choosing a variety of techniques to handle the different performanceproblems in the application, Ranker has been able to deliver and scale to thetraffic as it becomes more popular.