This document discusses the findilike system, which ranks hotels and summarizes reviews based on user preferences. It describes two components that can be evaluated: opinion-based entity ranking and review summarization. For evaluation, it proposes interleaving results from different ranking algorithms and randomly mixing phrases from summarization algorithms to observe user clicks, with more clicks indicating a better algorithm. It also describes a mini test bed for local evaluation and submitting new algorithms for online performance testing.

1. In Situ Evaluation of Entity Ranking
and Opinion Summarization
using
www.findilike.com
Kavita Ganesan & ChengXiang Zhai
University of Illinois @ Urbana Champaign

2. What is findilike?
• Preference – driven search engine
– Currently works in hotels domain
– Finds & ranks hotels based on user preferences:
Structured: price, distance
Unstructured: “friendly service”, “clean”, “good views”
(Based on existing user reviews)  UNIQUE
• Beyond search: Support for analysis of hotels
– Opinion summaries
– Tag cloud visualization of reviews

3. …What is findilike?
• Developed as part of PhD. Work – new system
(Opinion-Driven Decision Support System, UIUC, 2013)
• Tracked ~1000 unique users from Jan - Aug ‘13
– Working on speed & reaching out to more users

4. Evaluating Review Summarization
Mini Test-bed
• Base code to extend
• Set of sample sentences
• Gold standard summary for those sentences
• ROUGE toolkit to evaluate the results
• Data set based on - Ganesan et. al 2010

5. Evaluating Entity Ranking
Mini Test-bed
• Base code to extend
• Terrier Index of hotel reviews
• Gold standard ranking of hotels
• Code to generate nDCG scores.
• Raw unindexed data set for reference

6. Building a new ranking model
Extend Weighting
Model

7. DEMO

8. 2 Components that can be evaluated
through natural user interaction
1
Ranking entities based on
unstructured user preferences
Opinion-Based Entity Ranking
(Ganesan & Zhai 2012)
Summarization of reviews
Generating short phrases
summarizing key opinions
(Ganesan et. al 2010, 2012)
2

9. Evaluation of entity ranking
• Retrieval
– Interleave results
Balanced
interleaving
(T. Joachims, 2002)
Base
DirichletLM
A click indicates preference…
Base

10. Snapshot of pairwise comparison
results for entity ranking
# Queries
B is better
Algorithms
DirichletLM,
Base, PL2
# Queries
A is Better
A B CA > CB
(A Better)
CB > CA
(B Better)
CA = CB > 0
(Tie)
CA = CB = 0 Total
DLM Base 30 35 2 5 72
PL2 Base 10 28 3 7 48
… … … … … … …

11. Snapshot of pairwise comparison
results for entity ranking
A B CA > CB
(A Better)
CB > CA
(B Better)
Base model
better, but DLM
not too far behind
Base model
better CA = CB & > 0
PL2 not
(Tie)
CA = CB = 0 Total
too good
DLM Base 30 35 2 5 72
PL2 Base 10 28 3 7 48
… … … … … … …

12. Evaluation of review summarization
Randomly mix top N
phrases from two
algorithms
ALGO1
ALGO2 Monitor click-through
More clicks on phrases from Algo1 vs. Algo2 
Algo1 better
on per
entity basis

13. How to submit a new algorithm?
Submit code
Performance
report
A B CA > CB
(A Better)
… … … … CB > CA
(B Better)
DLM Base 30 35
PL2 Base 10 28
Online Performance
Test on mini test
bed
Sample Code
Test Data & Gold
Standard
Evaluator
(nDCG, ROUGE)
Mini Testbed
Local performance
Write Java
based code
Extend
existing code
Implementation

14. More information about evaluation…
eval.findilike.com

15. Thanks! Questions?
Links
• Evaluation: http://eval.findilike.com
• System: http://hotels.findilike.com/
• Related Papers: kavita-ganesan.com

16. References
• Ganesan, K. A., C. X. Zhai, and E. Viegas, Micropinion Generation: An Unsupervised
Approach to Generating Ultra-Concise Summaries of Opinions, Proceedings of the
21st International Conference on World Wide Web 2012 (WWW '12), 2012.
• Ganesan, K. A., and C. X. Zhai, Opinion-Based Entity Ranking, Information Retrieval,
vol. 15, issue 2, 2012
• Ganesan, K. A., C. X. Zhai, and J. Han, Opinosis: A Graph Based Approach to
Abstractive Summarization of Highly Redundant Opinions, Proceedings of the 23rd
International Conference on Computational Linguistics (COLING '10), 2010.
• T. Joachims. Optimizing search engines using clickthrough data. In Proceedings of
the eighth ACM SIGKDD international conference on Knowledge discovery and
data mining, KDD ’02, NY, 2002.