Opinosis: A Graph Based Approach to Abstractive Summarization of Highly Redundant Opinions<br />Kavita Ganesan, ChengXiang...
Opinion Summarization Today…<br />Opinion Summary for iPod<br />Existing methods: Generate structured ratings for an entit...
Opinion Summarization Today…<br />Opinion Summary for iPod<br />To know more: read many redundant sentences<br />structure...
Ideally, we need….Supporting textual summary!<br />
A good textual Opinion Summary should…<br /><ul><li>Summarize the major opinions
What are the major complaints/praise in an aspect?
Concise
Easily digestible
Viewable on smaller screens
Readable
Easily understood</li></li></ul><li>Important information summarized<br />Concise<br />Readable<br />“The iPhone’sbattery ...
How to generate such summaries?<br />
Widely studied for years[Radev et al.2000; Erkan & Radev, 2004; Mihalcea & Tarau, 2004…]<br /><ul><li>But, not suitable fo...
Bias:with limit on summary size
selected sentence may have missed critical info
Verbose:May contain irrelevant information
not suitable for smaller devices</li></ul>Extractive Summarization<br />
Widely studied for years[Radev et al.2000; Erkan & Radev, 2004; Mihalcea & Tarau, 2004…]<br /><ul><li>But, not suitable fo...
Bias:with limit on summary size
selected sentence may have missed critical info
Verbose:May contain irrelevant information
not suitable for smaller devices</li></ul>Extractive Summarization<br />Extractive<br />Abstractive<br />
Existing methods:<br />Some methods require manual effort [DeJong1982] [Radev and McKeown1998] [Finley and Harabagiu2002]<...
Impractical – high computational costs</li></ul>Abstractive Summarization - HARD!!<br />
<ul><li>`Shallow’ abstractive summarizer
Generates concise summaries using:
existing text
inherent redundancies
Uses minimal external knowledge
 lightweight</li></ul>Our Method: Opinosis<br />
Opinosis: High Level Overview<br />
Opinosis: High Level Overview<br />Set of sentences:<br /><ul><li>Topic specific (ex. battery life of ipod)
POS annotated</li></ul>Input <br />
Opinosis: High Level Overview<br />too<br />Set of sentences:<br /><ul><li>Topic specific (ex. battery life of ipod)
POS annotated</li></ul>phone <br />with<br />my <br />calls <br />frequently <br />drop <br />the <br />iphone<br />is <br...
Opinosis: High Level Overview<br />too<br />Set of sentences:<br /><ul><li>Topic specific (ex. battery life of ipod)
POS annotated</li></ul>phone <br />with<br />my <br />calls <br />frequently <br />drop <br />the <br />iphone<br />is <br...
Set of sentences:<br /><ul><li>Topic specific (ex. battery life of ipod)
POS annotated</li></ul>Opinosis: High Level Overview<br />too<br />phone <br />with<br />The iPhone is a great device, but...
Step 1: Building the Opinosis-Graph<br />
Assume:<br />2 sentences about “call quality of iphone”<br /><ul><li>Opinosis-Graph is empty</li></ul>Building Opinosis-Gr...
Building Opinosis-Graph<br />My phone calls drop frequently with the iPhone. <br />
Building Opinosis-Graph<br />My phone calls drop frequently with the iPhone. <br />my<br />my<br />unique(WORD + POS)  = n...
Building Opinosis-Graph<br />My phone calls drop frequently with the iPhone. <br />co-occurrence<br />my<br />phone<br />1...
Building Opinosis-Graph<br />My phone calls drop frequently with the iPhone. <br />my<br />phone<br />calls<br />1:1<br />...
Building Opinosis-Graph<br />My phone calls drop frequently with the iPhone. <br />my<br />phone<br />calls<br />drop<br /...
Building Opinosis-Graph<br />My phone calls drop frequently with the iPhone. <br />my<br />phone<br />calls<br />drop<br /...
Building Opinosis-Graph<br />My phone calls drop frequently with the iPhone. <br />my<br />phone<br />calls<br />drop<br /...
Building Opinosis-Graph<br />My phone calls drop frequently with the iPhone. <br />my<br />phone<br />calls<br />drop<br /...
Building Opinosis-Graph<br />My phone calls drop frequently with the iPhone. <br />my<br />phone<br />calls<br />drop<br /...
Building Opinosis-Graph<br />My phone calls drop frequently with the iPhone. <br />.<br />my<br />phone<br />calls<br />dr...
.<br />Building Opinosis-Graph<br />My phone calls drop frequently with the iPhone. <br />my<br />phone<br />calls<br />dr...
Building Opinosis-Graph<br />Great device, but the calls drop too frequently.<br />.<br />my<br />phone<br />calls<br />dr...
Building Opinosis-Graph<br />Great device, but the calls drop too frequently.<br />.<br />my<br />phone<br />calls<br />dr...
Building Opinosis-Graph<br />Great device, but the calls drop too frequently.<br />.<br />my<br />phone<br />calls<br />dr...
Building Opinosis-Graph<br />Great device, but the calls drop too frequently.<br />.<br />great<br />device<br />,<br />bu...
Building Opinosis-Graph<br />Great device, but the calls drop too frequently.<br />.<br />great<br />device<br />,<br />bu...
Building Opinosis-Graph<br />Great device, but the calls drop too frequently.<br />.<br />great<br />device<br />,<br />bu...
Building Opinosis-Graph<br />Great device, but the calls drop too frequently.<br />.<br />great<br />device<br />,<br />bu...
Building Opinosis-Graph<br />Great device, but the calls drop too frequently.<br />.<br />great<br />device<br />,<br />bu...
Building Opinosis-Graph<br />Great device, but the calls drop too frequently.<br />.<br />great<br />device<br />,<br />bu...
Graph is now ready for Step 2!<br />.<br />great<br />device<br />,<br />but<br />my<br />phone<br />drop<br />the<br />ip...
3 Important Properties of the Opinosis-Graph<br />
Property 1<br />Naturally captures redundancies<br />.<br />great<br />device<br />,<br />but<br />my<br />phone<br />drop...
Naturally captures redundancies<br />.<br />the<br />iphone<br />my<br />phone<br />great<br />device<br />,<br />but<br /...
Naturally captures redundancies<br />.<br />the<br />iphone<br />my<br />phone<br />great<br />device<br />,<br />but<br /...
Property 2<br />Captures gapped subsequences<br />1. My phone calls drop frequently with the iPhone. <br />2. Great device...
1. My phone calls drop frequently with the iPhone. <br />2. Great device, but the calls drop toofrequently.<br />Captures ...
Property 2<br />Captures gapped subsequences<br />.<br />great<br />device<br />,<br />but<br />the<br />iphone<br />my<br...
 discovery of new sentences</li></li></ul><li>Captures collapsible structures<br />1. Calls drop frequently with the iPhon...
Captures collapsible structures<br />1. Calls drop frequently with the iPhone<br />2. Calls drop frequently with the Black...
Step 2a: Generate Candidate Summaries<br />
Repeatedly search the Opinosis-Graph for a Valid Path<br />Generate Candidate Summaries<br />
Set of connected nodes<br />Has a Valid Start Node (VSN)<br />Natural starting point of a sentence<br />Opinosis uses avg....
Finding Candidate Summaries<br />drop<br />frequently<br />with<br />the<br />iphone<br />calls<br />,<br />    .<br />VSN...
Finding Candidate Summaries<br />drop<br />frequently<br />with<br />the<br />iphone<br />calls<br />,<br />    .<br />VSN...
Finding Candidate Summaries<br />drop<br />frequently<br />with<br />the<br />iphone<br />calls<br />,<br />    .<br />VSN...
Some paths are collapsible<br />Identify such paths through a collapsible node<br />Treat linking verbs (e.g. is, are)  as...
A Collapsible Structure<br />linking verb = collapsible node<br /> is<br />very<br />clear<br />the<br />screen <br />big<...
A Collapsible Structure<br /> is<br />very<br />screen <br />clear<br />the<br />big<br />collapse + merge<br />and<br />v...
CC after linking verbs: concatenate using commas<br />“The screen isvery clear,bright,big”<br />Better readability:<br />“...
Step 2b: Score Candidate Summaries<br />
Type 1:  High confidence summaries<br />Select candidateswithhigh redundancy<br /># of sentences sharing same path <br />c...
Gaps vary between sentences sharing nodes<br />Gap Threshold (σgap)<br />w2<br />w3<br />w1<br />Candidate X <br />Sentenc...
σgapenforces maximum allowed gap between two adjacent nodes<br />Gap Threshold (σgap)<br />w2<br />w1<br />Candidate X <br...
After candidate scoring:<br />Select top 2 scoring candidates <br />Most dissimilar candidates<br />Step 3: Final Opinosis...
Evaluation<br />
User Reviews:<br />Hotels: Tripadvisor.com <br />Products: Amazon.com<br />Cars: Edmunds.com<br />Data<br />
Summarization Task<br />ReviewsEdmunds Tripadvisor Amazon<br />Topic 51<br />sentence 1…..<br />sentence 2…..<br />sentenc...
Human composed summaries<br />Concise (<25 words)<br />Focus on summarizing major opinions<br />~4 human summaries per top...
Hard to find ‘general’ abstractive summarizer<br />Use MEAD - Extractive based method [Radev et al.2000]<br />Select 2 sen...
<ul><li>ROUGE(rouge-1, rouge-2, rouge-su4)</li></ul>Standard measure for summarization tasks<br />Readability Test<br />Me...
Results<br />
Estimate: How much one summary writer agrees with the rest<br />Human Performance<br />
Human Performance<br />Human summaries - semantically similar. Slight difference in word usage. <br />
Human vs. Opinosis vs. MEAD<br />
Human vs. Opinosis vs. MEAD<br />Highest recall<br />Lowest precision<br />Much longer sentences<br />ROUGE Recall<br />RO...
Upcoming SlideShare
Loading in …5
×

Opinosis Presentation @ Coling 2010: Opinosis - A Graph Based Approach to Abstractive Summarization of Highly Redundant Opinions

6,456 views
7,801 views

Published on

Presentation about generating textual summaries of opinions.

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
6,456
On SlideShare
0
From Embeds
0
Number of Embeds
4,788
Actions
Shares
0
Downloads
0
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide
  • Most work in opinion summarization focus on prediction the aspect based ratings for an entity.For example, for an ipod, you may predict that the appearance is 5 stars, ease of use is three stars..etc.
  • But the problem is, if you wanted to know more about each of these aspects, you would actually have to read many sentences, from thethousands of reviews, to have your questions answered.
  • For such textual summaries to be useful to users we first require that the summary actually summarizes the major opinions, is concise so that its viewable on smaller screens and of course should be reasonably readable.
  • For textual summaries to actually be useful to users we first require that the summary actually summarizes the major opinions, is concise so that its viewable on smaller screens and of course should be reasonably readable.
  • For such textual summaries to be useful to users we first require that the summary actually summarizes the major opinions, is concise so that its viewable on smaller screens and of course should be reasonably readable.
  • Miss out information during sentence selection
  • Miss out information during sentence selection
  • -Compared to extractive summarization, the study of abstractive summarization on the whole has been quite limited. -and existing methods have some shortcomings. First, some of the techniques require considerable amounts of manual effort to define templates and scripts that can later be filled with extracted information. Then, some methods rely very heavily on natural language understanding which can be computationally quite costly and the methods are often very domain specific.
  • Redundancies implies confidence
  • Redundancies implies confidence
  • Redundancies implies confidence
  • Redundancies implies confidence
  • Redundancies implies confidence
  • 3rd property is that the graph captures collapsible structures. These are 2 new sentences….The first part of the 2 sentences are common, the second part is different.This is a new subgraph for 2 sentences with a common structure. Calls drop freq….and Calls drop freq…..
  • Such structures are ideal for collapse and compression and such structures can be easily discovered using the OG
  • Such structures can be easily discovered using the Opinosis-Graph
  • Now, some paths may be collapsible. We need to identify such paths.
  • This structure can be potentially collapsed to form a merged structure such as this, where we have the anchor and the merged collapsed candidates. But the question is, how do we merge such structures?
  • So once we have the candidate summaries, we need to score the candidates, because not all are equally good.
  • First, since we want high confidence summaries, then we select candidates that have high level of redundancy. This redundancy depends on the number of sentences that share the same path and this count is controlled by a gap threshold which is explained in the next slide.Then, if we want summaries with good coverage, then ideally longer sentences would be better. So for this, we weigh the level of redundancy by the length ofThe candidate paths. So you will favor longer sentences that are reasonably redundant.
  • Each candidate summary is based on a set of sentences consisting of varying gaps between nodes. If we allow all gaps to be valid, then you risk generating ill formed sentences. So we restrict this, by introducing a gap threshold.
  • If a sentence is a member of two adjacent node, but does not satisfy the gap threshold, then this entry will not contribute to the redundancy counts that we will use for scoring. This avoids over-estimation of redundancy
  • The final step essentially involves choosing the candidates to be part of the summary.We choose the top 2 scoring candidates that are most dissimilar.
  • Our raw data is based on user reviews from Tripadvisor, amazon &amp; edmunds.
  • The input to the summarizer is a set of topic related sentences constructured from the raw reviews. Each of these is called a review document. And each review documentHas about 100 unordered sentences. The task is then to summarize each of these review documents.
  • Now we will look into the effects of gap setting.A small gap requires that the words in the graph are close together in the actual text. This graph shows the ROUGE-1 scores at different levels of gap threshold. A small gap requires that the position of words are close together in the actual text.
  • Now we will look into the effects of gap setting.A small gap requires that the words in the graph are close together in the actual text. This graph shows the ROUGE-1 scores at different levels of gap threshold. A small gap requires that the position of words are close together in the actual text.
  • Now we will look into the effects of gap setting.A small gap requires that the words in the graph are close together in the actual text. This graph shows the ROUGE-1 scores at different levels of gap threshold. A small gap requires that the position of words are close together in the actual text.
  • * So when the gap is one, the performance is lowest because strict adjacency does not allow redundancies to be easily discovered.
  • When the gap is slightly relaxed notice that there is a significant jump in performance. This is because we are able to discover more redundancies, and thus we are ableto generate higher confidence summaries.
  • As the gap increases, there is small amounts of improvement
  • When the gap is too large however, you risk generating ill formed sentences. So the best is to set the gap threshold between 2 and 5
  • This is the ROUGE-1 f-scores using different scoring functions. The orange line is for the scoring function that uses only redundancy, and the red and green line are for scoring functions that use both redundancy and path length
  • It is obvious that the scoring functions that use redundancy &amp; path length perform better. This is because, the summaries have better coverage and thus better performance.
  • For a given topic, we take the sentences from the opinosis summary and the corresponding human summaries to form a mixed set of sentences. From this set, we then ask a human assessor to pick 2 sentencesthat are least readable.
  • Opinosis Presentation @ Coling 2010: Opinosis - A Graph Based Approach to Abstractive Summarization of Highly Redundant Opinions

    1. 1. Opinosis: A Graph Based Approach to Abstractive Summarization of Highly Redundant Opinions<br />Kavita Ganesan, ChengXiang Zhai, Jiawei Han<br />University of Illinois @ Urbana Champaign<br />
    2. 2. Opinion Summarization Today…<br />Opinion Summary for iPod<br />Existing methods: Generate structured ratings for an entity[Lu et al., 2009; Lerman et al., 2009;..]<br />
    3. 3. Opinion Summarization Today…<br />Opinion Summary for iPod<br />To know more: read many redundant sentences<br />structured format useful, but not enough!<br />
    4. 4. Ideally, we need….Supporting textual summary!<br />
    5. 5. A good textual Opinion Summary should…<br /><ul><li>Summarize the major opinions
    6. 6. What are the major complaints/praise in an aspect?
    7. 7. Concise
    8. 8. Easily digestible
    9. 9. Viewable on smaller screens
    10. 10. Readable
    11. 11. Easily understood</li></li></ul><li>Important information summarized<br />Concise<br />Readable<br />“The iPhone’sbattery lasts long and is cheap but it’s bulky. <br />An Ideal Summary<br />
    12. 12. How to generate such summaries?<br />
    13. 13. Widely studied for years[Radev et al.2000; Erkan & Radev, 2004; Mihalcea & Tarau, 2004…]<br /><ul><li>But, not suitable for:</li></ul>Generating concise summaries <br />Summarizing highly redundant text<br /><ul><li>Problems
    14. 14. Bias:with limit on summary size
    15. 15. selected sentence may have missed critical info
    16. 16. Verbose:May contain irrelevant information
    17. 17. not suitable for smaller devices</li></ul>Extractive Summarization<br />
    18. 18. Widely studied for years[Radev et al.2000; Erkan & Radev, 2004; Mihalcea & Tarau, 2004…]<br /><ul><li>But, not suitable for:</li></ul>Generating concise summaries <br />Summarizing highly redundant text<br /><ul><li>Problems
    19. 19. Bias:with limit on summary size
    20. 20. selected sentence may have missed critical info
    21. 21. Verbose:May contain irrelevant information
    22. 22. not suitable for smaller devices</li></ul>Extractive Summarization<br />Extractive<br />Abstractive<br />
    23. 23. Existing methods:<br />Some methods require manual effort [DeJong1982] [Radev and McKeown1998] [Finley and Harabagiu2002]<br /><ul><li>Need to define templates to be filled</li></ul>Some methods rely heavily on NL understanding [Saggion and Lapalme2002] [Jing and McKeown2000]<br /><ul><li>Domain dependent
    24. 24. Impractical – high computational costs</li></ul>Abstractive Summarization - HARD!!<br />
    25. 25. <ul><li>`Shallow’ abstractive summarizer
    26. 26. Generates concise summaries using:
    27. 27. existing text
    28. 28. inherent redundancies
    29. 29. Uses minimal external knowledge
    30. 30. lightweight</li></ul>Our Method: Opinosis<br />
    31. 31. Opinosis: High Level Overview<br />
    32. 32. Opinosis: High Level Overview<br />Set of sentences:<br /><ul><li>Topic specific (ex. battery life of ipod)
    33. 33. POS annotated</li></ul>Input <br />
    34. 34. Opinosis: High Level Overview<br />too<br />Set of sentences:<br /><ul><li>Topic specific (ex. battery life of ipod)
    35. 35. POS annotated</li></ul>phone <br />with<br />my <br />calls <br />frequently <br />drop <br />the <br />iphone<br />is <br />a <br />Input <br />.<br />great <br />Step 1: Generate graph representation of<br />sentences (Opinosis-Graph)<br />device <br />
    36. 36. Opinosis: High Level Overview<br />too<br />Set of sentences:<br /><ul><li>Topic specific (ex. battery life of ipod)
    37. 37. POS annotated</li></ul>phone <br />with<br />my <br />calls <br />frequently <br />drop <br />the <br />iphone<br />is <br />a <br />Input <br />.<br />calls <br />frequently <br />drop <br />great <br />device <br />great <br />device <br />3.2<br />candidate sum1<br />2.5<br />candidate sum2<br />Step 2:Find promising paths (candidate summaries) & score these candidates<br />Step 1: Generate graph representation of<br />sentences (Opinosis-Graph)<br />
    38. 38. Set of sentences:<br /><ul><li>Topic specific (ex. battery life of ipod)
    39. 39. POS annotated</li></ul>Opinosis: High Level Overview<br />too<br />phone <br />with<br />The iPhone is a great device, but calls drop frequently. <br />my <br />calls <br />frequently <br />drop <br />the <br />iphone<br />is <br />a <br />Input <br />Step 3: Select top scoring candidates as final summary <br />.<br />calls <br />frequently <br />drop <br />great <br />device <br />3.2<br />candidate sum1<br />2.5<br />candidate sum2<br />great <br />Step 2:Find promising paths (candidate summaries) & score these candidates<br />Step 1: Generate graph representation of<br />sentences (Opinosis-Graph)<br />device <br />
    40. 40. Step 1: Building the Opinosis-Graph<br />
    41. 41. Assume:<br />2 sentences about “call quality of iphone”<br /><ul><li>Opinosis-Graph is empty</li></ul>Building Opinosis-Graph<br />My phone calls drop frequently with the iPhone.<br />Great device, but the calls drop too frequently.<br />
    42. 42. Building Opinosis-Graph<br />My phone calls drop frequently with the iPhone. <br />
    43. 43. Building Opinosis-Graph<br />My phone calls drop frequently with the iPhone. <br />my<br />my<br />unique(WORD + POS) = node <br />1:1<br />1:1<br />Positional Reference Information<br />SID<br />PID<br />
    44. 44. Building Opinosis-Graph<br />My phone calls drop frequently with the iPhone. <br />co-occurrence<br />my<br />phone<br />1:1<br />1:2<br />Positional Reference Information<br />SID<br />PID<br />
    45. 45. Building Opinosis-Graph<br />My phone calls drop frequently with the iPhone. <br />my<br />phone<br />calls<br />1:1<br />1:2<br />1:3<br />
    46. 46. Building Opinosis-Graph<br />My phone calls drop frequently with the iPhone. <br />my<br />phone<br />calls<br />drop<br />1:1<br />1:2<br />1:3<br />1:4<br />
    47. 47. Building Opinosis-Graph<br />My phone calls drop frequently with the iPhone. <br />my<br />phone<br />calls<br />drop<br />frequently<br />1:1<br />1:2<br />1:3<br />1:4<br />1:5<br />
    48. 48. Building Opinosis-Graph<br />My phone calls drop frequently with the iPhone. <br />my<br />phone<br />calls<br />drop<br />frequently<br />with<br />1:1<br />1:2<br />1:3<br />1:4<br />1:5<br />1:6<br />
    49. 49. Building Opinosis-Graph<br />My phone calls drop frequently with the iPhone. <br />my<br />phone<br />calls<br />drop<br />the<br />frequently<br />with<br />1:1<br />1:2<br />1:3<br />1:4<br />1:7<br />1:5<br />1:6<br />
    50. 50. Building Opinosis-Graph<br />My phone calls drop frequently with the iPhone. <br />my<br />phone<br />calls<br />drop<br />the<br />iphone<br />frequently<br />with<br />1:1<br />1:2<br />1:3<br />1:4<br />1:7<br />1:8<br />1:5<br />1:6<br />
    51. 51. Building Opinosis-Graph<br />My phone calls drop frequently with the iPhone. <br />.<br />my<br />phone<br />calls<br />drop<br />the<br />iphone<br />frequently<br />with<br />1:9<br />1:1<br />1:2<br />1:3<br />1:4<br />1:7<br />1:8<br />1:5<br />1:6<br />
    52. 52. .<br />Building Opinosis-Graph<br />My phone calls drop frequently with the iPhone. <br />my<br />phone<br />calls<br />drop<br />the<br />iphone<br />frequently<br />with<br />1:9<br />1:1<br />1:2<br />1:3<br />1:4<br />1:7<br />1:8<br />1:5<br />1:6<br />
    53. 53. Building Opinosis-Graph<br />Great device, but the calls drop too frequently.<br />.<br />my<br />phone<br />calls<br />drop<br />the<br />iphone<br />frequently<br />with<br />1:9<br />1:1<br />1:2<br />1:3<br />1:4<br />1:7<br />1:8<br />1:5<br />1:6<br />
    54. 54. Building Opinosis-Graph<br />Great device, but the calls drop too frequently.<br />.<br />my<br />phone<br />calls<br />drop<br />the<br />iphone<br />great<br />device<br />,<br />but<br />frequently<br />with<br />2:1<br />2:2<br />2:3<br />2:4<br />1:9<br />1:1<br />1:2<br />1:3<br />1:4<br />1:7<br />1:8<br />1:5<br />1:6<br />
    55. 55. Building Opinosis-Graph<br />Great device, but the calls drop too frequently.<br />.<br />my<br />phone<br />calls<br />drop<br />iphone<br />great<br />device<br />,<br />but<br />the<br />frequently<br />with<br />2:1<br />2:2<br />2:3<br />2:4<br />1:9<br />1:1<br />1:2<br />1:3<br />1:4<br />1:8<br />1:7<br />1:5<br />1:6<br />
    56. 56. Building Opinosis-Graph<br />Great device, but the calls drop too frequently.<br />.<br />great<br />device<br />,<br />but<br />my<br />phone<br />calls<br />drop<br />the<br />iphone<br />frequently<br />with<br />2:1<br />2:2<br />2:3<br />2:4<br />1:7,2:5<br />1:9<br />1:1<br />1:2<br />1:3<br />1:4<br />1:8<br />1:5<br />1:6<br />
    57. 57. Building Opinosis-Graph<br />Great device, but the calls drop too frequently.<br />.<br />great<br />device<br />,<br />but<br />my<br />phone<br />drop<br />the<br />iphone<br />calls<br />frequently<br />with<br />2:1<br />2:2<br />2:3<br />2:4<br />1:7,2:5<br />1:9<br />1:1<br />1:2<br />1:4<br />1:8<br />1:3,2:6<br />1:6<br />1:5<br />
    58. 58. Building Opinosis-Graph<br />Great device, but the calls drop too frequently.<br />.<br />great<br />device<br />,<br />but<br />my<br />phone<br />drop<br />the<br />iphone<br />calls<br />frequently<br />with<br />2:1<br />2:2<br />2:3<br />2:4<br />1:7,2:5<br />1:4, 2:7<br />1:9<br />1:1<br />1:2<br />1:8<br />1:3, 2:6<br />1:6<br />1:5<br />
    59. 59. Building Opinosis-Graph<br />Great device, but the calls drop too frequently.<br />.<br />great<br />device<br />,<br />but<br />my<br />phone<br />drop<br />the<br />iphone<br />calls<br />too<br />frequently<br />with<br />2:1<br />2:2<br />2:3<br />2:4<br />1:7,2:5<br />2:8<br />1:4, 2:7<br />1:9<br />1:1<br />1:2<br />1:8<br />1:3, 2:6<br />1:6<br />1:5<br />
    60. 60. Building Opinosis-Graph<br />Great device, but the calls drop too frequently.<br />.<br />great<br />device<br />,<br />but<br />my<br />phone<br />drop<br />the<br />iphone<br />calls<br />too<br />frequently<br />with<br />2:1<br />2:2<br />2:3<br />2:4<br />1:7,2:5<br />2:8<br />1:4, 2:7<br />1:9<br />1:1<br />1:2<br />1:8<br />1:3, 2:6<br />1:5, 2:9<br />1:6<br />
    61. 61. Building Opinosis-Graph<br />Great device, but the calls drop too frequently.<br />.<br />great<br />device<br />,<br />but<br />my<br />phone<br />drop<br />the<br />iphone<br />calls<br />too<br />frequently<br />with<br />1:9, 2:10<br />2:1<br />2:2<br />2:3<br />2:4<br />1:7,2:5<br />2:8<br />1:4, 2:7<br />1:1<br />1:2<br />1:8<br />1:3, 2:6<br />1:5, 2:9 <br />1:6<br />
    62. 62. Graph is now ready for Step 2!<br />.<br />great<br />device<br />,<br />but<br />my<br />phone<br />drop<br />the<br />iphone<br />calls<br />too<br />frequently<br />with<br />1:9, 2:10<br />2:1<br />2:2<br />2:3<br />2:4<br />1:7,2:5<br />2:8<br />1:4, 2:7<br />1:1<br />1:2<br />1:8<br />1:3, 2:6<br />1:5, 2:9 <br />1:6<br />Building Opinosis-Graph<br />
    63. 63. 3 Important Properties of the Opinosis-Graph<br />
    64. 64. Property 1<br />Naturally captures redundancies<br />.<br />great<br />device<br />,<br />but<br />my<br />phone<br />drop<br />the<br />iphone<br />calls<br />too<br />frequently<br />with<br />1:9, 2:10<br />2:1<br />2:2<br />2:3<br />2:4<br />1:7,2:5<br />2:8<br />1:4, 2:7<br />1:1<br />1:2<br />1:8<br />1:3, 2:6<br />1:5, 2:9 <br />1:6<br />
    65. 65. Naturally captures redundancies<br />.<br />the<br />iphone<br />my<br />phone<br />great<br />device<br />,<br />but<br />drop<br />calls<br />too<br />with<br />frequently<br />1:9, 2:10<br />1:7, 2:5<br />Property 1<br />2:1<br />2:2<br />2:3<br />2:4<br />2:8<br />1:8<br />1:1<br />1:2<br />1:4, 2:7<br />1:3, 2:6<br />1:6<br />1:5, 2:9 <br />Path shared by 2 sentences naturally captured by nodes<br />
    66. 66. Naturally captures redundancies<br />.<br />the<br />iphone<br />my<br />phone<br />great<br />device<br />,<br />but<br />drop<br />calls<br />too<br />with<br />frequently<br />1:9, 2:10<br />1:7, 2:5<br />Property 1<br />2:1<br />2:2<br />2:3<br />2:4<br />2:8<br />1:8<br />1:1<br />1:2<br />1:4, 2:7<br />1:3, 2:6<br />1:6<br />1:5, 2:9 <br />Easily discover redundanciesfor high confidence summaries<br />
    67. 67. Property 2<br />Captures gapped subsequences<br />1. My phone calls drop frequently with the iPhone. <br />2. Great device, but the calls drop toofrequently.<br />
    68. 68. 1. My phone calls drop frequently with the iPhone. <br />2. Great device, but the calls drop toofrequently.<br />Captures gapped subsequences<br />.<br />the<br />iphone<br />my<br />phone<br />Property 2<br />great<br />device<br />,<br />but<br />drop<br />calls<br />too<br />with<br />frequently<br />2:8<br />1:4, 2:7<br />1:9, 2:10<br />1:7, 2:5<br />2:1<br />2:2<br />2:3<br />2:4<br />1:8<br />1:1<br />1:2<br />1:3, 2:6<br />1:5, 2:9 <br />1:6<br />Gap between words = 2<br />
    69. 69. Property 2<br />Captures gapped subsequences<br />.<br />great<br />device<br />,<br />but<br />the<br />iphone<br />my<br />phone<br />drop<br />calls<br />too<br />with<br />frequently<br />2:8<br />1:4, 2:7<br />1:9, 2:10<br />2:1<br />2:2<br />2:3<br />2:4<br />1:7, 2:5<br />1:8<br />1:1<br />1:2<br />1:3, 2:6<br />1:5, 2:9 <br />1:6<br />Gapped subsequences allow:<br /><ul><li> redundancy enforcements
    70. 70. discovery of new sentences</li></li></ul><li>Captures collapsible structures<br />1. Calls drop frequently with the iPhone<br />2. Calls drop frequently with the Black Berry<br />calls<br />drop<br />frequently<br />with<br />the<br />iphone<br /> black berry<br />Property 3<br />Calls drop frequently with the iPhone and Black Berry<br />
    71. 71. Captures collapsible structures<br />1. Calls drop frequently with the iPhone<br />2. Calls drop frequently with the Black Berry<br />calls<br />drop<br />frequently<br />with<br />the<br />iphone<br /> black berry<br />Property 3<br />- Can easily be discovered using OG<br />- Ideal for collapse & compression<br />
    72. 72. Step 2a: Generate Candidate Summaries<br />
    73. 73. Repeatedly search the Opinosis-Graph for a Valid Path<br />Generate Candidate Summaries<br />
    74. 74. Set of connected nodes<br />Has a Valid Start Node (VSN)<br />Natural starting point of a sentence<br />Opinosis uses avg. positional information<br />Has a Valid End Node (VEN)<br />Point that completes a sentence<br />Opinosis uses punctuations & conjunctions<br />Valid Path<br />
    75. 75. Finding Candidate Summaries<br />drop<br />frequently<br />with<br />the<br />iphone<br />calls<br />,<br /> .<br />VSN<br />VEN<br />
    76. 76. Finding Candidate Summaries<br />drop<br />frequently<br />with<br />the<br />iphone<br />calls<br />,<br /> .<br />VSN<br />VEN<br />Candidate summary<br />
    77. 77. Finding Candidate Summaries<br />drop<br />frequently<br />with<br />the<br />iphone<br />calls<br />,<br /> .<br />VSN<br />VEN<br />Pool of candidate summaries<br />
    78. 78. Some paths are collapsible<br />Identify such paths through a collapsible node<br />Treat linking verbs (e.g. is, are) as collapsible nodes<br />Linking verbs have hub-like properties<br />Commonly used in opinion text<br />Collapsible Structures<br />
    79. 79. A Collapsible Structure<br />linking verb = collapsible node<br /> is<br />very<br />clear<br />the<br />screen <br />big<br />anchor<br />- Common structure<br />-High redundancy path<br />collapsed candidates (CC)<br />- Subgraphs to be merged<br />
    80. 80. A Collapsible Structure<br /> is<br />very<br />screen <br />clear<br />the<br />big<br />collapse + merge<br />and<br />very<br />clear<br />big<br />is<br />screen<br />the<br />CC2<br />anchor<br />CC1<br />
    81. 81. CC after linking verbs: concatenate using commas<br />“The screen isvery clear,bright,big”<br />Better readability:<br />“The screen is very clear,brightandbig”<br />How to Merge Structures?<br />CC1<br />CC2<br />CC3<br />Find last connector using hints from OG<br />
    82. 82. Step 2b: Score Candidate Summaries<br />
    83. 83. Type 1: High confidence summaries<br />Select candidateswithhigh redundancy<br /># of sentences sharing same path <br />controlled by gap threshold, σgap<br />Type 2: +Good coverage<br />Select longer candidates<br />redundancy * length of candidate paths<br />Favor longer but redundant candidates<br />Scoring Options<br />
    84. 84. Gaps vary between sentences sharing nodes<br />Gap Threshold (σgap)<br />w2<br />w3<br />w1<br />Candidate X <br />Sentence 1 <br />1 <br />2 <br />Sentence 2 <br />4 <br />4 <br />Sentence X<br />m<br />n<br />gap<br />gap<br />
    85. 85. σgapenforces maximum allowed gap between two adjacent nodes<br />Gap Threshold (σgap)<br />w2<br />w1<br />Candidate X <br />σgap<br />Sentence 1 <br />1 <br />< <br />σgap<br />> <br />Sentence 2 <br />4 <br />Sentence n<br />m<br />-Lower risk of ill-formed sentences <br />-Avoids over-estimation of redundancy <br />gap<br />
    86. 86. After candidate scoring:<br />Select top 2 scoring candidates <br />Most dissimilar candidates<br />Step 3: Final Opinosis Summaries<br />
    87. 87. Evaluation<br />
    88. 88. User Reviews:<br />Hotels: Tripadvisor.com <br />Products: Amazon.com<br />Cars: Edmunds.com<br />Data<br />
    89. 89. Summarization Task<br />ReviewsEdmunds Tripadvisor Amazon<br />Topic 51<br />sentence 1…..<br />sentence 2…..<br />sentence 3…..<br />sentence 4…..<br />Topic 1<br />sentence 1…..<br />sentence 2…..<br />sentence 3…..<br />sentence 4…..<br />Topic 2<br />sentence 1…..<br />sentence 2…..<br />sentence 3…..<br />sentence 4…..<br />~100 <br />unordered, <br />topic-related,<br />sentences<br />review document 1<br />review document 51<br />review document 2<br />summarize<br />summarize<br />summarize<br />
    90. 90. Human composed summaries<br />Concise (<25 words)<br />Focus on summarizing major opinions<br />~4 human summaries per topic<br />Gold Standard<br />
    91. 91. Hard to find ‘general’ abstractive summarizer<br />Use MEAD - Extractive based method [Radev et al.2000]<br />Select 2 sentences as the summary<br />Baseline<br />
    92. 92. <ul><li>ROUGE(rouge-1, rouge-2, rouge-su4)</li></ul>Standard measure for summarization tasks<br />Readability Test<br />Measures: How different Opinosis summaries are compared to human composed summaries?<br />Evaluation Measures<br />
    93. 93. Results<br />
    94. 94. Estimate: How much one summary writer agrees with the rest<br />Human Performance<br />
    95. 95. Human Performance<br />Human summaries - semantically similar. Slight difference in word usage. <br />
    96. 96. Human vs. Opinosis vs. MEAD<br />
    97. 97. Human vs. Opinosis vs. MEAD<br />Highest recall<br />Lowest precision<br />Much longer sentences<br />ROUGE Recall<br />ROUGE Precision<br />
    98. 98. Human vs. Opinosis vs. MEAD<br />Highest recall<br />Lowest precision<br />Overall: Baseline does not do well in generating concise summaries. <br />Much longer sentences<br />ROUGE Precision<br />ROUGE Recall<br />ROUGE Precision<br />
    99. 99. Human vs. Opinosis<br />similar<br />similar<br />ROUGE Recall<br />ROUGE Precision<br />
    100. 100. Human vs. Opinosis<br />similar<br />similar<br />Performance of Opinosis is reasonable  similar to Human performance<br />ROUGE Precision<br />ROUGE Recall<br />
    101. 101. Effect of Gap Threshold (σgap)<br />
    102. 102. Effect of Gap Threshold (σgap)<br />σgap<br />
    103. 103. Effect of Gap Threshold (σgap)<br />small σgap words in summary close together in original text<br />σgap<br />
    104. 104. Effect of Gap Threshold (σgap)<br />Lowest performance<br />Strict adjacency – disallows redundancies to be captured<br />σgap<br />
    105. 105. Effect of Gap Threshold (σgap)<br />Jump in performance<br />More redundancies are captured<br />σgap<br />
    106. 106. Effect of Gap Threshold (σgap)<br />Small improvements afterwards<br />σgap<br />
    107. 107. Effect of Gap Threshold (σgap)<br />Small improvements afterwards<br />σgap<br />
    108. 108. Effect of Gap Threshold (σgap)<br />Small improvements afterwards<br />σgap too large: ill formed sentences<br />Set σgap to low value<br />σgap<br />
    109. 109. Compare: Scoring Functions <br />
    110. 110. Compare: Scoring Functions <br />redundancy & path length<br />only redundancy<br />σgap<br />
    111. 111. Compare: Scoring Functions <br />redundancy & path length<br />only redundancy<br />redundancy & path length  summaries with better coverage <br />σgap<br />
    112. 112. Readability Test<br />Topic X<br />Topic X<br />Mixed Sentences<br />sentence 1…..<br />sentence 3…..<br />sentence 2…..<br />sentence 4…..<br />sentence 8…..<br />sentence 6…..<br />sentence 7…..<br />sentence 5…..<br />………………………<br />………………………<br />Opinosis Generated<br />sentence 1…..<br />sentence 2…..<br />MIX<br />Human Composed 1<br />sentence 1…..<br />sentence 2…..<br />sentence Y..<br />Human Composed 4<br />sentence 1…..<br />sentence 2…..<br />sentence Z…..<br />Pick at most 2 least readable sentences<br />
    113. 113. Assessor often picks:<br />Opinosis sentences - Opinosis summaries have readability issues<br />Non-Opinosis sentences or makes no picks - Opinosis summaries similar to human summaries<br />Readability Test<br />
    114. 114. Assessor picked:<br />34/102Opinosisgeneratedsentences as least readable<br />Our Readability Test<br />
    115. 115. Assessor picked:<br />34/102Opinosisgeneratedsentences as least readable<br />Our Readability Test<br />> 60% of Opinosis sentences are not very differentfrom humancomposed sentences<br />
    116. 116. A framework for summarizing highly redundant opinions<br />Use graph representation to generate concise abstractive summaries<br /><ul><li>General & lightweight: Can be used on any corpus with high redundancies(Twitter comments, Blog comments, etc)</li></ul>Summary<br />
    117. 117. Dataset and Demo Software is available<br />http://timan.cs.uiuc.edu/downloads.html<br />
    118. 118. [Barzilay and Lee2003] Barzilay, Regina and Lillian Lee. 2003. Learning to paraphrase: an unsupervised approach using multiple-sequence alignment. In NAACL ’03: Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, pages 16–23, Morristown, NJ, USA. <br />[DeJong1982] DeJong, Gerald F. 1982. An overview of the FRUMP system. In Lehnert, Wendy G. and Martin H. Ringle, editors, Strategies for Natural Language Processing, pages 149–176. Lawrence Erlbaum, Hillsdale, NJ.<br />[Erkan and Radev2004] Erkan, G¨unes and Dragomir R. Radev. 2004. Lexrank: graph-based lexical centrality as salience in text summarization. J. Artif. Int. Res.,22(1):457–479.<br />[Finley and Harabagiu2002] Finley, SandaHarabagiu and Sanda M. Harabagiu. 2002. Generating single and multi-document summaries with gistexter. In Proceedings of the workshop on automatic summarization, pages 30–38. <br />[Hu and Liu2004] Hu, Minqing and Bing Liu. 2004. Mining and summarizing customer reviews. In KDD, pages 168–177. <br />[Jing and McKeown2000] Jing, Hongyan and Kathleen R. McKeown. 2000. Cut and paste based text summarization. In Proceedings of the 1st North American chapter of the Association for Computational Linguistics conference, pages 178–185, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.<br />[Lerman et al.2009] Lerman, Kevin, Sasha Blair-Goldensohn, and Ryan Mcdonald. 2009. Sentiment summarization: Evaluating and learning user preferences. In 12th Conference of the European Chapter of the Association for Computational Linguistics (EACL-09).<br />[Lin and Hovy2003] Lin, Chin-Yew and Eduard Hovy. 2003. Automatic evaluation of summaries using ngram co-occurrence statistics. In Proc. HLT-NAACL, page 8 pages. <br />[LIN2004a] LIN, Chin-Yew. 2004a. Looking for a few good metrics : Rouge and its evaluation. proc. of the 4th NTCIR Workshops, 2004.<br />[Lin2004b] Lin, Chin-Yew. 2004b. Rouge: a package for automatic evaluation of summaries. In Proceedings of theWorkshop on Text Summarization Branches Out (WAS 2004), Barcelona, Spain.<br />[Lu et al.2009] Lu, Yue, ChengXiang Zhai, and Neel Sundaresan. 2009. Rated aspect summarization of short comments. In 18th International World WideWeb Conference (WWW2009), April.<br />References<br />
    119. 119. [Mihalcea and Tarau2004] Mihalcea, R. and P. Tarau. 2004. TextRank: Bringing order into texts. In Proceedings of EMNLP-04and the 2004 Conference on Empirical Methods in Natural Language Processing, July. <br />[Pang and Lee2004] Pang, Bo and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the ACL, pages 271–278. <br />[Pang et al.2002] Pang, Bo, Lillian Lee, and ShivakumarVaithyanathan. 2002. Thumbs up? Sentiment classification using machine learning techniques. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 79–86.<br />[Radev and McKeown1998]Radev, DR and K. McKeown. 1998. Generating natural language summaries from multiple on-line sources. Computational Linguistics, 24(3):469–500. <br />[Radev et al.2000] Radev, Dragomir, Hongyan Jing, and MalgorzataBudzikowska. 2000. Centroid-based summarization of multiple documents: Sentence extraction, utility-based evaluation, and user studies. In ANLP/NAACL Workshop on Summarization, pages 21–29. <br />[Radev et al.2002] Radev, Dragomir R., Eduard Hovy, and Kathleen McKeown. 2002. Introduction to the special issue on summarization.<br />[Saggion and Lapalme2002] Saggion, Horacio and Guy Lapalme. 2002. Generating indicative-informative summaries with sumum. Computational Linguistics, 28(4):497–526.<br />[Snyder and Barzilay2007] Snyder, Benjamin and Regina Barzilay. 2007. Multiple aspect ranking using the good grief algorithm. In In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics (HLT-NAACL), pages 300–307.<br />[Titov and Mcdonald2008] Titov, Ivan and Ryan Mcdonald. 2008. A joint model of text and aspect ratings for sentiment summarization. In Proceedings of ACL-08: HLT, pages 308–316, Columbus, Ohio, June. Association for Computational Linguistics.<br />..References<br />
    120. 120. Thank You…<br />

    ×