Searching Web Forums Talk given at Webdam workshop 9/30/13

1. Amélie Marian – Rutgers University09/30/2013
Searching Web Forums
Amélie Marian, Rutgers University
Joint work with Gayatree Ganu

2. Amélie Marian – Rutgers University09/30/2013
2
Forum Popularity and Search
• Forums with most traffic
[http://rankings.big-boards.com]
- BMW
- 50K uniq visitors/day
- 25M Posts
- 0.6M Members
- Filipino Community
- Subaru Impreza Owners
- Rome Total War
- …
- Pakistan Cricket Fan Site
- Prison Talk
- Online Money making
Despite popularity,
forums lack good
search capabilities

3. Amélie Marian – Rutgers University09/30/2013
3
Patient Emotion and stRucture Search
USer tool(PERSEUS) - Outline
Multi-Granularity Search
Challenges
- Unstructured text
- Background information omitted
- Discussion digression
Contributions
Return each results at varying focus
levels, allowing more or less
context. (CIKM 2013)
Egocentric Search
Challenges
- Multiple interpersonal relations
with varying importance
Contributions
Proposed a multidimensional user
similarity measure.
Use authorship for improving
personalized and keyword search.

4. Amélie Marian – Rutgers University09/30/2013
4
Hierarchical Model
• Hierarchy over objects at three searchable levels
– pertinent sentences, larger posts, entire discussions or threads
• Hierarchy
– captures strength of association, containment relationship
• Lower levels for
smaller objects
• Edge represents
containment
• Edge weight of 2
indicates that the text
in child was repeated
in the text of parent
Thread 1 Thread 2
Post 1 Post 2 Post 4Post 3
Sent 1 Sent 2 Sent 3 Sent 4 Sent 5 Sent 6
Dataset
Word 1 Word 2 Word 3 Word 4 Word 1
2
2
2

5. Amélie Marian – Rutgers University09/30/2013
5
Alternate Scoring Functions
Example Textual Results.
Query : hair loss
Top-4 Results
Post1: (A) Aromasin certainly caused my hair loss and the hair started falling 14 days after the
chemo. However, I bought myself a rather fashionable scarf to hide the baldness. I wear it everyday,
even at home. (B) Onc was shocked by my hair loss so I guess it is unusual on Aromasin. I had no
other side effects from Aromasin, no hot flashes, no stomach aches or muscle pains, no headaches or
nausea and none of the chemo brain.
Post2: (C) Probably everyone is sick of the hair loss questions, but I need help with this falling hair. I
had my first cemotherapy on 16th September, so due in one week for the 2nd treatment. (D) Surely
the hair loss can’t be starting this fast..or can it?. I was running my fingers at the nape of my neck
and about five came out in my fingers. Would love to hear from anyone else have AC done
(Doxorubicin and Cyclophosphamide) only as I am not due to have the 3rd drug (whatever that is - 12
weekly sessions) after the 4 sessions of AC. Doctor said that different people have different side
effects, so I wanted to know what you all went through. (E) Have n’t noticed hair loss elsewhere, just
the top hair and mainly at the back of my neck. (F) I thought the hair would start thining out
between 2nd and 3rd treatment, not weeks after the 1st one. I have very curly long ringlets past my
shoulders and am wondering if it would be better to just cut it short or completely shave it off. I am
willing to try anything to make this stop, does anyone have a good recommendation for a shampoo,
vitamins or supplements and (sadly) a good wig shop in downtown LA.
Post3: My suggestion is, don’t focus so much on organic. Things can be organic and very unhealthy. I
believe it when I read that nothing here is truly organic. They’re allowed a certain percentage. I think
5% of the food can not be organic and it still can carry the organic label. What you want is
nonprocessed, traditional foods. Food that comes from a farm or a farmer’s market. Small farmers are
not organic just because it is too much trouble to get the certification. Their produce is probably better
than most of the industrial organic stuff. (G) Sorry Jennifer, chemotherapy and treatment followed
by hair loss is extremely depressing and you cannot prepare enough for falling hair, especially hair
in clumps. (H) I am on femara and hair loss is non-stop, I had full head of thick hair.
tf*idf
Sent (E) (4.742)
Sent (A) (4.711)
Sent (C) (4.696)
Sent (G) (4.689)
BM25
Sent (D) (10.570)
Sent (B) (10.458)
Sent (H) (10.362)
Sent (E) (10.175)
HScore
Post2 (0.131)
Sent (G) (0.093)
Post1 (0.092)
Sent (H) (0.089)
Score tf*idf (t,d) = (1+log(tft,d)) * log(N/dft) * 1/CharLength

6. Amélie Marian – Rutgers University09/30/2013
6
Scoring Multi-Granularity Results
Goal: Unified scoring for objects at multiple granularity levels
– largely varying sizes
– with inherent containment relationship
Hierarchical Scoring Function (HScore)
Score for node i with respect to search term t and having j children:
… if i is a non-leaf node
= 1 … if i is a leaf node containing t
= 0 … if i is a leaf node not containing t
ewij = edge weight between parent i and child j
P(j) = number of parents of j
C(i) = number of children of i

7. Amélie Marian – Rutgers University09/30/2013
7
Effect of Size Weighting
Parameter  on HScore
• Parameter  controls the intermixing of granularities
0
2
4
6
8
10
12
14
16
18
20
0 0.1 0.2 0.3 0.4 0.5 BM25
Threads
Posts
Sentences
Size parameter 
Numberofresults
intop-20list
HScore

8. Amélie Marian – Rutgers University09/30/2013
8
Multi-Granularity Result Generation
Sorted Ordering:
Post3(2.5), Post1(2.1), Post2(2), Sent1(1.6), Sent2(1.5), Sent3(1.4), Sent4(1.3),
Sent6(0.4), Sent5(0.1), Post4(0.1), Thread1(0.1), Thread2(0.1)
For result size k=4, optimizing for the sum of scores:
• Overlap: {Post3, Post1, Post2, Sent1} Sum Score = 8.2 (minus 1.6?)
• Greedy: {Post3, Post1, Post2, Sent6} Sum Score = 7.0
• Best: {Post3, Post2, Sent1, Sent2} Sum Score = 7.6
33% sample queries had overlap amongst at least 3 of top-10 results
Thread 1 Thread 2
Post 1 Post 2 Post 4Post 3
Sent 1 Sent 2 Sent 3 Sent 4 Sent 5 Sent 6
0.1
2.1 2 2.5 0.1
0.1
0.1 0.41.6 1.5 1.4 1.3

9. Amélie Marian – Rutgers University09/30/2013
9
Multi-Granularity Result Generation
Goal: Generating a non-overlapping result set maximizing
“quality”
• Quality = Sum of scores of all results in the set
• Maximal independent set problem (NP Hard)
• Existing Algorithm: Lexicographic All Independent Sets (LAIS)
outputs maximal independent set with polynomial delay in
specific order

10. Amélie Marian – Rutgers University09/30/2013
10
Optimal Algorithm for k-set
(OAKS)
• Fix node ordering by decreasing scores
• Efficient OAKS Algorithm (typically k<<n):
– Start with k-sized first independent set, i.e., greedy
– Branch from nodes preceding kth node of the set, check if
maximal
– Find new k-sized maximal sets, save in priority queue
– Reject sets from priority queue where starting node occurs
after current best set’s kth node

11. Amélie Marian – Rutgers University09/30/2013
11
OAKS
Sorted Ordering:
Post3(2.5), Post1(2.1), Post2(2), Sent1(1.6), Sent2(1.5), Sent3(1.4), Sent4(1.3),
Sent6(0.4), Sent5(0.1), Post4(0.1), Thread1(0.1), Thread2(0.1)
For k=4, Greedy = {Post3, Post1, Post2, Sent6} SumScore=7.0
In the 1st iteration:
{Post3, Post2, Sent1, Sent2} SumScore = 7.6
{Post3 , Post1, Sent3, Sent4} SumScore = 7.3
Branches from nodes before Sent6,
i.e. Sent1, Sent2, Sent3, Sent4
Branch from Sent1, removing all adjacent to Sent1,  {Post3, Post2, Sent1}
Maximal on first 4 nodes? YES!
then complete to size k and insert in queue- {Post3, Post2, Sent1, Sent2}
Thread 1 Thread 2
Post 1 Post 2 Post 4Post 3
Sent 1 Sent 2 Sent 3 Sent 4 Sent 5 Sent 6
0.1
2.1 2 2.5 0.1
0.1
0.1 0.41.6 1.5 1.4 1.3

12. Amélie Marian – Rutgers University09/30/2013
12
Evaluating OAKS Algorithm
Comparing OAKS Runtime
Small overhead for practical k (=20)
• Scoring time = 0.96 sec
• OAKS Result set generation time = 0.09 sec
Word
Frequency
Sets Evaluated Run Time (sec)
LAIS OAKS LAIS OAKS
20-30 57.59 8.12 0.78 0.12
30-40 102.07 5.06 7.88 0.01
40-50 158.80 5.88 26.94 0.01
50-60 410.18 6.30 82.20 0.02
60-70 716.40 5.26 77.61 0.01
70-80 896.59 8.30 143.33 0.04
Comparing LAIS and OAKS
– 100 relatively infrequent queries
with corpus frequency in range
20-30, 30-40…
– OAKS is very efficient. Time
required by OAKS depends on k
OAKS improves over
Greedy SumScore in
31% queries @top20

13. Amélie Marian – Rutgers University09/30/2013
13
Dataset and Evaluation Setting
• Data collected from breastcancer.org
– 31K threads, 301K posts, 1.8M unique sentences, 46K keywords
• 18 Sample Queries
– e.g., broccoli, herceptin side effects, emotional meltdown, scarf or
wig, shampoo recommendation …
• Experimental Search Strategies – top20 results
- Mixed-Hierarchy : Optimal mixed granularity result.
- Posts-Hierarchy : Hierarchical scoring of posts only.
- Posts-tf*idf : Existing traditional search.
- Mixed-BM25

14. Amélie Marian – Rutgers University09/30/2013
14
Evaluating Perceived Relevance
Graded Relevance Scale
Exactly relevant answer,
Relevant but too broad,
Relevant but too narrow,
Partially relevant answer,
Not Relevant
Crowd Sourced Relevance
using Mechanical Turk
- Over 7 annotations
- Quality control -Honey pot
questions
- EM algorithm for consensus
Query = shampoo recommendation
 = 0.1  = 0.2  = 0.3  = 0.4
Rank = 1 Rel Broad Rel Broad Rel Broad Partial
2 Rel Broad Rel Broad Rel Broad Partial
3 Rel Broad Rel Broad Rel Broad Partial
4 Rel Broad Rel Broad Exactly Rel Rel Broad
5 Rel Broad Rel Broad Exactly Rel Partial
6 Exactly Rel Exactly Rel Rel Narrow Rel Narrow
7 Rel Broad Exactly Rel Rel Narrow Not Rel
8 Rel Broad Rel Broad Not Rel Partial
9 Rel Broad Rel Narrow Rel Broad Partial
10 Exactly Rel Rel Narrow Partial Rel Narrow
11 Rel Broad Rel Broad Exactly Rel Not Rel
12 Rel Broad Rel Broad Exactly Rel Not Rel
13 Rel Broad Exactly Rel Partial Not Rel
14 Not Rel Exactly Rel Rel Narrow Partial
15 Not Rel Exactly Rel Not Rel Rel Broad
16 Not Rel Rel Broad Rel Narrow Not Rel
17 Exactly Rel Rel Broad Exactly Rel Not Rel
18 Exactly Rel Exactly Rel Partial Partial
19 Not Rel Rel Broad Rel Narrow Not Rel
20 Not Rel Exactly Rel Partial Not Rel
Mixed-Hierarchy

15. Amélie Marian – Rutgers University09/30/2013
15
Evaluating Perceived Relevance
Mean Average Precision
Search
System
MAP
@
 =
0.1
 =
0.2
 =
0.3
 =
0.4
Mixed
Hierarchy
10 0.98 0.98 0.90 0.70
20 0.97 0.95 0.85 0.66
Posts-
Hierarchy
10 0.76 0.75 0.77 0.78
20 0.72 0.71 0.73 0.75
Posts-
tf*idf
10 0.76 0.73 0.76 0.76
20 0.74 0.72 0.72 0.73
Mixed
BM25
10 b=0.75 0.55
20 k=1.2 0.54
Clearly, Mixed-H
outperforms
post only methods
Users perceive higher
relevance of mixed
granularity results
0.00
5.00
10.00
15.00
20.00
25.00
30.00
35.00
α=0.1 α=0.2 α=0.3 α=0.4 b=0.75
k1=1.2
Discounted Cumulative Gain
@20
Mixed-hierarchy Posts-hierarchy
Posts-tf*idf Mixed-BM25

16. Amélie Marian – Rutgers University09/30/2013
16
EgoCentric Search
• Previous technique did not take the authorship of posts into
account
• Some forum participants are similar, sharing same topics of
interest or having the same needs, not necessarily at the
same time
– Rank similar author’s posts higher for personalized search
• Some forum participants are experts, prolific and
knowledgeable
– Expert opinions carry more weight in keyword search
• Author score to enhance personalized & keyword search

17. Amélie Marian – Rutgers University09/30/2013
17
Author Score
• Forum participants have several reasons to be linked
• Build a multidimensional heterogeneous graph over authors
incorporating many relations
• But, users assign different importance to different relations
auth 1
Topic 1
auth 2
auth n
Topic 2
Topic t
query 1
query 2
query n
W(a,t) W(q,t) author 1
author 2
author n
author 3
W(a1,a2)
User Profiles:
- Location
- Age
- Cancer stage
- Treatment
- …
-Co-participation
-Explicit References

18. Amélie Marian – Rutgers University09/30/2013
18
Contributions
Critical problem for leveraging authorship for search:
Incorporating multiple user relations with varying importance
learned egocentrically from user behavior
Outline:
• Author score computation using multidimensional graph
• Personalized predictions of user interactions: authors most
likely to provide answers
• Re-ranking results of keyword search using author expertise

19. Amélie Marian – Rutgers University09/30/2013
19
Multi-Dimensional Random
Walks (MRW)
• Random Walks (RW) for finding most influential users
– Pt+1 = M × Pt … till convergence
– M = α(A + D) + (1 − α)E … relation matrix A, D for dangling
nodes, uniform matrix E, α usually set to 0.85
• Rooted RW for node similarity
– Teleport back to root node with probability (1-α)
– Computes similarity of all nodes w.r.t root node
• Multidimensional RW– Heterogeneous Networks:
– Transition matrix computed as A = 1 * A1 +  2 * A2 + ... +  n * An
where i  i = 1 and all  i >= 0
– Egocentric weights -
For root node r :  i (r) = j ewAi (r, m)/ Ak j ewAk (r, j)
…  m  Ai and  j  Ak
a
b
c
2
3
A =
a b c
a 0 0 0
b 2 0 0
c 0 3 0
D =
a b c
a 0 0 0.33
b 0 0 0.33
c 0 0 0.33
E =
a b c
a .33 .33 .33
b .33 .33 .33
c .33 .33 .33

20. Amélie Marian – Rutgers University09/30/2013
20
Personalized Answer Search
• Link prediction by leveraging user similarities:
– Given participant behavior, find similar users to the user asking question
– Predict who will respond to this question
• Learn similarities from first 90% training threads
• Relations used:
– Topics covered in text, Co-participation in threads,
Signature profiles, Proximity of posts
• MRW similarity compared with baselines:
– Single relations
– PathSim:
• Existing approach for heterogeneous networks
• Predefined paths of fixed length
• No dynamic choice of path
Link prediction enables
suggesting which threads
or which users to follow

21. Amélie Marian – Rutgers University09/30/2013
21
Predicting User Interactions
0
0.1
0.2
0.3
0.4
0.5
10 20 30 40 50 60 70 80 90 100
MAP
Top-K similar participants
MAP for link prediction
Multidimensional RW
has best prediction
performance

22. Amélie Marian – Rutgers University09/30/2013
22
Predicting User Interactions
• Leverage content of the initial post to find users who are
experts on the question
– TopicScore computed as cosine similarity between author’s history and
initial post
– UserScore = β * MRWScore + (1- β) * TopicScore
Neighbors β = 0 β = 0.1 β = 0.2 β = 1
Top 5 0.52 0.64 (8%) 0.61 (4%) 0.59
Top 10 0.31 0.50 (8%) 0.49 (5%) 0.46
Top 15 0.24 0.43 (8%) 0.42 (6%) 0.40
Top 20 0.20 0.39 (6%) 0.39 (7%) 0.37
Purely MRW
Purely topical
expertise
% Improvement over purely MRW
MAP

23. Amélie Marian – Rutgers University09/30/2013
23
0.72
0.73
0.74
0.75
0.76
0.77
0.78
0.79
0.80
0.81
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
MAP@10
Tradeoff Parameter ω
IR Score λ=0.1
IR Score λ=0.2
Enhanced Keyword Search
• Non-rooted RW to find most influential expert users
• Re-rank top-k results of IR scoring using author scores
• Final score of post = ω*IR_score λ + (1- ω)*Authority_score
– Posts only, tf*idf scoring with size parameter 
Re-ranking search
results with author
score yields higher
MAP relevance
4% improvement
5%

24. Amélie Marian – Rutgers University09/30/2013
24
Patient Emotion and stRucture Search
USer tool(PERSEUS) - Conclusions
• Designed hierarchical model and score that allows generating
search results at several granularities of web forum objects.
• Proposed OAKS algorithm for best non-overlapping result.
• Conducted extensive user studies, show that mixed collection of
granularities yields better relevance than post-only results.
• Combined multiple relations linking users for computing similarities
• Enhanced search results using multidimensional author similarity
• Future Directions:
– Multi-granular search on web pages, blogs, emails. Dynamic focus level
selection.
– Search in and out of context over dialogue, interviews, Q&A.
– Optimal result set selection for targeted advertising, result diversification
– Time sensitive recommendations – Changing friendships, progressive
search needs.

25. Amélie Marian – Rutgers University09/30/2013
Thank you!