The document summarizes a research project that analyzed question and answer exchanges from three online Q&A sites to understand how question properties influence answer quantity, quality, and archive value. The researchers found that objective questions received more fulfilling answers and produced more archivable answers than subjective questions. However, subjective questions received many more total answers on average. The specific Q&A site a question was asked on was also found to be a significant predictor of answer fulfillment and archivability. The researchers concluded the sites exhibited different but successful ways to address varied audience goals based on question type.

1. ASKED AND ANSWERED:
On Qualities and Quantities of Answers in
Online Question and Answer Sites
John Logie, Joe Weinberg, F. Maxwell Harper, and Joseph A. Konstan
University of Minnesota — Twin Cities
@The Social Mobile Web Workshop/ICWSM 2011, Barcelona, SPAIN
GRANT: 0812148
“Understanding and Supporting Online Question-and-
Answer Sites”

2. ASKED AND ANSWERED:
On Qualities and Quantities of Answers in
Online Question and Answer Sites
?...!.... ?. . ! . . . .
?. . . . . ! . . ?.!.
?. ?. . ! . . ! . .
?. . ! . . ! . . . . . ?.!.....
John Logie, Joe Weinberg, F. Maxwell Harper, and Joseph A. Konstan
University of Minnesota — Twin Cities
@The Social Mobile Web Workshop/ICWSM 2011, Barcelona, SPAIN
GRANT: 0812148
“Understanding and Supporting Online Question-and-
Answer Sites”

3. PROJECT ADDRESSES SOCIAL Q&A SITES

4. PROJECT ADDRESSES SOCIAL Q&A SITES
. . . FOCUS IS ON Q&A EXCLUSIVELY (NOT SEARCH, SOCIAL NETWORK)

5. PROJECT GOALS

6. PROJECT GOALS
#1 ARE QUESTION PROPERTIES PREDICTIVE OF ANSWER
QUANTITY, QUALITY, OR ARCHIVE VALUE?

7. PROJECT GOALS
#1 ARE QUESTION PROPERTIES PREDICTIVE OF ANSWER
QUANTITY, QUALITY, OR ARCHIVE VALUE?
#2 DO BETTER QUESTIONS PRODUCE BETTER ANSWER
THREADS (AND WORSE, WORSE)?

8. PROJECT GOALS
#1 ARE QUESTION PROPERTIES PREDICTIVE OF ANSWER
QUANTITY, QUALITY, OR ARCHIVE VALUE?
#2 DO BETTER QUESTIONS PRODUCE BETTER ANSWER
THREADS (AND WORSE, WORSE)?
#3 TO WHAT EXTENT DO PROPERTIES OF Q&A SITES
INFLUENCE QUANTITY, QUALITY, OR ARCHIVE VALUE?

9. PROJECT BACKGROUND

10. PROJECT BACKGROUND
2 ? TYPES

11. PROJECT BACKGROUND
2 ? TYPES

12. PROJECT BACKGROUND MAGIC
ARISTOTLE
2 ? TYPES DUST!

13. PROJECT BACKGROUND
6 ? TYPES
ADVICE FACTUAL
IDENTIFICATION PRESCRIPTIVE
(DIS)APPROVAL QUALITY

14. PROJECT BACKGROUND
6 ? TYPES
ADVICE FACTUAL
IDENTIFICATION PRESCRIPTIVE
(DIS)APPROVAL QUALITY

15. PROJECT BACKGROUND
6 ? TYPES
COMPUTER SCIENCE
+
RHETORIC

16. PROJECT BACKGROUND
6 ? TYPES
ADVICE FACTUAL
IDENTIFICATION PRESCRIPTIVE
(DIS)APPROVAL QUALITY

17. PROJECT BACKGROUND
6 ? TYPES
Should I take the
TMB train to the
ADVICE BCN airport? Or FACTUAL
get a cab?
IDENTIFICATION PRESCRIPTIVE
(DIS)APPROVAL QUALITY

18. PROJECT BACKGROUND
6 ? TYPES
Should I take the Do Spanish
TMB train to the cash machines
ADVICE BCN airport? Or accept US FACTUAL
get a cab? debit cards?
IDENTIFICATION PRESCRIPTIVE
(DIS)APPROVAL QUALITY

19. PROJECT BACKGROUND
6 ? TYPES
Should I take the Do Spanish
TMB train to the cash machines
ADVICE BCN airport? Or accept US FACTUAL
get a cab? debit cards?
IDENTIFICATION PRESCRIPTIVE
(DIS)APPROVAL QUALITY
Can SPAIN
repeat as World
Cup champion?

20. PROJECT BACKGROUND
6 ? TYPES
Should I take the Do Spanish
TMB train to the cash machines
ADVICE BCN airport? Or accept US FACTUAL
get a cab? debit cards?
IDENTIFICATION PRESCRIPTIVE
(DIS)APPROVAL QUALITY How do I set my
iPhone to avoid
Can SPAIN expensive roaming
repeat as World charges while in
Cup champion? Europe?

21. PROJECT BACKGROUND
6 ? TYPES
ADVICE FACTUAL
IDENTIFICATION PRESCRIPTIVE
(DIS)APPROVAL QUALITY

22. PROJECT BACKGROUND
6 ? TYPES
ADVICE FACTUAL
IDENTIFICATION PRESCRIPTIVE
(DIS)APPROVAL QUALITY
What is
your
favorite x?

23. PROJECT BACKGROUND
6 ? TYPES
ADVICE FACTUAL
IDENTIFICATION PRESCRIPTIVE
(DIS)APPROVAL QUALITY
What is
What is the
your
best x?
favorite x?

24. PROJECT BACKGROUND
6 ? TYPES
ADVICE FACTUAL
IDENTIFICATION PRESCRIPTIVE
(DIS)APPROVAL QUALITY
What is
What is the
your
best x?
favorite x?

25. PROJECT BACKGROUND
6 ? TYPES
ADVICE FACTUAL
IDENTIFICATION PRESCRIPTIVE
(DIS)APPROVAL QUALITY
What is
What is the
your
best x?
favorite x?

26. PROJECT BACKGROUND
6 ? TYPES
SUBJECTIVE OBJECTIVE
ADVICE FACTUAL
IDENTIFICATION PRESCRIPTIVE
(DIS)APPROVAL QUALITY
What is
What is the
your
best x?
favorite x?

27. PROJECT BACKGROUND
6 ? TYPES
SUBJECTIVE OBJECTIVE
ADVICE FACTUAL
IDENTIFICATION PRESCRIPTIVE
(DIS)APPROVAL QUALITY
What is
What is the
your
best x?
favorite x?

28. PROJECT BACKGROUND
6 ? TYPES
SUBJECTIVE OBJECTIVE
ADVICE FACTUAL
IDENTIFICATION PRESCRIPTIVE
(DIS)APPROVAL QUALITY
What is
What is the
your
best x?
favorite x?

29. PROJECT METHODS
Dataset of 300 Q&A exchanges from
3 interesting Q&A sites
100 each from . . .
=
=
=

30. PROJECT METHODS
Dataset of 300 Q&A exchanges from
3 interesting Q&A sites
100 each from . . .
= (BIG)
=
=

31. PROJECT METHODS
Dataset of 300 Q&A exchanges from
3 interesting Q&A sites
100 each from . . .
= (BIG)
= (SMALL)
=

32. PROJECT METHODS
Dataset of 300 Q&A exchanges from
3 interesting Q&A sites
100 each from . . .
= (BIG)
= (SMALL)
=

33. PROJECT METHODS
Dataset of 300 Q&A exchanges from
3 interesting Q&A sites
100 each from . . .
= (BIG)
= (SMALL)
= (ODD!)

34. PROJECT METHODS
• 2,419 answers hand-coded by co-authors.
(Average of 8 answers per question)
• Completed answer threads evaluated against
STRUCTURAL and CODED question properties
• (e.g. question length is STRUCTURAL; but whether
coders believe a question will generate answers
worthy of archiving is a CODED property)

35. PROJECT METHODS
• Question coding by 2 co-authors, 7 undergrads
Answer coding by 2 co-authors
• Coders were blinded to the sites via a coding tool
• In additional to coding individual answers, co-authors
also evaluated each question’s complete response thread
assessing:
1) Likelihood that the questioner received at least one fulfilling answer
2) Whether the thread had produced archival value
3) Whether the thread reinforced community bonds within the Q&A site

36. PROJECT METHODS

37. PROJECT METHODS
CODER
CORRELATION
0.74
0.70
0.38

38. PROJECT BACKGROUND (REFRESHER)
6 ? TYPES
ADVICE FACTUAL
IDENTIFICATION PRESCRIPTIVE
(DIS)APPROVAL QUALITY

39. PROJECT BACKGROUND (REFRESHER)
6 ? TYPES
SUBJECTIVE OBJECTIVE
ADVICE FACTUAL
IDENTIFICATION PRESCRIPTIVE
(DIS)APPROVAL QUALITY

40. PROJECT RESULTS
OBJECTIVE QUESTIONS RECEIVE MORE FULFILLING ANSWERS
Subjective Objective
5
4
3 3.277 3.541
2
1
0
Average Fulfillment of Questioner’s Goals or Needs

41. PROJECT RESULTS
OBJECTIVE QUESTIONS PRODUCE MORE “ARCHIVABLE” ANSWERS
Subjective Objective
5
4
3
3.085
2
1.789
1
0
Average “Archivability” of Answer Sets

42. PROJECT RESULTS
OBJECTIVE QUESTIONS PRODUCE MORE “ARCHIVABLE” ANSWERS
Subjective Objective
5
4
3
3.085
2
What is 1.789
your 1
favorite fish
to catch 0
and Average “Archivability” of Answer Sets

43. PROJECT RESULTS
OBJECTIVE QUESTIONS PRODUCE MORE “ARCHIVABLE” ANSWERS
Subjective Objective
5
4
What do I
need to take
3
3.085 great
2 snowboard
What is 1.789
your 1
favorite fish
to catch 0
and Average “Archivability” of Answer Sets

44. PROJECT RESULTS
...BUT SUBJECTIVE QUESTIONS RECEIVE MANY MORE ANSWERS
Subjective Objective
12
AVG = 11.23
9 133 QUESTIONS
1,491 ANSWERS
6
AVG = 6.31
147 QUESTIONS
3
928 ANSWERS
0
Average Number of Answers Per Question

45. PROJECT RESULTS
IN GENERAL, QUALITIES OF QUESTIONS ARE NOT ESPECIALLY
}
PREDICTIVE OF FULFILLING ANSWER SETS
SITE
QUESTION LENGTH
QUESTION CATEGORY (6? TYPES)
MULTIPLE QUESTIONS
R = 0.28
2
NEEDED REVISIONS
LEVEL OF PERSONALIZATION
ANTICIPATED ARCHIVE VALUE
ANTICIPATED LASTING VALUE
EMOTIONAL APPEALS

46. PROJECT RESULTS
THE SITE WHERE A QUESTION IS ASKED IS A SIGNIFICANT
PREDICTOR OF ASKER FULFILLMENT
YAHOO ANSWERS 2.98
ANSWERBAG 3.02
ASK METAFILTER 4.29
0 1.25 2.5 3.75 5
Likelihood that An Asker Receives a Fulfilling Set of Answers

47. PROJECT RESULTS
THE SITE WHERE A QUESTION IS ASKED IS ALSO A
SIGNIFICANT PREDICTOR OF ANSWER ARCHIVABILITY
YAHOO ANSWERS 2.05
ANSWERBAG 2.14
ASK METAFILTER 3.25
0 1 2 3 4
Likelihood that An Answer Set Has Achival Value

48. PROJECT REFLECTIONS
ASK METAFILTER’S PERFORMANCE IS SUPERFICIALLY
STRONGER DESPITE AN ARGUABLY TERRIBLE INTERFACE
VS. ?!?!?

49. PROJECT REFLECTIONS
ASK METAFILTER’S PERFORMANCE IS SUPERFICIALLY
STRONGER DESPITE AN ARGUABLY TERRIBLE INTERFACE
FREE
VS. ?!?!?

50. PROJECT REFLECTIONS
ASK METAFILTER’S PERFORMANCE IS SUPERFICIALLY
STRONGER DESPITE AN ARGUABLY TERRIBLE INTERFACE
FREE
VS. ?!?!?
FREE

51. PROJECT REFLECTIONS
ASK METAFILTER’S PERFORMANCE IS SUPERFICIALLY
STRONGER DESPITE AN ARGUABLY TERRIBLE INTERFACE
FREE
$5
VS. ?!?!?
FREE

52. PROJECT REFLECTIONS
. . . BUT OUR QUESTION DATA SUGGESTS A HIGH DEGREE OF
SELF-SELECTION BY Q&A USERS AT THE INPUT STAGE
Yahoo Answers Answerbag Ask Metafilter
50
40
30
20
10
0
ADVICE IDENTIFICATION (DIS)APPROVAL QUALITY PRESCRIPTIVE FACTUAL
(GRAPH BASED ON 257 QUESTIONS WITH CLEAR MAJORITY TYPE)

53. PROJECT REFLECTIONS
. . . BUT OUR QUESTION DATA SUGGESTS A HIGH DEGREE OF
SELF-SELECTION BY Q&A USERS AT THE INPUT STAGE
Yahoo Answers Answerbag Ask Metafilter
50
40
30
20
10
0
ADVICE IDENTIFICATION (DIS)APPROVAL QUALITY PRESCRIPTIVE FACTUAL
(GRAPH BASED ON 257 QUESTIONS WITH CLEAR MAJORITY TYPE)

54. PROJECT REFLECTIONS
. . . BUT OUR QUESTION DATA SUGGESTS A HIGH DEGREE OF
SELF-SELECTION BY Q&A USERS AT THE INPUT STAGE
Yahoo Answers Answerbag Ask Metafilter
50
40
30
20
10
0
ADVICE IDENTIFICATION (DIS)APPROVAL QUALITY PRESCRIPTIVE FACTUAL
(GRAPH BASED ON 257 QUESTIONS WITH CLEAR MAJORITY TYPE)

55. PROJECT REFLECTIONS
WE HAVE NO WAY OF KNOWING, BUT THE SAME USER MIGHT CHOOSE
ANSWERBAG FOR ONE TYPE OF QUESTION AND ASK METAFILTER FOR
ANOTHER . . .AND THIS WOULD BE REASONABLE GIVEN THE SITES’ EMPHASES
88 questions 88 questions 81 questions
21%
35%
48% 52%
65%
79%
STRONGLY OVERWHELMINGLY
BALANCED
SUBJECTIVE OBJECTIVE

56. PROJECT REFLECTIONS
AND GIVEN THE HIGH LEVELS OF OVERALL FULFILLMENT (AND HIGH TRAFFIC)
WE PERHAPS DO WELL TO THINK OF THESE SITES AS EXHIBITING DIFFERENT
WAYS TO SUCCESSFULLY ADDRESS VARIED AUDIENCE GOALS
Subjective Objective
5
4
3 3.277 3.541
2
1
0
Average Fulfillment of Questioner’s Goals or Needs

57. PROJECT REFLECTIONS
AND GIVEN THE HIGH LEVELS OF OVERALL FULFILLMENT (AND HIGH TRAFFIC)
WE PERHAPS DO WELL TO THINK OF THESE SITES AS EXHIBITING DIFFERENT
WAYS TO SUCCESSFULLY ADDRESS VARIED AUDIENCE GOALS
Subjective Objective
5
4
3 3.277 3.541
2
12 ANSWERS
FULFILLMENT 1
4/5
ARCHIVABILITY 0
Average Fulfillment of Questioner’s Goals or Needs
1/1
(ANSWERBAG)

58. PROJECT REFLECTIONS
AND GIVEN THE HIGH LEVELS OF OVERALL FULFILLMENT (AND HIGH TRAFFIC)
WE PERHAPS DO WELL TO THINK OF THESE SITES AS EXHIBITING DIFFERENT
WAYS TO SUCCESSFULLY ADDRESS VARIED AUDIENCE GOALS
Subjective Objective
5
4
3 3.277 3.541
2
13 ANSWERS
12 ANSWERS
1
FULFILLMENT
FULFILLMENT
5/5
4/5 ARCHIVABILITY
ARCHIVABILITY 0 4/5
Average Fulfillment of Questioner’s Goals or Needs
1/1 (METAFILTER)
(ANSWERBAG)

59. PROJECT REFLECTIONS
AND GIVEN THE HIGH LEVELS OF OVERALL FULFILLMENT (AND HIGH TRAFFIC)
WE PERHAPS DO WELL TO THINK OF THESE SITES AS EXHIBITING DIFFERENT
WAYS TO SUCCESSFULLY ADDRESS VARIED AUDIENCE GOALS
Subjective Objective
5
What is
your 4
favorite fish
to catch 3 3.277 3.541
and
2
13 ANSWERS
12 ANSWERS
1
FULFILLMENT
FULFILLMENT
5/5
4/5 ARCHIVABILITY
ARCHIVABILITY 0 4/5
Average Fulfillment of Questioner’s Goals or Needs
1/1 (METAFILTER)
(ANSWERBAG)

60. PROJECT REFLECTIONS
AND GIVEN THE HIGH LEVELS OF OVERALL FULFILLMENT (AND HIGH TRAFFIC)
WE PERHAPS DO WELL TO THINK OF THESE SITES AS EXHIBITING DIFFERENT
WAYS TO SUCCESSFULLY ADDRESS VARIED AUDIENCE GOALS
Subjective Objective
5
What is What do I
your 4
need to take
favorite fish great
to catch 3 3.277 3.541 snowboard
and
2
13 ANSWERS
12 ANSWERS
1
FULFILLMENT
FULFILLMENT
5/5
4/5 ARCHIVABILITY
ARCHIVABILITY 0 4/5
Average Fulfillment of Questioner’s Goals or Needs
1/1 (METAFILTER)
(ANSWERBAG)

61. CONCLUSION
Any . . .

62. CONCLUSION
?
Any . . . S ?