2. План
• Почему «ответы на вопросы»?
• «Традиционный» вопросно ответный поиск
«Традиционный» вопросно‐ответный поиск
• Ответы на вопросы в Вебе
• Социальный вопросно‐ответный поиск
• Заключение
18.12.2010 Павел Браславский 2
4. Запросы – вопросы: 2‐3%
Запросы вопросы: 2 3%
http://company.yandex.ru/facts/researches/ya_search_2009.xml
Павел Браславский ‐ Анализ запросов 4
5. Близкие области
Близкие области
• ЕЯ‐интерфейс к БД
• Диалоговые системы
Диалоговые системы
18.12.2010 Павел Браславский 5
6. Примеры вопросов
Примеры вопросов TREC
1. Who is the author of the book, "The Iron Lady: A Biography of
1 Wh i th th f th b k "Th I L d A Bi h f
Margaret Thatcher"?
2. What was the monetary value of the Nobel Peace Prize in 1989?
3. What does the Peugeot company manufacture?
4. How much did Mercury spend on advertising in 1993?
5. What is the name of the managing director of Apricot Computer?
5. What is the name of the managing director of Apricot Computer?
6. Why did David Koresh ask the FBI for a word processor?
7. What debts did Qintex group leave?
8. What is the name of the rare neurological disease with symptoms
8 Wh i h f h l i l di ih
such as: involuntary movements (tics), swearing, and incoherent
vocalizations (grunts, shouts, etc.)?
18.12.2010 Павел Браславский 6
10. Ключевые компоненты
Ключевые компоненты
• ИПС (индексирование документов,
р р фр
извлечение и ранжирование фрагментов) )
• NER
• Классификатор вопросов ( гипотезы
ф (
ответа)
• Синтаксический и семантический анализ
• М
Машина вывода
textual inference/entailment/reasoning
g
18.12.2010 Павел Браславский 10
11. Пример
• Сегодня в возрасте восьмидесяти лет в
Переделкино умер глава русской
р д у р ру
православной церкви Алексий Второй.
• Патриарх Алексий II скончался 5 декабря
Патриарх Алексий II скончался 5 декабря
2008 года.
18.12.2010 Павел Браславский 11
12. CLEF 2009
CLEF 2009
• ResPubliQA: 500 natural language questions,
bliQ 00 ll i
systems must return the passage, multilingual
collection of legislation documents.
ll ti f l i l ti d t
• QAST: written and oral questions (factual and
definitional) in different languages are formulated
against a set of audio recordings.
• GikiCLEF: open list questions over Wikipedia that
require geographic reasoning, complex
information extraction, and cross‐lingual
processing.
18.12.2010 Павел Браславский 12
13. GikiCLEF
• EX01: Name Portuguese‐speaking
EX01 Name Portuguese speaking Nobel prize winners
• EX02: List Portuguese Pop/Rock groups created in the 90s.
• EX03: Which Brazilian football players play in clubs in the Iberian
Pensinsula?
Pensinsula?
• EX04: What capitals of Dutch provinces received their town privileges
during the sixteenth century?
• EX05: In
EX05: In which places did Italo Calvino live during adulthood?
adulthood?
• EX06: Name Mexican poets who published volumes with ballads until
1930.
• EX07: Name
EX07: Name authors born in Alaska and who wrote fiction about it it.
• EX08: What Belgians won the Tour de France exactly twice?
• EX09: Find Amazon tribes which have no written language
• EX10: Find Northern E
EX10 Fi d N h Europe companies which produce nano‐electronic
i hi h d l i
components for planes.
Павел Браславский 13
14. РОМИП
• К
Коллекция BY.WEB
BYWEB
• 10K запросов‐вопросов из лога поисковой машины
– g
gta san andreas как сделать машину призрак?
д у р р
– монгольские полевки как ухаживать?
– берут ли с экземой в армию?
– перелёт до екатеринбурга от москвы сколько по времени?
– черезсколько дней появляются корни у отростка традесканции?
– всем ли девушкам важны деньги?
– как заполучить парня своей мечты?
– где пройдет финал кубка уефа
где пройдет финал кубка уефа 2009?
• До 5 ответов системы: docID, краткий ответ, фрагмент
(до 300 символов)
http://romip.ru/ru/2010/tracks/qa.html
18.12.2010 Павел Браславский 14
18. Примеры систем
Примеры систем
• AnswerBus, PowerSet и
( p g g p
LLC (http://www.languagecomputer.com/) )
не работают
• EasyAsk AnswerLogic AnswerFriend Start
EasyAsk, AnswerLogic, AnswerFriend, Start,
Quasm, Mulder, Webclopedia, ISI TextMap,
etc. [
[Manning]
]
18.12.2010 Павел Браславский 18
19. Wolfram Alpha
Wolfram Alpha
18.12.2010 Павел Браславский 19
22. Разные данные/подходы
Разные данные/подходы
• Поиск ответа по коллекции текстов
• Поиск ответа в структурированных данных
Поиск ответа в структурированных данных
• Поиск ответа в коллекции вопросов и
ответов
– FAQ
– Онлайн консультации
– Форумы сообщества
Форумы, сообщества
– Специализированные социальные сервисы
18.12.2010 Павел Браславский 22
23. Социальный поиск
Социальный поиск
1. Поиск с помощью сообщества
2. Поиск с учетом социальных
Поиск с учетом социальных
взаимодействий пользователей
3. Поиск по контенту, который является
3 П й
результатом социальных взаимодействий
18.12.2010 Павел Браславский 23
27. Проблемы/задачи
• Качество контента
– Информация/общение
ф р ц / щ
• Устранение избыточности (смысловые
дубликаты)
• Релевантность/ранжирование
• Классификация запросов
• З
Запрос ВВопрос
18.12.2010 Павел Браславский 27
28. Finding High Quality Content in SM
Finding High Quality Content in SM
E. Agichtein, C. Castillo, D. Donato, A. Gionis, and G.
E A i h i C C ill D D A Gi i dG
Mishne, Finding High Quality Content in Social
Media, in WSDM 2008
Media in WSDM 2008
• Well‐written
• Interesting As judged by
• Relevant (answer)
e e a (a s e ) professional editors
professional editors
• Factually correct
• Popular?
• Provocative?
• Useful?
18.12.2010 Павел Браславский 28
[Agichtein]
36. Link Analysis for Authority Estimation
Link Analysis for Authority Estimation
Answer 1 User 3 User 3
Question 1
Q ti 1
User 1
User 1 User 4
User 4
Answer 2
User 5
Question 2 Answer 3 User 5
User 2 User 2 User 6
Answer 4
Answer 4 User 6
User 6
Question 3
Answer 5
A( jAnswer 6∑ H (i )
)=
i = 0.. M
H (i ) = ∑ A( j )
j = 0.. K Hub (asker)
H b( k ) Authority (answerer)
A th it ( )
36
[Agichtein]
39. Top Features for Question Classification
Top Features for Question Classification
• Ak
Asker popularity (“stars”)
l it (“ t ”)
• Punctuation density
• Topical category
• Page views
• KL Divergence from reference corpus LM
39
[Agichtein]
42. Dimensions of Quality
Dimensions of “Quality”
• Well‐written
ll i
• Interesting
g
• Relevant (answer)
• Factually correct
Factually correct
• Popular?
• Timely?
As judged by the asker (or community)
• Provocative?
• Useful?
42
[Agichtein]
43. Yahoo! Answers: The Good News
Yahoo! Answers: The Good News
• Active community of millions of users in many
g g
countries and languages
• Eff i f
Effective for subjective i f
bj i information needs
i d
– Great forum for socialization/chat
• C b i l bl f h d t fi d i f
Can be invaluable for hard‐to‐find information
ti
not available on the web
43
[Agichtein]
45. Yahoo! Answers: The Bad News
Yahoo! Answers: The Bad News
May have to wait a long time to get a satisfactory answer
40
1. FIFA World Cup
1 FIFA World Cup
35
2. Optical
30 3. Poetry
3. Poetry
25 4. Football (American)
20 5. Soccer
15 6. Medicine
10 7. Winter Sports
5 8. Special Education
8 Special Education
0 9. General Health Care
1 2 3 4 5 6 7 8 9 10 10. Outdoor Recreation
10. Outdoor Recreation
Time to close a question (hours)
May never obtain a satisfying answer
May never obtain a satisfying answer
45
[Agichtein]
46. Y. Liu, J. Bian, and E. Agichtein, in SIGIR 2008
Y Liu J Bian and E Agichtein in SIGIR 2008
Predicting Asker Satisfaction
Given a question submitted by an asker in CQA,
predict whether the user will be satisfied with the
predict whether the user will be satisfied with the
answers contributed by the community.
– “Satisfied” :
• The asker has closed the question AND
The asker has closed the question AND
• Selected the best answer AND
• Rated best answer >= 3 “stars” (# not important)
Rated best answer >= 3 stars (# not important)
– Else, “Unsatisfied
46
[Agichtein]
47. ASP: Asker Satisfaction Prediction
ASP: Asker Satisfaction Prediction
Answerer History
Answerer History
Answer
Category Text
Asker History
Question
Wikipedia
Classifier
News
asker is asker is not
satisfied satisfied
47
[Agichtein]
48. Experimental Setup: Data
Experimental Setup: Data
Crawled from Yahoo! Answers in early 2008
Questions Answers Askers Categories % Satisfied
216,170 1,963,615 158,515 100 50.7%
“Anonymized” dataset available at:
http://ir.mathcs.emory.edu/shared/
http://ir mathcs emory edu/shared/
1/2009: Yahoo! Webscope : “Comprehensive”
/ h ! b “ h ”
Answers dataset: ~5M questions & answers.
48
[Agichtein]
49. Satisfaction by Topic
Satisfaction by Topic
Topic Questions Answers A per Q Satisfied Asker Time to close
rating by k
b asker
2006 FIFA 1194 35,659 329.86 55.4% 2.63 47
World Cup
W ld C minutes
i
Mental 151 1159 7.68 70.9% 4.30 1.5 days
Health
H lth
Mathematics 651 2329 3.58 44.5% 4.48 33
minutes
Diet & 450 2436 5.41 68.4% 4.30 1.5 days
Fitness
49
[Agichtein]
50. Satisfaction Prediction: Human Judges
Satisfaction Prediction: Human Judges
• T th k ’ ti
Truth: asker’s rating
• A random sample of 130 questions
• Researchers
– Agreement: 0.82 F1: 0.45 2P*R/(P+R)
• Amazon Mechanical Turk
Amazon Mechanical Turk
– Five workers per question.
– Agreement: 0.9 F1: 0.61
Agreement: 0.9 F1: 0.61
– Best when at least 4 out of 5 raters agree
50
[Agichtein]
51. Performance: ASP vs. Humans (F1, Satisfied)
Performance: ASP vs Humans (F1 Satisfied)
Classifier With Text Without Text Selected Features
ASP_SVM 0.69 0.72 0.62
ASP_C4.5 0.75 0.76 0.77
ASP_RandomForest 0.70 0.74 0.68
ASP_Boosting 0.67 0.67 0.67
ASP_NB 0.61 0.65 0.58
Best Human Perf 0.61
Baseline (random) 0.66
Human F1 is lower than the random baseline!
Human F1 is lower than the random baseline!
ASP is significantly more effective than humans
g y
51
52. Top Features by Information Gain
Top Features by Information Gain
• 0.14
0 14 Q: Askers’ previous rating
Q Ak ’ i ti
• 0.14 Q: Average past rating by asker
• 0.10 UH: Member since (interval)
• 0.05 g yp Q
UH: Average # answers for by past Q
• 0.05 UH: Previous Q resolved for the asker
• 0.04
0 04 CA: Average asker rating for category
CA: Average asker rating for category
• 0.04 UH: Total number of answers
received
…
52
[Agichtein]
53. Ссылки
• В
Видео + транскрипт лекции Маннига про QA (курс NLP, лекция
+ М QA ( NLP
18) http://see.stanford.edu/see/courses.aspx
• Слайды лекции Маннига про QA
http://www.stanford.edu/class/cs224n/syllabus.html#lec18
htt // t f d d / l / 224 / ll b ht l#l 18
• РОМИП QA http://romip.ru/ru/2010/tracks/qa.html
• QA @ TREC http://trec.nist.gov/data/qamain.html
p g q
• CLEF http://www.clef‐campaign.org/
• AnswerBus http://answerbus.coli.uni‐sb.de/
• Ответы@mail ru http://otvety mail ru/
Ответы@mail.ru http://otvety.mail.ru/
• Yahoo! Answers http://answers.yahoo.com/
• Quora http://www.quora.com/
• Aardvark http://vark.com/
• WolframAlpha http://www.wolframalpha.com/
18.12.2010 Павел Браславский 53
54. Статьи
• Dmitri Roussinov, Weiguo Fan, and Jose Robles‐Flores. 2008. Beyond
Dmitri Roussinov Weiguo Fan and Jose Robles Flores 2008 Beyond
keywords: Automated question answering on the web. Commun. ACM 51,
9.
• Kwok C., Etzioni O. and Weld D.S. Scaling Question Answering to the Web.
Kwok C Etzioni O and Weld D S Scaling Question Answering to the Web
ACM TOIS, Vol. 19, No. 3, July 2001.
• Banko M. et al. AskMSR: Question Answering Using the Worldwide Web.
p g y p g
In Proc. of 2002 AAAI Spring Symposium on Mining Answers from Texts
and Knowledge Bases.
• Zhiping Zheng. 2003. Question answering using web news as knowledge
base. In Proceedings of the tenth conference on European chapter of the
Association for Computational Linguistics ‐ Volume 2 (
i i f i l i i i l (EACL '03), Vol. 2.
' ) l
Association for Computational Linguistics, Morristown, NJ, USA, 251‐254.
• E. Agichtein, C. Castillo, D. Donato, A. Gionis, and G. Mishne, Finding High
Quality Content in Social Media, in WSDM 2008
Quality Content in Social Media in WSDM 2008
• Y. Liu, J. Bian, and E. Agichtein, Predicting Asker Satisfaction, SIGIR 2008
18.12.2010 Павел Браславский 54
55. Спасибо за внимание!
Павел Браславский
Павел Браславский
pb@yandex‐team.ru
18.12.2010 Павел Браславский 55
