The presentation consists of 3 parts:
1. Introduction into conversational agents.
2. The Conversational Intelligence Challenge in which we are participating with our skill-based conversational agent.
3. Details of our solution for this challenge. Our conversational agent (bot#1337) took second place in ConvAI Human Evaluation Round (http://convai.io/1_round/).
In this presentation dialog system, conversational agent and chatbot means the same.
6. ConvAI
- 6 Teams (McGill, MIPT, University of Wroclaw, …)
- Human evaluation qualification round (July, 2017)
- 1st place: 2.386 of 5 (overall dialog quality)
- 2nd (Ours): 2.318 of 5
- Released dataset: http://convai.io/data/
- about 2k dialogs
- NIPS Final (December, 2017)
- Talk with bots and help to collect the data:
http://t.me/ConvaiBot
14. Evolution of skill classifier
● Baseline (done)
○ no conversational data
○ use classifier to select skills
● Model with scorer (in progress)
○ we have some data after human evaluation round
○ 2k dialogs (all bots)
○ we are mostly interested in our bot scores
● Model with improved scorer (to be done)
○ data from Mechanical Turk
15. What is done. Dialog evaluation scorer
- 2 evaluation scorers were built by using ConvAI human evaluation dataset
- Current utterance quality scorer: [context, utterance] => (poor, good)
- Word level GRU, sequence length is 50
- Overall dialog quality scorer: [overall dialog] => (poor, neutral, good)
- Word level GRU, sequence length is a whole dialog
16. Future work
● Improve classifier by using current utterance quality scorer
● Setup for human dialog evaluation (Amazon Mechanical Turk, Telegram)
● Bot with bot conversation using dialog scorer
● Improve skills
● New skills (summarization, retrieval based models)
17. Summary
- Skill: it is a narrow model (question generation/answerer, chit-chat, …)
- Conversational agent requires management of such skills
- Management can be done using FSM, but it is hard to maintain them
- Our approach helps to get rid of FSM and focus on skill implementation
- Main idea: use classifier which decides what skill to use
- Future work may lead to interesting results
- Talk with our Telegram bot here: https://t.me/IdrisConvaioTestBot
18. References
1. The Conversational Intelligence Challenge: http://convai.io/
2. Our bot: https://t.me/IdrisConvaioTestBot
3. Seo M. et al. Bidirectional attention flow for machine comprehension //arXiv preprint
arXiv:1611.01603. – 2016.
4. Zhou, Q., Yang, N., Wei, F., Tan, C., Bao, H., & Zhou, M. (2017). Neural Question Generation from
Text: A Preliminary Study. arXiv preprint arXiv:1704.01792.
5. Bordes, A., & Weston, J. (2016). Learning end-to-end goal-oriented dialog. arXiv preprint
arXiv:1605.07683.
6. Serban I. V. et al. A Deep Reinforcement Learning Chatbot //arXiv preprint arXiv:1709.02349. –
2017.
7. Lewis M. et al. Deal or No Deal? End-to-End Learning for Negotiation Dialogues //arXiv preprint
arXiv:1706.05125. – 2017.
