Eugene Klyuchnikov, Business Intelligence Lead, TourRadar ~You ask, we don't answer (yet). How to build a perfect ML-based question answering model which doesn't work.~

1. You ask, we don't answer (yet).
How to build a perfect ML-based
question answering solution
which doesn't work

2. In the beginning was the Idea...

3. In the beginning was the Idea...

4. In the beginning was the Idea...

5. Question Answering System
● Generalizable to different types of questions
● Oh, it must be awesome
● Autonomous doesn’t require human race to operate
● Trustworthy no hassles
● Scalable to infinity and beyond

6. Hackathon
Question Answering System

7. Question Answering System
Hackathon:
Suggested, but not limited to:
● Amazon Lex
● Microsoft QnA
1 day
(Friday)
5 teams,
different solutions

8. AWS Lex
Question
MS QnA
Analyze
response
Show Lex
Show QnA
Sorry we could not
process your question.
Score > 50
Lex is able to
recognize the intent
RUAVA
Reisen- und Abenteuervirtuellerassistent

9. Question
RUAVA (doh!)
Black
Box
Sorry we could not
process your question.
50%
50%

10. AWS Lex
Question
MS QnA
Analyze
response
Show _
Show QnA
Sorry we could not
process your question.
RUAVA
I know
everything!
I’m not sure...
Score > 50
Lex is able to
recognize the intent

11. AWS Lex
Question
MS QnA
Analyze
response
Show _
Show QnA
Sorry we could not
process your question.
Lex is able to
recognize the intent
(of course!)
RUAVA
I know
everything!
I’m not sure...

12. RUAVA
● Bad results
○ Intents were very sensitive to new / inaccurate utterances
○ Intents were hard to train (what keywords should we use to trigger intent?)
○ Lex intents were overtaking responds with incorrect answers
○ QnA Score > 50 was a hard limit
● Improvements
○ Train better, automate training phase
○ Use more data
○ Focus on one platform…
○ … or use a completely different approach

13. Completely Different Approach
cosine similarity
TF-IDF, lemmatization
Natural Language Toolkit
Answer from
the closest question
becomes an answer
to original one

14. Completely Different Approach
cosine similarity model
TF-IDF, lemmatization
Natural Language Toolkit
cover by tests,
create api,
ensure security,
dockerize...
add logging,
monitoring,
alerting...
PROD

15. Completely Different Approach
PROD

16. Completely Different Approach
Expectations

17. Completely Different Approach
Expectations Reality

18. Happy End
cosine similarity
TF-IDF, lemmatization
Natural Language Toolkit
Answer from
the closest question
becomes an answer
to original one

19. Happy End
cosine similarity
TF-IDF, lemmatization
Natural Language Toolkit
Answer from
the closest question
becomes an answer
to original one

20. Happy End
cosine similarity
TF-IDF, lemmatization
Natural Language Toolkit
Answer from
the closest question
becomes an answer
to original one
Internal
knowledge base /
onboarding tool

21. Happy End
● Automating customer support roster
● Working on recommendation systems
● Personalizing SERPs
● Doing reviews sentiment analysis / language detection
● Generating the product descriptions
● Predicting the intent to book
● Segmenting customers

22. Thank you!
TourRadar
https://www.tourradar.com/
https://www.facebook.com/TourRadar/
https://www.instagram.com/tourradar/
https://www.youtube.com/tourradar
https://www.linkedin.com/company/tourradar/
Eugene Klyuchnikov
eugene@tourradar.com