Deephack report Kaggle Allen AI QA

1. Kaggle Allen AI
Resolventa team

2. Competition
• Question and 4 Answers
• Common science
• 2500 train, ~8.000 validate, ~22.000 test

3. In which way is the orbit
of a comet different from
the orbit of Earth?

4. As we used to solve
competitions?
DATA
XGBoost

5. As we used to solve
competitions?
DATA
XGBoost

7. Wiki
Ck-12
Lucene
*2vec
Custom LSTM
Graph of Words
Other
XGBoost
Custom Linear Model
Averaging
Voting

8. Lucene
maybe deeper
than most of the neural networks in this competition

9. Lucene benchmark
top-50
01.01.2016

11. • similarity: tf-idf, bm25, lmd, …
• docs: by sentences, by paragraph, by articles
• minimal size: 50, 100, 150, 200, 300
• stemming: yes, no
• stop words: yes, no
• synonyms: yes, no
• corpus: ck12, science wiki, all wiki, …

12. • similarity: tf-idf, bm25, lmd
• docs: by sentences, by paragraph, by articles
• minimal size: 50, 100, 150, 200, 300
• stemming: yes, no
• stop words: yes, no
• synonyms: yes, no
• corpus: ck12, science wiki, all wiki, …Let’s try all we can

13. >3000 indexes
<1 min
create index, 10.000 requests, remove index*
* ~ 20.000 articles

20. “BETONOMESHALKA”
relevance id
weight

21. • Which of the following is not a real chemical
element?
• Hydrogen
• Oxygen
• Gold
• Watermelon

22. • A widow's peak is inherited by anyone with a
dominant allele, while a straight hairline is inherited
only when the offspring receives recessive alleles
from both parents. Which of the following genotypes
is MOST likely to have a straight hairline phenotype?
• ww
• wW
• Ww
• WW

23. • A widow's peak is inherited by anyone with a
dominant allele, while a straight hairline is inherited
only when the offspring receives recessive alleles
from both parents. Which of the following genotypes
is MOST likely to have a straight hairline phenotype?
• ww -> ww
• wW -> ww
• Ww -> ww
• WW -> ww

24. Architecture Inspiration
LSTM-BASED DEEP LEARNING MODELS FOR NONFACTOID
ANSWER SELECTION

25. Dataset is a bitch
• NNs require lots of data
• 2.5k q&a

26. Ideas
• Use pre trained embeddings - word2vec
• Use dumb networks
• Get more data - data augmentation
• Add IR results as memory (MemN2N)

27. Best Architecture
• Simplified architecture
• Few neurons: embedding=10, lstm=32
• Max pooling
• Wrong answers shuffling
• Allen AI 4th grade datasets

28. Results unclear
• Overfitted on training thru data leakages
• Could not get >0.31 with honest CV

29. Graph of words

30. Graph of words

31. Questions -> Word2vec ->
t-SNE

32. Resolventa team

33. respect