SlideShare a Scribd company logo
The Allen AI Science
Challenge &
DeepHack.Q&A
St. Petersburg Data Science Meetup #6, Feb 19th, 2016
Q: When athletes begin to exercise, their heart rates and respiration rates
increase. At what level of organization does the human body coordinate these
functions?
A. at the tissue level
B. at the organ level
C. at the system level
D. at the cellular level
Wed 7 Oct 2015 – Sat 13 Feb 2016
Stage 1: 800 teams (>1000 participants),
Stage 2: 170 teams
https://www.kaggle.com/c/the-allen-ai-science-challenge
2700 questions - train set
8132 questions - validation set
21298 questions - final test set
DeepHack Q&A qa.deephack.me/
Qualification round: Top-50 participants with the
highest scores
Rough competition: Kaggle Top-40 to get to the Top-50 o_O
Winter ML school + hackathon: 31st, Jan - 5th Feb, 2016
GP team created at Jan, 31st from the four teams
The final 30 minutes of the hackathon: https:
//www.youtube.com/watch?v=tCKL5vbiHuo
Pavel Kalaidin (VK)
Marat Zainutdinov (Quantbrothers)
Roman Trusov (ITMO University)
Artyom Korkhov (Zvooq)
Igor Shilov (Zvooq)
Timur Luguev (Clevapi)
Ilyas Luguev (Clevapi)
Team Generation Gap
DeepHack: 1st, ~0.556
Allen AI: 7th, 0.55059
Datasets
ck12.org
wikipedia.org (science subset)
flashcards: studystack.com,
quizlet.com
Topic at the forum: https://www.kaggle.com/c/the-allen-ai-science-challenge/forums/t/16877/external-data-
repository
Hail to Lucene
Lucene
Question
a) ans1
b) ans2
c) ans3
d) ans4
Question ans1
Question ans4
Question ans3
Question ans2
0.5
0.4
0.02
0.01
...
0.5
0.4
0.02
0.01
...
0.5
0.4
0.02
0.01
...
0.5
0.4
0.02
0.01
...
Wiki ck12
quizlets
Stemming,stopwords
Custom queries rule
Lucene scores: https://lucene.apache.
org/core/3_5_0/api/core/org/apache/lucene/search/Similarity.html
AdaGram (a.k.a Reptil)
Breaking Sticks and Ambiguities with Adaptive Skip-gram: http:
//arxiv.org/abs/1502.07257
Reference implementation in Julia: https://github.
com/sbos/AdaGram.jl
reptil art cultur final play
signific role folklor
religion popular cultur moch
peopl noun coldblood anim
scale general move stomach
short leg exampl snake lizard
turtl noun aw person
Model trained like this: sh train.sh --min-freq 20 --window 5 --workers 40 --epochs 5
--dim 300 --alpha 0.1 corpus.txt adam.dict adam.model
Number of prototypes is 5 by default.
AdaGram (a.k.a Reptil) approach
[0.42, 0.55, 0.08, …]
N-grams PMI
x, y - Ngrams
Example 1-gram -> 1-gram
unit -> state
magnet -> field
carbon -> dioxid
million -> year
year -> ago
amino -> acid
Example 1-gram -> 3-gram
around -> million year ago
period -> million year ago
forc -> van der waal
fossil -> million year ago
nobel -> prize physiolog medicin
date -> million year ago
mercuri -> venus earth mar
N-grams PMI
greatest contributor air pollut unit state
What is the greatest contributor to air pollution
in the United States?
greatest
contributor
air
...
greatest contributor
contributor air
air pollut
...
1-grams 2-grams 3-grams
Power plants
power plant
power
plant
1-grams
power plant
2-grams
...
Scores
Fail Story
TL;DR wasted tons of time, got ~0.3 in
almost all approaches
LSA + Lucene
Corpus
LSA
TI_1
TI_2
TI_n
Lucene
qa pair 1
qa pair 2
qa pair 3
qa pair 4
Queries in
topic
indices
Result: for each qa pair,
max(s1...sn)
Gave 1% improvement over
basic Lucene; but took
EXTREMELY long time to
process :(
Syntax co-occurrence
nobel chemistry prize 517
national science academy 445
long time period 340
also role play 306
nobel physic prize 279
national medical library 273
carbon water dioxide 261
second thermodynamics law 247
speed sound of_pobj
density population compound
take place dobj
link external compound 0.3 :(
word2vec combinations
Wanted to capture the intersection of meanings, but didn’t know
how to combine word2vec representations
TFIDFqa pairs
Combinations of
question tokens
Combinations of
answer tokens
Cosine
similarity
Max score ~ 0.3 :( even with careful kw filtering
word2gauss didn’t help too
Averaging Neural networks (1st encounter)
w2v_dim = 300
vec_q = mean(w2v(Q))
vec_c = mean(w2v(Ac))
vec_w = mean(w2v(Aw))
cos_sim(vec_q, vec_c) > cos_sim(vec_q, vec_w)
Averaging Neural networks (1st encounter)
w2v_dim = 300
vec_q = mean(w2v(Q))
vec_c = mean(w2v(Ac))
vec_w = mean(w2v(Aw))
a = CNN(w2v(X))
vec_x = mean(w2v(X) * a)
cos_sim(vec_q, vec_c) > cos_sim(vec_q, vec_w)
Semantic Neural networks (2nd encounter)
+ Paragraphs
LSTM = LSTM(w2v)
LSTM(s1 | s2) > LSTM(s1 | s3) if s1 and s2 are from the same
paragraph, while s1 and s3 are not
LSTM(a, b) is low then a and b are from the same paragraph (energy
based learning)
Loss = max(0, M - LSTM(s1, s2) + LSTM(s1, s3))
Score: 0.26
Siamese architecture
Hinge Loss
Margin
Reading Neural networks (3rd encounter)
+ Lots of paragraphs
+ Search Engine
+ A survey:
- bigrams are not accounted
- main idea (keywords) of a sentence is not recognized
Reading Neural networks (3rd encounter)
+ Lots of paragraphs
+ Search Engine
+ A survey:
- bigrams are not accounted
- main idea (keywords) of a sentence is not recognized
Reading Neural networks (3rd encounter)
All we want is to know if a sentence is from a paragraph to be
able to rerank lucene scores.
Hinge Loss
Margin
LSTM(P)
LSTM(s1)
LSTM(s2)
Reading Neural networks (3rd encounter)
sentences -> LSTM -> Dense NN -> Embedding
w2v -> LSTM -> Dense NN -> Embedding
w2v -> Mean -> Dense NN -> Embedding
Neural networks. Learned lessons.
Start as small as possible
Corruption is important for siamese networks
Learning curve is misleading in NLP
Lessons learned
Start early - wasted two first months of the competition (but had a week
of 24/7 hackathon at the end)
No stickers in the team channel (except with Yann LeCun on a good submit)
Common toolbox is nice
A dedicated server is a good thing to have (no need in AWS spot instances)
Experiment fast, fail early
Team work means a lot

More Related Content

Similar to The Allen AI Science Challenge

Introduction to Algorithms
Introduction to AlgorithmsIntroduction to Algorithms
Introduction to Algorithms
Venkatesh Iyer
 
2.03.Asymptotic_analysis.pptx
2.03.Asymptotic_analysis.pptx2.03.Asymptotic_analysis.pptx
2.03.Asymptotic_analysis.pptx
ssuser1fb3df
 
String kmp
String kmpString kmp
String kmp
thinkphp
 
Computing k-rank Answers with Ontological CP-nets
Computing k-rank Answers with Ontological CP-netsComputing k-rank Answers with Ontological CP-nets
Computing k-rank Answers with Ontological CP-nets
Oana Tifrea-Marciuska
 
A Network-Aware Approach for Searching As-You-Type in Social Media
A Network-Aware Approach for Searching As-You-Type in Social MediaA Network-Aware Approach for Searching As-You-Type in Social Media
A Network-Aware Approach for Searching As-You-Type in Social Media
INRIA-OAK
 
Computer Science Engineering : Data structure & algorithm, THE GATE ACADEMY
Computer Science Engineering : Data structure & algorithm, THE GATE ACADEMYComputer Science Engineering : Data structure & algorithm, THE GATE ACADEMY
Computer Science Engineering : Data structure & algorithm, THE GATE ACADEMY
klirantga
 
Machine Learning
Machine LearningMachine Learning
Machine Learning
butest
 
Lecture 2 data structures & algorithms - sorting techniques
Lecture 2  data structures & algorithms - sorting techniquesLecture 2  data structures & algorithms - sorting techniques
Lecture 2 data structures & algorithms - sorting techniques
Dharmendra Prasad
 
Software-Praktikum SoSe 2005 Lehrstuhl fuer Maschinelles ...
Software-Praktikum SoSe 2005 Lehrstuhl fuer Maschinelles ...Software-Praktikum SoSe 2005 Lehrstuhl fuer Maschinelles ...
Software-Praktikum SoSe 2005 Lehrstuhl fuer Maschinelles ...
butest
 
Stack squeues lists
Stack squeues listsStack squeues lists
Stack squeues lists
James Wong
 
Stacks queues lists
Stacks queues listsStacks queues lists
Stacks queues lists
Harry Potter
 
Stacks queues lists
Stacks queues listsStacks queues lists
Stacks queues lists
Tony Nguyen
 
Stacks queues lists
Stacks queues listsStacks queues lists
Stacks queues lists
Luis Goldster
 
Stacksqueueslists
StacksqueueslistsStacksqueueslists
Stacksqueueslists
Fraboni Ec
 
Stacks queues lists
Stacks queues listsStacks queues lists
Stacks queues lists
Young Alista
 
algo1-aa-gist_typed.pdf
algo1-aa-gist_typed.pdfalgo1-aa-gist_typed.pdf
algo1-aa-gist_typed.pdf
prajjwalbajpai1
 
time_complexity_list_02_04_2024_22_pages.pdf
time_complexity_list_02_04_2024_22_pages.pdftime_complexity_list_02_04_2024_22_pages.pdf
time_complexity_list_02_04_2024_22_pages.pdf
SrinivasaReddyPolamR
 
Introduction to computing Processing and performance.pdf
Introduction to computing Processing and performance.pdfIntroduction to computing Processing and performance.pdf
Introduction to computing Processing and performance.pdf
TulasiramKandula1
 
introduction to algorithm for beginneer1
introduction to algorithm for beginneer1introduction to algorithm for beginneer1
introduction to algorithm for beginneer1
ranjankumarbehera14
 
Lecture38
Lecture38Lecture38
Lecture38
David Evans
 

Similar to The Allen AI Science Challenge (20)

Introduction to Algorithms
Introduction to AlgorithmsIntroduction to Algorithms
Introduction to Algorithms
 
2.03.Asymptotic_analysis.pptx
2.03.Asymptotic_analysis.pptx2.03.Asymptotic_analysis.pptx
2.03.Asymptotic_analysis.pptx
 
String kmp
String kmpString kmp
String kmp
 
Computing k-rank Answers with Ontological CP-nets
Computing k-rank Answers with Ontological CP-netsComputing k-rank Answers with Ontological CP-nets
Computing k-rank Answers with Ontological CP-nets
 
A Network-Aware Approach for Searching As-You-Type in Social Media
A Network-Aware Approach for Searching As-You-Type in Social MediaA Network-Aware Approach for Searching As-You-Type in Social Media
A Network-Aware Approach for Searching As-You-Type in Social Media
 
Computer Science Engineering : Data structure & algorithm, THE GATE ACADEMY
Computer Science Engineering : Data structure & algorithm, THE GATE ACADEMYComputer Science Engineering : Data structure & algorithm, THE GATE ACADEMY
Computer Science Engineering : Data structure & algorithm, THE GATE ACADEMY
 
Machine Learning
Machine LearningMachine Learning
Machine Learning
 
Lecture 2 data structures & algorithms - sorting techniques
Lecture 2  data structures & algorithms - sorting techniquesLecture 2  data structures & algorithms - sorting techniques
Lecture 2 data structures & algorithms - sorting techniques
 
Software-Praktikum SoSe 2005 Lehrstuhl fuer Maschinelles ...
Software-Praktikum SoSe 2005 Lehrstuhl fuer Maschinelles ...Software-Praktikum SoSe 2005 Lehrstuhl fuer Maschinelles ...
Software-Praktikum SoSe 2005 Lehrstuhl fuer Maschinelles ...
 
Stack squeues lists
Stack squeues listsStack squeues lists
Stack squeues lists
 
Stacks queues lists
Stacks queues listsStacks queues lists
Stacks queues lists
 
Stacks queues lists
Stacks queues listsStacks queues lists
Stacks queues lists
 
Stacks queues lists
Stacks queues listsStacks queues lists
Stacks queues lists
 
Stacksqueueslists
StacksqueueslistsStacksqueueslists
Stacksqueueslists
 
Stacks queues lists
Stacks queues listsStacks queues lists
Stacks queues lists
 
algo1-aa-gist_typed.pdf
algo1-aa-gist_typed.pdfalgo1-aa-gist_typed.pdf
algo1-aa-gist_typed.pdf
 
time_complexity_list_02_04_2024_22_pages.pdf
time_complexity_list_02_04_2024_22_pages.pdftime_complexity_list_02_04_2024_22_pages.pdf
time_complexity_list_02_04_2024_22_pages.pdf
 
Introduction to computing Processing and performance.pdf
Introduction to computing Processing and performance.pdfIntroduction to computing Processing and performance.pdf
Introduction to computing Processing and performance.pdf
 
introduction to algorithm for beginneer1
introduction to algorithm for beginneer1introduction to algorithm for beginneer1
introduction to algorithm for beginneer1
 
Lecture38
Lecture38Lecture38
Lecture38
 

Recently uploaded

SAP BW4HANA Implementagtion Content Document
SAP BW4HANA Implementagtion Content DocumentSAP BW4HANA Implementagtion Content Document
SAP BW4HANA Implementagtion Content Document
newdirectionconsulta
 
Namma-Kalvi-11th-Physics-Study-Material-Unit-1-EM-221086.pdf
Namma-Kalvi-11th-Physics-Study-Material-Unit-1-EM-221086.pdfNamma-Kalvi-11th-Physics-Study-Material-Unit-1-EM-221086.pdf
Namma-Kalvi-11th-Physics-Study-Material-Unit-1-EM-221086.pdf
22ad0301
 
Salesforce AI + Data Community Tour Slides - Canarias
Salesforce AI + Data Community Tour Slides - CanariasSalesforce AI + Data Community Tour Slides - Canarias
Salesforce AI + Data Community Tour Slides - Canarias
davidpietrzykowski1
 
一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理
bmucuha
 
一比一原版莱斯大学毕业证(rice毕业证)如何办理
一比一原版莱斯大学毕业证(rice毕业证)如何办理一比一原版莱斯大学毕业证(rice毕业证)如何办理
一比一原版莱斯大学毕业证(rice毕业证)如何办理
zsafxbf
 
Econ3060_Screen Time and Success_ final_GroupProject.pdf
Econ3060_Screen Time and Success_ final_GroupProject.pdfEcon3060_Screen Time and Success_ final_GroupProject.pdf
Econ3060_Screen Time and Success_ final_GroupProject.pdf
blueshagoo1
 
reading_sample_sap_press_operational_data_provisioning_with_sap_bw4hana (1).pdf
reading_sample_sap_press_operational_data_provisioning_with_sap_bw4hana (1).pdfreading_sample_sap_press_operational_data_provisioning_with_sap_bw4hana (1).pdf
reading_sample_sap_press_operational_data_provisioning_with_sap_bw4hana (1).pdf
perranet1
 
Module 1 ppt BIG DATA ANALYTICS NOTES FOR MCA
Module 1 ppt BIG DATA ANALYTICS NOTES FOR MCAModule 1 ppt BIG DATA ANALYTICS NOTES FOR MCA
Module 1 ppt BIG DATA ANALYTICS NOTES FOR MCA
yuvarajkumar334
 
How To Control IO Usage using Resource Manager
How To Control IO Usage using Resource ManagerHow To Control IO Usage using Resource Manager
How To Control IO Usage using Resource Manager
Alireza Kamrani
 
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
9gr6pty
 
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
eudsoh
 
一比一原版(UofT毕业证)多伦多大学毕业证如何办理
一比一原版(UofT毕业证)多伦多大学毕业证如何办理一比一原版(UofT毕业证)多伦多大学毕业证如何办理
一比一原版(UofT毕业证)多伦多大学毕业证如何办理
exukyp
 
A gentle exploration of Retrieval Augmented Generation
A gentle exploration of Retrieval Augmented GenerationA gentle exploration of Retrieval Augmented Generation
A gentle exploration of Retrieval Augmented Generation
dataschool1
 
社内勉強会資料_Hallucination of LLMs               .
社内勉強会資料_Hallucination of LLMs               .社内勉強会資料_Hallucination of LLMs               .
社内勉強会資料_Hallucination of LLMs               .
NABLAS株式会社
 
一比一原版(uom毕业证书)曼彻斯特大学毕业证如何办理
一比一原版(uom毕业证书)曼彻斯特大学毕业证如何办理一比一原版(uom毕业证书)曼彻斯特大学毕业证如何办理
一比一原版(uom毕业证书)曼彻斯特大学毕业证如何办理
osoyvvf
 
Bangalore ℂall Girl 000000 Bangalore Escorts Service
Bangalore ℂall Girl 000000 Bangalore Escorts ServiceBangalore ℂall Girl 000000 Bangalore Escorts Service
Bangalore ℂall Girl 000000 Bangalore Escorts Service
nhero3888
 
一比一原版(lbs毕业证书)伦敦商学院毕业证如何办理
一比一原版(lbs毕业证书)伦敦商学院毕业证如何办理一比一原版(lbs毕业证书)伦敦商学院毕业证如何办理
一比一原版(lbs毕业证书)伦敦商学院毕业证如何办理
ywqeos
 
一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理
一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理
一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理
uevausa
 
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
hqfek
 
Call Girls Lucknow 0000000000 Independent Call Girl Service Lucknow
Call Girls Lucknow 0000000000 Independent Call Girl Service LucknowCall Girls Lucknow 0000000000 Independent Call Girl Service Lucknow
Call Girls Lucknow 0000000000 Independent Call Girl Service Lucknow
hiju9823
 

Recently uploaded (20)

SAP BW4HANA Implementagtion Content Document
SAP BW4HANA Implementagtion Content DocumentSAP BW4HANA Implementagtion Content Document
SAP BW4HANA Implementagtion Content Document
 
Namma-Kalvi-11th-Physics-Study-Material-Unit-1-EM-221086.pdf
Namma-Kalvi-11th-Physics-Study-Material-Unit-1-EM-221086.pdfNamma-Kalvi-11th-Physics-Study-Material-Unit-1-EM-221086.pdf
Namma-Kalvi-11th-Physics-Study-Material-Unit-1-EM-221086.pdf
 
Salesforce AI + Data Community Tour Slides - Canarias
Salesforce AI + Data Community Tour Slides - CanariasSalesforce AI + Data Community Tour Slides - Canarias
Salesforce AI + Data Community Tour Slides - Canarias
 
一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理
 
一比一原版莱斯大学毕业证(rice毕业证)如何办理
一比一原版莱斯大学毕业证(rice毕业证)如何办理一比一原版莱斯大学毕业证(rice毕业证)如何办理
一比一原版莱斯大学毕业证(rice毕业证)如何办理
 
Econ3060_Screen Time and Success_ final_GroupProject.pdf
Econ3060_Screen Time and Success_ final_GroupProject.pdfEcon3060_Screen Time and Success_ final_GroupProject.pdf
Econ3060_Screen Time and Success_ final_GroupProject.pdf
 
reading_sample_sap_press_operational_data_provisioning_with_sap_bw4hana (1).pdf
reading_sample_sap_press_operational_data_provisioning_with_sap_bw4hana (1).pdfreading_sample_sap_press_operational_data_provisioning_with_sap_bw4hana (1).pdf
reading_sample_sap_press_operational_data_provisioning_with_sap_bw4hana (1).pdf
 
Module 1 ppt BIG DATA ANALYTICS NOTES FOR MCA
Module 1 ppt BIG DATA ANALYTICS NOTES FOR MCAModule 1 ppt BIG DATA ANALYTICS NOTES FOR MCA
Module 1 ppt BIG DATA ANALYTICS NOTES FOR MCA
 
How To Control IO Usage using Resource Manager
How To Control IO Usage using Resource ManagerHow To Control IO Usage using Resource Manager
How To Control IO Usage using Resource Manager
 
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
一比一原版(uob毕业证书)伯明翰大学毕业证如何办理
 
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
一比一原版马来西亚博特拉大学毕业证(upm毕业证)如何办理
 
一比一原版(UofT毕业证)多伦多大学毕业证如何办理
一比一原版(UofT毕业证)多伦多大学毕业证如何办理一比一原版(UofT毕业证)多伦多大学毕业证如何办理
一比一原版(UofT毕业证)多伦多大学毕业证如何办理
 
A gentle exploration of Retrieval Augmented Generation
A gentle exploration of Retrieval Augmented GenerationA gentle exploration of Retrieval Augmented Generation
A gentle exploration of Retrieval Augmented Generation
 
社内勉強会資料_Hallucination of LLMs               .
社内勉強会資料_Hallucination of LLMs               .社内勉強会資料_Hallucination of LLMs               .
社内勉強会資料_Hallucination of LLMs               .
 
一比一原版(uom毕业证书)曼彻斯特大学毕业证如何办理
一比一原版(uom毕业证书)曼彻斯特大学毕业证如何办理一比一原版(uom毕业证书)曼彻斯特大学毕业证如何办理
一比一原版(uom毕业证书)曼彻斯特大学毕业证如何办理
 
Bangalore ℂall Girl 000000 Bangalore Escorts Service
Bangalore ℂall Girl 000000 Bangalore Escorts ServiceBangalore ℂall Girl 000000 Bangalore Escorts Service
Bangalore ℂall Girl 000000 Bangalore Escorts Service
 
一比一原版(lbs毕业证书)伦敦商学院毕业证如何办理
一比一原版(lbs毕业证书)伦敦商学院毕业证如何办理一比一原版(lbs毕业证书)伦敦商学院毕业证如何办理
一比一原版(lbs毕业证书)伦敦商学院毕业证如何办理
 
一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理
一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理
一比一原版加拿大渥太华大学毕业证(uottawa毕业证书)如何办理
 
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
一比一原版爱尔兰都柏林大学毕业证(本硕)ucd学位证书如何办理
 
Call Girls Lucknow 0000000000 Independent Call Girl Service Lucknow
Call Girls Lucknow 0000000000 Independent Call Girl Service LucknowCall Girls Lucknow 0000000000 Independent Call Girl Service Lucknow
Call Girls Lucknow 0000000000 Independent Call Girl Service Lucknow
 

The Allen AI Science Challenge

  • 1. The Allen AI Science Challenge & DeepHack.Q&A St. Petersburg Data Science Meetup #6, Feb 19th, 2016
  • 2. Q: When athletes begin to exercise, their heart rates and respiration rates increase. At what level of organization does the human body coordinate these functions? A. at the tissue level B. at the organ level C. at the system level D. at the cellular level Wed 7 Oct 2015 – Sat 13 Feb 2016 Stage 1: 800 teams (>1000 participants), Stage 2: 170 teams https://www.kaggle.com/c/the-allen-ai-science-challenge 2700 questions - train set 8132 questions - validation set 21298 questions - final test set
  • 3. DeepHack Q&A qa.deephack.me/ Qualification round: Top-50 participants with the highest scores Rough competition: Kaggle Top-40 to get to the Top-50 o_O Winter ML school + hackathon: 31st, Jan - 5th Feb, 2016 GP team created at Jan, 31st from the four teams The final 30 minutes of the hackathon: https: //www.youtube.com/watch?v=tCKL5vbiHuo
  • 4. Pavel Kalaidin (VK) Marat Zainutdinov (Quantbrothers) Roman Trusov (ITMO University) Artyom Korkhov (Zvooq) Igor Shilov (Zvooq) Timur Luguev (Clevapi) Ilyas Luguev (Clevapi) Team Generation Gap DeepHack: 1st, ~0.556 Allen AI: 7th, 0.55059
  • 5. Datasets ck12.org wikipedia.org (science subset) flashcards: studystack.com, quizlet.com Topic at the forum: https://www.kaggle.com/c/the-allen-ai-science-challenge/forums/t/16877/external-data- repository
  • 6. Hail to Lucene Lucene Question a) ans1 b) ans2 c) ans3 d) ans4 Question ans1 Question ans4 Question ans3 Question ans2 0.5 0.4 0.02 0.01 ... 0.5 0.4 0.02 0.01 ... 0.5 0.4 0.02 0.01 ... 0.5 0.4 0.02 0.01 ... Wiki ck12 quizlets Stemming,stopwords
  • 7. Custom queries rule Lucene scores: https://lucene.apache. org/core/3_5_0/api/core/org/apache/lucene/search/Similarity.html
  • 8. AdaGram (a.k.a Reptil) Breaking Sticks and Ambiguities with Adaptive Skip-gram: http: //arxiv.org/abs/1502.07257 Reference implementation in Julia: https://github. com/sbos/AdaGram.jl
  • 9. reptil art cultur final play signific role folklor religion popular cultur moch peopl noun coldblood anim scale general move stomach short leg exampl snake lizard turtl noun aw person
  • 10. Model trained like this: sh train.sh --min-freq 20 --window 5 --workers 40 --epochs 5 --dim 300 --alpha 0.1 corpus.txt adam.dict adam.model Number of prototypes is 5 by default.
  • 11. AdaGram (a.k.a Reptil) approach [0.42, 0.55, 0.08, …]
  • 12. N-grams PMI x, y - Ngrams Example 1-gram -> 1-gram unit -> state magnet -> field carbon -> dioxid million -> year year -> ago amino -> acid Example 1-gram -> 3-gram around -> million year ago period -> million year ago forc -> van der waal fossil -> million year ago nobel -> prize physiolog medicin date -> million year ago mercuri -> venus earth mar
  • 13. N-grams PMI greatest contributor air pollut unit state What is the greatest contributor to air pollution in the United States? greatest contributor air ... greatest contributor contributor air air pollut ... 1-grams 2-grams 3-grams Power plants power plant power plant 1-grams power plant 2-grams ...
  • 15. Fail Story TL;DR wasted tons of time, got ~0.3 in almost all approaches
  • 16. LSA + Lucene Corpus LSA TI_1 TI_2 TI_n Lucene qa pair 1 qa pair 2 qa pair 3 qa pair 4 Queries in topic indices Result: for each qa pair, max(s1...sn) Gave 1% improvement over basic Lucene; but took EXTREMELY long time to process :(
  • 17. Syntax co-occurrence nobel chemistry prize 517 national science academy 445 long time period 340 also role play 306 nobel physic prize 279 national medical library 273 carbon water dioxide 261 second thermodynamics law 247 speed sound of_pobj density population compound take place dobj link external compound 0.3 :(
  • 18. word2vec combinations Wanted to capture the intersection of meanings, but didn’t know how to combine word2vec representations TFIDFqa pairs Combinations of question tokens Combinations of answer tokens Cosine similarity Max score ~ 0.3 :( even with careful kw filtering word2gauss didn’t help too
  • 19. Averaging Neural networks (1st encounter) w2v_dim = 300 vec_q = mean(w2v(Q)) vec_c = mean(w2v(Ac)) vec_w = mean(w2v(Aw)) cos_sim(vec_q, vec_c) > cos_sim(vec_q, vec_w)
  • 20. Averaging Neural networks (1st encounter) w2v_dim = 300 vec_q = mean(w2v(Q)) vec_c = mean(w2v(Ac)) vec_w = mean(w2v(Aw)) a = CNN(w2v(X)) vec_x = mean(w2v(X) * a) cos_sim(vec_q, vec_c) > cos_sim(vec_q, vec_w)
  • 21. Semantic Neural networks (2nd encounter) + Paragraphs LSTM = LSTM(w2v) LSTM(s1 | s2) > LSTM(s1 | s3) if s1 and s2 are from the same paragraph, while s1 and s3 are not LSTM(a, b) is low then a and b are from the same paragraph (energy based learning) Loss = max(0, M - LSTM(s1, s2) + LSTM(s1, s3)) Score: 0.26
  • 24. Reading Neural networks (3rd encounter) + Lots of paragraphs + Search Engine + A survey: - bigrams are not accounted - main idea (keywords) of a sentence is not recognized
  • 25. Reading Neural networks (3rd encounter) + Lots of paragraphs + Search Engine + A survey: - bigrams are not accounted - main idea (keywords) of a sentence is not recognized
  • 26. Reading Neural networks (3rd encounter) All we want is to know if a sentence is from a paragraph to be able to rerank lucene scores.
  • 27.
  • 29. Reading Neural networks (3rd encounter) sentences -> LSTM -> Dense NN -> Embedding w2v -> LSTM -> Dense NN -> Embedding w2v -> Mean -> Dense NN -> Embedding
  • 30. Neural networks. Learned lessons. Start as small as possible Corruption is important for siamese networks Learning curve is misleading in NLP
  • 31. Lessons learned Start early - wasted two first months of the competition (but had a week of 24/7 hackathon at the end) No stickers in the team channel (except with Yann LeCun on a good submit) Common toolbox is nice A dedicated server is a good thing to have (no need in AWS spot instances) Experiment fast, fail early Team work means a lot