Real-Time Open-Domain QA with Dense-Sparse Phrase Index
•
2 likes•401 views
2019/09/05 LGCNS AI Tech Talk for NLU (feat.KorQuAD) - Naver Clova AI 서민준 님 - accepted for 2019 ACL - Real-Time Open-Domain QA with Dense-Sparse Phrase Index
Recommended
More Related Content
Recently uploaded
Recently uploaded (20)
Featured
Featured (20)
Real-Time Open-Domain QA with Dense-Sparse Phrase Index
- 1. Real-Time Open-Domain QA with Dense-Sparse Phrase Index Minjoon Seo*, Jinhyuk Lee*, Tom Kwiatkowski, Ankur Parikh, Ali Farhadi, Hannaneh Hajishirzi ACL 2019 presented at LG CNS AI Tech Talk on Sep 5, 2019 * denotes equal contribution
- 3. Some Model When was Obama born? 1961 5 Million documents 3 Billion tokens
- 4. Information Retrieval Reader (Model) When was Obama born? 1961 Chen et al., 2017 Retrieve & Read TF-IDF, BM-25, LSA 1. Error propagation: reading only 5-10 docs 2. Query-dependent encoding: 30s+ per query
- 5. We want… • To ”read” entire Wikipedia • 5-10 docs 5 Million docs • Reach long-tail answers • Fast inference on CPUs • 35s 0.5s • Maintain high accuracy HOW?
- 6. Our approach: index phrases!
- 7. Barack Obama … … (1961-present … … 44th President … … United States. Who is the 44th President of the U.S.?Nearest neighbor search When was Obama born? “Barack Obama (1961-present) was the 44th President of the United States.” Phrase encoding Question encoding Phrase Indexing Seo et al., 2018
- 8. [-3, 0.1, …] [0.3, -0.2, …] [0.5, 0.1, …] [0.7, -0.4, …] [0.5, 0.0, …] [3.3, -2.2, …] When was Obama born? Nearest neighbor search [0.5, 0.1, …] Document Indexing - Locality Sensitive Hashing (LSH) - aLSH (Shrivastava & Li, 2014) - HNSW (Malkov & Yashunin, 2018)
- 9. 𝑎 = argmax 𝑎 𝐹𝜃 𝑎, 𝑞, 𝑑 𝑎 = argmax 𝑎 𝐺 𝜃(𝑞) ∙ 𝐻 𝜃(𝑎, 𝑑) Model phrase question document Query-Agnostic Decomposition Question encoder Phrase encoder
- 10. Phrase (and question) Representation • Dense representation • Can utilize deep neural networks • great for capturing semantic and syntactic information • Not great for disambiguating ”Einstein” vs “Tesla” • Sparse representation (bag-of-word) • Great for capturing lexical information • Represent each phrase with a concatenation of both
- 11. Dense-Sparse Phrase Index (DenSPI) … When was Barack Obama born? Reader Model 1961 When was Barack Obama born? … Query vector for documentDense 1961 Sparse Retrieve & Read (Chen et al., 2017) Phrase Index Document Index DenSPI Ours N = 60 Billion N = 5 Million
- 12. Dense Representation for Phrases According Text Encoder (BERT) to the American Library Association dot Start vector End vector Coherency vector Coherency scalar We want encoding for this phrase phrase vector
- 13. Dense Representation for Questions [CLS] Text Encoder (BERT) When was Barack Obama born? Start vector End vector Coherency vector Coherency scalar1 question vector
- 14. Sparse Representation • TF-IDF document & paragraph vector, computed over Wikipedia • Unigram & Bigram (vocab size = 17 Million) • Adopted DrQA’s vocab/TF-IDF (Chen et al., 2017)
- 15. Beware of the scale… • 60 Billion phrases in Wikipedia! • Training • Softmax on 60 Billion phrases? • Storage • 60 Billion phrases x 4 KB per phrase = 240 TB? • Search • Exact search on 60 Billion phrases? We want to be open-research-friendly 4 GPUs, 64 GB RAM, 2 TB Storage, One Week
- 16. Training • Close-domain QA dataset: the model can easily overfit • e.g. ”who” question when only one named entity in the context • Negative sampling and concatenation • Sampling strategy is crucial • Use query encoder to associate similar questions in training set • Concatenate the context that the similar question belongs to
- 17. Storage • 60 Billion phrases x 4 KB per phrase = 240 TB! 1. Pointer: share start and end vectors • 240 TB 12 TB 2. Filter: 1-layer classifier on phrase vectors • 12 TB 4.5 TB 3. Scalar Quantization: 4 bytes 1 byte per dim • 4.5 TB 1.5 TB
- 18. Search • An open-source library for large-scale dense+sparse nearest neighbor search is non-existent • Dense-first search (DFS) • faiss (Johnson et al., 2017) • Sparse-first search (SFS) • similar to retrieve & read • Hybrid
- 19. Experiments
- 20. Weaver (Raison et al., 2018) BERTserini (Yang et al., 2019) 42% EM 39% EM DrQA (Chen et al., 2017) 30% EM DenSPI (Ours) 36% EM 35 s/Q Open-Domain SQuAD Red color is query- agnostic. 0.8 s/Q 144x 44x Multi-step reasoner (Das et al., 2019) 32% EM MINIMAL (Min et al., 2018) 35% EM 115 s/Q
- 21. Qualitative Comparisons Q: What can hurt a teacher’s mental and physical health? … and poor mental health can lead to problems such as substance abuse. Teachers face several occupational hazards in their line of work, including occupational stress… Mental health Teacher Retrieve & Read (Chen et al., 2017) DenSPI (Ours)
- 22. Q: Who was Kennedy’s science adviser that opposed manned spacecraft flights? Kennedy’s science advisor Jerome Wiesner, … his opposition to manned spaceflight … Apollo program … and the sun by NASA manager Abe Silverstein, who later said that … Apollo program Although Grumman wanted a second unmanned test, George Low decided … be manned. Apollo program Kennedy’s science advisor Jerome Wiesner, … his opposition to manned spaceflight … Apollo program Jerome Wiesner of MIT, who served as a … advisor to … Kennedy, … opponent of manned Space Race … science advisor Jerome Wiesner … strongly opposed to manned space exploration, … John F. Kennedy
- 23. Q: What is the best thing to do when bored? I’m nearly bored to death Bored to Death (song) The twin tunnels were bored by … tunnel boring machine (TBM) … Waterview Connection It’s easier to say you’re bored, or to be angry, than it is to be sad. Bored to Death (song)
- 25. Q: What is the best thing to do when bored? I’m nearly bored to death Bored to Death (song) The twin tunnels were bored by … tunnel boring machine (TBM) … Waterview Connection It’s easier to say you’re bored, or to be angry, than it is to be sad. Bored to Death (song) When bored, she enjoys drawing. Big Brother 2 he can think of a much more fun thing he can do while on his back: painting. Angry Kid She is a live music goer, and her hobby is watching movies. Pearls Before Swine
- 27. Conclusion • “Read” entire Wikipedia in 0.5s with CPUs • Query-agnostic, indexable phrase representations • Utilize both dense (BERT-based) and sparse (bag-of-word) representations for encoding lexical, syntactic, and semantic information • 6,000x lower computational cost with higher accuracy for exact search • At least 44x faster open-domain QA with higher accuracy • (query-agnostic) decomposability gap still exists (6-10%); we hope future research can close the gap
- 28. Thank you! Questions? • http://nlp.cs.washington.edu/denspi @seo_minjoon