4th Natural Language Interface over the Web of Data (NLIWoD) workshop and QALD-9 Question Answering over Linked Data Challenge

1. 4th Natural Language Interface over the Web of Data (NLIWoD) workshop and QALD-9 Question Answering over Linked Data Challenge Presenter: Prof. Key-Sun Choi and Dr. Muhammad Saleem NLIWoD 4 and QALD-9 @ ISWC 2018 Monterey, USA Horizon 2020, GA No 688227 9th October 2018 Usbeck (DICE Data Science Group, University Paderborn) NLIWoD and QALD-9 9th October 2018 1 / 17

2. Organization committee Key-Sun Choi KAIST, Korea Jin-Dong Kim Database Center for Life Science, Japan Axel-Cyrille Ngonga Ngomo Paderborn University, Germany Muhammad Saleem Leipzig University, Germany Ricardo Usbeck Paderborn University, Germany Usbeck (DICE Data Science Group, University Paderborn) NLIWoD and QALD-9 9th October 2018 2 / 17

3. Program committee Kody Moodley, Maastricht University Grigorios Tzortzis, NCSR Demokritos Vanessa Lopez, IBM Dennis Diefenbach, University Jean Monet Kuldeep Singh, Fraunhofer IAIS Edgard Marx, Leipzig University of Applied Sciences (HTWK) Raghava Mutharaju, IIIT-Delhi Subhabrata Mukherjee, Max Planck Institute for Informatics Varish Mulwad, GE Global Research Roberto Garcia, Universitat de Lleida Giorgos Giannopoulos, Imis Institute, "Athena" R.C. Usbeck (DICE Data Science Group, University Paderborn) NLIWoD and QALD-9 9th October 2018 3 / 17

4. Overview NLIWoD-3 Time Author Title 09.00-09.05 Key-Sun Choi Introduction 09.05-09.45 Peter F. Patel-Schneider Keynote: "Connecting Industrial NL Applications to Knowledge (in Nuance)" 09.45-10.15 Richard Frost and Shane Peelar An Extensible Natural-Language Query Interface to an Event- Based Semantic Web 10.15-10.30 Younggyun Hahm, Jiho Kim, Sangmin An, Minho Lee and Key-Sun Choi Chatbot Who Wants to Learn the Knowledge: KB-Agent 10:30-11:00 Coﬀee Break 11.00-11.20 Muhammad Saleem QALD 9 Challenge Overview and Evaluation 11.20-11.50 Jiho Kim, Sangha Nam and Key- Sun Choi Open Relation Extraction by Matrix Factorization and Univer- sal Schemas 11.50 - end Kyriaki Zafeiroudi, Leah Eck- man and Rebecca Passonneau Best Paper: Testing a Knowl- edge Inquiry System on Ques- tion Answering Usbeck (DICE Data Science Group, University Paderborn) NLIWoD and QALD-9 9th October 2018 4 / 17

5. Keynote Please welcome Peter F. Patel Schneider to his keynote: ’Connecting Industrial NL Applications to Knowledge (in Nuance)’ Usbeck (DICE Data Science Group, University Paderborn) NLIWoD and QALD-9 9th October 2018 5 / 17

6. Keynote Coﬀee break! Usbeck (DICE Data Science Group, University Paderborn) NLIWoD and QALD-9 9th October 2018 6 / 17

7. Overview Question Answering Question answering systems mediate between An user expressing an information need in natural language RDF-modelled data Usbeck (DICE Data Science Group, University Paderborn) NLIWoD and QALD-9 9th October 2018 7 / 17

8. Overview QALD QALD is a series of evaluation campaigns that provide a benchmark for comparing diﬀerent approaches and systems Get a picture of their strengths and shortcomings Gain insight into how we can develop approaches that deal with Semantic Web data as a knowledge source QALD-1 @ ESWC 2011 (3) QALD-2 @ ESWC 2012 (4) QALD-3 @ CLEF 2013 (6) QALD-4 @ CLEF 2014 QA track (9) QALD-5 @ CLEF 2015 QA track (7) QALD-6 @ ESWC 2016 (13) QALD-7 @ ESWC 2017 (3) QALD-8 @ ISWC 2017 (8/3) QALD-9 @ ISWC 2018 (6/5) Usbeck (DICE Data Science Group, University Paderborn) NLIWoD and QALD-9 9th October 2018 8 / 17

9. Tasks Overall task Given a natural language question, retrieve the correct answer(s) from a given RDF repository. Types of challenges (speciﬁc tasks): 1 Multilingual 2 Wikidata Usbeck (DICE Data Science Group, University Paderborn) NLIWoD and QALD-9 9th October 2018 9 / 17

10. Tasks Overall task Given a natural language question, retrieve the correct answer(s) from a given RDF repository. Types of challenges (speciﬁc tasks): 1 Multilingual 2 Wikidata Canceled Usbeck (DICE Data Science Group, University Paderborn) NLIWoD and QALD-9 9th October 2018 10 / 17

11. Task 1 - Multilingual questions Dataset: DBpedia 2016-10 (with multilingual labels) Challenge: Lexical and structural gap between natural language expressions and data, e.g. high → elevation have inhabitants → populationTotal graduate from → almaMater Usbeck (DICE Data Science Group, University Paderborn) NLIWoD and QALD-9 9th October 2018 11 / 17

12. Task 1 - Multilingual questions Dataset: DBpedia 2016-10 (with multilingual labels) Challenge: Lexical and structural gap between natural language expressions and data, e.g. high → elevation have inhabitants → populationTotal graduate from → almaMater Questions: 413 training, 150 test questions (out of which 50 are novel) Provided in diﬀerent languages Can be answered with respect to the provided RDF data Annotated with corresponding SPARQL queries and answers QALD 9 test stems partly form chatbot logs http://chat.dbpedia.org The largest QALD ever! Usbeck (DICE Data Science Group, University Paderborn) NLIWoD and QALD-9 9th October 2018 11 / 17

13. Example Which book has the most pages? Welches Buch hat die meisten Seiten? Quale libro ha il maggior numero di pagine? Quel livre a le plus de pages? ¿Que libro tiene el mayor numero de paginas? . . . Usbeck (DICE Data Science Group, University Paderborn) NLIWoD and QALD-9 9th October 2018 12 / 17

14. Data experts for creating QALD-9 Rricha Jalota, Paderborn University, Germany Paramjot Kauer, Paderborn University, Germany Abdullah Ahmed, Paderborn University, Germany Danish Ahmed, Paderborn University, Germany Nikit Srivasta, Paderborn University, Germany Michael Röder, Paderborn University, Germany Jan Reineke, Paderborn University, Germany Alexander Bigerl, Paderborn University, Germany Afshin Amini, Paderborn University, Germany Geraldo De Souza, Paderborn University, Germany Felix Conrads, Paderborn University, Germany and InfAI e.V. Leipzig Usbeck (DICE Data Science Group, University Paderborn) NLIWoD and QALD-9 9th October 2018 13 / 17

15. Participants in Task 1 - English Dennis Diefenbach - Université Jean Monnet, Saint-Étienne WDAqua-core1: DBpedia http://wdaqua.eu/qa Task 1, English, French Sen Hu - School of Electronics Engineering and Computer Science, Peking University gAnswer http://ganswer.gstore-pku.com/api/qald.jsp? Task 1, English Peter Nancke et al. - Leipzig University, Germany TeBaQA http://139.18.2.39:8187/ Task 1, English Szabó Bence et al. - Paderborn University, Germany Elon http://qald-beta.cs.upb.de:443/ Task 1, English Lukas Blübaum and Nick Düsterhus - Paderborn University Germany QASystem http://qald-beta.cs.upb.de:80/ Task 1, English Usbeck (DICE Data Science Group, University Paderborn) NLIWoD and QALD-9 9th October 2018 14 / 17

16. And the winner... Curve Balls: Real world queries (dirty!), new query forms (nasty!), weak deﬁned answer types (not nice!) Usbeck (DICE Data Science Group, University Paderborn) NLIWoD and QALD-9 9th October 2018 15 / 17

17. And the winner... Curve Balls: Real world queries (dirty!), new query forms (nasty!), weak deﬁned answer types (not nice!) All QA systems were run on QALD-9 train and test dataset in English and GERBIL QA version 0.2.3 Usbeck (DICE Data Science Group, University Paderborn) NLIWoD and QALD-9 9th October 2018 15 / 17

18. And the winner... Curve Balls: Real world queries (dirty!), new query forms (nasty!), weak deﬁned answer types (not nice!) All QA systems were run on QALD-9 train and test dataset in English and GERBIL QA version 0.2.3 More details at: www.semantic-web-journal.net/content/ benchmarking-question-answering-systems FAIR experiment data for training and test dataset at http://w3id.org/gerbil/qa/experiment?id=201810080002 http://w3id.org/gerbil/qa/experiment?id=201810060001 Usbeck (DICE Data Science Group, University Paderborn) NLIWoD and QALD-9 9th October 2018 15 / 17

19. And the winner... ...is gAnswer! Annotator Macro Precision Macro Recall Macro F1 Error Count Average Time/Doc ms Macro F1 QALD Elon (WS) 0.049 0.053 0.050 2 219 0.100 QASystem (WS) 0.097 0.116 0.098 0 1014 0.200 TeBaQA (WS) 0.129 0.134 0.130 0 2668 0.222 wdaqua-core1 (DBpedia) 0.261 0.267 0.250 0 661 0.289 gAnswer (WS) 0.293 0.327 0.298 1 3076 0.430 Usbeck (DICE Data Science Group, University Paderborn) NLIWoD and QALD-9 9th October 2018 16 / 17

20. That’s all Folks! Thank you! Questions? Data Science@UPB Follow us on Twitter @DiceResearch Usbeck (DICE Data Science Group, University Paderborn) NLIWoD and QALD-9 9th October 2018 17 / 17

4th Natural Language Interface over the Web of Data (NLIWoD) workshop and QALD-9 Question Answering over Linked Data Challenge

Holistic Benchmarking of Big Linked Data

4th Natural Language Interface over the Web of Data (NLIWoD) workshop and QALD-9 Question Answering over Linked Data Challenge