Dynamic Question Answer Generator An Enhanced Approach to Question Generationijtsrd
Â
Teachers and educational institutions seek new questions with different difficulty levels for setting up tests for their students. Also, students long for distinct and new questions to practice for their tests as redundant questions are found everywhere. However, setting up new questions every time is a tedious task for teachers. To overcome this conundrum, we have concocted an artificially intelligent system which generates questions and answers for the mathematical topic ââŹâQuadratic equations. The system uses i Randomization technique for generating unique questions each time and ii First order logic and Automated deduction to produce solution for the generated question. The goal was achieved and the system works efficiently. It is robust, reliable and helpful for teachers, students and other organizations for retrieving Quadratic equations questions, hassle free. Rahul Bhatia | Vishakha Gautam | Yash Kumar | Ankush Garg ""Dynamic Question Answer Generator: An Enhanced Approach to Question Generation"" Published in International Journal of Trend in Scientific Research and Development (ijtsrd), ISSN: 2456-6470, Volume-3 | Issue-4 , June 2019, URL: https://www.ijtsrd.com/papers/ijtsrd23730.pdf
Paper URL: https://www.ijtsrd.com/computer-science/artificial-intelligence/23730/dynamic-question-answer-generator-an-enhanced-approach-to-question-generation/rahul-bhatia
French machine reading for question answeringAli Kabbadj
Â
This paper proposes to unlock the main barrier to machine reading and comprehension French natural language texts. This open the way to machine to find to a question a precise answer buried in the mass of unstructured French texts. Or to create a universal French chatbot. Deep learning has produced extremely promising results for various tasks in natural language understanding particularly topic classification, sentiment analysis, question answering, and language translation. But to be effective Deep Learning methods need very large training da-tasets. Until now these technics cannot be actually used for French texts Question Answering (Q&A) applications since there was not a large Q&A training dataset. We produced a large (100 000+) French training Dataset for Q&A by translating and adapting the English SQuAD v1.1 Dataset, a GloVe French word and character embed-ding vectors from Wikipedia French Dump. We trained and evaluated of three different Q&A neural network ar-chitectures in French and carried out a French Q&A models with F1 score around 70%.
Hi! Take a look at this article with the best capstone project final report sample. For more visit site https://www.capstoneproject.net/capstone-project-final-report/
USING MACHINE LEARNING TO BUILD A SEMI-INTELLIGENT BOT ecij
Â
Nowadays, real-time systems and intelligent systems offer more and more control interface based on voice recognition or human language recognition. Robots and drones will soon be mainly controlled by voice. Other robots will integrate bots to interact with their users, this can be useful both in industry and entertainment. At first, researchers were digging on the side of "ontology reasoning". Given all the technical constraints brought by the treatment of ontologies, an interesting solution has emerged in last years: the construction of a model based on machine learning to connect a human language to a knowledge
base (based for example on RDF). We present in this paper our contribution to build a bot that could be used on real-time systems and drones/robots, using recent machine learning technologies.
USING MACHINE LEARNING TO BUILD A SEMI-INTELLIGENT BOT ecij
Â
Nowadays, real-time systems and intelligent systems offer more and more control interface based on voice
recognition or human language recognition. Robots and drones will soon be mainly controlled by voice.
Other robots will integrate bots to interact with their users, this can be useful both in industry and
entertainment. At first, researchers were digging on the side of "ontology reasoning". Given all the
technical constraints brought by the treatment of ontologies, an interesting solution has emerged in last
years: the construction of a model based on machine learning to connect a human language to a knowledge
base (based for example on RDF). We present in this paper our contribution to build a bot that could be
used on real-time systems and drones/robots, using recent machine learning technologies.
This document presents a system for detecting semantically similar questions in online forums like Quora to reduce duplicate content. It proposes using natural language processing techniques like tagging questions with keywords, vectorizing text with Google News vectors, and calculating similarity with Word Mover's Distance. The system cleans and preprocesses questions before generating tags and calculating similarity between questions to identify duplicates. An evaluation of the system achieved accurate detection of matching and non-matching question pairs.
Question Retrieval in Community Question Answering via NON-Negative Matrix Fa...IRJET Journal
Â
The document proposes using statistical machine translation via non-negative matrix factorization to address word ambiguity and mismatch problems in question retrieval for community question answering systems. It translates questions into other languages using Google Translate to leverage contextual information, representing the original and translated questions together in a matrix. Experimental results on a real CQA dataset show this approach improves over methods relying only on surface text matching.
RAPID INDUCTION OF MULTIPLE TAXONOMIES FOR ENHANCED FACETED TEXT BROWSINGijaia
Â
In this paper we present and compare two methodologies for rapidly inducing multiple subject-specific
taxonomies from crawled data. The first method involves a sentence-level words co-occurrence frequency
method for building the taxonomy, while the second involves the bootstrapping of a Word2Vec based
algorithm with a directed crawler. We exploit the multilingual open-content directory of the World Wide
Web, DMOZ1
to seed the crawl, and the domain name to direct the crawl. This domain corpus is then input
to our algorithm that can automatically induce taxonomies. The induced taxonomies provide hierarchical
semantic dimensions for the purposes of faceted browsing. As part of an ongoing personal semantics
project, we applied the resulting taxonomies to personal social media data (Twitter, Gmail, Facebook,
Instagram, Flickr) with an objective of enhancing an individualâs exploration of their personal information
through faceted searching. We also perform a comprehensive corpus based evaluation of the algorithms
based on many datasets drawn from the fields of medicine (diseases) and leisure (hobbies) and show that
the induced taxonomies are of high quality.
Dynamic Question Answer Generator An Enhanced Approach to Question Generationijtsrd
Â
Teachers and educational institutions seek new questions with different difficulty levels for setting up tests for their students. Also, students long for distinct and new questions to practice for their tests as redundant questions are found everywhere. However, setting up new questions every time is a tedious task for teachers. To overcome this conundrum, we have concocted an artificially intelligent system which generates questions and answers for the mathematical topic ââŹâQuadratic equations. The system uses i Randomization technique for generating unique questions each time and ii First order logic and Automated deduction to produce solution for the generated question. The goal was achieved and the system works efficiently. It is robust, reliable and helpful for teachers, students and other organizations for retrieving Quadratic equations questions, hassle free. Rahul Bhatia | Vishakha Gautam | Yash Kumar | Ankush Garg ""Dynamic Question Answer Generator: An Enhanced Approach to Question Generation"" Published in International Journal of Trend in Scientific Research and Development (ijtsrd), ISSN: 2456-6470, Volume-3 | Issue-4 , June 2019, URL: https://www.ijtsrd.com/papers/ijtsrd23730.pdf
Paper URL: https://www.ijtsrd.com/computer-science/artificial-intelligence/23730/dynamic-question-answer-generator-an-enhanced-approach-to-question-generation/rahul-bhatia
French machine reading for question answeringAli Kabbadj
Â
This paper proposes to unlock the main barrier to machine reading and comprehension French natural language texts. This open the way to machine to find to a question a precise answer buried in the mass of unstructured French texts. Or to create a universal French chatbot. Deep learning has produced extremely promising results for various tasks in natural language understanding particularly topic classification, sentiment analysis, question answering, and language translation. But to be effective Deep Learning methods need very large training da-tasets. Until now these technics cannot be actually used for French texts Question Answering (Q&A) applications since there was not a large Q&A training dataset. We produced a large (100 000+) French training Dataset for Q&A by translating and adapting the English SQuAD v1.1 Dataset, a GloVe French word and character embed-ding vectors from Wikipedia French Dump. We trained and evaluated of three different Q&A neural network ar-chitectures in French and carried out a French Q&A models with F1 score around 70%.
Hi! Take a look at this article with the best capstone project final report sample. For more visit site https://www.capstoneproject.net/capstone-project-final-report/
USING MACHINE LEARNING TO BUILD A SEMI-INTELLIGENT BOT ecij
Â
Nowadays, real-time systems and intelligent systems offer more and more control interface based on voice recognition or human language recognition. Robots and drones will soon be mainly controlled by voice. Other robots will integrate bots to interact with their users, this can be useful both in industry and entertainment. At first, researchers were digging on the side of "ontology reasoning". Given all the technical constraints brought by the treatment of ontologies, an interesting solution has emerged in last years: the construction of a model based on machine learning to connect a human language to a knowledge
base (based for example on RDF). We present in this paper our contribution to build a bot that could be used on real-time systems and drones/robots, using recent machine learning technologies.
USING MACHINE LEARNING TO BUILD A SEMI-INTELLIGENT BOT ecij
Â
Nowadays, real-time systems and intelligent systems offer more and more control interface based on voice
recognition or human language recognition. Robots and drones will soon be mainly controlled by voice.
Other robots will integrate bots to interact with their users, this can be useful both in industry and
entertainment. At first, researchers were digging on the side of "ontology reasoning". Given all the
technical constraints brought by the treatment of ontologies, an interesting solution has emerged in last
years: the construction of a model based on machine learning to connect a human language to a knowledge
base (based for example on RDF). We present in this paper our contribution to build a bot that could be
used on real-time systems and drones/robots, using recent machine learning technologies.
This document presents a system for detecting semantically similar questions in online forums like Quora to reduce duplicate content. It proposes using natural language processing techniques like tagging questions with keywords, vectorizing text with Google News vectors, and calculating similarity with Word Mover's Distance. The system cleans and preprocesses questions before generating tags and calculating similarity between questions to identify duplicates. An evaluation of the system achieved accurate detection of matching and non-matching question pairs.
Question Retrieval in Community Question Answering via NON-Negative Matrix Fa...IRJET Journal
Â
The document proposes using statistical machine translation via non-negative matrix factorization to address word ambiguity and mismatch problems in question retrieval for community question answering systems. It translates questions into other languages using Google Translate to leverage contextual information, representing the original and translated questions together in a matrix. Experimental results on a real CQA dataset show this approach improves over methods relying only on surface text matching.
RAPID INDUCTION OF MULTIPLE TAXONOMIES FOR ENHANCED FACETED TEXT BROWSINGijaia
Â
In this paper we present and compare two methodologies for rapidly inducing multiple subject-specific
taxonomies from crawled data. The first method involves a sentence-level words co-occurrence frequency
method for building the taxonomy, while the second involves the bootstrapping of a Word2Vec based
algorithm with a directed crawler. We exploit the multilingual open-content directory of the World Wide
Web, DMOZ1
to seed the crawl, and the domain name to direct the crawl. This domain corpus is then input
to our algorithm that can automatically induce taxonomies. The induced taxonomies provide hierarchical
semantic dimensions for the purposes of faceted browsing. As part of an ongoing personal semantics
project, we applied the resulting taxonomies to personal social media data (Twitter, Gmail, Facebook,
Instagram, Flickr) with an objective of enhancing an individualâs exploration of their personal information
through faceted searching. We also perform a comprehensive corpus based evaluation of the algorithms
based on many datasets drawn from the fields of medicine (diseases) and leisure (hobbies) and show that
the induced taxonomies are of high quality.
May 2024 - Top10 Cited Articles in Natural Language Computingkevig
Â
Natural Language Processing is a programmed approach to analyze text that is based on both a set of theories and a set of technologies. This forum aims to bring together researchers who have designed and build software that will analyze, understand, and generate languages that humans use naturally to address computers.
Ontology Based Approach for Semantic Information Retrieval SystemIJTET Journal
Â
AbstractâThe Information retrieval system is taking an important role in current search engine which performs searching operation based on keywords which results in an enormous amount of data available to the user, from which user cannot figure out the essential and most important information. This limitation may be overcome by a new web architecture known as the semantic web which overcome the limitation of the keyword based search technique called the conceptual or the semantic search technique. Natural language processing technique is mostly implemented in a QA system for asking userâs questions and several steps are also followed for conversion of questions to the query form for retrieving an exact answer. In conceptual search, search engine interprets the meaning of the userâs query and the relation among the concepts that document contains with respect to a particular domain that produces specific answers instead of showing lists of answers. In this paper, we proposed the ontology based semantic information retrieval system and the Jena semantic web framework in which, the user enters an input query which is parsed by Standford Parser then the triplet extraction algorithm is used. For all input queries, the SPARQL query is formed and further, it is fired on the knowledge base (Ontology) which finds appropriate RDF triples in knowledge base and retrieve the relevant information using the Jena framework.
HyperQA: A Framework for Complex Question-AnsweringJinho Choi
Â
This abstract describes the overall framework of our question-answering system designed to answer various types of complex questions. Our framework makes heavy use of natural language processing techniques for the retrieval, ranking, and generation of correct answers. Our approach has been tested on answering arithmetic questions requiring logical reasoning as well as higher-order factoid questions aggregating information across different documents.
This document provides a thorough survey of recent advances in open-domain textual question answering systems based on deep learning. It begins with introducing the background and motivation for open-domain question answering and deep learning approaches. The document then defines the scope of open-domain textual question answering. The main contributions of the survey are summarized as systematically reviewing open-domain question answering models based on deep learning, introducing recent models and comparing their performance, and discussing challenges and future directions. The survey is organized into sections on the overview of open-domain question answering systems, key components of these systems, datasets and metrics, followed by conclusions and future work.
LSTM Model for Semantic Clustering of User-Generated Content Using AI Geared ...IRJET Journal
Â
The document describes a system called Smart Reply that uses machine learning to automatically generate short message responses on wearable devices. It uses an LSTM neural network model trained on large amounts of conversation data to predict likely responses given an incoming message. It addresses challenges of response quality, diversity, scalability and privacy. The system clusters messages into semantic groups using a graph-based semi-supervised learning technique to ensure responses cover different intents. It has been successfully deployed and handles 10% of message replies on wearable devices.
Question Answering has been a well-researched NLP area over recent years. It has become necessary for
users to be able to query through the variety of information available - be it structured or unstructured. In
this paper, we propose a Question Answering module which a) can consume a variety of data formats - a
heterogeneous data pipeline, which ingests data from product manuals, technical data forums, internal
discussion forums, groups, etc. b) addresses practical challenges faced in real-life situations by pointing to
the exact segment of the manual or chat threads which can solve a user query c) provides segments of texts
when deemed relevant, based on user query and business context. Our solution provides a comprehensive
and detailed pipeline that is composed of elaborate data ingestion, data parsing, indexing, and querying
modules. Our solution is capable of handling a plethora of data sources such as text, images, tables,
community forums, and flow charts. Our studies performed on a variety of business-specific datasets
represent the necessity of custom pipelines like the proposed one to solve several real-world document
question-answering
The document describes an intelligent question answering system that can leverage heterogeneous datasets including product manuals, technical forums, and discussion threads. The system includes four main modules: 1) A document parser that can parse different data types including text, images, tables, and forums using deep learning models. 2) A document indexer that indexes documents for retrieval. 3) A document retriever that handles query processing and identifies relevant text segments. 4) A document reader that provides answers by analyzing relevant text segments. The system aims to reduce the time needed to find answers across different data sources by automatically identifying the most relevant information for a given question.
Exploring Semantic Question Generation Methodology and a Case Study for Algor...IJCI JOURNAL
Â
Assessment of student performance is one of the most important tasks in the educational process. Thus, formulating questions and creating tests takes the instructor a lot of time and effort. However, the time spent for learning acquisition and on exam preparation could be utilized in better ways. With the technical development in representing and linking data, ontologies have been used in academic fields to represent the terms in a field by defining concepts and categories classifies the subject. Also, the emergence of such methods that represent the data and link it logically contributed to the creation of methods and tools for creating questions. These tools can be used in existing learning systems to provide effective solutions to assist the teacher in creating test questions. This research paper introduces a semantic methodology for automating question generation in the domain of Algorithms. The primary objective of this approach is to support instructors in effectively incorporating automatically generated questions into their instructional practice, thereby enhancing the teaching and learning experience.
MULTI-LEARNING SPECIAL SESSION / EDUCON 2018 / EMADRID TEAMeMadrid network
Â
1) The document proposes an approach to assist course creators in generating or restructuring courses by exploiting text mining techniques, semantic information from DBpedia, and linking educational resources.
2) The approach was implemented as a prototype that retrieves online courses, identifies key elements from text, formulates queries to other courses, and returns related courses to help creators generate mashups.
3) Preliminary tests on 265 computer science courses showed promising results, though future work is needed to improve similarity measures and generate concept maps between related courses.
This document proposes a model for automatically clustering Thai students' online homework assignments before teachers grade them. The model uses five parts: 1) Thai word segmentation, 2) stop-word removal, 3) term weighting, 4) document clustering using k-means, and 5) performance evaluation. The model was tested on 1,000 student assignments and achieved high accuracy, purity, and F-measure scores similar to human grading, allowing teachers to grade assignments more efficiently.
This document is a thesis that proposes using word embeddings to improve information retrieval by addressing term mismatch issues. It discusses word2vec, a technique for learning word embeddings from large text corpora that capture semantic relationships between words. The thesis proposes two approaches: 1) incorporating word embedding similarities into a probabilistic language model for retrieval and 2) a vector space model. Due to time constraints, only the first approach is implemented, which integrates word embeddings into ALMasri and Chevallet's probabilistic language model. Experiments are conducted to evaluate the impact of using semantic features from word embeddings on retrieval effectiveness.
The document describes a study that used large language models (LLMs) like GPT-3 and GPT-4 to complete knowledge from Wikidata. The researchers developed a pipeline called LLMKE that combines knowledge probing and Wikidata entity mapping. They were able to achieve a macro-averaged F1-score of 0.701 on the ISWC 2023 LM-KBC Challenge, with scores ranging from 1.00 to 0.328 depending on the domain. The results show that LLMs have varying knowledge depending on the domain and more work is needed to determine when they can be used for automatic knowledge base completion and correction.
Robotics-Based Learning in the Context of Computer ProgrammingJacob Storer
Â
This document is a project report for research into whether robotics-based learning or simulation-based learning is more effective for teaching programming. It describes the objectives of developing tutorials for both an Arduino robot and visual basic simulation. Programming tasks for moving forwards/backwards and along shapes were developed. Tutorials and programs were implemented to teach these tasks. Surveys were given to test groups after using each method to collect data on their effectiveness for comparison. While results were mixed, all indicated learning was improved with a teacher. Due to the small sample size, no conclusive answer could be provided.
AUTOMATIC QUESTION GENERATION USING NATURAL LANGUAGE PROCESSINGIRJET Journal
Â
The document describes a proposed method for automatic question generation using natural language processing and T5 text-to-text transfer transformer models. The method uses T5 models trained on the Stanford Question Answering Dataset to generate questions from paragraphs of text without requiring extensive grammar rules. The proposed system aims to assist students in learning by generating questions to test their understanding from provided materials.
Categorization of Protean Writers by Exploitation of Raspberry Piijtsrd
Â
Raspberry Pi is a computer though smaller but has versatile functionality. These are useful in assisting variety of educational institutions for teaching and other investigational indagations. In this paper, three prominent Bangladeshi writer's works were catalogued by using Raspberry Pi 3. The significance of this research pivots on low cost computational teaching in different institutions. Mathematica was used for this purpose and it's comprised of two modules which effectively communicate and shape an effective interpreter. It's free on Raspberry. After gathering literature and amassing all text files in BD writer's database, application of variety of algorithms were done for categorization. Then experimental analysis was done. Among different categorizers Markov and NaĂÂŻve Bayes have high precision and have best training times. This research will help in further distinguishing of languages by using an economical approach and will assist in further investigational studies for better understanding. Pritom Sarker | Jannatul Ferdous | Nakib Aman Turzo | Biplob Kumar | Jyotirmoy Ghose "Categorization of Protean Writers by Exploitation of Raspberry Pi" Published in International Journal of Trend in Scientific Research and Development (ijtsrd), ISSN: 2456-6470, Volume-4 | Issue-4 , June 2020, URL: https://www.ijtsrd.com/papers/ijtsrd31332.pdf Paper Url :https://www.ijtsrd.com/engineering/computer-engineering/31332/categorization-of-protean-writers-by-exploitation-of-raspberry-pi/pritom-sarker
Project MLExAI: Machine Learning Experiences in AIbutest
Â
The project aims to develop a framework for teaching core AI topics with a focus on machine learning. It involves hands-on term projects where students design and implement learning systems for applications like web document classification, web user profiling, and character recognition. Preliminary results found students had positive experiences applying machine learning concepts to real-world problems.
Project MLExAI: Machine Learning Experiences in AIbutest
Â
The project aims to develop a framework for teaching core AI topics with a focus on machine learning. It involves hands-on term projects where students design and implement learning systems for applications like web document classification, user profiling, and character recognition. Preliminary results found students had positive experiences applying machine learning concepts to real-world problems.
Naver learning to rank question answer pairs using hrde-ltcNAVER Engineering
Â
The automatic question answering (QA) task has long been considered a primary objective of artificial intelligence.
Among the QA sub-systems, we focused on answer-ranking part. In particular, we investigated a novel neural network architecture with additional data clustering module to improve the performance in ranking answer candidates which are longer than a single sentence. This work can be used not only for the QA ranking task, but also to evaluate the relevance of next utterance with given dialogue generated from the dialogue model.
In this talk, I'll present our research results (NAACL 2018), and also its potential use cases (i.e. fake news detection). Finally, I'll conclude by introducing some issues on previous research, and by introducing recent approach in academic.
Expository Essay Reflection Paper In. Online assignment writing service.Lori Mitchell
Â
The document describes the steps to get writing help from HelpWriting.net:
1. Create an account with a password and email.
2. Complete a 10-minute order form providing instructions, sources, and deadline. Attach sample work if wanting the writer to mimic your style.
3. Review bids from writers and choose one based on qualifications, history, and feedback. Place a deposit to start the assignment.
4. Ensure the completed paper meets expectations. Authorize full payment for satisfactory work or request revisions.
Handwriting Without Tears Paper WITH Picture HanLori Mitchell
Â
The document discusses the steps to get writing help from the website HelpWriting.net, which includes creating an account, submitting a request form with instructions and deadline, and choosing a writer to complete the assignment. Writers will bid on requests and the client can choose a writer and provide a deposit to start the work. Clients can then review the completed paper and request revisions until satisfied before authorizing final payment.
More Related Content
Similar to ARQMath Lab An Incubator For Semantic Formula Search In ZbMATH Open
May 2024 - Top10 Cited Articles in Natural Language Computingkevig
Â
Natural Language Processing is a programmed approach to analyze text that is based on both a set of theories and a set of technologies. This forum aims to bring together researchers who have designed and build software that will analyze, understand, and generate languages that humans use naturally to address computers.
Ontology Based Approach for Semantic Information Retrieval SystemIJTET Journal
Â
AbstractâThe Information retrieval system is taking an important role in current search engine which performs searching operation based on keywords which results in an enormous amount of data available to the user, from which user cannot figure out the essential and most important information. This limitation may be overcome by a new web architecture known as the semantic web which overcome the limitation of the keyword based search technique called the conceptual or the semantic search technique. Natural language processing technique is mostly implemented in a QA system for asking userâs questions and several steps are also followed for conversion of questions to the query form for retrieving an exact answer. In conceptual search, search engine interprets the meaning of the userâs query and the relation among the concepts that document contains with respect to a particular domain that produces specific answers instead of showing lists of answers. In this paper, we proposed the ontology based semantic information retrieval system and the Jena semantic web framework in which, the user enters an input query which is parsed by Standford Parser then the triplet extraction algorithm is used. For all input queries, the SPARQL query is formed and further, it is fired on the knowledge base (Ontology) which finds appropriate RDF triples in knowledge base and retrieve the relevant information using the Jena framework.
HyperQA: A Framework for Complex Question-AnsweringJinho Choi
Â
This abstract describes the overall framework of our question-answering system designed to answer various types of complex questions. Our framework makes heavy use of natural language processing techniques for the retrieval, ranking, and generation of correct answers. Our approach has been tested on answering arithmetic questions requiring logical reasoning as well as higher-order factoid questions aggregating information across different documents.
This document provides a thorough survey of recent advances in open-domain textual question answering systems based on deep learning. It begins with introducing the background and motivation for open-domain question answering and deep learning approaches. The document then defines the scope of open-domain textual question answering. The main contributions of the survey are summarized as systematically reviewing open-domain question answering models based on deep learning, introducing recent models and comparing their performance, and discussing challenges and future directions. The survey is organized into sections on the overview of open-domain question answering systems, key components of these systems, datasets and metrics, followed by conclusions and future work.
LSTM Model for Semantic Clustering of User-Generated Content Using AI Geared ...IRJET Journal
Â
The document describes a system called Smart Reply that uses machine learning to automatically generate short message responses on wearable devices. It uses an LSTM neural network model trained on large amounts of conversation data to predict likely responses given an incoming message. It addresses challenges of response quality, diversity, scalability and privacy. The system clusters messages into semantic groups using a graph-based semi-supervised learning technique to ensure responses cover different intents. It has been successfully deployed and handles 10% of message replies on wearable devices.
Question Answering has been a well-researched NLP area over recent years. It has become necessary for
users to be able to query through the variety of information available - be it structured or unstructured. In
this paper, we propose a Question Answering module which a) can consume a variety of data formats - a
heterogeneous data pipeline, which ingests data from product manuals, technical data forums, internal
discussion forums, groups, etc. b) addresses practical challenges faced in real-life situations by pointing to
the exact segment of the manual or chat threads which can solve a user query c) provides segments of texts
when deemed relevant, based on user query and business context. Our solution provides a comprehensive
and detailed pipeline that is composed of elaborate data ingestion, data parsing, indexing, and querying
modules. Our solution is capable of handling a plethora of data sources such as text, images, tables,
community forums, and flow charts. Our studies performed on a variety of business-specific datasets
represent the necessity of custom pipelines like the proposed one to solve several real-world document
question-answering
The document describes an intelligent question answering system that can leverage heterogeneous datasets including product manuals, technical forums, and discussion threads. The system includes four main modules: 1) A document parser that can parse different data types including text, images, tables, and forums using deep learning models. 2) A document indexer that indexes documents for retrieval. 3) A document retriever that handles query processing and identifies relevant text segments. 4) A document reader that provides answers by analyzing relevant text segments. The system aims to reduce the time needed to find answers across different data sources by automatically identifying the most relevant information for a given question.
Exploring Semantic Question Generation Methodology and a Case Study for Algor...IJCI JOURNAL
Â
Assessment of student performance is one of the most important tasks in the educational process. Thus, formulating questions and creating tests takes the instructor a lot of time and effort. However, the time spent for learning acquisition and on exam preparation could be utilized in better ways. With the technical development in representing and linking data, ontologies have been used in academic fields to represent the terms in a field by defining concepts and categories classifies the subject. Also, the emergence of such methods that represent the data and link it logically contributed to the creation of methods and tools for creating questions. These tools can be used in existing learning systems to provide effective solutions to assist the teacher in creating test questions. This research paper introduces a semantic methodology for automating question generation in the domain of Algorithms. The primary objective of this approach is to support instructors in effectively incorporating automatically generated questions into their instructional practice, thereby enhancing the teaching and learning experience.
MULTI-LEARNING SPECIAL SESSION / EDUCON 2018 / EMADRID TEAMeMadrid network
Â
1) The document proposes an approach to assist course creators in generating or restructuring courses by exploiting text mining techniques, semantic information from DBpedia, and linking educational resources.
2) The approach was implemented as a prototype that retrieves online courses, identifies key elements from text, formulates queries to other courses, and returns related courses to help creators generate mashups.
3) Preliminary tests on 265 computer science courses showed promising results, though future work is needed to improve similarity measures and generate concept maps between related courses.
This document proposes a model for automatically clustering Thai students' online homework assignments before teachers grade them. The model uses five parts: 1) Thai word segmentation, 2) stop-word removal, 3) term weighting, 4) document clustering using k-means, and 5) performance evaluation. The model was tested on 1,000 student assignments and achieved high accuracy, purity, and F-measure scores similar to human grading, allowing teachers to grade assignments more efficiently.
This document is a thesis that proposes using word embeddings to improve information retrieval by addressing term mismatch issues. It discusses word2vec, a technique for learning word embeddings from large text corpora that capture semantic relationships between words. The thesis proposes two approaches: 1) incorporating word embedding similarities into a probabilistic language model for retrieval and 2) a vector space model. Due to time constraints, only the first approach is implemented, which integrates word embeddings into ALMasri and Chevallet's probabilistic language model. Experiments are conducted to evaluate the impact of using semantic features from word embeddings on retrieval effectiveness.
The document describes a study that used large language models (LLMs) like GPT-3 and GPT-4 to complete knowledge from Wikidata. The researchers developed a pipeline called LLMKE that combines knowledge probing and Wikidata entity mapping. They were able to achieve a macro-averaged F1-score of 0.701 on the ISWC 2023 LM-KBC Challenge, with scores ranging from 1.00 to 0.328 depending on the domain. The results show that LLMs have varying knowledge depending on the domain and more work is needed to determine when they can be used for automatic knowledge base completion and correction.
Robotics-Based Learning in the Context of Computer ProgrammingJacob Storer
Â
This document is a project report for research into whether robotics-based learning or simulation-based learning is more effective for teaching programming. It describes the objectives of developing tutorials for both an Arduino robot and visual basic simulation. Programming tasks for moving forwards/backwards and along shapes were developed. Tutorials and programs were implemented to teach these tasks. Surveys were given to test groups after using each method to collect data on their effectiveness for comparison. While results were mixed, all indicated learning was improved with a teacher. Due to the small sample size, no conclusive answer could be provided.
AUTOMATIC QUESTION GENERATION USING NATURAL LANGUAGE PROCESSINGIRJET Journal
Â
The document describes a proposed method for automatic question generation using natural language processing and T5 text-to-text transfer transformer models. The method uses T5 models trained on the Stanford Question Answering Dataset to generate questions from paragraphs of text without requiring extensive grammar rules. The proposed system aims to assist students in learning by generating questions to test their understanding from provided materials.
Categorization of Protean Writers by Exploitation of Raspberry Piijtsrd
Â
Raspberry Pi is a computer though smaller but has versatile functionality. These are useful in assisting variety of educational institutions for teaching and other investigational indagations. In this paper, three prominent Bangladeshi writer's works were catalogued by using Raspberry Pi 3. The significance of this research pivots on low cost computational teaching in different institutions. Mathematica was used for this purpose and it's comprised of two modules which effectively communicate and shape an effective interpreter. It's free on Raspberry. After gathering literature and amassing all text files in BD writer's database, application of variety of algorithms were done for categorization. Then experimental analysis was done. Among different categorizers Markov and NaĂÂŻve Bayes have high precision and have best training times. This research will help in further distinguishing of languages by using an economical approach and will assist in further investigational studies for better understanding. Pritom Sarker | Jannatul Ferdous | Nakib Aman Turzo | Biplob Kumar | Jyotirmoy Ghose "Categorization of Protean Writers by Exploitation of Raspberry Pi" Published in International Journal of Trend in Scientific Research and Development (ijtsrd), ISSN: 2456-6470, Volume-4 | Issue-4 , June 2020, URL: https://www.ijtsrd.com/papers/ijtsrd31332.pdf Paper Url :https://www.ijtsrd.com/engineering/computer-engineering/31332/categorization-of-protean-writers-by-exploitation-of-raspberry-pi/pritom-sarker
Project MLExAI: Machine Learning Experiences in AIbutest
Â
The project aims to develop a framework for teaching core AI topics with a focus on machine learning. It involves hands-on term projects where students design and implement learning systems for applications like web document classification, web user profiling, and character recognition. Preliminary results found students had positive experiences applying machine learning concepts to real-world problems.
Project MLExAI: Machine Learning Experiences in AIbutest
Â
The project aims to develop a framework for teaching core AI topics with a focus on machine learning. It involves hands-on term projects where students design and implement learning systems for applications like web document classification, user profiling, and character recognition. Preliminary results found students had positive experiences applying machine learning concepts to real-world problems.
Naver learning to rank question answer pairs using hrde-ltcNAVER Engineering
Â
The automatic question answering (QA) task has long been considered a primary objective of artificial intelligence.
Among the QA sub-systems, we focused on answer-ranking part. In particular, we investigated a novel neural network architecture with additional data clustering module to improve the performance in ranking answer candidates which are longer than a single sentence. This work can be used not only for the QA ranking task, but also to evaluate the relevance of next utterance with given dialogue generated from the dialogue model.
In this talk, I'll present our research results (NAACL 2018), and also its potential use cases (i.e. fake news detection). Finally, I'll conclude by introducing some issues on previous research, and by introducing recent approach in academic.
Similar to ARQMath Lab An Incubator For Semantic Formula Search In ZbMATH Open (20)
Expository Essay Reflection Paper In. Online assignment writing service.Lori Mitchell
Â
The document describes the steps to get writing help from HelpWriting.net:
1. Create an account with a password and email.
2. Complete a 10-minute order form providing instructions, sources, and deadline. Attach sample work if wanting the writer to mimic your style.
3. Review bids from writers and choose one based on qualifications, history, and feedback. Place a deposit to start the assignment.
4. Ensure the completed paper meets expectations. Authorize full payment for satisfactory work or request revisions.
Handwriting Without Tears Paper WITH Picture HanLori Mitchell
Â
The document discusses the steps to get writing help from the website HelpWriting.net, which includes creating an account, submitting a request form with instructions and deadline, and choosing a writer to complete the assignment. Writers will bid on requests and the client can choose a writer and provide a deposit to start the work. Clients can then review the completed paper and request revisions until satisfied before authorizing final payment.
My Mother Childhood Essay. Essay On My MotLori Mitchell
Â
The document provides instructions for requesting an assignment to be written by a writing service. It outlines a 5-step process: 1) Create an account with necessary information. 2) Complete a 10-minute order form providing instructions, sources, deadline, and an optional sample. 3) Review bids from writers and choose one based on qualifications. 4) Receive the paper and authorize payment if pleased. 5) Request revisions to ensure satisfaction, with a refund option for plagiarized work.
The bidding system matches requests with qualified writers. Customers provide the necessary assignment details and deadline. Writers then submit bids that customers can review based on history and feedback
Stephen King Quote If You Want To Be A Writer, You Must Do Two ThingsLori Mitchell
Â
Here are the key steps to conduct statistical analysis using SPSS:
1. Enter your data into SPSS. This involves creating variable names for each column of data and inputting the actual values.
2. Select the appropriate statistical test based on your research question and data type (e.g. t-test for comparing two groups, ANOVA for more than two groups, correlation for relationship between variables).
3. Select "Analyze" from the top menu, then choose the specific statistical test (e.g. "Compare Means" then "Independent Samples T-Test").
4. Move the appropriate variables representing your groups/factors/variables of interest into the correct boxes in the test window.
5
500 Word Essay - Example, Length And Writing Tips AtLori Mitchell
Â
The document provides instructions for writing a 500-word essay with the help of the HelpWriting.net service. It outlines a 5-step process: 1) Create an account; 2) Complete an order form with instructions and deadline; 3) Review bids from writers and choose one; 4) Review the paper and authorize payment; 5) Request revisions until satisfied. It emphasizes that the service provides original, high-quality content and refunds plagiarized work.
Pin On Educational Purposes. Online assignment writing service.Lori Mitchell
Â
The document provides instructions for requesting writing assistance from HelpWriting.net. It outlines a 5-step process: 1) Create an account with a password and email. 2) Complete a 10-minute order form providing instructions, sources, and deadline. 3) Review bids from writers and choose one based on qualifications. 4) Review the completed paper and authorize payment if satisfied. 5) Request revisions to ensure satisfaction, and HelpWriting.net guarantees original, high-quality work or a full refund.
English Essay Form 1 - JerryldOneal. Online assignment writing service.Lori Mitchell
Â
The document provides instructions for using the HelpWriting.net service to have essays written. It outlines a 5-step process: 1) Create an account with an email and password. 2) Complete a form with assignment details and deadline. 3) Review bids from writers and select one. 4) Review the completed paper and authorize payment. 5) Request revisions until satisfied with the paper. The service aims to provide original, high-quality content and offers refunds for plagiarized work.
Nurse Practitioner Personal Statement Sample That CanLori Mitchell
Â
The document provides instructions for requesting writing assistance from HelpWriting.net. It is a 5-step process: 1) Create an account with a password and email. 2) Complete a 10-minute order form providing instructions, sources, and deadline. 3) Review bids from writers and choose one based on qualifications. 4) Receive the paper and authorize payment if satisfied. 5) Request revisions to ensure needs are fully met. HelpWriting.net uses a bidding system and promises original, high-quality content with refunds for plagiarism.
As An English Teacher, Writin. Online assignment writing service.Lori Mitchell
Â
The document provides instructions for creating an account and submitting assignment requests on the HelpWriting.net website. It explains that writers will bid on requests and the customer can choose a writer, make a deposit, and receive the completed paper, with options for revisions. The site promises original, high-quality work and refunds for plagiarized content.
How To Write An Good Literature Essays OnlineLori Mitchell
Â
This document provides instructions for writing literature essays online through the HelpWriting.net website. It outlines a 5-step process: 1) Create an account with a password and email; 2) Complete an order form with instructions, sources, and deadline; 3) Review writer bids and choose one to start the assignment; 4) Review and approve the completed paper or request revisions; 5) Request multiple revisions to ensure satisfaction and receive a refund for plagiarized work.
This document provides instructions for creating an account and submitting assignment requests on the HelpWriting.net website. It outlines a 5-step process: 1) Create an account with a password and email. 2) Complete a form with assignment details, sources, and deadline. 3) Review bids from writers and choose one. 4) Review the completed paper and authorize payment. 5) Request revisions until satisfied, with a refund option for plagiarism. The document promises original, high-quality work and support through the writing process.
College Essay Essay In High School. Online assignment writing service.Lori Mitchell
Â
The document provides instructions for completing an assignment writing request on the HelpWriting.net website. It outlines a 5-step process: 1) Create an account with a password and email. 2) Complete an order form providing instructions, sources, and deadline. 3) Review bids from writers and choose one based on qualifications. 4) Review the completed paper and authorize payment. 5) Request revisions to ensure satisfaction, with a full refund option for plagiarized work. The purpose is to guide users through obtaining writing help services from HelpWriting.net.
Cheap Essay Writing Services - Avail Best Essay Help AuthorSTLori Mitchell
Â
The document describes a writing service that allows users to request essay help online in 5 steps:
1. Create an account with a password and email.
2. Complete an order form providing instructions, sources, and deadline.
3. Review bids from writers and choose one based on qualifications.
4. Review the completed paper and authorize payment if satisfied.
5. Request revisions until fully satisfied, with a refund option for plagiarized work.
Scientific Research Paper Format Template - WritinghtLori Mitchell
Â
The document provides instructions for requesting writing assistance from HelpWriting.net in 5 steps:
1. Create an account with a password and email.
2. Complete a 10-minute order form providing instructions, sources, deadline, and attaching a sample for style imitation.
3. Review bids from writers for the request, choose one based on qualifications, order history, and feedback, then pay a deposit.
4. Review the completed paper and authorize final payment if satisfied, or request free revisions.
5. Choose HelpWriting.net confidently knowing needs will be fully met, with original work or a full refund for plagiarism.
Tips For Writing A Research Paper Research4LifeLori Mitchell
Â
I apologize, upon further reflection I do not feel comfortable providing a case study analysis without more context about the specific organization, goals of the study, and methodology. Case studies should be carefully designed and implemented to ethically and accurately evaluate organizations. Perhaps we could discuss this topic at a higher level without focusing on a specific case.
The document summarizes the direct method observed in a Catalan language classroom. The teacher spoke only Catalan from the beginning of class. She introduced vocabulary words using gestures, drawings on the board, and actions. She read aloud from a book in Catalan and used illustrations and gestures to explain the text. Students then read sentences aloud, which the teacher clarified through examples in Catalan. The direct method immersed students in the target language from the start of the lesson.
Samples Of Persuasive Essays For High School StLori Mitchell
Â
The document discusses the four major sociological paradigms: structural functional, social conflict, symbolic interaction, and postmodern perspectives. Structural functional paradigm views society as a system that is stable, with shared values and expectations. Social conflict paradigm sees society as an arena of inequality that generates conflict and change. Symbolic interaction paradigm focuses on daily social interactions and the meanings derived from symbols. Postmodern paradigm is skeptical of objective truths and universal explanations.
The Elements Of Writing A Song Songwriting Basics,Lori Mitchell
Â
The document discusses the benefits a student can gain from studying philosophy, including learning to evaluate arguments, analyze ideas, and draw their own conclusions. It explains the Socratic method of teaching uses examination, analysis and evaluation to bring questions into sharper focus and provide foundations for answers. Critical thinking used to analyze philosophical issues engages rational deliberation, investigation of facts and reasons, and evaluation of arguments.
If They Give You Lined Paper, Write Sideways. By Daniel QuinnLori Mitchell
Â
The document provides instructions for requesting writing assistance from HelpWriting.net. It outlines a 5-step process: 1) Create an account with a password and email. 2) Complete a 10-minute order form providing instructions, sources, and deadline. 3) Review bids from writers and select one. 4) Review the completed paper and authorize payment if satisfied. 5) Request revisions until fully satisfied, with a refund option for plagiarized content. The document promises original, high-quality work meeting all needs.
Robot Writing Paper - 3 Styles By Pink Posy PaperiLori Mitchell
Â
This document provides instructions for requesting writing assistance from the website HelpWriting.net in 5 steps:
1. Create an account with a password and email.
2. Complete a 10-minute order form with instructions, sources, deadline, and attach a sample work.
3. Review bids from writers and choose one based on qualifications.
4. Review the completed paper and authorize payment if satisfied.
5. Request revisions until fully satisfied, with a refund option for plagiarized work.
How to Manage Reception Report in Odoo 17Celine George
Â
A business may deal with both sales and purchases occasionally. They buy things from vendors and then sell them to their customers. Such dealings can be confusing at times. Because multiple clients may inquire about the same product at the same time, after purchasing those products, customers must be assigned to them. Odoo has a tool called Reception Report that can be used to complete this assignment. By enabling this, a reception report comes automatically after confirming a receipt, from which we can assign products to orders.
A Free 200-Page eBook ~ Brain and Mind Exercise.pptxOH TEIK BIN
Â
(A Free eBook comprising 3 Sets of Presentation of a selection of Puzzles, Brain Teasers and Thinking Problems to exercise both the mind and the Right and Left Brain. To help keep the mind and brain fit and healthy. Good for both the young and old alike.
Answers are given for all the puzzles and problems.)
With Metta,
Bro. Oh Teik Bin đđ¤đ¤đĽ°
Gender and Mental Health - Counselling and Family Therapy Applications and In...PsychoTech Services
Â
A proprietary approach developed by bringing together the best of learning theories from Psychology, design principles from the world of visualization, and pedagogical methods from over a decade of training experience, that enables you to: Learn better, faster!
Andreas Schleicher presents PISA 2022 Volume III - Creative Thinking - 18 Jun...EduSkills OECD
Â
Andreas Schleicher, Director of Education and Skills at the OECD presents at the launch of PISA 2022 Volume III - Creative Minds, Creative Schools on 18 June 2024.
Philippine Edukasyong Pantahanan at Pangkabuhayan (EPP) CurriculumMJDuyan
Â
(đđđ đđđ) (đđđŹđŹđ¨đ§ đ)-đđŤđđĽđ˘đŚđŹ
đđ˘đŹđđŽđŹđŹ đđĄđ đđđ đđŽđŤđŤđ˘đđŽđĽđŽđŚ đ˘đ§ đđĄđ đđĄđ˘đĽđ˘đŠđŠđ˘đ§đđŹ:
- Understand the goals and objectives of the Edukasyong Pantahanan at Pangkabuhayan (EPP) curriculum, recognizing its importance in fostering practical life skills and values among students. Students will also be able to identify the key components and subjects covered, such as agriculture, home economics, industrial arts, and information and communication technology.
đđąđŠđĽđđ˘đ§ đđĄđ đđđđŽđŤđ đđ§đ đđđ¨đŠđ đ¨đ đđ§ đđ§đđŤđđŠđŤđđ§đđŽđŤ:
-Define entrepreneurship, distinguishing it from general business activities by emphasizing its focus on innovation, risk-taking, and value creation. Students will describe the characteristics and traits of successful entrepreneurs, including their roles and responsibilities, and discuss the broader economic and social impacts of entrepreneurial activities on both local and global scales.
CapTechTalks Webinar Slides June 2024 Donovan Wright.pptxCapitolTechU
Â
Slides from a Capitol Technology University webinar held June 20, 2024. The webinar featured Dr. Donovan Wright, presenting on the Department of Defense Digital Transformation.
CapTechTalks Webinar Slides June 2024 Donovan Wright.pptx
Â
ARQMath Lab An Incubator For Semantic Formula Search In ZbMATH Open
1. Copyright Š 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution
4.0 International (CC BY 4.0). CLEF 2020, 22-25 September 2020, Thessaloniki, Greece.
ARQMath Lab: An Incubator for Semantic Formula
Search in zbMATH Open?
Philipp Scharpf1
, Moritz Schubotz2,3
, AndrĂŠ Greiner-Petter2
,
Malte Ostendorff1
, Olaf Teschke3
, and Bela Gipp2
1
University of Konstanz, Konstanz, Germany
{first.last}@uni-konstanz.de
2
University of Wuppertal, Wuppertal, Germany
andre.greiner-petter@zbmath.org, {last}@uni-wuppertal.de
3
FIZ Karlsruhe, Karlsruhe, Germany
{first.last}@fiz-karlsruhe.de
Abstract. The zbMATH database contains more than 4 million bibliographic
entries. We aim to provide easy access to these entries. Therefore, we maintain
different index structures, including a formula index. To optimize the findability
of the entries in our database, we continuously investigate new approaches to
satisfy the information needs of our users. We believe that the findings from the
ARQMath evaluation will generate new insights into which index structures are
most suitable to satisfy mathematical information needs. Search engines, recom-
mender systems, plagiarism checking software, and many other added-value ser-
vices acting on databases such as the arXiv and zbMATH need to combine natu-
ral and formula language. One initial approach to address this challenge is to
enrich the mostly unstructured document data via Entity Linking. The ARQMath
Task at CLEF 2020 aims to tackle the problem of linking newly posted questions
from Math Stack Exchange (MSE) to existing ones that were already answered
by the community. To deeply understand MSE information needs, answer-, and
formula types, we performed manual runs for tasks 1 and 2. Furthermore, we
explored several formula retrieval methods: For task 2, such as fuzzy string
search, k-nearest neighbors, and our recently introduced approach to retrieve
Mathematical Objects of Interest (MOI) with textual search queries. The task re-
sults show that neither our automated methods nor our manual runs archived good
scores in the competition. However, the perceived quality of the hits returned by
the MOI search particularly motivates us to conduct further research about MOI.
Keywords: Information Retrieval, Mathematical Information Retrieval,
Question Answering, Semantic Search, Machine Learning, Mathematical Ob-
jects of Interest, ARQMath Lab
Scharpf et al., âARQMath Lab: An Incubator for Semantic Formula Search
in zbMATH Open?â, ARQMath Lab @ CLEF 2020, Virtual Event
Preprint from: https://www.gipp.com/pub/
2. 1 Introduction
In 2013 the first prototype of formula-search in zbMATH was announced [1], which
became an integral part of the zbMATH interface by now. At the beginning of 2021,
zbMATH will transform its business model from a subscription-based service to a pub-
licly funded open service. In this context, we evaluate novel approaches to include
mathematical formulae as first-class citizens in our mathematical information retrieval
infrastructure. Despite the standard search that targets abstract, review, and publication
meta-data, zbMATH also traces incoming links from the Question Answering platform
MathOverflow and provides backlinks from scientific articles to MathOverflow links,
mentioning the publication [1]. We hypothesize that federating information from
zbMATH and MathOverflow will enhance the zbMATH search experience signifi-
cantly. The ARQMath Lab at CLEF 2020 aims to tackle the problem of linking newly
posted questions from Math Stack Exchange to existing ones that were already an-
swered by the community [2]. Using question postings from a test collection (extracted
by the ARQMath organizers from an MSE Internet Archive Snapshot1
until 2018) as
queries, the goal is to retrieve relevant answer posts, containing both text and at least
one formula. The test collection created for the task is intended to be used by research-
ers as a benchmark for mathematical retrieval tasks that involve both natural and math-
ematical language. The ARQMath Lab consists of two separate subtasks. Task 1 â An-
swer poses the challenge to retrieve relevant community answer post given a question
from Math Stack Exchange (MSE). Task 2 â Formulas poses the challenge to retrieve
relevant formulas from question and answer posts. Specifically, the aim of Task 1 is to
be able to find old answers to new questions to speed up the community answer process.
The aim of Task 2 is to find a ranked list of relevant formulae in old questions and
answers to match to a query formula from the new question. This task design seems to
a good fit for our research interest, since the information needs are related. Moreover,
MathOverflow and math.stackexchange use the same data-format, which enables us to
reuse software developed during this competition and to transform it into production
software later on. On the other hand, the mathematical level of questions on Math Stack
Exchange is less sophisticated and thus not all relevant rankings might be suitable for
our use-case.
1.1 ARQMath Lab
The ARQMath lab was motivated by the fact that Mansouri et al. discovered âthat 20%
of the mathematical queries in general-purpose search engines were expressed as well-
formed questionsâ [2], [3]. Furthermore, with the increasing public interest in Commu-
nity Question Answering sites such as MSE2
and MathOverflow3
, it will be beneficial
to develop computational methods to support human answerers. Particularly, the âtime-
1
https://archive.org/download/stackexchange
2
https://math.stackexchange.com
3
https://mathoverflow.net
3. to-answerâ should be shortened by linking to related answers already provided on the
platform, which can potentially lead to the answer more quickly. This will be of great
help since most of the time the question is urgent and related â sometimes even directly
exact â existing answers are available. However, the task is challenging because both
questions and answers can be a combination of natural and mathematical language,
involving words and formulae. ARQMath lab at CLEF 2020 will be the first in a three-
year sequence through which the organizers âaim to push the state of the art in evalua-
tion design for math-aware IRâ [2]. The task starts with the domain of mathematics
involving formula language. The goal is to later extend the task to other domains (e.g.,
chemistry or biology), which employ other types of special notation.
1.2 Math Stack Exchange
Stack Exchange is an online platform with a host of Q&A forums [4]. The Stack Ex-
change network consists of 177 Q&A communities including Stack Overflow, which
claims to be âthe largest, most trusted online community for developers to learn and
share their knowledgeâ2
. The different topic sites include Q&A on computer issues,
math, physics, photography, etc. Users can rank questions and answers by voting them
up or down according to their quality assessment. Stack Exchange provides its content
publicly available in XML format under the Creative Commons license [4]. The Math
Stack Exchange collection for the ARQ lab tasks comprises Q&A postings extracted
from data dumps from the Internet Archive4
. Currently, over 1 million questions are
included [2].
2 Related Work
2.1 Mathematical Question Answering
Already in 1974, Smith [5] describes a project investigating the understanding of natu-
ral language by computers. He develops a theoretical model of natural language pro-
cessing (NLP) and algorithmically implements his theory. Specifically, he chooses the
domain of elementary mathematics to construct a Q&A system for unrestricted natural
language input. However, for some time later, there was little interest and progress in
the field of mathematical question answering. In 2012, Nguyen et al. [6] present a math-
aware search engine for a math question answering system. Their system handles both
textual keywords as well as mathematical expressions. The math feature extraction is
designed to encode the semantics of math expressions via a Finite State Machine model.
They tested their approach against three classical information retrieval strategies on
math documents crawled from Math OverďŹow, claiming to outperform them by more
than 9%. In 2017, Bhattacharya et al. [7] publish a survey of question answering for
math and science problems. They explore the current achievements towards the goal of
making computers smart enough to pass math and science tests. They conclude claim-
ing that âthe smartest AI could not pass high schoolâ. In 2018, Gunawan et al. [8]
4
https://archive.org
4. present an Indonesian question answering system for solving arithmetic word problems
using pattern matching. Their approach is integrated into a physical humanoid robot.
For auditive communication with the robot, the userâs Indonesian question must be
translated into English text. They employ NLP using the NLTK toolkit5
, specifically
co-referencing, question parsing, and preprocessing. They conclude claiming that the
Q&A system achieves an accuracy between 80% and 100%. However, they state that
the response time is rather slow with average about more than one minute. Also in 2018,
Schubotz et al. [9] present MathQA6
, an open-source math-aware question answering
system based on Ask Platypus7
. The system returns as a single mathematical formula
for a natural language question in English or Hindi. The formulae are fetched from the
open knowledge-base Wikidata8
. With numeric values for constants loaded from Wik-
idata, the user can do computations using the retrieved formula. It is claimed that the
system outperforms a popular computational mathematical knowledge-engine by 13%.
In 2019, Hopkins et al. [10] report on the SemEval 2019 task on math question answer-
ing. The derived a question set from Math SAT practice exams, including 2778 training
questions and 1082 test questions. According to their study, the top system correctly
answered 45% of the test questions, with a random guessing baseline at 17%. Beyond
the domain of math Q&A, Pineau [11] and Abdi et al. [12] present first approaches to
answer questions on physics.
2.2 Mathematical Document Subject Class Classification
For open-domain question redirection, it is beneficial to classify a given mathematical
question by its domain, e.g. geometry, calculus, set theory, physics, etc. There have
been several approaches to perform categorization or subject class classification for
mathematical documents. In 2017, Suzuki and Fujii [13] test classification methods on
collections built from MathOverflow9
and the arXiv10
paper preprint repository. The
user tags include both keywords for math concepts and categories form the Mathemat-
ical Subject Classification (MSC) 201011
top and second-level subjects. In 2020,
Scharpf et al. [9] investigate how combining encodings of natural and mathematical
language affect the classification and clustering of documents with mathematical con-
tent. They employ sets of documents, sections, and abstracts from the arXiv10,
labeled
by their subject class (mathematics, computer science, physics, etc.) to compare differ-
ent encodings of text and formulae and evaluate the performance and runtimes of se-
lected classification and clustering algorithms. Also in 2020, Schubotz et al. [14] ex-
plore whether it is feasible to automatically assign a coarse-grained primary classifica-
tion using the MSC scheme using multi-class classification algorithms. They claim to
achieve a precision of 81% for the automatic article classification. We conclude that
5
https://www.nltk.org
6
http://mathqa.wmflabs.org
7
https://askplatyp.us
8
https://www.wikidata.org
9
https://mathoverflow.net
10
https://arxiv.org
11
http://msc2010.org
5. for math Q&A systems, the classification needs to be performed at the sentence level.
If MSE questions contain several sentences, the problem could potentially also be
framed as an abstract classification problem.
2.3 Connecting Natural and Mathematical Language
For mathematical question answering, mathematical information needs to be connected
to natural language queries. Yang & Ko [15] present a search engine for formulae in
MathML12
using a plain word query. Mansouri et al. [3] investigate how queries for
mathematical concepts are performed in search engines. They conclude âthat math
search sessions are typically longer and less successful than general search sessionsâ.
For non-mathematical queries, search engines like Google13
or DuckDuckGo14
already
provide entity cards with a short encyclopedic description of the searched concept [16].
For mathematical concepts, however, there is an urgent need to connect a natural lan-
guage query to a formula representing the keyword. Dmello [16] proposes integrating
entity cards into the math-aware search interface MathSeer15
. Scharpf et al. [17] pro-
pose a Formula Concept Retrieval challenge for Formula Concept Discovery (FCD)
and Formula Concept Recognition (FCR) tasks. They present first machine learning
based approaches for retrieving formula concepts from the NTCIR 11/12 arXiv da-
taset16
.
2.4 Semantic Annotations
To connect mathematical formulae and symbols to natural language keywords, seman-
tic annotations are an effective means. So far there are only a few annotation systems
available for mathematical documents. Dumitru et al. [18] present a browser-based an-
notation tool (âKAT systemâ) for linguistic/semantic annotations in structured
(XHTML5) documents. Scharpf et al. [19] present âAnnoMathTeXâ, a recommender
system for formula and identifier annotation of Wikipedia articles using Wikidata17
QID item tags. The annotations can be integrated into the MathML markup using
MathML Wikidata Content Dictionaries18
[20], [21], [22].
3 Summary of Our Approach
We tackle the ARQMath lab tasks (Task 1 â answer retrieval, Task 2 â formula re-
trieval) using manual run selection benchmarking. Therefore, we create, populate, and
12
https://www.w3.org/TR/MathML3
13
https://www.google.com
14
https://duckduckgo.com
15
https://www.cs.rit.edu/~dprl/mathseer
16
http://ntcir-math.nii.ac.jp
17
https://www.wikidata.org
18
https://www.openmath.org
6. employ a Wiki19
with pages for normal (Task 1) and formula (Task 2) topics. The main
objective of our experiments was to explore methods to enable automatic answer as-
signment recommendations to question postings on Mathematics Stack Exchange
(MSE). We tested the following approaches or methods: 1) manual run annotation using
Google and MSE search, 2) formula TF-IDF or Doc2vec20
encodings [23] using the
Python libraries Scikit-learn21
[24] and Gensim22
[25], 3) fuzzy string comparison or
matching using rapidfuzz23
, 4) k-nearest neighbors algorithm, and 5) discovering of
Mathematical Objects of Interest (MOI) with textual search queries [26].
As result, we obtained a relevant MSE answer(s) ID for each query in the sample of
Task 1, and a ranked list of most relevant formulae for each query in the sample of Task
2 (if available). Finally, we analyzed our results using a manual consistency and quality
check.
4 Workflow of Our Approach
The workflow of our approach is illustrated in Fig. 1. It can be logically divided into
three stages: 1) the creation of a Wiki with pages for normal and formula topics, 2)
methods to tackle Task 1, and 3) methods to tackle Task 2.
Fig. 1. Workflow of our approach to retrieve answer and formula candidates for Tasks 1 and 2.
In the following, we describe the stages with their subtasks in more detail.
19
https://arq20.formulasearchengine.com
20
Also known as âParagraph Vectorsâ, as introduced in [23].
21
https://scikit-learn.org
22
https://radimrehurek.com/gensim
23
https://github.com/maxbachmann/rapidfuzz
Wiki
â˘Retrieval of URLs using Google and MSE search
â˘Creation of Wiki at arq20.formulasearchengine.com
â˘Creation of Wiki pages for normal and formula topics
Task 1
â˘Insert links to math.stackexchange.com/questions/xxx on Wikipedia page
â˘Manual run selection of the most suitable answer
â˘Insert links to https://math.stackexchange.com/a/xxx as ârelevant answersâ
property on Wikidata item for normal topics
Task 2
â˘Manual run selection of the most suitable formula(e)
â˘LaTeX string as âdefining formulaâ property as subproperty of ârelevant
answersâ on Wikidata item for formula topics
7. 4.1 Setup Wiki Framework
The initial preparation step for our approach to tackle Task 1 and 2 was to create, pop-
ulate, and employ a MediaWiki environment connected to a mathoid [27] rendering
service with pages for normal and formula topics. For each query, there is a Wikibase
item with the following properties: âmath-stackexchange-categoryâ (P10), âtopic-idâ
(P12), âpost-typeâ (P9), âmath stackexcange post idâ (P5), and ârelevant answersâ (P14).
Having set up the Wiki, we manually retrieved the question URLs using Google and
MSE search and inserted them as values for the âmath stackexchange post idâ on the
respective question pages. Unfortunately by doing so some post 2019 new post-ids were
entered because we did not check the date carefully enough. The âmath-stackexchange-
categoryâ values were automatically retrieved from the question tags. The âtopic-idâ
(e.g., A.50) was transferred from the task dataset, the âpost-typeâ set to âQuestionâ.
Unfortunately, as we discovered later, the use of Google and MSE search led to results
outside the task dataset. This means that the answer that was accepted as the best answer
by the questioner was often not included in the task dataset. However, our aim was to
establish the âcorrectâ answer as semantic reference in our MediaWiki.
4.2 Populate Topic Answers (Task 1)
The first part in our experimental pipeline was a manual run selection of the most suit-
able answer from the MSE question posting page (preferably the one selected by the
questioner, if available). Subsequently, we inserted links to the answers, i.e.,
math.stackexchange.com/a/xxx to the ârelevant answersâ property of the query item
normal topics page.
4.3 Populate Formula Answers (Task 2)
The second part in our experimental pipeline was a manual run selection of the most
suitable formula per question or answer. The chosen formula was considered to answer
the given question as concise as possible. Thus, we did interpret Task 2 as having to
find formula answers to the question and only not similar formulae. We inserted the
extracted LaTeX string to the âdefining formulaâ property, as a subproperty of ârelevant
answersâ on the Wikidata item for formula topics.
4.4 Preparing Data for Experiments and Submission
After having populated our Wiki database, we used a SPARQL query (Fig. 2) to have
an overview of its content. The query fetches all Wikidata question items, displaying
their âtopic-idâ (e.g. A.1 or B.1), âpost-idâ (e.g., 3063081), and the formula LaTeX
string. With the list of normal and formula topic insertions, we performed a quality
check, correcting wrong or missing values.
8. Fig. 2. SPARQL query to retrieve our manually inserted data containing topic answer links (Task
1 - Answer) and formula LaTeX strings (Task 2 - Formulas). The query properties are âmath-
stackexchange-categoryâ (P10), âtopic-idâ (P12), âpost-typeâ (P9), âmath stackexcange post idâ
(P5), and ârelevant answersâ (P14).
4.5 Discovering Mathematical Objects of Interest
The previously developed MOI search engine [26] allows us to search meaningful
mathematical expressions by a given textual search query. This workflow can be used
to solve Task 2, but it requires some substantial updates. Essentially, Task 2 requests
relevant formula IDs for a given input formula ID. Each formula ID is mapped to the
corresponding post ID. Hence, we can take the entire post of a formula ID as the input
for our MOI search engine. However, there are two main problems with the existing
approach: (i) the MOI search engine was developed and tested only to search for key-
words, thus, entering entire posts at once may harm the accuracy, and (ii) every re-
trieved MOI is by design a subexpression and, thus, has probably no designated formula
ID. To overcome these issues, we need to understand the current system. The MOI
search system retrieves MOIs in two steps. The first step retrieves relevant documents
from an elasticsearch24
instance for the input query. Hence, we first indexed all
ARQMath posts in elasticsearch. To index the content of each post appropriately, we
set up the standard English stemmer, stopword filtering, HTML stripping (filters out
HTML tags but preserves the content of each tag), and enable ASCII folding (converts
alphabetic, numeric, and symbolic characters to their ASCII equivalence, e.g., âĂĄâ is
replaced by âaâ). For the search query, we used the standard match query system but
boosted every mathematical expression in the input. This tells elasticsearch to focus
more on the math expressions in a search query, rather than the actual text. With this
setup, we overcome the mentioned issue (i) and can search for relevant posts by enter-
ing an entire content of a post. In the second step of the MOI search engine, the engine
24
https://www.elastic.co
9. disassembles all formulae in the retrieved documents and calculates the mBM25 score
[26] for each of these subexpressions (MOI)
s(đĄ, đ) â
(đ + 1)IDF(đĄ)ITF(đĄ, đ)TF(đĄ, đ)
max
đĄâ˛âđ|đ(đĄ)
TF(đĄâ˛, đ) + đ (1 â đ +
đAVGDL
|đ|AVGc
)
,
mBM25(đĄ, đˇ) â max
đâđˇ
s(đĄ, đ),
where mBM25(đĄ, đˇ) is a modified version of the BM25 relevance score [28] with đˇ as
the entire ARQMath corpus, IDF(đĄ) is the inverse document frequency of the term đĄ,
TF(đĄ, đ) the term frequency of the term đĄ in the document đ â đˇ, ITF(đĄ, đ) the inverse
term frequency (calculated the same way as IDF(đĄ) but on the document level for the
document đ), AVGDL the average document length of đˇ and AVGC the average com-
plexity of đˇ (see [26] for a more detailed description). The top-scored expressions will
be returned. The mBM25 score requires the global term and document frequencies of
every subexpression. Hence, we first calculated these global values for every subex-
pression of every formula in the ARQMath dataset. Table 1 shows the statistics of this
MOI database in comparison to the previously generated databases for arXiv and
zbMATH. A document in ARQMath is a post from MSE. The dataset only includes
MathML representations. The complexity of a formula is the maximum depth of the
Presentation MathML representation of the formula. As Table 1 shows, the ARQMath
database can be interpreted as a hybrid between the full research papers in arXiv and
relatively short review discussions in zbMATH (mainly containing reviews of mathe-
matical articles).
Table 1. The MOI database statistics of ARQMath compared to the existing databases for arXiv
and zbMATH. The document length is the number of subexpressions.
arXiv zbMATH ARQMath
Documents 841,008 1,349,297 2,058,866
Formulae 294,151,288 11,747,860 26,074,621
Subexpressions 2,508,620,512 61,355,307 143,317,218
Unique Subexpressions 350,206,974 8,450,496 16,897,129
Avg. Doc. Length 2,982.87 45.47 69.69
Avg. Complexity 5.01 4.89 5.00
Max. Complexity 218 26 188
Table 2 lists the machine specification for the MOI retrieval and runtime for example
query B.1.
Table 2. Machine hardware specification and example runtime for query B.1.
Machine Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz - 4 Cores / 8 Threads
RAM 32GB 2133 MHz
Disk 1TB SSD
Required Diskspace 7.8 GB (Posts) + 3 GB (MOIs) = 10.8 GB
Runtime 6.0 s / query (average over all queries)
10. Considering that every formula in the ARQMath dataset has its own ID and the system
needs to preserve the ID during computation, we need to attach the ID to every gener-
ated MOI. However, this would result in a massive overload. For example, the single
identifier đĽ appears 7.6 million times in ARQMath and thus will have millions of dif-
ferent formula IDs. The entire ARQMath dataset has 16.8 million unique MOIs. Handle
this number of different IDs is impractical. Hence, we choose a different approach to
get the formula IDs for every MOI. Since the search engine retrieves the relevant doc-
uments first, we only need to consider formula IDs that exist in these retrieved docu-
ments. To achieve this, we attached the formula IDs to every post in the elasticsearch
database rather than to the MOIs itself. A single document in elasticsearch now contains
the post ID, the textual content, and a list of MOIs with local term frequencies (how
often the MOI appears in the corresponding post) and formula IDs. Note that most MOI
still has multiple formula IDs, since a subexpression may appear multiple times in a
single post, but the number of different IDs reduced drastically. Since the IDs are now
attached to each post but are not used in the search query, the performance of retrieving
relevant documents from elasticsearch stays the same. With this approach, we may cal-
culate multiple but different mBM25 scores for a single formula ID, since a single
unique formula ID can be attached to multiple MOIs. To calculate the final score for a
formula ID, we calculated the average of all mBM25 scores for a formula ID. For ex-
ample, consider we would retrieve the document with the ID 2759760. This post con-
tains the formula ID 25466124
đ
đĽ6
,
which would be disassembled into its subexpressions đ, đĽ6
, and đĽ. Hence, we would
calculate three mBM25 scores for đ/đĽ6
. The average of these scores would be the score
for the formula ID.
We used this updated MOI search engine to retrieve results for Task 2. Note that the
approach might be a bit unorthodox, since the MOI search engine takes the entire post
of the given formula ID rather than the formula ID alone. We interpreted Task 2 to
retrieve answer formulae for a given question formula, rather than retrieving visually
or semantically similar formulae. Based on this interpretation, it makes sense to use the
entire post of a formula ID to search for relevant answers. In other words, we interpreted
Task 2 as an extension and math specific version of Task 1. In summary, the key steps
of the MOI search engine to solve Task 2 were the following:
1. Take the entire post of the given formula ID.
2. Search for posts similar to the retrieved post in step 1.
3. Extract all MOI from all retrieved posts in step 2.
4. Calculate mBM25 scores for all MOIs of step 3.
5. Group the MOIs by their associated formula IDs (every formula ID has now
multiple mBM25 scores).
6. Average the mBM25 scores for each formula ID.
11. For Task 2, we retrieved 107,476 MOIs. We used the provided annotation dataset to
evaluate the retrieved results. For a better comparison, we calculated the nDCGp
â˛
(nDCG-prime) score, as the task organizers did [29]. Note the nDCGp
â˛
removes un-
judged documents before calculating the score. Since these were post-experiment cal-
culations, there is not much correlation between the retrieved MOI documents and the
judged formula IDs. We found 179 formula IDs that were retrieved by our MOI engine
and contained a judgment by the annotators of the ARQMath task. Based on these 179
judges, we retrieved an nDCGp
â˛
value of 0.374, which is in the midrange compared to
the other competitors.
4.6 Data Integration of Query and Pool Formulae
We tested two other approaches for Task 2: Formula pool retrieval via k-nearest neigh-
bors and fuzzy string matching. For both methods, we first needed to integrate the pool
of formulae (the task dataset) with our query set, consisting of the formulae, which we
âmanuallyâ chose from the candidate answers to be a formula answer to the question
asked.
Fig. 3. Workflow for Task 2 â formula answer candidate retrieval. Manually selected âqueryâ
formulae must be integrated with the task dataset pool before testing k-nearest neighbors or fuzzy
string formula candidate retrieval.
Data
integration
query & pool
K-nearest
neighbors
retrieval
Fuzzy string
candidates
retrieval
12. Fig. 4. Workflow of the data integration (DI) and formula candidate retrieval via k-nearest neigh-
bors (kNN) and (fuzzy) string similarity matching for Task 2.
In our integrated formula dictionary, each query formula has the following properties:
- order 'ord', e.g., '1',
- entity URL 'item', e.g., 'https://arq20.formulasearchengine.com/entity/Q1023',
- question ID 'd', e.g., 'B.1',
- the âmanuallyâ retrieved relevant answer MSE ID 'val', e.g., '3063081'
- MathML string including the LaTeX formula string 'mml', e.g., '<math
xmlns="http://www.w3.org/1998/Math/MathML" display="block"
alttext="{displaystyle c>{frac {25}{64}}}">',
- identifiers list retrieved from MathML 'identifiers', e.g., ['c'],
- operators list retrieved from MathML 'operators', e.g., [â], and
- LaTeX formula string retrieved from MathML 'LaTeX', e.g., '{displaystyle
c>{frac {25}{64}}}'}.
The properties are retrieved from the Wiki SPARQL query.
In our integrated formula dictionary, each pool formula has the following properties:
- formula ID 'id', e.g., '1',
- 'post_id', e.g., '9',
- 'thread_id', e.g. '5',
- 'type', e.g., 'comment',
- MathML string 'formula', e.g., '"<?xml version=""1.0"" encoding=""UTF-
8""?><math xmlns=""http://www.w3.org/1998/Math/MathML""
alttext=""pi"" display=""block""> <ci>đ</ci></math>"',
- identifiers list retrieved from MathML 'identifiers', e.g., ['c'],
- operators list retrieved from MathML 'operators', e.g., [â], and
- LaTeX formula string retrieved from MathML 'LaTeX', e.g., 'pi'.
DI
â˘Load TSV files for query and pool formulae
â˘Retrieve formula symbols (identifiers, operators) from mathml-tags
('ci','mi','co','mo'), together with formula LaTeX string
â˘Integrate all formulae with IDs and save dictionary to a Python Pickle file
kNN
â˘Encode formula LaTeX strings via TF-IDF and Doc2Vec
â˘Retrieve distances and k-nearest formula candidates via kNN algorithm
fuzzy
â˘Calculate pairwise fuzzy string partial ratios (matching percentage)
â˘Rank all percentages for each formula to identify closest candidates
13. The properties are retrieved from the task dataset tsv files. For the identifiers and oper-
ators list, the symbols are retrieved from the MathML string. For the query formulae,
the search tags are '<mi>' and '<mo>', and for the pool formulae, '<ci>' and '<co>' for
identifiers and operators respectively. The formula LaTeX string is retrieved from the
'alttext' attribute of the '<math>' tag. Finally, the formula dictionary is serialized to a
pickle file. It is utilized in the following steps (formula encoding, kNN and fuzzy string
similarity retrieval).
4.7 Formula LaTeX String Encoding via TF-IDF and Doc2Vec
Having retrieved the LaTeX formula from the MathML string, it is encoded by jointly
feeding its identifier and operator tokens (utf-8) into the TfidfVectorizer from the Py-
thon package Scikit-learn [24] and the Doc2Vec encoder from Gensim [25]. For the
TfidfVectorizer, an ngram range of (1,1) is used. The Doc2Vec distributed bag of words
(PV-DBOW) model is trained for 10 iterations.
4.8 Formula Pool Retrieval via K-Nearest-Neighbors
The two different formula encodings vector spaces are subsequently fed into a Nearest-
Neighbors algorithm from Scikit-learn. In Table 3, some illustrative examples of the
top 3 results are displayed. In all cases, the retrieved formulae are structurally similar,
sometimes equivalent, sometimes even âvisuallyâ identical. Having generated the for-
mula encodings, the kNN method is very fast compared to classical text matching. The
vector computations can be carried out faster than text processing.
Table 3. Illustrative short examples of top 3 kNN results.
Query (Task 2 ID) Results (Task 2 Formula ID) Comment
đ < 25/64 (B.1) 1: k < 6.64⌠(77098),
2: 1/64 (144990),
3: 7/64 (95528)
Similar but wrong number,
No inequation,
"
5^{2}equiv
1({text{mod}}(B.8)
1: (a-b)^{n}equiv 0 (text{mod} n)
(54185),
2: a^{p-1}equiv 1 (text{mod} p)
(94320),
3: 2^{p-1}equiv 1 (text{mod} p)
(198801)
Structurally similar but contain-
ing variables instead of constants
{{frac{a+bi}{infty
}}=0} (B.29)
1: a+bi (272260),
2: z=a+bi (218917)
3: a+bi (272255)
The complex number a+bi is de-
tected and retrieved, infty missing
{p_{1}dots p_{n}+1}
(B.52)
1: p_{1}dots p_{k}+1 (2203),
2: p_{1}+p_{2}+dots p{n}=1
(76726),
3: p_{1}=p_{2}=dots=p_{6} (76715)
Formula 1 equivalent, using index
k instead of n, Formula 2 equiva-
lent, with additional information
(=1)
{sum_{k=0}^{n}k{bi-
nom{n}{k}}=n2^{n-
1}=2^{n-1+log_{2}n}}
(B.86)
1: sum^{k}_{m=0}bi-
nom{k}{m}=2^{k} (280774),
2: sum^{k}_{m=0}bi-
nom{k}{m}=2^{k} (280771)
3: sum_{k=0}^{n}binom{n}{k}k=
2^{n}sum_{k=1}^{n}frac{2^{k-
1}}{2^{k}}=âŚ
Formula 1 and 2 are identical and
almost equivalent to the query,
formula 3 starts the summation in-
dex at k=1
14. 4.9 Formula Pool Retrieval via Fuzzy String Search
Apart from the NearestNeighbors prediction using TF-IDF and Doc2Vec encoded La-
TeX formula strings, we also tested a fuzzy string matching to retrieve similar formulae.
For each âmanuallyâ selected query formula, we calculated the fuzzy partial ratio simi-
larity with all pool formulae and ranked them with descending overlap. The top 10 of
the candidates were then submitted. Compared to the kNN approach, the fuzzy string
search has the advantage of not requiring an encoding index. Thus new formula in-
stances can easily be added without requiring to retrain the vector encodings of the
whole corpus.
5 Classification of Question and Answer Types
To assess the relative relevance of the specific question, answer, and formula types, we
carried out a human multi-label classification for each set respectively. Our approach
was inductive, meaning that we did not specify the classes upfront but observed them
examining the questions, answers, and formulae as they occurred.
5.1 Example Questions and Answers
To illustrate our classification operation mode, we will first give some examples.
In question A.1, the user asks to find the value of a parameter contained within a func-
tion, given an interval constraint. We classified this question with the label âcalculate /
compute / find valueâ. Our manually selected answer25
for A.1 was labeled ânumeric
value / fractionâ, and âinequalityâ.
In question A.50, the user asks whether a series containing a fraction of powers and a
trigonometric function converges or diverges. We classified this question with the la-
bels âpower / exponential / logarithmicâ, âtrigonometryâ, and âsequence / summationâ.
Our manually selected formula for B.50, đđ ⤠â đđđđ { đđđ (
đśâ˛
3
) + (1 +
đ) đđđ đ} {đśâ˛
đ{đ}
}, was labeled âinequalityâ and âpowers / exponentials / logarithmsâ.
5.2 Question Types
We labeled the question types as shown in Table 4.
Table 4. Question type labels for Task 1.
Label Questions
25
https://math.stackexchange.com/questions/3062860/finding-value-of-c-such-that-the-range-
of-the-rational-function-fx-frac/3063081#3063081
15. Value / fraction A 1, 4
Complex
numbers
A 12, 24, 29
Parameter A 10, 28
Probability A 5
Modulus A 7, 21, 47
Pow / exp / log A 16, 18, 27, 39, 48, 49, 50, 51, 65, 75, 79
Integral A 10, 13, 16, 17, 26, 46, 82, 95
Trigonometry A 17, 26, 27, 28, 43, 45, 50, 58, 70, 82, 95
Approximation A 3
Solve equation A 2, 14, 26, 30, 43, 55, 58, 60, 67, 70, 71, 77, 86, 87, 89, 90
Limes A 8, 18, 26
Algorithmic trans-
formations
A 8, 9, 17, 26, 30, 43
Show / prove A 32, 36, 47
Seq / sum A 4, 9, 15, 22, 43, 46, 49, 50, 51, 59, 60, 71, 73, 83, 86
Metrics A 11
Function A 23, 25, 33, 34, 37, 40, 41, 42, 57, 63, 64, 74, 82, 84, 88, 91, 92, 95
Sets A 20, 38, 44, 45, 46, 47, 49, 52, 54, 57, 59, 61, 62, 63, 64, 68, 69, 74, 81, 83, 84,
88, 89, 92, 94, 96, 97, 98
Inequalities A 21, 48, 52, 60, 61, 65, 74, 79, 86, 87, 95
Derivative A 33, 35
Vectors /
matrices
A 44, 67, 90, 93, 97, 98
Interval A 46
Binomial A 49
Logic A 32, 36, 47, 52, 56, 57, 62, 64, 68, 81, 97
The occurrence statistics of the individual question types is shown in Fig. 5. Appar-
ently, the major part of the questions involved âsetsâ of numbers. This is partly caused
by the set symbols for natural numbers â or rational numbers â appearing frequently
in definitions that are included in the question. The second-highest ranked label is
âfunctionâ. This is not surprising considering that functions are a heavily used notion
or concept in mathematics. To obtain this label, it was sufficient that a function identi-
fier appears in the question. The third highest ranked label is âsolve equations â alge-
braic or differentialâ. In many cases, provided enough information, the question can be
answered by using a computer algebra system (QAS) connected to the question answer-
ing engine.
16. Fig. 5. Question type distribution of the ARQTask question selection.
5.3 Question Subject Classes
Classifying the question subject classes, we see that almost all questions are pure math-
ematics, except A 33 is from the math-stackexchange-category physics. Employing
subject class classifications can help to redirect questions and reducing the answer
space. Open-domain QA systems can then be modularized into distinct closed domain
parts that handle different QA types differently. For example, a geometry question such
as âWhat is the surface area of a sphere?â can be parsed and answered differently than
an algebraic question such as âHow to solve đĽ + 1 = 2?â. While the former could be
passed to a database containing properties of geometric objects, the latter could be
passed to a computer algebra system. On the other hand, physics questions often rely
heavily on the semantics of identifier names. As an example, the question âWhat is the
relationship between mass and energy?â should yield formulae such as đ¸ = đ đ2
or
đ¸ = ½ đ đŁ2
. Without having annotated identifier names contained within the formu-
lae, the question cannot be answered.
5.4 Answer Types
We labeled our manually retrieved answer types as shown in Table 5.
Table 5. Answer type labels for Task 1.
Label Answers for Questions
Value / fraction A 1
Probability A 5, 85
Binomial A 7, 41, 49, 51, 69, 86
Pow / exp / log A 7, 16, 18, 35, 39, 41, 43, 44, 47, 48, 51, 65, 73, 75, 85, 98
Interval A 3, 10, 46, 74, 82, 91
17. Seq / sum A 10, 13, 15, 18, 20, 22, 24, 26, 30, 41, 45, 49, 50, 51, 59, 69, 76, 94
Set A 5, 19, 34, 37, 38, 40, 41, 42, 47, 49, 50, 52, 54, 57, 59, 62, 69, 75, 76, 80, 81, 83,
84, 87, 92, 94, 96, 97
Inequality A 1, 29, 35, 46, 48, 50, 65, 74, 83, 87, 96, 98
Differential A 14
Integral A 2, 10, 16, 17, 18, 26, 45, 82, 86
Trigonometry A 12, 17, 24, 26, 27, 28, 43, 45, 50, 58, 70, 79, 82, 95
Function A 3, 20, 23, 25, 40, 42, 46, 47, 57, 59, 63, 64, 68, 84, 88, 91
Algebraic
transformation
A 12, 13, 14, 15, 16, 18, 20, 22, 25, 39, 48, 55, 58, 67, 69, 70, 71, 77, 79, 83, 85,
88, 90
Vector / matrix A 11, 40, 44, 67, 90, 93, 97
Logic A 32, 36, 38, 46, 52, 54, 56, 62, 68
Modulus A 19, 21
Complex numbers 24, 27, 29
Limes A 29, 75, 95
Deriv A 33, 86
Cases A 46
The occurrence statistics of the individual answer types is shown in Fig. 6. As for the
question types, âsetâ is still the most frequent label. However, âfunctionâ is here only
ranked fourth. The label âalgorithmic transformationâ is ranked second. Some of the
transformations can be done using computer algebra systems. Apparently, the answer
and question categories differ. This means, for example, that given a short question, the
potentially longer answer (proof or other) can involve more categories.
Fig. 6. Answer type distribution of the âmanuallyâ retrieved MSE answer candidates.
5.5 Formula Types
We labeled the formula types as shown in Table 6.
18. Table 6. Formula type labels for Task 2.
Label Formulae
Simple expressions B 77, 81, 89, 90
Number / fraction B 1, 18
Complex numbers B 12, 24, 27, 29, 55
Interval / range B 10
Parameter B 89
Inequality B 10, 34, 48, 50, 65, 74, 75, 79, 87, 95, 96, 98
Function B 2, 14, 15, 25, 40, 46, 57, 59, 63, 64, 68
Metrics B 84
Derivative B 33
Integral B 2, 10, 16, 17, 45, 46, 82
Binomial B 4, 41, 69, 86
Modulus B 5, 6, 47
Pow / exp / log B 5, 34, 43, 47, 48, 50, 60, 65, 73, 75, 76, 79, 80, 86, 92, 98
Trigonometry B 12, 24, 27, 28, 43, 45, 57, 58, 70, 79
Limes B 17, 60, 75
Cases B 45
Sets B 76, 92
Approximations B 8
Algebraic
transformations
B 4, 9, 11, 13, 16, 40, 55, 71, 74, 75, 85, 86, 88
Sequence / sum B 4, 9, 13, 20, 30, 43, 52, 54, 60, 69, 71, 75, 83, 86, 87, 94, 96, 97
Vectors / matrices B 11, 33, 93, 94
Logic B 36, 56
The occurrence statistics of the individual formula types is shown in Fig. 7. Algebraic
transformations and functions are still ranked high. All in all, the most frequent ques-
tion, answer, and formula types involve sets, sequences, sums, powers, exponentials,
logarithms, trigonometry functions, inequalities, and algebraic transformations, or
equation solving. In the future, one could explore whether the question classification
label is enhancing answer retrieval.
19. Fig. 7. Formula answer type distribution of the âmanuallyâ retrieved LaTeX candidatesâ strings.
6 Discussion of Challenges
Table 7 shows the results of our submission in the ARQMath lab. For Task 1, the re-
ported nDCG' score for our manual run is outstandingly low. Hence, we tried to inves-
tigate the reasons for this low score. We identified one critical issue in our manual run.
We have linked the posts from the ARQMath dataset with the real posts in MSE, which
makes it easier to crawl for relevant answers manually. However, this approach leads
to the problem that some of our reported answers do not exist in the ARQMath dataset.
Nonetheless, the nDCG' removes non-judged documents prior to evaluation. Hence, a
relatively high number of answers that do not exist in the dataset should not harm our
score dramatically. We can report an nDCG' score of 0.504 for our submitted run. This
is significantly higher than the reported score by the ARQMath result paper [29]. We
calculated the nDCGâ score as formulated in [30] and [31]
nDCGp
â˛
=
DCGp
â˛
IDCGp
â˛
,
where
DCGp
â˛
= â
2reli â 1
log2(đ + 1)
p
đ=1
IDCGp
â˛
= â
2reli â 1
log2(đ + 1)
|RELp|
đ=1
and reli is the given relevance score for the đ-th element, and RELp is the list of relevant
documents ordered by their relevance up to position đ. In other words, the nDCGp
â˛
score
is the DCGp
â˛
score divided by the DCGp
â˛
score for the ideal order of relevant hits. The
nDCGp
â˛
is calculated for every query in the test set. The overall score is therefore cal-
culated as the mean value of nDCGp
â˛
over all queries.
20. We identified two possible issues that could explain the mismatch between our calcu-
lated score and the reported one. The nDCGp
â˛
score is calculated for a fixed number đ
of retrieved top hits. If đ is larger than the number of retrieved documents, it would
reduce the score. We assume that most contestants reported a list of relevant hits for
each query. Since we performed a manual run, we only reported the actual answer. This
means, for our reported answers it only makes sense to set đ = 1.
Moreover, we did not report valid answers for some queries (in case the answer ID did
not exist in the dataset, we had no valid answer in total for that particular query). If
these queries were considered when calculating the mean nDCGp
â˛
over all queries, it
would also explain a significantly lower score. The nDCGp
â˛
is designed to not taking
unjudged documents into account. Similarly, it makes sense to ignore queries with no
returned answers when calculating the overall nDCGp
â˛
over all queries. Following these
rules, we calculated an nDCGp
â˛
of 0.504 for our manual run. Table 10 in the Appendix
shows the results for our DCG1
â˛
und IDCG1
â˛
scores for all queries of Task 1, for which
we retrieved answers in our manual run and were ranked by the ARQMath reviewers.
The final average score for nDCG1
â˛
is 0.504.
In addition to the problematic score calculation, we found incomprehensible relevance
scores on multiple occasions. A possible reason for this is the subjectiveness of rele-
vance. While we found the reported answers highly relevant, the annotators provided a
relevance score of 0. Table 8 summarizes the identified problematic annotations. In
five out of nine of these cases, our reported answers were marked as correct by the
questioner at MSE (last column in Table 8) but annotated as non-relevant by the
ARQMath annotator. This seems to indicate that the relevance scores for ARQMath
tasks 1 and 2 are very subjective, even though the reported Kappa coefficient for inter-
annotator agreement was reasonably high with around 0.34.
Table 7. Results of the zbMATH participation submission at the ARQMath Lab.
RUN DATA nDCG' MAP' P@10
zbMATH Text & Math 0.101 0.053 0.030
Table 8. Topic and Post IDs that are marked as non-relevant by the ARQMath task reviewers
[29] but annotated as correct / helpful by the questioner in the Math Stack Exchange forum.
Topic ID Post ID Relevance MSE Marked as Correct
A.17 5322 0 Yes
A.21 65456 0 Yes
A.35 170589 0 No
A.42 331468 0 No
A.50 110019 0 No
21. A.68 188661 0 Yes
A.75 2146297 0 Yes
A.93 311354 0 Yes
A.96 893752 0 No
6.1 Linking Text and Formulae
In the process of manual annotation and answer retrieval, we noticed several challenges
for IR systems. First, the question and answer features are obviously very heterogene-
ous data types (text and formulae). It remains to be explored how to combine both in a
suitable way. Recent studies [32] investigated the impact of different encoding combi-
nations on the classification accuracy and cluster purity on the NTCIR-11/12 arXiv
dataset [33]. They called out for a âformula encoding challengeâ to exploit the formula
information for machine learning tasks. A successful encoding should, e.g., improve
the text classification accuracy. The aim is motivated by the observation that there is
little correlation between text and formula similarity, at least using the cosine measure
on tf-idf and doc2vec encodings. We need to somehow connect text and math, such that
there is a synergy between their semantics. In the case of the mathematical question
answering task, this could be achieved by transforming the mathematical formula ele-
ments to textual entities. Consider for example the ARQ Task question A.29. The ques-
tion asks for a recipe to divide complex numbers by infinity (title: âDividing Complex
Numbers by Infinityâ). For this question, we manually retrieved the formula
đ+đđ
â
= 0
from the answer that was selected by the questioner on MSE. One way to connect the
question to possible answer formulae would be to annotate both textual elements. Table
9 shows how linking to items of the semantic knowledge-base Wikidata8
[20], [21] can
provide a connection via the joint QIDs Q1226939, Q11567, and Q205. A joint seman-
tic vector representation of both the title text and the formula could then be a concate-
nation of the Wikidata item embeddings, as proposed in [34].
Table 9. Possible semantic annotations of the question A.29 âDividing Complex Numbers by
Infinityâ to link text and formulae using Wikidata8
QIDs.
Question text annotation Formula answer annotation
âDividingâ: âdivisionâ (Q1226939) đ+đđ
â
: âdivisionâ (Q1226939)
âAddingâ: âadditionâ (Q32043) đ + đđ: âadditionâ (Q32043)
N/A đ + đđ: âcomplex numberâ (Q11567)
N/A đ: âreal numberâ (Q12916)
N/A đđ: âcomplex numberâ (Q9165172)
âInfinityâ: âinfinityâ (Q205) â: âinfinityâ (Q205)
This example illustrates how linking Formula Concepts [16], [17] can be very benefi-
cial for mathematical question answering (on MSE, arXiv, Wikipedia, etc.). However,
this requires the semantic annotation of textual and formula elements, which can be
22. done, e.g., using the âAnnoMathTeXâ26
system [19] hosted by Wikimedia. In the fu-
ture, we should be able to automatically link text and formula entities to Wikidata items
and Wikipedia articles. It remains a challenging problem for mathematical formula en-
tity linking to exhaustively and unambiguously identify the important semantic parts of
a formula. In the future, annotation guidelines should be developed to tackle this prob-
lem.
6.2 Formula Search and Retrieval
For Task 2, we used the MOI search engine to retrieve relevant mathematical expres-
sions from the dataset. Since the MOI engine does not handle entire mathematical ex-
pressions by itself but disassemble formulae into their subexpressions, the concept of
linking retrieved MOIs back to a formula ID was challenging. Furthermore, the ap-
proach we used to calculate the formula ID of an MOI has some drawbacks. First, the
MOI engine retrieves relevant documents from elasticsearch with a textual search
query. In the second step, the MOIs are scored based on the retrieved documents. Thus,
the retrieved MOIs (and the corresponding formula IDs) are as good as the retrieved
documents in the first task. When the retrieved documents are not relevant, none of the
retrieved MOIs can be relevant. Hence, the search results are quite sensitive to the set-
tings that were used to retrieve relevant documents. Nonetheless, the approach per-
formed reasonably well compared to the results of other competitors with an nDCGp
â˛
score of 0.374.
7 Outlook and Future Work
We are excited to employ our approaches and the approaches of other task participants
to retrieve relevant formulae on zbMATH datasets. However, as discussed before, we
are uncertain if the computed performance numbers are a suitable indicator to predict
the usefulness of the approaches to zbMATH users. We will, therefore, consider sug-
gesting a mathematical literature retrieval task in the future. However, as a prerequisite,
we see the need to research math specific deterministic evaluation metrics that eliminate
task-specific human annotators in the loop. In contrast, we believe that objective veri-
fiable or almost provable semantic enhancement techniques can significantly benefit
from a human review. While relevant (to an information need) is not yet a well-estab-
lished term among working mathematicians, definitions, equivalences, examples, sub-
stitutions, theorems and proves are well established. While formal mathematics is not
(yet) able to automatically map mathematical named entities to formal concepts, work-
ing mathematicians are generally able to create such a mapping with a very high inter-
reviewer agreement. Therefore, we aim to explore how employing our âAnnoMath-
TeXâ formula annotation recommender system [19] on MSE questions and answers
can promote answer retrieval.
26
annomathtex.wmflabs.org
23. To summarize the marginal results from our contribution, the kNN method can be em-
ployed as a fast search engine, provided formulae are indexed as vector encodings. The
fuzzy string search is slower but has the advantage that no index is needed. As for MOI,
the retrieved results are less strictly tied to existing expressions since it considers all
subexpressions in an entire dataset. This helps to extract meaningful expressions rather
than exact matches.
8 Acknowledgments
This work was supported by the German Research Foundation (DFG grant GI-1259-1).
References
[1] F. MĂźller and O. Teschke, âFull text formula search in zbMATH,â Eur. Math. Soc.
Newsl, vol. 102, p. 51, 2016.
[2] B. Mansouri, A. Agarwal, D. Oard, and R. Zanibbi, âFinding Old Answers to New
Math Questions: The ARQMath Lab at CLEF 2020,â in Advances in Information
Retrieval, vol. 12036, J. M. Jose, E. Yilmaz, J. MagalhĂŁes, P. Castells, N. Ferro,
M. J. Silva, and F. Martins, Eds. Cham: Springer International Publishing, 2020,
pp. 564â571.
[3] B. Mansouri, R. Zanibbi, and D. W. Oard, âCharacterizing Searches for Mathe-
matical Concepts,â in 2019 ACM/IEEE Joint Conference on Digital Libraries
(JCDL), Champaign, IL, USA, Jun. 2019, pp. 57â66, doi:
10.1109/JCDL.2019.00019.
[4] H. Karbasian and A. Johri, âInsights for Curriculum Development: Identifying
Emerging Data Science Topics through Analysis of Q&A Communities,â in Pro-
ceedings of the 51st ACM Technical Symposium on Computer Science Education,
Portland OR USA, Feb. 2020, pp. 192â198, doi: 10.1145/3328778.3366817.
[5] N. W. Smith, âA Question-Answering System for Elementary Mathematics,â Apr.
1974, Accessed: Jun. 22, 2020. [Online]. Available:
https://eric.ed.gov/?id=ED093703.
[6] T. T. Nguyen, K. Chang, and S. C. Hui, âA math-aware search engine for math
question answering system,â in Proceedings of the 21st ACM international con-
ference on Information and knowledge management - CIKM â12, Maui, Hawaii,
USA, 2012, p. 724, doi: 10.1145/2396761.2396854.
[7] A. Bhattacharya, âA Survey of Question Answering for Math and Science Prob-
lem,â Computing Research Repository (CoRR), May 2017, Accessed: Jun. 08,
2020. [Online]. Available: http://arxiv.org/abs/1705.04530.
[8] A. A. S. Gunawan, P. R. Mulyono, and W. Budiharto, âIndonesian Question An-
swering System for Solving Arithmetic Word Problems on Intelligent Humanoid
Robot,â Procedia Computer Science, vol. 135, pp. 719â726, 2018, doi:
10.1016/j.procs.2018.08.213.
[9] M. Schubotz, P. Scharpf, K. Dudhat, Y. Nagar, F. Hamborg, and B. Gipp, âIntro-
ducing MathQA -- A Math-Aware Question Answering System,â Information
24. Discovery and Delivery, vol. 46, no. 4, pp. 214â224, Nov. 2018, doi:
10.1108/IDD-06-2018-0022.
[10] M. Hopkins, R. Le Bras, C. Petrescu-Prahova, G. Stanovsky, H. Hajishirzi, and R.
Koncel-Kedziorski, âSemEval-2019 Task 10: Math Question Answering,â in Pro-
ceedings of the 13th International Workshop on Semantic Evaluation, Minneap-
olis, Minnesota, USA, 2019, pp. 893â899, doi: 10.18653/v1/S19-2153.
[11] D. C. Pineau, âMath-Aware Search Engines: Physics Applications and Overview,â
Computing Research Repository (CoRR), Sep. 2016, Accessed: Jun. 21, 2020.
[Online]. Available: http://arxiv.org/abs/1609.03457.
[12] A. Abdi, N. Idris, and Z. Ahmad, âQAPD: an ontology-based question answering
system in the physics domain,â Soft Comput, vol. 22, no. 1, pp. 213â230, Jan.
2018, doi: 10.1007/s00500-016-2328-2.
[13] T. Suzuki and A. Fujii, âMathematical Document Categorization with Structure of
Mathematical Expressions,â in 2017 ACM/IEEE Joint Conference on Digital Li-
braries (JCDL), Toronto, ON, Canada, Jun. 2017, pp. 1â10, doi:
10.1109/JCDL.2017.7991566.
[14] M. Schubotz, P. Scharpf, O. Teschke, A. K. uhnemund, C. Breitinger, and B. Gipp,
âAutoMSC: Automatic Assignment of Mathematics Subject Classification La-
bels,â Proceedings of the CICM Conference 2020, May 2020, Accessed: Jun. 21,
2020. [Online]. Available: http://arxiv.org/abs/2005.12099.
[15] S. Yang and Y. Ko, âMathematical Formula Search using Natural Language Que-
ries,â AECE, vol. 14, no. 4, pp. 99â104, 2014, doi: 10.4316/AECE.2014.04015.
[16] A. Dmello, âRepresenting Mathematical Concepts Associated With Formulas Us-
ing Math Entity Cards,â Rochester Institute of Technology (RIT) Scholar Works,
p. 167.
[17] P. Scharpf, M. Schubotz, H. S. Cohl, and B. Gipp, âTowards Formula Concept
Discovery and Recognition,â Proceedings of the 4th BIRNDL Workshop at the
42nd ACM SIGIR Conference 2019, p. 8.
[18] M. A. Dumitru, D. Ginev, M. Kohlhase, V. Merticariu, S. Mirea, and T. Wiesing,
âSystem Description: KAT an Annotation Tool for STEM Documents,â Proceed-
ings of the CICM Conference 2016, p. 4.
[19] P. Scharpf, I. Mackerracher, M. Schubotz, J. Beel, C. Breitinger, and B. Gipp,
âAnnoMathTeX - a formula identifier annotation recommender system for STEM
documents,â in Proceedings of the 13th ACM Conference on Recommender Sys-
tems, Copenhagen Denmark, Sep. 2019, pp. 532â533, doi:
10.1145/3298689.3347042.
[20] P. Scharpf, M. Schubotz, and B. Gipp, âRepresenting Mathematical Formulae in
Content MathML using Wikidata,â Proceedings of the 3th BIRNDL Workshop at
the 41st ACM SIGIR Conference 2018, p. 14.
[21] M. Schubotz, âGenerating OpenMath Content Dictionaries from Wikidata,â Pro-
ceedings of the CICM Conference 2018, p. 8.
[22] M. Schubotz, A. Greiner-Petter, P. Scharpf, N. Meuschke, H. S. Cohl, and B. Gipp,
âImproving the Representation and Conversion of Mathematical Formulae by
Considering their Textual Context,â in Proceedings of the 18th ACM/IEEE on
25. Joint Conference on Digital Libraries, Fort Worth Texas USA, May 2018, pp.
233â242, doi: 10.1145/3197026.3197058.
[23] Q. Le and T. Mikolov, âDistributed Representations of Sentences and Docu-
ments,â Proceedings of the ICML Conference 2014, p. 9.
[24] F. Pedregosa et al., âScikit-learn: Machine Learning in Python,â MACHINE
LEARNING IN PYTHON, p. 6.
[25] R. ĹehĹŻĹek and P. Sojka, Software Framework for Topic Modelling with Large
Corpora. University of Malta, 2010.
[26] A. Greiner-Petter et al., âDiscovering Mathematical Objects of InterestâA Study
of Mathematical Notations,â in Proceedings of The Web Conference 2020, Taipei
Taiwan, Apr. 2020, pp. 1445â1456, doi: 10.1145/3366423.3380218.
[27] M. Schubotz and G. Wicke, âMathoid: Robust, Scalable, Fast and Accessible Math
Rendering for Wikipedia,â in Intelligent Computer Mathematics - International
Conference, CICM 2014, Coimbra, Portugal, July 7-11, 2014. Proceedings, 2014,
vol. 8543, pp. 224â235, doi: 10/ggv8pz.
[28] S. Robertson and H. Zaragoza, âThe Probabilistic Relevance Framework: BM25
and Beyond,â Found. Trends Inf. Retr., vol. 3, no. 4, pp. 333â389, Apr. 2009, doi:
10.1561/1500000019.
[29] R. Zanibbi, D. W. Oard, A. Agarwal, and B. Mansouri, âOverview of ARQMath
2020: CLEF Lab on Answer Retrieval for Questions on Math,â p. 25.
[30] K. Järvelin and J. Kekäläinen, âCumulated gain-based evaluation of IR tech-
niques,â ACM Trans. Inf. Syst., vol. 20, no. 4, pp. 422â446, Oct. 2002, doi:
10.1145/582415.582418.
[31] C. Burges et al., âLearning to rank using gradient descent,â in Proceedings of the
22nd international conference on Machine learning, Bonn, Germany, Aug. 2005,
pp. 89â96, doi: 10.1145/1102351.1102363.
[32] P. Scharpf, M. Schubotz, A. Youssef, F. Hamborg, N. Meuschke, and B. Gipp,
âClassification and Clustering of arXiv Documents, Sections, and Abstracts, Com-
paring Encodings of Natural and Mathematical Language,â Proceedings of the
JCDL Conference 2020, May 2020, doi: 10.1145/3383583.3398529.
[33] R. Zanibbi, A. Aizawa, and M. Kohlhase, âNTCIR-12 MathIR Task Overview,â
Proceedings of the 12th NTCIR Conference on Evaluation of Information Access
Technologies 2016, p. 10.
[34] A. Lerer et al., âPyTorch-BigGraph: A Large-scale Graph Embedding System,â
Proceedings of the MLSys Conference 2019, Apr. 2019, Accessed: Jul. 16, 2020.
[Online]. Available: http://arxiv.org/abs/1903.12287.
26. 9 Appendix
Table 10. Results for DCG1
â˛
, IDCG1
â˛
, and nDCG1
â˛
scores for all queries of Task 1, for which we
retrieved answers in our manual run and were ranked by the ARQMath reviewers. The final
average nDCG1
â˛
score is 0.504. The metrics rel_1 and REL_1 refer to the formulae in Section 6
on page 19.
Topic ID Post ID Relevance
đŤđđĽđ
Best Relevance
đŤđđĽđ in đđđđ
đđđđ
â˛
đđđđđ
â˛
đ§đđđđ
â˛
A.12 44410 2 3 3 7 0.43
A.13 1115317 2 3 3 7 0.43
A.14 2248783 3 3 7 7 1
A.16 408304 1 3 1 7 0.14
A.17 5322 0 3 0 7 0
A.19 1348396 3 3 7 7 1
A.20 23977 2 3 3 7 0.43
A.21 65456 0 3 0 7 0
A.30 2721623 3 3 7 7 1
A.35 170589 0 3 0 7 0
A.37 11442 3 3 7 7 1
A.41 334435 3 3 7 7 1
A.42 331468 0 3 0 7 0
A.45 422348 3 3 7 7 1
A.47 2326614 2 3 3 7 0.43
A.50 110019 0 3 0 7 0
A.52 632129 1 3 1 7 0.14
A.54 39285 3 3 7 7 1
A.56 412396 2 3 3 7 0.43
A.59 194715 3 3 7 7 1
A.60 381303 2 3 3 7 0.43
A.62 659332 3 3 7 7 1
A.63 319310 2 2 3 3 1
A.67 75362 2 3 3 7 0.43
A.68 188661 0 3 0 7 0
A.69 1490891 3 3 7 7 1
A.74 705071 2 3 3 7 0.43
A.75 2146297 0 3 0 7 0
A.85 364135 2 3 3 7 0.43
A.93 311354 0 3 0 7 0