The document provides an overview of the NTCIR-14 CENTRE Task, which aims to examine the replicability and reproducibility of results from past CLEF, NTCIR, and TREC evaluations. It describes the task specifications, including the replicability and reproducibility subtasks that asked participants to replicate or reproduce past run pairs. It also discusses the additional relevance assessments that were collected and the evaluation measures used, such as root mean squared error and effect ratio. The only participating team was able to mostly replicate the effects observed in the original NTCIR runs for the replicability subtask.
Overview of the NTCIR-15 We Want Web with CENTRE (WWW-3) Task
by
Tetsuya Sakai, Sijie Tao, Zhaohao Zeng
Yukun Zheng, Jiaxin Mao, Zhumin Chu, Yiqun Liu
Maria Maistro
Zhicheng Dou
Nicola Ferro
Ian Soboroff
Faster Evolutionary Multi-Objective Optimization via GALE: the Geometric Acti...Joe Krall
Abstract Goal optimization has long been a topic of great interest in computer science. The literature contains many thousands of papers that discuss methods for the search of optimal solutions to complex problems. In the case of multi-objective optimization, such a search yields iteratively improved approximations to the Pareto frontier, i.e. the set of best solutions contained along a trade-off curve of competing objectives.
To approximate the Pareto frontier, one method that is ubiquitous throughout the field of optimization is stochastic search. Stochastic search engines explore solution spaces by randomly mutating candidate guesses to generate new solutions. This mutation policy is employed by the most commonly used tools (e.g. NSGA-II, SPEA2, etc.), with the goal of a) avoiding local optima, and b) expand upon diversity in the set of generated approximations. Such "blind" mutation policies explore many sub-optimal solutions that are discarded when better solutions are found. Hence, this approach has two problems. Firstly, stochastic search can be unnecessarily computationally expensive due to evaluating an overwhelming number of candidates. Secondly, the generated approximations to the Pareto frontier are usually very large, and can be difficult to understand.
To solve these two problems, a more-directed, less-stochastic approach than standard search tools is necessary. This thesis presents GALE (Genetic Active Learning). GALE is an active learner that finds approximations to the Pareto frontier by spectrally clustering candidates using a near-linear time recursive descent algorithm that iteratively divides candidates into halves (called leaves at the bottom level). Active learning in GALE selects a minimally most-informative subset of candidates by only evaluating the two-most different candidates during each descending split; hence, GALE only requires at most, $2LogN$ evaluations per generation. The candidates of each leaf are thereafter non-stochastically mutated in the most promising directions along each piece. Those leafs are piece-wise approximations to the Pareto frontier.
The experiments of this thesis lead to the following conclusion: a near-linear time recursive binary division of the decision space of candidates in a multi-objective optimization algorithm can find useful directions to mutate instances and find quality solutions much faster than traditional randomization approaches. Specifically, in comparative studies with standard methods (NSGA-II and SPEA2) applied to a variety of models, GALE required orders of magnitude fewer evaluations to find solutions. As a result, GALE can perform dramatically faster than the other methods, especially for realistic models.
Design and evaluation of a genomics variant analysis pipeline using GATK Spar...Paolo Missier
A paper presented at the annual Italian Database conference (SEBD): http://sisinflab.poliba.it/sebd/2018/
here is the paper: http://sisinflab.poliba.it/sebd/2018/papers/June-27-Wednesday/1-Big-Data/SEBD_2018_paper_23.pdf
The talk I gave at the Stream Reasoning workshop in TU Berlin on December 8. I give an overview of RSEP-QL and how it can capture and formalise the behaviour of existing RSP engines, e.g. CSPARQL, EP-SPARQL, CQELS, SPARQLstream
Overview of the NTCIR-15 We Want Web with CENTRE (WWW-3) Task
by
Tetsuya Sakai, Sijie Tao, Zhaohao Zeng
Yukun Zheng, Jiaxin Mao, Zhumin Chu, Yiqun Liu
Maria Maistro
Zhicheng Dou
Nicola Ferro
Ian Soboroff
Faster Evolutionary Multi-Objective Optimization via GALE: the Geometric Acti...Joe Krall
Abstract Goal optimization has long been a topic of great interest in computer science. The literature contains many thousands of papers that discuss methods for the search of optimal solutions to complex problems. In the case of multi-objective optimization, such a search yields iteratively improved approximations to the Pareto frontier, i.e. the set of best solutions contained along a trade-off curve of competing objectives.
To approximate the Pareto frontier, one method that is ubiquitous throughout the field of optimization is stochastic search. Stochastic search engines explore solution spaces by randomly mutating candidate guesses to generate new solutions. This mutation policy is employed by the most commonly used tools (e.g. NSGA-II, SPEA2, etc.), with the goal of a) avoiding local optima, and b) expand upon diversity in the set of generated approximations. Such "blind" mutation policies explore many sub-optimal solutions that are discarded when better solutions are found. Hence, this approach has two problems. Firstly, stochastic search can be unnecessarily computationally expensive due to evaluating an overwhelming number of candidates. Secondly, the generated approximations to the Pareto frontier are usually very large, and can be difficult to understand.
To solve these two problems, a more-directed, less-stochastic approach than standard search tools is necessary. This thesis presents GALE (Genetic Active Learning). GALE is an active learner that finds approximations to the Pareto frontier by spectrally clustering candidates using a near-linear time recursive descent algorithm that iteratively divides candidates into halves (called leaves at the bottom level). Active learning in GALE selects a minimally most-informative subset of candidates by only evaluating the two-most different candidates during each descending split; hence, GALE only requires at most, $2LogN$ evaluations per generation. The candidates of each leaf are thereafter non-stochastically mutated in the most promising directions along each piece. Those leafs are piece-wise approximations to the Pareto frontier.
The experiments of this thesis lead to the following conclusion: a near-linear time recursive binary division of the decision space of candidates in a multi-objective optimization algorithm can find useful directions to mutate instances and find quality solutions much faster than traditional randomization approaches. Specifically, in comparative studies with standard methods (NSGA-II and SPEA2) applied to a variety of models, GALE required orders of magnitude fewer evaluations to find solutions. As a result, GALE can perform dramatically faster than the other methods, especially for realistic models.
Design and evaluation of a genomics variant analysis pipeline using GATK Spar...Paolo Missier
A paper presented at the annual Italian Database conference (SEBD): http://sisinflab.poliba.it/sebd/2018/
here is the paper: http://sisinflab.poliba.it/sebd/2018/papers/June-27-Wednesday/1-Big-Data/SEBD_2018_paper_23.pdf
The talk I gave at the Stream Reasoning workshop in TU Berlin on December 8. I give an overview of RSEP-QL and how it can capture and formalise the behaviour of existing RSP engines, e.g. CSPARQL, EP-SPARQL, CQELS, SPARQLstream
Getting the Most out of Transition-based Dependency ParsingJinho Choi
This paper suggests two ways of improving transition-based, non-projective dependency parsing. First, we add a transition to an existing non-projective parsing algorithm, so it can perform either projective or non-projective parsing as needed. Second, we present a boot- strapping technique that narrows down discrepancies between gold-standard and automatic parses used as features. The new addition to the algorithm shows a clear advantage in parsing speed. The bootstrapping technique gives a significant improvement to parsing accuracy, showing near state-of-the- art performance with respect to other parsing approaches evaluated on the same data set.
DISCLAIMER: These are not my slides, but these slides are for the IPAW paper by Quan Pham, Tanu Malik, and Ian Foster: Auditing and Maintaining Provenance in Software Packages
Capturing and querying fine-grained provenance of preprocessing pipelines in ...Paolo Missier
a talk given at the VLDB 2021 conference, August, 2021, presenting our paper:
Capturing and Querying Fine-grained Provenance of Preprocessing Pipelines in Data Science. Chapman, A., Missier, P., Simonelli, G., & Torlone, R. PVLDB, 14(4):507–520, January, 2021.
http://doi.org/10.14778/3436905.3436911
The increasing amount of valuable semi-structured data has become available online. In this talk, we overview the state of the art in entity ranking over structured data ("linked data").
Webinar: What's New in Pipeline Pilot 8.5 Collection Update 1?BIOVIA
Collection Update 1 for Pipeline Pilot 8.5 includes key new features for the Pipeline Pilot Client, as well as the Imaging, Next Gen Sequencing, Chemistry, Documents and Text, and Statistics and Modeling collections. An exciting new feature for the Pipeline Pilot Client is Protocol Comparison – the ability to compare protocols, or versions of protocols, allowing you to see and resolve differences between them.
Heaven: Supporting Systematic Comparative Research of RDF Stream Processing E...Riccardo Tommasini
Stream Reasoning (SR) research field is grown enough to prove that reasoning upon rapidly changing information is possible. RDF Stream Processing (RSP) Engines, systems capable to handle at semantic level RDF-encoded information flows, are increasing in number of implemented solutions. Now the Stream Reasoning community is working on the standardization of the methods and tools that supported their development.
Many Computer Science (CS) research fields shown their interest for a deeper comprehension of their own work nature. Studies like [46, 51] investi- gated the publications in those field, highlighting that the majority of them are allied to an Engineering epistemology. However, they also evinced and criticized the concrete differences with other engineering research areas, which focus on evaluation of the proposed systems and not only on their design and development.
The lacks of an empirical approach can be ascribed to the complex nature of the software systems. However, it is possible to face such studies that can not be easily modeled, reducing the complexity of the analysis keeping intact the relevance of each involved system. In social science and economy, where researchers deal with cross case studies, it is commonly used a System- atic Comparative Research Approach (SCRA) within an experimental setting, which grants properties like repeatability, reproducibility and comparability to build the evaluation upon.
The SR community agreed that it is mandatory evaluating RSP Engines, understanding how these systems perform in real uses cases. Recent works in the filed [53, 41, 19] pursued this goal, providing benchmarks for RSP Engines evaluation. Further analysis pointed out the challenges involved by the Stream Reasoning research and posed the basis for a proper RSP Engines evaluation, describing in detail where previous works have failed and how the can be
improved [44].
The limitations of the existing benchmarking proposals proved that the
empirical evaluation of RSP Engines is just at the beginning. What is still missing in an infrastructure that allows to compare, possibly automatically, the performances of many RSP Engines and that grants the properties of an experimental setting. In this thesis we brace this challenge borrowing from the aerospace engineering the idea of an engine test stand, which is an automatic facility for engine testing and development.
A test stand allows to design experiments and to execute them, evaluat- ing engines in a controlled environment. Thus, we formulate the following research question: ”Can an engine test stand, together with queries, datasets and methods, support Systematic Comparative Research Approach for Stream Reasoning? ”
In this thesis we propose Heaven, an open source framework that enables the Systematic Comparative Approach in the Stream Reasoning research field. Heaven consists of: an RSP Engine Test Stand, which emulates the aerospace engineering facility in the Stream Rea
Effective and Efficient Entity Search in RDF dataRoi Blanco
Triple stores have long provided RDF storage as well as data access using expressive, formal query languages such as SPARQL. The new end users of the Semantic Web, however, are mostly unaware of SPARQL and overwhelmingly prefer imprecise, informal keyword queries for searching over data. At the same time, the amount of data on the Semantic Web is approaching the limits of the architectures that provide support for the full expressivity of SPARQL. These factors combined have led to an increased interest in semantic search, i.e. access to RDF data using Information Retrieval methods. In this work, we propose a method for effective and efficient entity search over RDF data. We describe an adaptation of the BM25F ranking function for RDF data, and demonstrate that it outperforms other state-of-the-art methods in ranking RDF resources. We also propose a set of new index structures for efficient retrieval and ranking of results. We implement these results using the open-source MG4J framework.
Developers often wonder how to implement a certain functionality
(e.g., how to parse XML files) using APIs. Obtaining
an API usage sequence based on an API-related natural
language query is very helpful in this regard. Given a query,
existing approaches utilize information retrieval models to
search for matching API sequences. These approaches treat
queries and APIs as bags-of-words and lack a deep understanding
of the semantics of the query.
We propose DeepAPI, a deep learning based approach to
generate API usage sequences for a given natural language
query. Instead of a bag-of-words assumption, it learns the
sequence of words in a query and the sequence of associated
APIs. DeepAPI adapts a neural language model named
RNN Encoder-Decoder. It encodes a word sequence (user
query) into a fixed-length context vector, and generates an
API sequence based on the context vector. We also augment
the RNN Encoder-Decoder by considering the importance
of individual APIs. We empirically evaluate our approach
with more than 7 million annotated code snippets collected
from GitHub. The results show that our approach generates
largely accurate API sequences and outperforms the related
approaches.
A relatively short Introduction to R as presented at the Belgian Software Craftmanship meetup group.
The goal of this presentation is to give you an introduction to:
• The style of the language
• It's ecosystem
• How common things like data manipulation and visualization work
• How to use it for machine learning
• Webdevelopment and report generation in R
• Integrating R in your system
License:
Introduction To R by Samuel Bosch
To the extent possible under law, the person who associated CC0 with Introduction To R has waived all copyright and related or neighboring rights
to Introduction To R.
http://creativecommons.org/publicdomain/zero/1.0/
Pickens, J., Cooper, M., and Golovchinsky, G. Reverted Indexing for Expansion and Feedback. In Proc. CIKM 2010, Toronto, Canada, ACM Press. See http://fxpal.com/?p=abstract&abstractID=581
Presented on 4 Dec 2014
@Jeju, South Korea
Title: An efficient method for assessing the impact of refactoring candidates on maintainability based on matrix computation
Getting the Most out of Transition-based Dependency ParsingJinho Choi
This paper suggests two ways of improving transition-based, non-projective dependency parsing. First, we add a transition to an existing non-projective parsing algorithm, so it can perform either projective or non-projective parsing as needed. Second, we present a boot- strapping technique that narrows down discrepancies between gold-standard and automatic parses used as features. The new addition to the algorithm shows a clear advantage in parsing speed. The bootstrapping technique gives a significant improvement to parsing accuracy, showing near state-of-the- art performance with respect to other parsing approaches evaluated on the same data set.
DISCLAIMER: These are not my slides, but these slides are for the IPAW paper by Quan Pham, Tanu Malik, and Ian Foster: Auditing and Maintaining Provenance in Software Packages
Capturing and querying fine-grained provenance of preprocessing pipelines in ...Paolo Missier
a talk given at the VLDB 2021 conference, August, 2021, presenting our paper:
Capturing and Querying Fine-grained Provenance of Preprocessing Pipelines in Data Science. Chapman, A., Missier, P., Simonelli, G., & Torlone, R. PVLDB, 14(4):507–520, January, 2021.
http://doi.org/10.14778/3436905.3436911
The increasing amount of valuable semi-structured data has become available online. In this talk, we overview the state of the art in entity ranking over structured data ("linked data").
Webinar: What's New in Pipeline Pilot 8.5 Collection Update 1?BIOVIA
Collection Update 1 for Pipeline Pilot 8.5 includes key new features for the Pipeline Pilot Client, as well as the Imaging, Next Gen Sequencing, Chemistry, Documents and Text, and Statistics and Modeling collections. An exciting new feature for the Pipeline Pilot Client is Protocol Comparison – the ability to compare protocols, or versions of protocols, allowing you to see and resolve differences between them.
Heaven: Supporting Systematic Comparative Research of RDF Stream Processing E...Riccardo Tommasini
Stream Reasoning (SR) research field is grown enough to prove that reasoning upon rapidly changing information is possible. RDF Stream Processing (RSP) Engines, systems capable to handle at semantic level RDF-encoded information flows, are increasing in number of implemented solutions. Now the Stream Reasoning community is working on the standardization of the methods and tools that supported their development.
Many Computer Science (CS) research fields shown their interest for a deeper comprehension of their own work nature. Studies like [46, 51] investi- gated the publications in those field, highlighting that the majority of them are allied to an Engineering epistemology. However, they also evinced and criticized the concrete differences with other engineering research areas, which focus on evaluation of the proposed systems and not only on their design and development.
The lacks of an empirical approach can be ascribed to the complex nature of the software systems. However, it is possible to face such studies that can not be easily modeled, reducing the complexity of the analysis keeping intact the relevance of each involved system. In social science and economy, where researchers deal with cross case studies, it is commonly used a System- atic Comparative Research Approach (SCRA) within an experimental setting, which grants properties like repeatability, reproducibility and comparability to build the evaluation upon.
The SR community agreed that it is mandatory evaluating RSP Engines, understanding how these systems perform in real uses cases. Recent works in the filed [53, 41, 19] pursued this goal, providing benchmarks for RSP Engines evaluation. Further analysis pointed out the challenges involved by the Stream Reasoning research and posed the basis for a proper RSP Engines evaluation, describing in detail where previous works have failed and how the can be
improved [44].
The limitations of the existing benchmarking proposals proved that the
empirical evaluation of RSP Engines is just at the beginning. What is still missing in an infrastructure that allows to compare, possibly automatically, the performances of many RSP Engines and that grants the properties of an experimental setting. In this thesis we brace this challenge borrowing from the aerospace engineering the idea of an engine test stand, which is an automatic facility for engine testing and development.
A test stand allows to design experiments and to execute them, evaluat- ing engines in a controlled environment. Thus, we formulate the following research question: ”Can an engine test stand, together with queries, datasets and methods, support Systematic Comparative Research Approach for Stream Reasoning? ”
In this thesis we propose Heaven, an open source framework that enables the Systematic Comparative Approach in the Stream Reasoning research field. Heaven consists of: an RSP Engine Test Stand, which emulates the aerospace engineering facility in the Stream Rea
Effective and Efficient Entity Search in RDF dataRoi Blanco
Triple stores have long provided RDF storage as well as data access using expressive, formal query languages such as SPARQL. The new end users of the Semantic Web, however, are mostly unaware of SPARQL and overwhelmingly prefer imprecise, informal keyword queries for searching over data. At the same time, the amount of data on the Semantic Web is approaching the limits of the architectures that provide support for the full expressivity of SPARQL. These factors combined have led to an increased interest in semantic search, i.e. access to RDF data using Information Retrieval methods. In this work, we propose a method for effective and efficient entity search over RDF data. We describe an adaptation of the BM25F ranking function for RDF data, and demonstrate that it outperforms other state-of-the-art methods in ranking RDF resources. We also propose a set of new index structures for efficient retrieval and ranking of results. We implement these results using the open-source MG4J framework.
Developers often wonder how to implement a certain functionality
(e.g., how to parse XML files) using APIs. Obtaining
an API usage sequence based on an API-related natural
language query is very helpful in this regard. Given a query,
existing approaches utilize information retrieval models to
search for matching API sequences. These approaches treat
queries and APIs as bags-of-words and lack a deep understanding
of the semantics of the query.
We propose DeepAPI, a deep learning based approach to
generate API usage sequences for a given natural language
query. Instead of a bag-of-words assumption, it learns the
sequence of words in a query and the sequence of associated
APIs. DeepAPI adapts a neural language model named
RNN Encoder-Decoder. It encodes a word sequence (user
query) into a fixed-length context vector, and generates an
API sequence based on the context vector. We also augment
the RNN Encoder-Decoder by considering the importance
of individual APIs. We empirically evaluate our approach
with more than 7 million annotated code snippets collected
from GitHub. The results show that our approach generates
largely accurate API sequences and outperforms the related
approaches.
A relatively short Introduction to R as presented at the Belgian Software Craftmanship meetup group.
The goal of this presentation is to give you an introduction to:
• The style of the language
• It's ecosystem
• How common things like data manipulation and visualization work
• How to use it for machine learning
• Webdevelopment and report generation in R
• Integrating R in your system
License:
Introduction To R by Samuel Bosch
To the extent possible under law, the person who associated CC0 with Introduction To R has waived all copyright and related or neighboring rights
to Introduction To R.
http://creativecommons.org/publicdomain/zero/1.0/
Pickens, J., Cooper, M., and Golovchinsky, G. Reverted Indexing for Expansion and Feedback. In Proc. CIKM 2010, Toronto, Canada, ACM Press. See http://fxpal.com/?p=abstract&abstractID=581
Presented on 4 Dec 2014
@Jeju, South Korea
Title: An efficient method for assessing the impact of refactoring candidates on maintainability based on matrix computation
Uk Research Infrastructure Workshop E-infrastructure Juan BicarreguiInnovate UK
Uk Research Infrastructure Workshop E-infrastructure Juan Bicarregui
How to build a successful EU project
by Juan Bicarregui
Scientific Computing Department STFC
Analysis of Educational Robotics activities using a machine learning approachLorenzo Cesaretti
These slides present the preliminary results through the utilisation of machine learning techniques for the analysis of Educational Robotics activities. An experimentation with 197 secondary school students from Italy was con-ducted, through updating Lego Mindstorms EV3 programming blocks in order to record log files containing the coding sequences designed by the students (within team work), during the resolution of a preliminary Robotics’ exercise. We utilised four machine learning techniques (logistic regression, support vec-tor machine, K-nearest neighbors and random forests) to predict the students’ performance, comparing a supervised approach (using twelve indicators ex-tracted from the log files as input for the algorithms) and a mixed approach (ap-plying a k-means algorithm to calculate the machine learning features). The re-sults have highlighted that SVM with the mixed approach outperformed the other techniques, and that three learning styles were predominantly emerged from the data mining analysis.
• Cooperated with fellow colleagues in a lab environment and experimented on the science of fluid flow through various types of piping and fittings.
• Researched the head loss that is caused in different piping including Venturi pipe, orifice plate, and elbow pipe fittings.
List from 1 – 10 1. Chanel Black Dress 2. Women’s suit 3. .docxcroysierkathey
List from 1 – 10
1. Chanel Black Dress
2. Women’s suit
3. Sports wear
4. Lace
5. High wasted jeans
6. Athletic shoes
7. Fitted clothing (like the kardashians)
8. Yoga attire
9. Oversized jackets
10. Colorful shoes
BN208 Networked Applications
Prepared by Dr Shaleeza Sohail Moderated by: Dr Fariza Sabrina Apr 2018
Assessment Details and Submission Guidelines
Unit Code BN208
Unit Title Networked Applications
Assessment Type Individual Assignment 2 (T1_2018)
Assessment Title Analyse Network Performance (Assignment Two)
Purpose of the
assessment (with
ULO Mapping)
This assignment assesses the following Unit Learning Outcomes; students should
be able to demonstrate their achievements in them.
d) Discuss performance and deployment issues for networked
applications;
e) Utilise appropriate industry tools and techniques to manage
networked applications
Weight 15%
Total Marks 50
Word limit 1000 – 1500
Due Date 1st of June 2018 (23:55) Friday Week 11
Submission
Guidelines
All work must be submitted on Moodle by the due date along with a
completed Assessment Cover Page.
The assessment must be in MS Word format, 1.15 spacing, 12-pt Calibri
(Body) font and normal margins on all four sides of your page with
appropriate section headings.
Reference sources must be cited in the text of the report, and listed
appropriately at the end in a reference list using IEEE referencing style.
The report should be in MS word format submitted on Moodle on time.
Extension If an extension of time to submit work is required, a Special Consideration
Application must be submitted directly to the School's Administration Officer,
in Melbourne on Level 6 or in Sydney on Level 7. You must submit this
application three working days prior to the due date of the assessment. Further
information is available at:
http://www.mit.edu.au/about-mit/institute-publications/policies-
procedures-and-guidelines/specialconsiderationdeferment
Academic
Misconduct
Academic Misconduct is a serious offence. Depending on the seriousness of the
case, penalties can vary from a written warning or zero marks to exclusion from
the course or rescinding the degree. Students should make themselves familiar
with the full policy and procedure available at:
http://www.mit.edu.au/about-mit/institute-publications/policies-
procedures-and-guidelines/Plagiarism-Academic-Misconduct-Policy-
Procedure.
http://www.mit.edu.au/about
http://www.mit.edu.au/about
http://www.mit.edu.au/about-mit/institute-publications/policies-procedures-and-guidelines/special-considerationdeferment
http://www.mit.edu.au/about-mit/institute-publications/policies-procedures-and-guidelines/special-considerationdeferment
http://www.mit.edu.au/about-mit/institute-publications/policies-procedures-and-guidelines/special-considerationdeferment
http://www.mit.edu.au/about-mit/institute-publications/polic ...
Towards explanations for Data-Centric AI using provenance recordsPaolo Missier
In this presentation, given to graduate students at Universita' RomaTre, Italy, we suggest that concepts well-known in Data Provenance can be exploited to provide explanations in the context of data-centric AI processes. Through use cases (incremental data cleaning, training set pruning), we build up increasingly complex provenance patterns, culminating in an open question:
how to describe "why" a specific data item has been manipulated as part of data processing, when such processing may consist of a complex data transformation algorithm.
Naver learning to rank question answer pairs using hrde-ltcNAVER Engineering
The automatic question answering (QA) task has long been considered a primary objective of artificial intelligence.
Among the QA sub-systems, we focused on answer-ranking part. In particular, we investigated a novel neural network architecture with additional data clustering module to improve the performance in ranking answer candidates which are longer than a single sentence. This work can be used not only for the QA ranking task, but also to evaluate the relevance of next utterance with given dialogue generated from the dialogue model.
In this talk, I'll present our research results (NAACL 2018), and also its potential use cases (i.e. fake news detection). Finally, I'll conclude by introducing some issues on previous research, and by introducing recent approach in academic.
Presentation made during the Intelligent User-Adapted Interfaces: Design and Multi-Modal Evaluation Workshop (IUadaptME) workshop conducted as part of UMAP 2018
Learning by example: training users through high-quality query suggestionsClaudia Hauff
A presentation given at UvA in September 2015, discussing joint work with Morgan Harvey and David Elsweiler.
Full paper: http://dl.acm.org/citation.cfm?id=2767731
Object Oriented Programming Lab Manual Abdul Hannan
Object oriented programing Lab manual for practicing and improve the coding skills of object oriented programming.
Published by Mohammad Ali Jinnah University Islamabad.
Sakai, T: The Effect of Inter-Assessor Disagreement on IR System Evaluation: A Case Study with Lancers and Students, Proceedings of EVIA 2017, pp.31-38, 2017.
http://ceur-ws.org/Vol-2008/paper_5.pdf
- Sakai, T: Towards Automatic Evaluation of Multi-Turn Dialogues: A Task Design that Leverages Inherently Subjective Annotations, Proceedings of EVIA 2017, pp.24-30, 2017.
http://ceur-ws.org/Vol-2008/paper_4.pdf
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex ProofsAlex Pruden
This paper presents Reef, a system for generating publicly verifiable succinct non-interactive zero-knowledge proofs that a committed document matches or does not match a regular expression. We describe applications such as proving the strength of passwords, the provenance of email despite redactions, the validity of oblivious DNS queries, and the existence of mutations in DNA. Reef supports the Perl Compatible Regular Expression syntax, including wildcards, alternation, ranges, capture groups, Kleene star, negations, and lookarounds. Reef introduces a new type of automata, Skipping Alternating Finite Automata (SAFA), that skips irrelevant parts of a document when producing proofs without undermining soundness, and instantiates SAFA with a lookup argument. Our experimental evaluation confirms that Reef can generate proofs for documents with 32M characters; the proofs are small and cheap to verify (under a second).
Paper: https://eprint.iacr.org/2023/1886
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfPeter Spielvogel
Building better applications for business users with SAP Fiori.
• What is SAP Fiori and why it matters to you
• How a better user experience drives measurable business benefits
• How to get started with SAP Fiori today
• How SAP Fiori elements accelerates application development
• How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities
• How SAP Fiori paves the way for using AI in SAP apps
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
1. Overview of the
NTCIR-14 CENTRE Task
Tetsuya Sakai Nicola Ferro Ian Soboroff
Zhaohao Zeng Peng Xiao Maria Maistro
Waseda University, Japan
University of Padua, Italy
NIST, USA
http://www.centre-eval.org/
Twitter: @_centre_
June 11 and 13, 2019@NTCIR-14, Tokyo.
3. Motivation
• Researcher A publishes a paper; says
Algo X > Algo Y on test data D.
⇒ Researcher B tries Algo X and Y on D but finds
Algo X < Algo Y. A replicability problem.
• Researcher A publishes a paper; says
Algo X > Algo Y on test data D.
⇒ Researcher B tries Algo X and Y on D’ but finds
Algo X < Algo Y. A reproducibility problem.
Can the IR community do better?
4. Research questions
• Can results from CLEF, NTCIR, and TREC be
replicated/reproduced easily? If not, what are the
difficulties? What needs to be improved?
• Can results from (say) TREC be reproduced with
(say) NTCIR data? Why or why not?
• [Participants may have their own research
questions]
5. CENTRE = CLEF/NTCIR/TREC
(replicability and) reproducibility
CLEF
Conferences
and Labs of the
Evaluation
Forum
NTCIR
NII Testbeds and
Community for
Information
access Research
TREC
Text REtrieval
Conference
Data, code, wisdom
Replicability and
Reproducibility experiments
Data, code, wisdom
Replicability and
Reproducibility experiments
Data, code, wisdom
Replicability and Reproducibility experiments
Europe and more;
Annual
US and more;
Annual
Asia and more;
Sesquinnual
6. Who should participate in CENTRE?
• IR evaluation nerds
• Students willing to replicate/reproduce SOTA
methods in IR, so that they will eventually be able
to propose their own methods and demonstrate
their superiority over the SOTA
• Those who want to be involved in more than one
evaluation communities across the globe
• Those who want to use data from more than one
evaluation conferences
8. CENTRE@NTCIR-14: How it works (1)
Past CLEFs
Past TRECs
Past NTCIRs
A, B: target
run pairs
A: advanced
B: baseline
A, B: target
run pairs
A, B: target
run pairs
IR literature
A, B: target
run pairs
Selected by
organisers
Selected by
each team
9. CENTRE@NTCIR-14: How it works (2)
Past CLEFs
Past TRECs
Past NTCIRs
A, B: target
run pairs
T1{A,B}:
replicated run pairs
T1: replicability subtask
(same NTCIR data)
A: advanced
B: baseline
A, B: target
run pairs
A, B: target
run pairs
IR literature
A, B: target
run pairs
Selected by
organisers
Selected by
each team
Evaluation measures:
• RSME of the topicwise deltas
• Pearson correlation of the
topicwise deltas
• Effect Ratio (ER)
10. CENTRE@NTCIR-14: How it works (2)
Past CLEFs
Past TRECs
Past NTCIRs
A, B: target
run pairs
T1{A,B}:
replicated run pairs
T1: replicability subtask
(same NTCIR data)
A: advanced
B: baseline
A, B: target
run pairs
A, B: target
run pairs
T2: reproducibility subtask
IR literature
T2CLEF-{A,B}
Run pairs
reproduced on
NTCIR data
T2TREC-{A,B}
T2OPEN-{A,B}
A, B: target
run pairs
Selected by
organisers
Selected by
each team
Evaluation measure:
Effect Ratio (ER)
T2OPEN runs evaluated only
with effectiveness measures
Past CLEF runs not considered at CENTRE@NTCIR-14
11. Target runs for CENTRE@NTCIR-14
• We focus on ad hoc monolingual web search.
• For T1 (replicability): A pair of runs from the NTCIR-13 We Want Web
task
- Overview paper:
http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings13/pdf/ntcir/01-NTCIR13-OV-WWW-
LuoC.pdf
• For T2 (reproducibility): A pair of runs from the TREC 2013 web track
- Overview paper:
https://trec.nist.gov/pubs/trec22/papers/WEB.OVERVIEW.pdf
• All of the above target runs use clueweb12 as the search corpus
http://www.lemurproject.org/clueweb12.php/
• Both the target NTCIR and TREC runs are implemented using Indri
https://www.lemurproject.org/indri/
12. Target NTCIR runs and data
for T1 (replicability)
• Test collection:
NTCIR-13 WWW English (clueweb12-B13, 100 topics)
• Target NTCIR run pairs:
From the NTCIR-13 WWW RMIT paper [Gallagher+17]
http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings13/pdf/ntcir/02-NTCIR13-WWW-GallagherL.pdf
A: RMIT-E-NU-Own-1 (sequential dependency model: SDM)
B: RMIT-E-NU-Own-3 (full dependency model: FDM)
• SDM and FDM are from [Metzler+Croft SIGIR05]
https://doi.org/10.1145/1076034.1076115
13. Target TREC runs and data
for T2 (reproduciblity)
• Test collection:
TREC 2013 Web Track (clueweb12 category A, 50 topics)
• Target TREC run pairs:
From the TREC 2013 Web Track U Delaware paper [Yang+13] :
https://trec.nist.gov/pubs/trec22/papers/udel_fang-web.pdf
A: UDInfolabWEB2 (selects semantically related terms using
Web-based working sets)
B: UDInfolabWEB1 (selects semantically related terms using
collection-based working sets)
• The working set construction method is from
[Fang+SIGIR06]
https://doi.org/10.1145/1148170.1148193
14. How participants were expected to
submit T1 (replicability) runs
0. Obtain clueweb12-B13 (or the full data)
http://www.lemurproject.org/clueweb12.php/
and Indri https://www.lemurproject.org/indri/
1. Register to the CENTRE task and obtain the NTCIR-13 WWW
topics and qrels from the CENTRE organisers
2. Read the NTCIR-13 WWW RMIT paper [Gallagher+17]
http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings13/pdf/ntcir/02-NTCIR13-WWW-GallagherL.pdf
and try to replicate the A-run and the B-run on the NTCIR-13
WWW-1 test collection
A: RMIT-E-NU-Own-1
B: RMIT-E-NU-Own-3
3. Submit the replicated runs by the deadline
15. How participants were expected to
submit T2TREC (reproducibility) runs
0. Obtain clueweb12-B13 (or the full data)
http://www.lemurproject.org/clueweb12.php/
and (optionally) Indri https://www.lemurproject.org/indri/
1. Register to the CENTRE task and obtain the NTCIR-13 WWW topics and
qrels from the CENTRE organisers
2. Read the TREC 2013 Web Track U Delaware paper [Yang+13] :
https://trec.nist.gov/pubs/trec22/papers/udel_fang-web.pdf
and try to reproduce the A-run and the B-run on the NTCIR-13 WWW-1
test collection
A: UDInfolabWEB2
B: UDInfolabWEB1
3. Submit the reproduced runs by the deadline
16. How participants were expected to
submit T2OPEN (reproducibility) runs
0. Obtain clueweb12-B13 (or the full data)
http://www.lemurproject.org/clueweb12.php/
and (optionally) Indri https://www.lemurproject.org/indri/
1. Register to the CENTRE task and obtain the NTCIR-13 WWW topics and
qrels from the CENTRE organisers
2. Pick any pair of runs A (advanced) and B (baseline) from the IR literature,
and try to reproduce the A-run and the B-run on the NTCIR-13 WWW-1
test collection
3. Submit the reproduced runs by the deadline
T2OPEN not discussed further in this overview.
For more, see:
Andrew Yates: MPII at the NTCIR-14 CENTRE Task
17. Submission instructions
• Each team can submit only one run pair per subtask
(T1, T2TREC, T2OPEN). So at most 6 runs in total.
• Run file format: same as WWW-1; see
http://www.thuir.cn/ntcirwww/ (Run Submissions
format). In short, it’s a TREC run file plus a SYSDESC
line.
• Run file names:
CENTRE1-<teamname>-<subtaskname>-[A,B]
Advanced Baseline
20. Additional relevance assessments
• NTCIR-13 WWW-1 English qrels expanded based on
the new CENTRE runs (i.e. 6 runs from MPII).
• Depth-30 pools were created, following WWW-1.
• topic-doc pairs previously unseen (2,617) were
judged.
NTCIR-13
WWW E
CENTRE
@NTCIR-14
100 topics
original qrels
Based on 13 runs from three teams
100 topics
augmented
qrels
22,912 topic-doc pairs 22,912 + 2,617 = 25,529 topic-doc pairs
23. T1:examining the topicwise deltas
original topicwise improvement replicated topicwise improvement
Covariance
Standard deviations
Pearson correlation
Collection
Measure
RMSE
Root Mean Squared Error
24. T1 and T2: Effect Ratio (1)
• T1 (replicability)
• T2TREC (reproducibility)
original mean improvement
reproduced mean improvement
replicated mean improvement
original mean improvement
25. T1 and T2: Effect Ratio (2)
• ER compares the sample (unstandardised) effect
size in the “rep” experiment against that in the
original experiment. “Can we still observe the same
effect?”
• ER>1 ⇒ effect larger in “rep” experiment
• ER=1 ⇒ effect identical to original experiment
• 0 < ER < 1 ⇒ effect smaller in “rep” experiment
• ER <= 0 ⇒ FAILURE: A-run does not outperform B-
run in the “rep” experiment
32. Reproducibility: w/o additional assessments (1)
• UDel said “web-based working sets (A) outperform
collection-based working sets (B)” for TREC
• MPII confirms this for NTCIR
• A statistically significantly outperforms B even with Q!
• Effect sizes for Q (Orig vs Repli) quite similar.
34. Reproducibility: WITH additional assessments
Larger ER values!
New relevant docs were found by the CENTRE runs; addition worthwhile!
Smaller
p-values!
36. CENTRE@NTCIR-14 conclusions
• We focussed on: “Can an effect (gain of A over B)
replicated/reproduced?”
• Replication of the RMIT effect on WWW-1 data:
successful at the ER level; not so successful at the
topic level.
• Reproduction of the TREC UDel effect on WWW-1
data: successful at the ER level; additional
relevance assessments further improved the ERs.
• Only one participating team…this needs to change.
37. SPECIAL THANKS
• To Andrew Yates (Max Planck Institute for
Informatics, Germany)
• MPII at the NTCIR-14 CENTRE Task
39. WWW and CENTRE join forces
• WWW-3 task will have three subtasks:
(1) Adhoc Chinese
(2) Adhoc English (WWW-2 + new WWW-topics)
(3) CENTRE (WWW-2 + new WWW-3 topics)
• CENTRE will accept two run types:
(a) Revived runs from WWW-2 (Tsinghua and RUC)
(b) REP (REPlicated and REProduced) runs: Tsinghua
and RUC algorithms re-implemented by other
teams
40. WWW-3 CENTRE
revived run 1
(Team A)
WWW-2 topics
WWW-2
target run 2
(Team B)
WWW-2
target run 1
(Team A)
WWW-3 CENTRE
revived run 2
(Team B)
revived (same team)
revived (same team)
New WWW-3 topics
Processed by
WWW-2 runs
Processed by
WWW-3 runs
WWW-3
Adhoc
(Team D)
Measuring progress:
WWW-2 top run → WWW-3 top run
COMPARE
Use the WWW-3 topics
(since WWW-3 systems
may be tuned with
WWW-2 topics)
41. WWW-3 CENTRE
revived run 1
(Team A)
WWW-2 topics
WWW-2
target run 2
(Team B)
WWW-2
target run 1
(Team A)
WWW-3 CENTRE
revived run 2
(Team B)
revived (same team)
revived (same team)
New WWW-3 topics
Processed by
WWW-2 runs
Processed by
WWW-3 runs
Replicated/reproduced
(different team)
Replicated/reproduced
(different team)
WWW-3 CENTRE
REP run 1-1
(Team C)
WWW-3 CENTRE
REP run 1-2
(Team C)
Measuring replicability on WWW-2 topics
COMPARE Use the WWW-2 topics
42. WWW-3 CENTRE
revived run 1
(Team A)
WWW-2 topics
WWW-2
target run 2
(Team B)
WWW-2
target run 1
(Team A)
WWW-3 CENTRE
revived run 2
(Team B)
revived (same team)
revived (same team)
New WWW-3 topics
Processed by
WWW-2 runs
Processed by
WWW-3 runs
Replicated/reproduced
(different team)
Replicated/reproduced
(different team)
WWW-3 CENTRE
REP run 1-1
(Team C)
WWW-3 CENTRE
REP run 1-2
(Team C)
Measuring reproducibility:
WWW-2-topics → WWW-3 topics
COMPARE
Use the WWW-2 topics Use the WWW-3 topics
44. CENTRE@CLEF 2019
CLEF NTCIR TREC
REproducibility
Nicola Ferro, Norbert Fuhr, Maria Maistro, Tetsuya
Sakai, and Ian Soboroff
April 15, 2019, Cologne
http://www.centre-eval.org/clef2019/
45. CENTRE@CLEF 2019
TasksThe CENTRE@CLEF 2019 lab will offer three tasks:
• Task 1 - Replicability: the task will focus on the
replicability of selected methods on the same
experimental collections;
• Task 2 - Reproducibility: the task will focus on the
reproducibility of selected methods on the different
experimental collections;
• Task 3 - Generalizability: the task will focus on collection
performance prediction and the goal is to rank (sub-
)collections on the basis of the expected performance
over them.
46. Task 1 & 2: Selected
Papers
For Task 1 - Replicability and Task 2 - Reproducibility we
selected the following papers from TREC Common
Core Track 2017 and 2018:
• [Grossman et al, 2017] Grossman, M. R., and
Cormack, G. V. (2017). MRG_UWaterloo and
Waterloo Cormack Participation in the TREC 2017
Common Core Track. In TREC 2017.
• [Benham et al, 2018] Benham, R., Gallagher, L.,
Mackenzie, J., Liu, B., Lu, X., Scholer, F., Moffat, A.,
and Culpepper, J. S. (2018). RMIT at the 2018 TREC
CORE Track. In TREC 2018.
47. Task 3 - Generalizability
• Training: participants need to run plain BM25 and
they need to identify features of the corpus and
topics that allow them to predict the system score
with respect to Average Precision (AP).
• Validation: participants can validate their method
and determine which set of features represent the
best choice for predicting AP score for each system.
• Test (submission): participants submit a run for
each system (BM25 and their own system) and an
additional file (one for each system) including the
AP score predicted for each topic.
48. SIGIR 2019 Open-Source IR
Replicability Challenge (OSIRRC 2019)
• Repeatability, replicability, and reproducibility:
• Good science!
• Helps sustain cumulative empirical progress in the field
• OSIRRC is a workshop at SIGIR with three goals:
• Develop a common Docker interface to capture ad hoc
retrieval experiments
• Build a library of Docker images that implements the
interface
• Explore this approach to additional tasks and evaluation
methodologies
49. OSIRRC 2019: Current Status
• We’ve converged on a Docker specification for ad
hoc retrieval experiments – called “the jig”
• All materials at https://github.com/osirrc
• Initial library of 12 images from 9 different groups
under development
• It’s not too late if you still want to participate!
• Wrapping an existing retrieval system in “the jig” isn’t
too difficult a task…
• See you at SIGIR 2019!
50. Nicola Ferro
@frrncl
University of Padua, Italy
The 3rd Strategic Workshop on Information Retrieval in Lorne (SWIRL III)
14 February 2018, Lorne, Australia
ACM Artifact Badging - SIGIR
Survey
51. ACM Badging: Artifacts Evaluated
This badge is applied to papers whose associated artifacts have
successfully completed an independent audit. Artifacts need not
be made publicly available to be considered for this badge.
However, they do need to be made available to reviewers.
Artifacts Evaluated – Functional
The artifacts associated with the research are found to be documented,
consistent, complete, exercisable, and include appropriate evidence of
verification and validation.
Artifacts Evaluated – Reusable
The artifacts associated with the paper are of a quality that significantly
exceeds minimal functionality. That is, they have all the qualities of the
Artifacts Evaluated – Functional level, but, in addition, they are very
carefully documented and well-structured to the extent that reuse and
repurposing is facilitated. In particular, norms and standards of the
research community for artifacts of this type are strictly adhered to.
5
3
52. ACM Badging: Artifacts Available
This badge is applied to papers in which
associated artifacts have been made
permanently available for retrieval.
Author-created artifacts relevant to this paper
have been placed on a publicicly accessible
archival repository. A DOI or link to this
repository along with a unique identifier for
the object is provided
Repositories used to archive data should have a
declared plan to enable permanent accessibility.
Personal web pages are not acceptable for this
purpose
Artifacts do not need to have been formally
evaluated in order for an article to receive this badge.
5
4
53. ACM Badging: Results Validated
This badge is applied to papers in which the main results of the
paper have been successfully obtained by a person or team
other than the author
Results Replicated
The main results of the paper have been obtained in a subsequent
study by a person or team other than the authors, using, in part,
artifacts provided by the author
Results Reproduced
The main results of the paper have been independently obtained in a
subsequent study by a person or team other than the authors,
without the use of author supplied artifacts.
Exact replication or reproduction of results is not required,
or even expected. In particular, differences in the results
should not change the main claims made in the paper
5
5
54. SIGIR Badging Task Force
Nicola Ferro (chair)
University of Padua, Italy
Diane Kelly (ex-officio)
The University of Tennessee,
Knoxville, USA
Maarten de Rijke (ex-officio)
University of Amsterdam, The
Netherlands
Leif Azzopardi
University of Strathclyde, UK
Peter Bailey
Microsoft, USA
Hannah Bast
University of Freiburg, Germany
Rob Capra
University of North Carolina at
Chapel Hill, USA
Norbert Fuhr
University of Duisburg-Essen,
Germany
Yiqun Liu
Tsinghua University, China
Martin Potthast
Leipzig University, Germany
Filip Radlinski
Google London, UK
Tetsuya Sakai
Waseda University, Japan
Ian Soboroff
National Institute of Standards and
Technology, USA
Arjen de Vries
Radboud University, The
Netherlands
5
6