We propose an online access panel to support the evaluation process of Interactive Information Retrieval (IIR) systems. By maintaining an online access panel with users of IIR systems we assume that the recurring effort to recruit participants for web-based as well as for lab studies can be minimized. We target on using the online access panel not only for our own development processes but to open it for other interested researchers in the field of IIR. In this paper we present the concept of the online access panel as well as first implementation details.
Establishing an Online Access Panel for Interactive Information Retrieval Research
1. Establishing an Online Access Panel for Interactive
Do you know this situation? Prototype and user study are ready to go …
… but you don‘t know how to locate a sufficient number of suitable participants to test your system?
Mission:
Supporting a user centered design process for developing
Interactive Information Retrieval (IIR) systems
Main problem of user studies:
Enormous recurring effort to recruit suitable participants for
user studies
Approach:
IIRpanel - an online access panel for IIR user studies
A pool of participants for web surveys and lab studies
Benefits:
Interested participants sign up only once with the panel
Researchers send participation calls to all or to a sample of panelists
Stable population of users - repeated examinations easily possible
Open questions:
How to address representativeness?
How to balance participants’ workload?
How to control participants’ learning effects?
How to make study data available for other researchers?
Strategic goal:
Establishing an international consortium of IIR researchers
interested in a joined management of the panel
IIR Researcher
Dagmar Kern, Peter Mutschke, Philipp Mayr
GESIS – Leibniz Institute for the Social Sciences, Cologne, Germany
Contact: dagmar.kern@gesis.org
Yeah, I have a
great idea for a
new IIR system.
Let‘s start
implementing a first
prototype.
Hi, how is it going
with your project?
Great! Now it‘s
time for the first
user study.
That shouldn’t
be a problem.
Arrrg, we need
participants!!!!
Information Retrieval Research
2 weeks later
Urrg …
Rejection
no 37
6 weeks later
Good luck finding
participants!
(6) select
and contact
panelists
(7) provide
study meta
data (e.g.
response rate)
(5) notification
of acceptance
Panelist DB IIRpanel
www.gesis.org/iirpanel
Panelists Consortium
Oh, you don’t have
time, ok. Sigh …