Experimental psychology, cognitive science or, more recently, cognitive neuroscience, is the main framework to place human information processing under extensive empirical scrutiny. The last decade has seen a surge of interest in the application of psychological measurements for evaluating increasingly complex human-technology interactions. While most welcome from the psychological perspective, we propose that the use of these methodologies should not rely only on the application of sophisticated measurement tools, but also on the application of contemporary knowledge on psychological phenomena and dynamics of human information processing. In addition, we argue that the latest developments in multimodal signals and data mining techniques offer a unique opportunity to extend psychological methodologies to large scale testing grounds. Thus, the application of psychological knowledge to information retrieval research will not only be beneficial for the latter, but for the former as well, inasmuch as information retrieval provides a real field of application for its hypotheses about human information processing.

1. On Human Information Processing in Information Retrieval
Alexandre Pereda-Baños (Eurecat), Ioannis Arapakis (Yahoo Labs), Miguel Barreda-Ángeles (Eurecat)

2. Human Information Processing (HIP)
§ We are not consciously aware of the
mental processes determining our
behaviour
§ Such unconscious influences reach
from basic or low-level mental
processes to high-level psychological
processes like motivations,
preferences, or complex behaviours

3. Human Information Processing (HIP)
§ The search for information is often led by a human brain
§ HIP is the field of study of experimental psychology and cognitive
neuroscience

4. Psychological Variables
§ The most interesting psychological variables and processes for
the study of IR are those related to attentional and emotional
phenomena
Selective attention
Cognitive
effort / arousal
Emotional reactions

5. Psychophysiological Measures of HIP
§ Standardised questionnaires for measuring
perceptual aspects, perceived usability, cognitive
working load, or affective
§ Online measures of user behavior and cognitive
states that are often unavailable for conscious
reports:
§ Behavioral
§ psychophysiological

6. Characteristics of Psychological Methods
§ Helpful in unveiling attentional and emotional reactions not
consciously available to us
§ Offer high temporal and spatial resolution
§ Robust against cognitive biases (e.g., social desirability bias*)
§ Always provide “honest” responses
§ No direct question to the subject, no direct answer
§ The information on the research questions has to be inferred
from the variations on the physiological signals and the way they
are related to psychological constructs
* The tendency of survey respondents to answer questions in a manner that will be viewed favorably by others.

7. Electrodermal Activity (EDA)
§ Changes in conductivity of the skin due to
activation of sweat glands by activation of the
autonomous nervous system (sympathetic
division)
§ Reflects general activation both for attentional
and emotional measures (in fact, it is calibrated
by having participants perform complex math
calculations)
§ It’s the basis of the “truth machine”, though not
as effective as fiction has led us to believe…

9. Electrodermal Activity (EDA)
§ Unconscious Physiological Effects of Search
Latency on Users and Their Click Behaviour
(SIGIR 2015)
• Although the latency effects did not produce
changes on the self-reported data, their
impact on users’ physiological responses is
evident
• Even when short latency increases of under
500ms are not consciously perceived, they
have sizeable physiological effects that can
contribute to the overall user experience
-0.4
-0.2
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1 2 3 4 5 6 7 8 9 10
µS
Time after query onset (in seconds)
0ms
500ms
750ms
1000ms
15.0
15.2
15.4
15.6
15.8
16.0
16.2
16.4
16.6
16.8
17.0
0 1 2 3 4 5 6 7 8 9 10 11 12
µS
Time after stimulus onset (in seconds)

10. Electrodermal Activity (EDA)
§ A large-scale query log analysis ascertained the effect on the
clicking behaviour of users and revealed a significant decrease
in users’ engagement with the search result page, even at
small increases in latency
0
0.05
0.10
0.15
0.20
0 500 750 1000
0
0.5
1.0
1.5
2.0
Fractionofquerypairs
Click-more-on-fast/Click-more-on-slow
Latency difference (in milliseconds)
Click-more-on-fast
Click-more-on-slow
Ratio
0
0.05
0.10
0.15
0.20
0 500 750 1000
0
0,5
1.0
1.5
2.0
Fractionofquerypairs
Click-on-fast/Click-on-slow
Latency difference (in milliseconds)
Click-on-fast
Click-on-slow
Ratio

11. HIP Dynamics
§ Human information processing is both serial and parallel
§ Cognitive science has provided large amounts of evidence that
conscious information processing is mainly serial
§ When processing information in situations that require to shift the
focus of attention between different tasks and/or stimuli, this
results in an increase in the effort required to process that
information
§ Simon effect

12. HIP Dynamics (Serial Processing)

13. HIP Dynamics (Serial Processing)
§ Switching tasks
§ Try to read the word in odd trials
and name the color on even
trials!
Green
Red
Blue
Red
Green
Yellow

14. HIP Dynamics (Parallel Processing)
§ Simon effect: Hit the left key if there is an A on screen and the
right if there is a B

15. HIP Dynamics (Parallel Processing)
§ The effect is still there with crossed hands!

16. Multimodal Behaviour Analysis
§ Behaviour measurements in ecological conditions
§ Behaviour understanding through cameras and microphones
§ Aggregating various online measures gives an accurate picture
of the user’s experience
§ Robust real-time behavior analyses, information that can then be
used for the purpose of research on human behavior and user
experience
§ The opportunity is ripe to move beyond experimental laboratory
settings, in which response has been traditionally measures, into
real large-scale controlled studies

17. Conclusions
§ The use of neuro-physiological methods in IR research is
essential in order to obtain a complete picture of the mental
processes underlying user search behaviour
§ The collaboration between psychological and IR research can go
far beyond the application of sophisticated measuring
methodologies
§ Introduce actual knowledge on the dynamics of human
information processing into a real-world testing ground
§ The use of multimodal signals holds the promise of allowing
large-scale, controlled studies that will undoubtedly foster the
progress of both research fields

18. Thank you for your attention!
iarapakis
http://www.slideshare.net/iarapakis/sigir15-NeuroIR