2015-02-12 research seminar

1. !"#$%&'()*+',-.*-/0-1'(23)'
3'
New applications based on implicit,
physiology based interaction!!
Dr. Aleksander Väljamäe
Marie Sk!odowska-Curie Fellow
Decision, Emotion and Perception Lab
Department of Behavioural Sciences and Learning
Linköping University, Linköping, Sweden
&
Laboratory of psychophysiology
Department of Higher Nervous Activity and Psychophysiology
SPbSU, Saint Petersburg, Russia
&
NeuroGenetics Media Lab, CIANT, Prague, Czech Republic

2. !"#$%&'()*+',-.*-/0-1'(23)'
('

3. !"#$%&'()*+',-.*-/0-1'(23)'
4'
Born in Tallinn, USSR, in 1978
1996-2000 First degree (B.Sc.) – Telecommunication, Tallinn
TU, Tallinn, Estonia
2001-2003 Second degree (M.Sc.) – Telecommunication,
Chalmers University of Technology, Gothenburg, Sweden
2003-2007 Ph.D. on multisensory virtual reality – Chalmers
University of Technology, Gothenburg, Sweden
2007-2010 First postdoc – neuroscience lab SPECS,
University of Pompeu Fabra, Barcelona, Spain
2011-2012 Second postdoc – Graz BCI lab, TU Graz &
Neuropsychology Laboratory, University of Graz, Graz,
Austria
Short CV

4. !"#$%&'()*+',-.*-/0-1'(23)'
)'

5. !"#$%&'()*+',-.*-/0-1'(23)'
5'
from Wolpaw et al. 2002

6. !"#$%&'()*+',-.*-/0-1'(23)'
6'
from Wolpaw et al. 2002

7. !"#$%&'()*+',-.*-/0-1'(23)'
7'
'Implicit, physiology based interaction
Physiological computing provides new channel to the
conventional Human Computer Interaction paradigm.
My interest is implicit interaction with multisensory media
and using it for health and art applications.
1. Subtopic one: Cinema and neuroscience: recent
research developments and future applications
2. Subtopic two: B-Reactable: multimodal tabletop system
for collaborative physiology monitoring and training

8. !"#$%&'()*+',-.*-/0-1'(23)'
8'
Media as a constructed...
but "real" experience

9. !"#$%&'()*+',-.*-/0-1'(23)'
9'
MEDIA AS CONSTRUCTED EXPERIENCES
Q1: What is neurocinema and enactive cinema?
Q2: How in can be used in art and research?

10. !"#$%&'()*+',-.*-/0-1'(23)'
32'
neuro
cinematics

11. !"#$%&'()*+',-.*-/0-1'(23)'
33'
Nishimoto et al., 2011, Current Biology
Work by Gallant Lab at UC Berkeley

12. !"#$%&'()*+',-.*-/0-1'(23)'
3('
Measuring Inter-Subject Correlation - ISC
Neurocinematics –
"The ability to measure the
effect of films on viewers’
brains with high spatial and
temporal precisions can
provide a new analytical
paradigm for assessing and
analyzing different aspects of
films, film genres, and
cinematic styles."
Hasson et al., Projections 2008

13. !"#$%&'()*+',-.*-/0-1'(23)'
34'
Inter-Subject Correlations for different films (1)
Bang! You're Dead (Hitchcock, 1961)
Hasson et al.,
Projections, 2008

14. !"#$%&'()*+',-.*-/0-1'(23)'
3)'
Inter-Subject Correlations for different films (2)
Hasson et al.,
Projections, 2008

15. !"#$%&'()*+',-.*-/0-1'(23)'
35'
Interactive
cinema

16. !"#$%&'()*+',-.*-/0-1'(23)'
36'
Creating interactive narrative
!!Script-Based Interactive Narrative:
- all scenes are pre-scripted
- linear or non-linear pre-dermined branchining structure
!!Emergence-Based Interactive Narrative:
- agents autonomy (goal-directed and reactive actions)
- story is created by agents and user interaction
!!Plan-Based Interactive Narrative:
- agents autonomy
- narrative manager dynamically evaluates the actions
of agents and user and channels them towards a
coherent story

17. !"#$%&'()*+',-.*-/0-1'(23)'
37'
'Affective chronometry and database cinema

18. !"#$%&'()*+',-.*-/0-1'(23)'
38'
Alexander Ponomarev,
INFO-SHOWER,
Russian Pavillon,
Venice Bienalle 2007
Pulling the armatures one can
regulate the speed of the
pictureflood. The content itself
is mixed with news,
entertainment, porn,
musicvideos etc.
Database media and cinema
http://tagr.tv http://www.youtube.com/watch?v=NEfVjAbMsZQ

19. !"#$%&'()*+',-.*-/0-1'(23)'
39'
Interactive
cinema
neuro
cinematics
enactive
cinema
Interactive cinema based on physiological sensing

20. !"#$%&'()*+',-.*-/0-1'(23)'
(2'
Using film for studying socio-emotional interaction
Tikka et al., 2012

21. !"#$%&'()*+',-.*-/0-1'(23)'
(3'
Implicit interaction: wearable systems
www.neurosky.com
www.emotiv.com
www.smartex.it
www.starlab.es

22. !"#$%&'()*+',-.*-/0-1'(23)'
(('
"Room of desires" installation (Smetana, 1996)
http://www.smetana.cz

23. !"#$%&'()*+',-.*-/0-1'(23)'
(4'
"Room of desires" installation – system architecture

24. !"#$%&'()*+',-.*-/0-1'(23)'
()'
Enactive cinema: "Obsession" installation, 2005
Tikka et al., 2006

25. !"#$%&'()*+',-.*-/0-1'(23)'
(5'
"Obsession" installation (Tikka, 2005)
Tikka et al., 2006

26. !"#$%&'()*+',-.*-/0-1'(23)'
(6'
Kaipainen et al., 2011
Enactive Cinema project: ontospace narrative engine

27. !"#$%&'()*+',-.*-/0-1'(23)'
(7'
SPECS-UPF research lines

28. !"#$%&'()*+',-.*-/0-1'(23)'
(8'
'Affective narratives as guided experience

29. !"#$%&'()*+',-.*-/0-1'(23)'
(9'
EDA-based modulation of video effects

30. !"#$%&'()*+',-.*-/0-1'(23)'
42'
'Neurofeedback in XIM – self-motion paradigm

31. !"#$%&'()*+',-.*-/0-1'(23)'
43'
Controlled stimulus – affective databases
HAU
LAU
HAP
LAP

32. !"#$%&'()*+',-.*-/0-1'(23)'
4('
Studying transitions using subsequent picture-pairs
HAU
LAP

33. !"#$%&'()*+',-.*-/0-1'(23)'
44'
'(A) Low arousal pleasant -> High arousal unpleasant
3 s pause between pictures 6 s pause between pictures
Patients: 22 migraineurs; age and gender matched controls: 12
Controls show positive valence supression, but patients do not

34. !"#$%&'()*+',-.*-/0-1'(23)'
4)'

35. !"#$%&'()*+',-.*-/0-1'(23)'
45'
Can enactive cinema be used in science?
Topics that can be addressed by enactive cinema:
- temporal dynamics of cognition, emotion and
perception
- unconscious processing
- decision making
- emotional synchronization, collective experience
- imaging genetics

36. !"#$%&'()*+',-.*-/0-1'(23)'
46'
Conclusions / Topics for discussions
Enactive cinema might have a big potential as a
scientific tool

37. !"#$%&'()*+',-.*-/0-1'(23)'
47'
Conclusions / Topics for discussions
Enactive cinema might have a big potential as a
scientific tool
What are the areas of common interest:
- research questions (creating a demand)
- research methodologies (e.g. ERPs during film
presentation)
- common video databases and related infrastructure
(e.g. "narratives" engine)
- ...

38. !"#$%&'()*+',-.*-/0-1'(23)'
48'
Radically new neurocinematic experiences?
Gerson et al., IEEE Trans. Neural Syst. 2006
Rapid Serial Visual Presentation – RSVP- sequence
for "human visual processors"
100 ms per image

39. !"#$%&'()*+',-.*-/0-1'(23)'
49'
:;<='<>'?@1A'0B'$11<C'D<11EF&'(224'

40. !"#$%&'()*+',-.*-/0-1'(23)'
)2'
Brain to Brain communication
.-1F<G@C'
@H-IJK-'
LEIJ<G@1B'
.-1F<G@C'
@H-IJK-'
LEIJ<G@1B'

41. !"#$%&'()*+',-.*-/0-1'(23)'
)3'
New installation in progress

42. !"#$%&'()*+',-.*-/0-1'(23)'
)('
B-Reactable project: 2013-2015

43. !"#$%&'()*+',-.*-/0-1'(23)'
)4'
Collaborations:
- Aleksei Ossadchi, Saint Petersburg State University
- Prof. Juri D. Kropotov, Natalia Shemyakina, Institute of
Human Brain, Russian Academy of Sciences
- Prof. Daniel Västfjäll, Linköping University, Linköping,
Sweden
- Prof. Sergi Jorda, Sebastian Mealla, University of
Pompeu Fabra, Barcelona, Spain
“B-Reactable: multimodal tabletop system for collaborative
physiology monitoring and training"
https://vimeo.com/14675468
B-Reactable project: 2013-2015

44. !"#$%&'()*+',-.*-/0-1'(23)'
))'
'First B-Reactable experiments

45. !"#$%&'()*+',-.*-/0-1'(23)'
)5'
'Collaborative tasks using EEG monitoring

46. !"#$%&'()*+',-.*-/0-1'(23)'
)6'
'Testing new setup at Sankt-Petersburg

47. !"#$%&'()*+',-.*-/0-1'(23)'
)7'
'New B-Reactable experiments: system architecture

48. !"#$%&'()*+',-.*-/0-1'(23)'
)8'
'New B-Reactable experiments: test conditions

49. !"#$%&'()*+',-.*-/0-1'(23)'
)9'
'B-Reactable work in progress

50. !"#$%&'()*+',-.*-/0-1'(23)'
52'
'New B-Reactable experiments: results

51. !"#$%&'()*+',-.*-/0-1'(23)'
53'
Thank you for your attention!
aleksander.valjamae@gmail.com
Web: http://www.ibl.liu.se/medarbetare/valjamae-aleksander?l=en