Keynote ICAT-EGVE 2020 (https://icat-egve-2020.org)

1. Alvaro CASSINELLI
Associate Professor, School Of Creative Media (CityU Hong Kong)
Co-Director eXtended Reality Lab (XRL)
Being-in-the-World in Times of Virtuality:
An Art and Science Perspective
ICAT-EGVE 2020 (VIRTUAL)
Hastings Contemporary’s robot, in front of Graham Sutherland’s
“Thorn Head.”Credit: Estate of Graham Sutherland; via Daniel Katz
Gallery; Hastings Contemporary

2. Forgetting purpose: answers from an art & science practice
VR, AR and Telexistence: a tool? an extension of the self? a new reality?
A better framework? reflective/reflexive technologies

4. YOU WORLD
Engineering is goal-oriented
Naive Realism operational framework
Technology designed to enhance natural organs of perception and action
World is the object of interest
Goal: more efficient work, more compelling experiences
AR, VR, Tele-existence as tools

5. Credit: TelepresenceRobots.com
Credit: AppAdvice
HOWEVER...
now we can talk with
friends and see them on
a phone, nice
okay, very useful for
attending venues in times
of COVID

6. Health Frontier conference, Stefan Wabner, David Orban, (2017)
Speaker doubling with a mime (telepresence robot alternative)
Flying Head (Flying Telepresence)
Jun Rekimoto Lab (2012)
Mechanical Turk?
makes total sense!
More natural...
Nice! no more
physical constraints,
a disembodied
viewer!

7. STELARC: THE HUMAN BODY IS OBSOLETE
Redesigning the Physical Body
Fascinating!
Imagine what can
we do...
The Machine to be Another
-Embodied Virtual Reality System
really, *what*
can we do with
this???

8. Any technology has “easter eggs”, revealed through imperfections,
side effects, glitches, malfunction
When technology fails to be invisible
YOU WORLD
First shift: from observed object to medium/interface itself

9. ...and it only re-appears when
the tool breaks down in use
Heidegger: a hammer withdraws from conscious attention
when we concentrate on the task ‘in hand’ (nailing)

10. AR, VR, MR is a wonderful hammer, nailing reality
What can we learn before trying to fix the interface?
When it breaks, illusions become evident, fascinating, magical

11. Examples in traditional and NEW MEDIA ART
Reappropriating the “shortcomings” of the medium
Visible brush, TV glitch, time distortion, perspective, trompe l’oeil...
Nam June Paik,
Magnet TV, 1965
Jackson Pollock drip painting session M.C. Escher, Relativity. 1953 Pere Borrel del Caso, Escaping criticism,1874

12. Henri Lartigue photography of a race car: use of focal
place distortion while panning
Toshio Iwai with NHK Science & Technical Research and
Labs. Morphovision: Distorted House (2005)

13. Khronos Projector, 2005

14. (the opportunity of the glitch in Science)
Image from: “In Search Of Schrodinger's Cat”
by John Gribbin
Optical Illusion:
how to fix this??
Experiment: search for a glitch in the matrix
Better model of the world and the observer
Serendipity is precious
https://leonbijelic.com/amazing-great-optical-illusion-artwork/
Wilhelm Conrad Roentgen, first ever X-ray (1895)
Gedankenexperiment: we even
dream with “glitches“ !
Bummer: my photographic
plate is wasted!

15. Why it is so easy to veer toward uncharted territory here?
Many technologies operate in ‘naive realism’
Painting:
From immortalizing aristocracy, documenting, narrating to art.
In control of the “tool”
... is this tool larger than life?
Tachi Lab,Telexistence AVATAR robot TELESAR VI
Telepresence, immersion:
From manipulation at a distance to...
Bayeux Tapestry, 1066
Diptych of Federico da Montefeltro,
Piero della Francesca, (1465~72)
Violin and Candlestick, Georges
Braque, (1910)
This new medium entangles observed and observer

16. And a third shift: from awareness of the medium to awareness of the observer
YOU WORLD
Yes.
Manipulating phenomenology produces big “glitches” and opportunities
“Reflective/Reflexive technologies”: a window into the self
A better framework: embodied cognition

17. A simple mirror is reflexive technology (understood beyond the literal meaning)
• Intended function: edit our social image (clothing, makeup)
• Unexpected result: how much can I edit myself and still be “me”? is there a “me” at all? where
I am standing? where is my “self”? Affected body image and schema...
La reproduction interdite,
Rene Magritte, (1937)
Public Space/Two Audiences’’, Dan Graham (1976)
Reflective/Reflexive tech. Examples:
Mirror therapy (phantom limb pain treatment)

18. Arnolfini Portrait, Jan van Eyke, 1434
By the way, the Milgram Continuum was there all along, mirrors are powerful!
(some would say magic)
Eerie early AV (augmented virtuality)
VR: immersion is inevitable right?
Un bar aux Folies Bergère, Édouard Manet (1882)
(not only that: space is impossible!)

19. Artificially out-
of-body
experience
Manipulation the perception of the SELF!
Boxed Ego, 2008
But what it is to “be somewhere”?
Toward “supermirrors”?

21. Augmented Reality is reflexive/reflective technology:
Intended function: training, augment our perception, surgery, transform our social image
Unexpected result: how much info can I take? How different can I be? Is that really real (*)? Who am I
without “makeup”?
(*) Christian Sandor, Martin Fuchs, Álvaro Cassinelli, Hao Li, Richard A. Newcombe, Goshiro Yamamoto, Steven K. Feiner:
Breaking the Barriers to True Augmented Reality. CoRR abs/1512.05471 (2015)
Hyper-Reality (concept film by Keiichi Matsuda)

22. Laser Aura
Laser Arrays
Programmable
Makeup
Haptic Mask
NOTE: my thing is Spatial Augmented Reality. Different modus operandi...

23. Minimal displays & supernormal stimuli
‘Here’s a thought experiment: suppose you hold up a smartphone and say, “where’s my shoe?” and a laser-
pointer jumps out of it and focuses a hot red dot on your lost shoe […]. Maybe we need a definitional category
that involves humble little “augment apps” that don’t inflict major interventions on “reality.”
Bruce Sterling, (Wired Blog)

24. Virtual Reality is reflexive technology:
Intended function: Simulation, training, entertainment, medical treatment...
Unexpected result: where am I? who I am immersing? do I need a body? Is the world a simulation?
An experimental study of a virtual reality counselling paradigm
using embodied self-dialogue, Mel Slater (2019)
Red or Blue pill, scene from The Matrix, (1999)

25. Telepresence is reflexive technology (Telexistence)
Intended function: videoconferencing, remote collaboration, virtual tourism, telesurgery, maintenance robots, etc.
Unexpected result: Where am I? Do I need to be somewhere? can I have a better body? Augmentation!
Room2room: Enabling life-size telepresence in a projected augmented reality environment,
T Pejsa, J Kantor, H Benko, E Ofek, A Wilson, Proceedings of the 19th ACM conference on
computer-supported cooperative work and social computing.
Marriot Hotels “Teleporter Booth”
ChameleonMask: Embodied Physical and Social Telepresence
using Human Surrogates, Rekimoto Lab, 2015
Can we teleport emotions?
Simultago, A.Cassinelli, (2013)
Stephane, P., Riva G. and Cassinelli A.: Creation
of Sympathetic Media Content, DAP’08 with
UbiComp, Seoul, South Korea [21/9.2008]

26. Awareness of presence, distorted feeling of presence, embodiment, body ownership,
artificial out-of-body experiences, redirected walking...
Jitter, Lag, Motion sickness
VR Uncanny Valley
Boxed Ego, (2008)
Redirected Walking in VR
OBE experiment (Olaf Blanke)
Estimation of Detection Thresholds for Redirected Walking
Techniques, Frank Steinicke, et al. IEEE Transactions on
Visualization and Computer Graphics, (2010)
Exploring Perceptual and Cognitive Effects of Extreme Augmented Reality Experiences
Daniel Eckhoff, Alvaro Cassinelli, Christian Sandor
cross-modal

27. Volumetric Histories: Blackout
Blackout is a virtual reality documentary
inviting New Yorkers to share stories in
their own voice. Blackout illuminates the
moments when our lives intersect in a
space where differences are most visible
– the subway. It is an ongoing,
participatory project gathering real
experiences from this political instant
https://vimeo.com/304005035
A work demonstrating a clever understanding of the
medium...
Tribeca Immersive Storyscapes and Virtual Arcade

28. Da Vinci surgical robot
“Robomecia”, a paramecium robotic avatar, 2008
Note that physical scale is irrelevant
Looking Glass holographic display

29. Note that matter itself is irrelevant
Poor’s man quantum teleportation
Tele-ping pong: proof-of-principle of an IT-engineered
wormhole [2005]

30. eeyee [Eric Siu]
2008 | Waerable Device
When Telepresence? partial telepresence?

31. Seeing without eyes: extending the body for spatial awareness
• new sensorial modality (this is not TVSS)
• Extension of the body (sensorial extension, 360 degrees)

32. Haptic Radar [2005~]
Experiments
Haptic immersion

33. • Feeling of immersion surprisingly high
• Strategies to explore real spaces where easily transferred to explore virtual spaces
• We can study the real device through this kind of simulations

34. Haptic Radar
Travel aid for the blind
Earlids
voluntary control of
auditory gain
Haptic Cat
Artificial cat whiskers
Haptic Car
full car “distal attribution”
New sensorial modalities based in the action-perception theory

35. “Interfaces”… or extensions of the Self?
interfaces for instincts?
(with S. Heirtrich)
“High Level”
artificial Cross -
Modalities

36. What it is like to be a cat? (with Joanna Lumley)

37. VR, AR,
Telexistence, digital
embodiment, BMI
YOU WORLD
AN INEVITABLE LAST SHIFT?
MIXED REALITY
MIXED SELF
Transhumanism
From being in
the world to
being the
world
HOLISTIC MINDSET

38. Artists / scientists contemplate reality leaving aside
teleological or technological considerations
Art: opportunities, language of the medium
Science: serendipity, discovery, new questions
taking advantage
demands a non-goal
oriented mindset

39. EMPOWERMENT OR ESCAPISM?
Neuromancer (William Gibson, 1984): people can teleport to cyberspace
The Matrix: people live in cyberspace (but is not good)
Tron: people are trapped in cyberspace
The grass is greener in my other body/reality...
Teleported where? why?
Becoming an avatar, transformed to whom?
Transform, manipulate, augment a (boring?) world?
Transhumanism, Ethical concerns

40. CONCLUSIONS
Realize that these are “reflective” technologies
Tool function eclipsed by its unexpected affordances
Its presence reveals easily as it “breaks” often (Heidegger)
Neurosciences, Psychology, Cognitive Sciences
Art & science journals (ex: Leonardo)
Humanities (ex: Computing & Philosophy conf.)
Change hats: you can publish in other fields, collaborate
Don’t fixate on function or fixing things
Don’t fixate on making it “invisible” (industry will anyway)
A perfect mirror does not belong to a funhouse
We are in the Victorian Era regarding this tech (pre-cinematographic)
Remember Louis Lumière: “cinema is an invention without a future” (1985)
The “tool” is designed to fuse with us
Create experiences but also enhance self-awareness
thaumatrope
phenakistiscope

41. Some good readings

42. अवतार