The document outlines the research directions of the Watanabe Laboratory's Meta Perception Group, led by Alvaro Cassinelli, focusing on integrating technology with human interaction through tangible data and augmented reality. It discusses enabling technologies such as zero-delay spatial AR and unique interfaces for augmented expression and perception. The goal is to create a seamless blend of the physical and digital worlds to enhance human experience and interaction.

1. Watanabe Laboratory

2. Ishikawa-Watanabe lab: meta perception group
By Alvaro Cassinelli, Assistant Professor
(Meta Perception Group Leader)

4. I. The physical cloud (tangible gigabytes)
• Space and objects as scaffold for data
• Proprioceptive interfaces / Deformable interfaces
• Multi-Modal Augmented Reality
Latest research areas:
II. Enabling technologies (sensing/projection, I/O)
• Zero-delay, zero-mismatch spatial AR
• Minimal, ubiquitous & context aware interactive displays
• Laser displays, pneumatic slow displays, roboptics
III. Mediated Self / Augmented Perception /Prosthetics
• Augmented sensing & expression
• Electronic travel aids, Wearables
how to interact
with the digital
world?
“enabling”…
new realities
Integrate
technology and
human life
art research

5. I. The Physical cloud: data haunted reality
I. The physical cloud (tangible gigabytes)
• Space and objects as scaffold for data
• Proprioceptive interfaces / Deformable interfaces
• Multi-Modal Augmented Reality
II. Enabling technologies (sensing/projection, I/O)
• Zero-delay, zero-mismatch spatial AR
• Minimal, ubiquitous & context aware interactive displays
• Laser displays, pneumatic slow displays, roboptics
III. Mediated Self / Augmented Perception /Prosthetics
• Augmented sensing & expression
• Electronic travel aids, Wearables

6. (b) Data on Objects
(objects as intuitive handles or “icons” : beyond “PUI”)
(a) Data on 3D space
Experiments on psychology
– space perception and
organization
• The body as a “bookshelf”
or body mnemonics(*)
• Shared “Memory Palace”
(interpersonal spatialized
database)
• The city, public spaces,
etc. as a 3d “bookshelf”
Experiments on
psychology
• background picture as
organizational
scaffold for files and
folders (Takashita-kun)
• Association by
perceived affordances
(Invoked computing)
How to interact with “just
coordinates” in space?
• Propioceptive interfaces
(VSD, Krhonos, Virtual HR)
• Deformable workspace
metahor Spidar Screen
• Virtual BookShelf
(Takashita-san)
• Real time projection
mapping
• twitterBrain (Philippe, Jordi)
How to interact with
objects that do not
have I/O interfaces?
• Invoked Computing /
function projector
• Objects with “memory” of
action (thermal camera)
• On the flight I/O interfaces
(LSD technology)
Research Concept
Enabling Technology
(*) Jussi Angesleva, 2004
Memory Blocks (2011~)
Laser Sensing Display
(create the sense of “presence” / mixed reality)
“Physical Cloud”?

7. (a) Interacting with “floating” data

8. … the Interaction Metaphor

9. KHRONOS PROJECTOR [2005]
• Deformable screen / fixed in space
• Screen as a controller

10. KHRONOS PROJECTOR (media arts, 2004), video excerpt
• Fixed screen / deformable (passive haptic feedback)
• Screen as a membrane between real and virtual

11. Precise control possible [2008]
…a physical attribute makes manipulation more precise,
even if this is PASSIVE FORCE FEEDBACK

12. • Rigid screen / moving on space (proprioception)
• Screen as a controller
Volumetric data visualization & interaction [2006]

13. - retro-reflective paper to reflect IR light
Technical details: pose estimation markers
- used both for pose estimation and control (drag, slice, zoom...)
- Off-the-shelf, camera-projector setup

14. Future directions…
• Fish Tank VR with head tracking
• FTIR multitouch
• force feedback display?

15. Memory block application: the Portable Desk [2013]
Classic Jazz
Rock

16. Application to Neurosciences: twitterBrain [2013~]
Multimodal virtual
presence of data
Extension: portable “memory
block” to store personal data
(music, books, etc)
Room, public space as “virtual
bookshelf”
• Common virtual support to query academic publications
• Real time communication between researchers
• Proprioception, spatial memory, procedural memory
A shared “physical” database

17. Spatialized database (for neurosciences)
• “Spatialized” academic database
• “Spatialized” social network
• big data visualization techniques
• “spatialized” sountrack library
Augmented Memories [J. Puig, 2012)

18. From Deformable workspace to…
Deformable User Interface [2014]
• Skeuomorph,
• Shape and
meaning…
(beyond the “soap bar smartphone”)

19. Haptic interaction
with virtual 3d
objects
“embedded” in
the real world…
“force
field”
Virtual Haptic Radar [2008]
- Importance of multi-modal immersion (or partial immersion) -

20. High Speed Gesture UI for 3d Display (zSpace) [2013]
- Importance of instant feedback to feel virtual objects as “real” -

21. Multimodal Spatial Augmented Reality (MSAR)
…How to generate a sense of co-located presence for this
“information layer” beyond projection-mapping?
• Projection mapping (image)
• Sound, vibration, temperature…
• Minimal, zero-delay, zero-mismatch interaction…
…Towards a Function Projector capable of projecting “affordances”!

22. (b) Interacting with (augmented) objects?
Towards a
function
projector

23. Invoked Computing (Function Projector) [2011]
“Augmented Reality as the graphic front-end of Ubiquity.
And Ubiquity as the killer-app of Sustainability.”
Bruce Sterling (Wired Blog on “Invoked Computing”)

24. Invoked Computing (video excerpt)

25. Visual and Tactile + high speed interaction [2012]

26. Physical affordances as services??
rechargeable
hammer
erasable
book
...utopian or dystopian future?

27. II. Enabling technologies
I. The physical cloud (tangible gigabytes)
• Space and objects as scaffold for data
• Proprioceptive interfaces / Deformable interfaces
• Multi-Modal Augmented Reality
II. Enabling technologies (sensing/projection, I/O)
• Zero-delay, zero-mismatch spatial AR
• Minimal, ubiquitous & context aware interactive displays
• Laser displays, pneumatic slow displays, roboptics
III. Mediated Self / Augmented Perception /Prosthetics
• Augmented sensing & expression
• Electronic travel aids, Wearables

28. Zero-delay, zero-mismatch for MSAR

29. The importance of real time in HCI
高速・低遅延ジェスチャーＵＩ 1000fps 30ms遅延
→ 没入感、自己認識の向上による制御性能の向上
（通常ジェスチャーUI 30fps 200ms遅延）
Computer
or Game

30. OmniTouch (Microsoft research)
Enabling Technology : Smart Laser Sensing [2003~]
• no delay, no misalignment
• projection on mobile, deformable surfaces
• Real time sensing
Vision based
Smart sensing
vs.
Skin Games
What?
How?
• Smart sensing
• Laser Sensing Display

31. AR surveying (distances,
angles, depth...)
ubiquitous display
medical
imaging (IR,
polarization...)
image enhancement
(contrast compensation,
color..).
Lots of applications demonstrated…

32. Camera-less
active tracking
principle
Markerless laser tracking (I/O interface) [2003]
2004
2004

34. • artificial synesthesia
• real-time interaction
• new interfaces for musical expression
scoreLight: a human sized pick-up head [2009]
(in collaboration with Daito Manabe)

35. Skin Games
Body as a controller (kinect) & body as a
display surface…

36. Cameraless interactive display (no calibration needed)
Laser GUI: minimal interface [2013~]

37. Text projection-mapping at around 200 fps

38. Minimal, interactive displays vs. pixelated projection mapping?
WORKSHOP at CHI 2013:
Steimle J., Benko H., Cassinelli A., Ishii H., Leithinger D., Maes P., Poupyrev I.: Displays Take New Shape: An Agenda for
Future Interactive Surfaces. CHI’13 Extended Abstracts on Human Factors in Computing, ACM Press, 2013.

39. …minimal displays by no means imply small
Laserinne [2009]

40. Real-world special effects…
…Real world “shader”?
Saccade Display + laser sensing [2012~]
Why laser?
• Stronger persistence of vision effect
• Very long distances! (on a car, etc…)
example of a
“context aware
display”
[+DIC]
video

41. Light “pet”
• Can sense, measure
• Draw, print
• Signal (danger), indicate
• Play
• Robot made of light
• Extension of the self
• Minimal interactive display

42. III. Mediated Self / Augmented Perception
I. The physical cloud (tangible gigabytes)
• Space and objects as scaffold for data
• Proprioceptive interfaces / Deformable interfaces
• Multi-Modal Augmented Reality
II. Enabling technologies (sensing/projection, I/O)
• Zero-delay, zero-mismatch spatial AR
• Minimal, ubiquitous & context aware interactive displays
• Laser displays, pneumatic slow displays, roboptics
III. Mediated Self / Augmented Perception /Prosthetics
• Augmented sensing & expression
• Electronic travel aids, Wearables

43. What is an interface? extension of the Self?

44. Haptic Radar for extended spatial awareness [2006]
• optical “antennae” for human, not devices
• New sensorial modality (this is not TVSS)
• Extension of the body, 360 degrees

45. Haptic Radar & HaptiKar (video excerpts)

46. Ongoing experiments (50 blind people in Brasil)
In collaboration with Professor Eliana Sampaio (CNAM, Paris)
• Quantitative measures (using a simulator – calibrated magnetic compass,
and virtual reality environment)
• Production?
Presently we are doing:
• Qualitative results were extraordinary (semi-structured interviews &
ANOVA analysis of anxiety trait/state).

47. …work on wearable computing

48. Laser Aura: externalizing emotions [2011]
Example of a “minimal display” inspired by manga graphical conventions…

49. • Real space as an opportunity to organize data (vs. the “cloud”)
• Real objects as “handles” to trigger computing functions
• The vision: Physical Cloud (computing)
Summarizing…
• Minimal, context aware displays (less is more!)
• Engineered Intuitive physics (generalization of “tangible” interfaces)
• Intentional stance (“live” agents instead of control knobs)
• Real time, zero-delay, zero-mismatch Spatial AR will produce a paradigm shift
making the digital “analog” again…
• Enabling technology?
• Possible instantiation: the light-pet vs. pixelated screens
• an alternative vision to the pixelated screen (including HMD)
• avatar robot made of light
• communication enhancement (laser aura)
• enhanced spatial awareness (signaling, etc)

50. Thanks!
For more:
www.alvarocassinelli.com