19 OCT 2010 @ Le Garage
        telepresence etc...




                      Christian Jacquemin, LIMSI-CNRS
            ...
purpose of the talk
   → present some research projects
in arts & science in which I have been
                           ...
LIMSI by numbers

 → 120 permanent researchers and
              administrative staff


            → 80 PhDs & postdocs

...
LIMSI by topics


→ Spoken, written and gestural language processing


           → Multimodal perception and interaction
...
Virtuality, Interaction,
                Design, & Art (VIDA)
                                           → seminars:
  200...
augmented virtuality
augmented
       virtuality
trompe-l'oeil artwork
           → multiple cameras

        for multiple projections

       ...
augmented
          virtuality
stereoscopic artwork

 → double camera for stereo rendering



                      Bertra...
augmented
                         virtuality
                      realistic shadow

(1) natural shadows through shadow m...
augmented
      virtuality
augmenting VR with
           shadow
      → video-based shadowing



                Bertrand ...
augmented
  virtuality

3D reconstruction for shadowing
augmented
                      virtuality
          augmenting VR with
                     shadow

                     ...
augmented
          virtuality
           some issues
in perceptual continuity
(static) augmented
             reality
AR for the
        performing arts
                         distortion

  distortion for geometry compensation
  (1) proje...
AR for open air
   installations


          → virtual texture

                  mapping



          Bertrand Planes

  ...
multimedia
  architectural AR
              multimodal
          synchronization

     → real-time sound analysis
      an...
multimedia
architectural AR
multimedia
architectural AR
multimedia
architectural AR
   → audio & graphic multi-channeling

                           Rami Ajaj
             Chris...
audio AR
       multi-listener
    spatialized audio

→write and play a spatialized
          audio composition
       for...
audio AR
                   multi-listener
                spatialized audio
→ single geometry vs. multimodal outputs
    ...
audio AR

→ virtual listeners


      SoundDelta
       2007-2009
     Wai Kit Chan
        Brian Katz
 Perrine Monjaux
  ...
(in situ) augmented
              reality
in situ AR

                                       → Genius Loci project
recycling 3D digital heritage into in situ Augmen...
in situ AR

→ historical hypotheses =
     3D scans of buildings,
     furniture, sculptures...
            + 3D models of...
in situ AR

→ contemporary rendering =
        3D scans of buildings,
       furniture, sculptures...
          + non-phot...
in situ AR

                                           → issues
                                        - calibration
    ...
mobile augmented
           reality
mobile AR
  June 5, 2009 - Futur en
                    Seine
augmentation of an urban
   environment in motion
          ...
mobile augmented
           reality
       technical setup
mobile AR

considered effects

        → image filters


     → physical models


            → particles
mobile AR
 multipass GPU
rendering chain
mobile augmented
           reality
 augmentation of urban
         environment
        → aesthetics issues
             w...
mobile augmented
           reality


           Bertrand Planes
             Wai Kit Chan
        Matthieu Courgeon
     ...
shadow mobile AR
Toute La Lumière
        Sur L'Ombre
shadow/silhouette detection
        and ornamentation
              → IR camera captu...
Toute La Lumière
    Sur L'Ombre


              set-up

    → image analysis:
  shadow & silhouette

  → image synthesis:...
Toute La Lumière
    Sur L'Ombre

        sample FXs
Toute La Lumière
    Sur L'Ombre
                      → scientific issues

 smart projection & environment-aware AR

    ...
Toute La Lumière
        Sur L'Ombre

                                        → partners

didascalie.net: digital platform...
hybrid interfaces
physical world
      augmentation
            physical layers

  → physical layers for animation
     from performer's cap...
physical world
augmentation

                 → multiple

             physical layers



     La Pluralité des mondes

  ...
physical world
                                                          augmentation



                                 ...
physical world
   augmentation
       physical models

    image-based animation
           of physical models
    → live ...
physical world
augmentation

          image-based

         MSS animation



           four set-ups

            with li...
physical world
                         augmentation
                                                    → related work

-...
dance gesture
                               analysis for physical
                                    models control
    ...
dance gesture
analysis for physical
     models control


                       → some scientific issues

               ...
multimodal interfaces
audiovisual magnifier
                lens
        multimodal distortion
              for multimedia
          informatio...
audiovisual
magnifier lens
multimodal distortion
      for multimedia
  information access
      → geometrically coherent
...
audiovisual
      magnifier lens
      multimodal distortion
            for multimedia
        information access

      ...
µ-surface, µ-view,
   µ-modal interface

               → 2D/3D combination
                for architectural design


   ...
µ-surface, µ-view,
              µ-modal interface

→ 2D/3D interface for acoustic rendering control & visualization

    ...
sonographic space
                              plumage
                         → sound analysis
spatialization according...
sonographic space
             plumage



             → read heads
         + free trajectories
           or elliptic pa...
sonographic space
         the concatenator

→ live sound analysis and replay
for musical performance or teaching


      ...
Topophonie:
                                granular audio-
                            graphical rendering
              ...
Topophonie:
                            granular rendering
                                      → LIMSI scientific issues...
affective computing
  & performing arts
capture of emotions
                                  physiological signals

                                 → intrusiven...
capture of emotions
                                → live audio & graphics from physiological signals



                ...
capture of emotions

                                                       → related work:

                             ...
tangibility
anthropomorphic
           interface
        tangible & affective
→ attractiveness of anthropomorphic
   interfaces for af...
anthropomorphic
                 interface
                tangible & affective
                             → related wor...
anthropomorphic
       interface
 Pogany head-shaped
           interface



           → image-based
         tangible in...
anthropomorphic
       interface
 Pogany head-shaped
           interface

              → used for
              the cont...
anthropomorphic
       interface
 application in remote
  emotion expression
      → experiment through
       a game for ...
augmented jewel @
      Fraunhofer
previous work
 Virtual Mirror



       → Virtual Mirror:
       try-on installation
     for virtual garments
  (shoes, o...
previous work
 Virtual Mirror
          → Virtual Mirror:
     work by Anna Hillsman
               & Peter Eisert
       ...
extension:
   virtual jewels


→ based on deformable surface
   tracking and virtual objects
                    rendering...
environment-based
        rendering

           → 3D multipass
                rendering:
                 - reflection
  ...
image-based
    rendering
→ from real jewel capture
        & reconstruction
             (Peter Eisert)
shadowing +
         compositing

                   → shadowing:
      ambient rendering + scaling
       + blur + semi-t...
synthesis and perspectives

       → current: cloth tracking + EBR/IBR
        flexible technique for cloth registered
   ...
synthesis and perspectives

                                             → virtual jewels open research avenues
          ...
perspectives for remote
corporate collaboration
telepresence
                                                                             → 3 factors:
                   ...
telepresence
                    → some issues:
technical choices: viewpoint choice,
               interaction modalities...
telepresence



  → some measures:
     spatial presence
         engagement
   emotional response
    ecological validity...
tangible user
                                                                         interfaces
                        ...
tangible user
                                                                interfaces
                                 ...
tangible user
                                                                             interfaces

                   ...
perceptual continuity

                          → beyond technological continuity
                                     cf...
multi-modality

                               → tactile, haptics, shadow etc.:
vision, a predominant sense in the 21st ce...
emotional expression
                                                           → emotional communication
- emotion expres...
augmented self


                               → augmented instrument:
                          - augmented musical inst...
art as a creative
                                  experimental setup
                                               → hy...
art fairs & galleries:
                       a creative market place
                                                → ar...
Upcoming SlideShare
Loading in …5
×

Jacquemin

942 views
885 views

Published on

Telepresence and Augmented Reality.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
942
On SlideShare
0
From Embeds
0
Number of Embeds
26
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Jacquemin

  1. 1. 19 OCT 2010 @ Le Garage telepresence etc... Christian Jacquemin, LIMSI-CNRS and University Paris 11, Orsay, France
  2. 2. purpose of the talk → present some research projects in arts & science in which I have been involved → offer synthesis and perspectives on telepresence, tangible interfaces, emotions...
  3. 3. LIMSI by numbers → 120 permanent researchers and administrative staff → 80 PhDs & postdocs → 2/3 computer science 1/3 physics
  4. 4. LIMSI by topics → Spoken, written and gestural language processing → Multimodal perception and interaction → Augmented and Virtual Reality
  5. 5. Virtuality, Interaction, Design, & Art (VIDA) → seminars: 2008-09: 4 seminars on simulation & materialization 2009-10: 4 seminars on Cloud and simulation → research projects: you will see some of them → collaborations: with artists, culture centers, associations, companies...
  6. 6. augmented virtuality
  7. 7. augmented virtuality trompe-l'oeil artwork → multiple cameras for multiple projections → viewer's location is constrained Espaces Indicibles Incidents Mémorables May 2007
  8. 8. augmented virtuality stereoscopic artwork → double camera for stereo rendering Bertrand Planes Gate:3.5 Artcore 2007
  9. 9. augmented virtuality realistic shadow (1) natural shadows through shadow mapping (2) video-based shadowing → increase the feeling of presence → allow for non tactile contact
  10. 10. augmented virtuality augmenting VR with shadow → video-based shadowing Bertrand Planes Mar3D La Paz 2006 Rami Ajaj & Fabien Pigeaud
  11. 11. augmented virtuality 3D reconstruction for shadowing
  12. 12. augmented virtuality augmenting VR with shadow → related works: - virtual shadows in VR (Naemura et al. 2002) - shadows in augmented realtiy (Haller et al 2003)
  13. 13. augmented virtuality some issues in perceptual continuity
  14. 14. (static) augmented reality
  15. 15. AR for the performing arts distortion distortion for geometry compensation (1) projection on non planar surfaces (2) projection on non orthogonal planes → keystone and virtual texture mapping → perspective removal
  16. 16. AR for open air installations → virtual texture mapping Bertrand Planes Bump it! Superflux Lyon Lumieres 2006
  17. 17. multimedia architectural AR multimodal synchronization → real-time sound analysis and network communication → audio & graphic multi-channeling → audio & graphic special effects (display of peak values)
  18. 18. multimedia architectural AR
  19. 19. multimedia architectural AR
  20. 20. multimedia architectural AR → audio & graphic multi-channeling Rami Ajaj Christophe d'Alessandro Markus Noisternig Bertrand Planes Organ & AR Church Ste Elisabeth Paris May 2008
  21. 21. audio AR multi-listener spatialized audio →write and play a spatialized audio composition for up to 200 listeners through wireless network
  22. 22. audio AR multi-listener spatialized audio → single geometry vs. multimodal outputs → virtual listeners → multipurpose graphical interface: - spatialized sonic composition - audio rendering computation - graphical advanced rendering
  23. 23. audio AR → virtual listeners SoundDelta 2007-2009 Wai Kit Chan Brian Katz Perrine Monjaux Nick Mariette Marcus Noisternig
  24. 24. (in situ) augmented reality
  25. 25. in situ AR → Genius Loci project recycling 3D digital heritage into in situ Augmented Reality Partners MAP-GAMSAU (Livio de Luca) Laboratoire Victor Vérité (Veronique Caye) LIMSI-CNRS (Emmanuelle Frenoux Sonia Dahdouh) Chartreuse-lès-Avignon
  26. 26. in situ AR → historical hypotheses = 3D scans of buildings, furniture, sculptures... + 3D models of lost items recycled into AR interactive installations
  27. 27. in situ AR → contemporary rendering = 3D scans of buildings, furniture, sculptures... + non-photorealistic rendering + special FXs for in situ AR interactive installations
  28. 28. in situ AR → issues - calibration - data recycling (scans, models, textures...) t - media merging (video, 3D synthesis, audio...) - interactivity (tracking, lighting, scenarios...) © MAP GAMSAU
  29. 29. mobile augmented reality
  30. 30. mobile AR June 5, 2009 - Futur en Seine augmentation of an urban environment in motion → IR camera capture with IR lighting to avoid video feedback → real-time image processing (GPU) → approximate calibration (keystone)
  31. 31. mobile augmented reality technical setup
  32. 32. mobile AR considered effects → image filters → physical models → particles
  33. 33. mobile AR multipass GPU rendering chain
  34. 34. mobile augmented reality augmentation of urban environment → aesthetics issues what looks nice? → technical issues lag, accuracy, lighting → scientific issues prepare symbolic mobile AR
  35. 35. mobile augmented reality Bertrand Planes Wai Kit Chan Matthieu Courgeon Sonia Dadouh Emmanuelle Frenoux Emmanuel Blachon
  36. 36. shadow mobile AR
  37. 37. Toute La Lumière Sur L'Ombre shadow/silhouette detection and ornamentation → IR camera capture with/without IR lighting shadow + silhouette detection → real-time image synthesis masked by shadow and/or silhouette
  38. 38. Toute La Lumière Sur L'Ombre set-up → image analysis: shadow & silhouette → image synthesis: masking & special FX
  39. 39. Toute La Lumière Sur L'Ombre sample FXs
  40. 40. Toute La Lumière Sur L'Ombre → scientific issues smart projection & environment-aware AR → social issues how shadow can inform us on our relationship with our environment → artistic issues can we have a portable installation for the street the library, & the stage
  41. 41. Toute La Lumière Sur L'Ombre → partners didascalie.net: digital platform for performing arts & theater company L'Ange Carasuelo: theater company Mediathèque Taverny LIMSI-CNRS (coordinator) → funding: Région Île de France
  42. 42. hybrid interfaces
  43. 43. physical world augmentation physical layers → physical layers for animation from performer's capture (postural or physiological sensors) → multiple transparent layers for scene depth → visual changes through layer removal
  44. 44. physical world augmentation → multiple physical layers La Pluralité des mondes Jacques Roubaud Incidents Mémorables December 2006
  45. 45. physical world augmentation → related works: La pluralité des Mondes, Incidents Mémorables - augmented performance (Sparacino et al 1999) - computer theater (Pinhanez 1997) - mass-spring models (Provot 1995)
  46. 46. physical world augmentation physical models image-based animation of physical models → live or prerecorded video → real-time 2D/3D graphics allows for real-time interaction with a mass-spring system
  47. 47. physical world augmentation image-based MSS animation four set-ups with live or synthesis video
  48. 48. physical world augmentation → related work - mass-spring systems through GPGPU (Georgii & Westermann 2005) - particles for cloning motion (Wolford & Guedes 2007)
  49. 49. dance gesture analysis for physical models control → collaboration IRCAM/LIMSI - gesture analysis - physical model rendering - control and expressiveness Co-supervisor: Frederic Bevilacqua (IRCAM) PhD student: Sarah Fdili Alaoui (LIMSI & IRCAM) http://sarah.alaoui.free.fr/ Funding: University Paris Sud 11
  50. 50. dance gesture analysis for physical models control → some scientific issues - movement quality recognition - physical model rendering and control - contribution to dance performance and pedagogy
  51. 51. multimodal interfaces
  52. 52. audiovisual magnifier lens multimodal distortion for multimedia information access → apply geometrically coherent distortions to visual and audio output → evaluate the gain in speed and comfort → study dissimila occlusions or distorsions
  53. 53. audiovisual magnifier lens multimodal distortion for multimedia information access → geometrically coherent distortions to visual and audio output Tifanie Bouchara Brian Katz
  54. 54. audiovisual magnifier lens multimodal distortion for multimedia information access → related work: - on demand video browsing (de Rooj 2007) - comparison AV, audio, & visual navigation in 3D (Gröhn et al. 2003) - fish eye views (Gutwin 2002)
  55. 55. µ-surface, µ-view, µ-modal interface → 2D/3D combination for architectural design → shared degrees of freedom between the modalities → tool for interface design & modeling modalities ⨯ devices ⨯ I/O
  56. 56. µ-surface, µ-view, µ-modal interface → 2D/3D interface for acoustic rendering control & visualization Rami Ajaj, Markus Noisternig, Lauri Savioja 3D visualization
  57. 57. sonographic space plumage → sound analysis spatialization according to sound parameters → interaction navigation into a soundscape → synthesis granular synthesis from sound grains
  58. 58. sonographic space plumage → read heads + free trajectories or elliptic paths
  59. 59. sonographic space the concatenator → live sound analysis and replay for musical performance or teaching Roland Cahen – ENSCI Diemo Schwarz – IRCAM Etienne Brunet - Musician
  60. 60. Topophonie: granular audio- graphical rendering → consortium ENSCI/IRCAM/LIMSI/Navidis/SmallLab - granular concatenative sound rendering - graphic particle rendering - audio-graphical representation and synchronization → funding Agence Nationale de la Recherche http://www.topophonie.fr/ Coordination: Roland Cahen (ENSCI)
  61. 61. Topophonie: granular rendering → LIMSI scientific issues - 3D audio-graphic scene representation (XML): extension of X3D - LOD computation of audio-graphic granular scenes: cf LOD for point-based graphics University of Bielefeld - audio-graphic synchronization
  62. 62. affective computing & performing arts
  63. 63. capture of emotions physiological signals → intrusiveness of the capture → complex signal processing: photography:Yann Le Herissé - multiple time-scales, - noise & signal overlay, - multivariate analysis required
  64. 64. capture of emotions → live audio & graphics from physiological signals Rami Ajaj Ivan Chabanaud Loïc Kessous Renaud Rubian Performer: photography: Perrine Monjaux Laurent Chanel Artistic design: Benedicte Adessi
  65. 65. capture of emotions → related work: - signal processing for emotion recognition (Kim et al. 2004, Haag et al. 2004) photo credit, didascalie.net - affective computing (Picard, 2002, Rani et al 2005) - emotion in voice (Banse & Scherer 1996)
  66. 66. tangibility
  67. 67. anthropomorphic interface tangible & affective → attractiveness of anthropomorphic interfaces for affective computing → body schema helps to cope with complex percepts → intuitive associations between action and effects
  68. 68. anthropomorphic interface tangible & affective → related works - Doll's head interface (Hinckley et al. 1994) - SenToy (Paiva et al. 2002) a full body doll
  69. 69. anthropomorphic interface Pogany head-shaped interface → image-based tangible interface
  70. 70. anthropomorphic interface Pogany head-shaped interface → used for the control of expressive facial animation
  71. 71. anthropomorphic interface application in remote emotion expression → experiment through a game for conveying an expression through facial animation
  72. 72. augmented jewel @ Fraunhofer
  73. 73. previous work Virtual Mirror → Virtual Mirror: try-on installation for virtual garments (shoes, or cloth texture and color)
  74. 74. previous work Virtual Mirror → Virtual Mirror: work by Anna Hillsman & Peter Eisert on garment tracking and texture overlay (clothes) or substitution (shoes) → based on deformable surface tracking or object tracking
  75. 75. extension: virtual jewels → based on deformable surface tracking and virtual objects rendering: two techniques for virtual jewel augmentation
  76. 76. environment-based rendering → 3D multipass rendering: - reflection - transparency - blending through environment mapping
  77. 77. image-based rendering → from real jewel capture & reconstruction (Peter Eisert)
  78. 78. shadowing + compositing → shadowing: ambient rendering + scaling + blur + semi-transparency → compositing: video + shadows + jewel rendering
  79. 79. synthesis and perspectives → current: cloth tracking + EBR/IBR flexible technique for cloth registered jewels (pendant or brooch) → ongoing: body tracking for necklace, ear-rings, bracelets... how far can go through markerless capture? can we make jewel substitution?
  80. 80. synthesis and perspectives → virtual jewels open research avenues to social and affective computing → ongoing collaboration with ENSCI has proposed the following tracks: - jewel as interactive diary - jewel as personal avatar - extravagant & dream jewel Solène Borrat (ENSCI) Flavie Papin (ENSCI) Christophe Luxereau, Gecom. Rhizomes. 2006
  81. 81. perspectives for remote corporate collaboration
  82. 82. telepresence → 3 factors: objective telepresence: stimuli for interaction subjective telepresence: cognitive factors to suspend disbelief social telepresence: sense of coexistence of other human beings → video conferencing examples: immersive systems emotional communication HowStuffWorks: Classroom video conferencing metaverse and video-conferencing brings the rainforest to the students in this lesson.
  83. 83. telepresence → some issues: technical choices: viewpoint choice, interaction modalities, environment awareness... personality traits: imagination, emotions, and willing suspension of disbelief (user self-transcendence style: self-forgetfulness, self-awareness)
  84. 84. telepresence → some measures: spatial presence engagement emotional response ecological validity task performance self efficacy
  85. 85. tangible user interfaces → a physical interface with the digital world 2D/3D w or wo videoprojection similarities with traditional interactors (tools, instruments, furniture...) → go well with - ambient computing, - I-room, - augmented & mixed realities... ReacTable: S. Jordà, M. Kaltenbrunner, G. Geiger and M. Alonso
  86. 86. tangible user interfaces → tabletop interfaces - dedicated GUI & interactions - multitouch vs single touch & user identification - retro- vs. direct projection - 3D augmentation - combination with phicons... → tabletop applications - distant desktop sharing - brainstorming and creativity TeamViz: R. Ajaj, L. Saviioja, F. Vernier, C. Jacquemin
  87. 87. tangible user interfaces → an example of 2nd generation TUI: SandScape seamless coupling of sensing & display user interact with sand gets immediate analysis & effects in the 3D representation + projection on the interface SandScape: Y. Wang, A. Biderman, B. Piper, C. Ratti, and H. Ishii
  88. 88. perceptual continuity → beyond technological continuity cf Milgram schema about the real to virtual continuum: - trompe-l'oeil - light & shadow continuity - tactile continuity (?) - environment consideration: ecological validity Company Adrien Mondot
  89. 89. multi-modality → tactile, haptics, shadow etc.: vision, a predominant sense in the 21st century is not the only one multimodality enhances presence, emotion, belief, engagement... → real-world mimicking: “natural” interactions for interfaces, add reality to interfaces without loss of power: < augmented instruments, everyday gestures, anthropomorphic interfaces... Novint Falcon
  90. 90. emotional expression → emotional communication - emotion expression (posture & facial expression of avatars) - social signal processing - anthropomorphic I/O → joint research: HCI & experimental psychology Pascale Barret Oh peer, my Teddy! MARC expressive 3D face Matthieu Courgeon
  91. 91. augmented self → augmented instrument: - augmented musical instruments, - augmented brush, - augmented body (eg dataglove) → virtual self - metaverse - social networks Eduardo Kac exoskeleton
  92. 92. art as a creative experimental setup → hybrid objects: creative play at the frontier between art and technology, innovative combinations of medias and tools → art installation: a specific usage context that generally involves an audience and large scale Speaker Lamp by Bertrand Planes testing environments
  93. 93. art fairs & galleries: a creative market place → art curators: art market is a tradeoff between corporate and creative world → diffusion: art events can appeal medias and promote project visibility → issue: how to display a work-in-progress as an art work Damien Hirst auction

×