Bi Manual 3D Painting - An Interaction Paradigm for Augmented Reality Live Performance - slideshare
Upcoming SlideShare
Loading in...5
×
 

Bi Manual 3D Painting - An Interaction Paradigm for Augmented Reality Live Performance - slideshare

on

  • 264 views

The rise of gestural interaction led artists to produce shows, or instal-lations based on this paradigm. We present the first stages of the « Sculpture numérique » (Virtual Sculpture) project. This ...

The rise of gestural interaction led artists to produce shows, or instal-lations based on this paradigm. We present the first stages of the « Sculpture numérique » (Virtual Sculpture) project. This project was born from a collabo-ration with dancers. Its goal is to propose bi-manual interactions in a large augmented space: we aim at giving dancers the possibility to generate and ma-nipulate virtual elements on stage using their hands. The first set of interactions we present in this paper is 3D painting, where the user can generate 3D virtual matter from his hands. The movement of the hand defines a stroke, which shape is controlled by the shape of the hand. Changing the shape and orientation of the hand allows switching between three interaction modes to produce volumes, surfaces or curves in space. We explore the applicative case of dance, with the goal of producing a plastic creation from choreography.

Statistics

Views

Total Views
264
Views on SlideShare
264
Embed Views
0

Actions

Likes
0
Downloads
2
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Bi Manual 3D Painting - An Interaction Paradigm for Augmented Reality Live Performance - slideshare Bi Manual 3D Painting - An Interaction Paradigm for Augmented Reality Live Performance - slideshare Presentation Transcript

    • Alexis Clay - Jean-Christophe Lombardo - Nadine Couture - Julien Conan Bi-manual An interaction paradigm for augmented reality live performanceHuman-Computer Interaction, Tourism and Cultural Heritage (HCITOCH 2012)
    • 4th March 2011 Movement Interaction ▪ Emotion Recognition ▪ Cultural Applications3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 2
    • 4th March 2011Feedback from the audience: 80% return rate 90% found it innovative 120 submitted forms Movement Interaction ▪ Emotion Recognition ▪ Cultural Applications3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 3
    • 4th March 2011 Feedback from the audience:Audience bright interest for 3Dribbons following the dancer’s hands 120 submitted forms Movement Interaction ▪ Emotion Recognition ▪ Cultural Applications 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 4
    • 4th March 2011Feedback from the audience: >90% wish to witness the evolution of the show 120 submitted forms Movement Interaction ▪ Emotion Recognition ▪ Cultural Applications3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 5
    • Summary1 VIRTUAL SCULPTURE Hand Interactions Stereoscopic Display2 MANUAL INTERACTIONS3 CAVE 3D PAINTING Prototype1 CAVE and development tools4 STAGE 3D PAINTING Prototype25 PERSPECTIVES Live Performance Application 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 6
    • 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D 5 PERSPECTIVES SCULPTURE INTERACTIONS PAINTING PAINTINGState of the Art Sketching Surfaces BLUIsculpt system Deisinger et al - 2000 Arctic Region Super Computing Center - 20023D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 7
    • 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D 5 PERSPECTIVES SCULPTURE INTERACTIONS PAINTING PAINTING Outcomes1 Direct Metaphor  Virtual matter seems to come out right under the user’s hand Surface Drawing Schkölne, Pruett, Schröder- 2001 Freehand Interaction Large Space Interaction and visualization2 Large Space Painting  Dance application Sketching Surfaces Deisinger et al - 2000 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 8
    • 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D 5 PERSPECTIVESSCULPTURE INTERACTIONS PAINTING PAINTINGOur Objective Hands’ Joints Path Schkolne et al. Application to DanceSurface Generation Live Performance3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 9
    • 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D 5 PERSPECTIVESSCULPTURE INTERACTIONS PAINTING PAINTINGOur Objective Credits: Samsung Galaxy SIII advertisement - 20123D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 10
    • 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D 5 PERSPECTIVESSCULPTURE INTERACTIONS PAINTING PAINTING Metaphor Generate Virtual Objects Basic Information Hand conformation Wrists’ positions Articulations and tip positions 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 11
    • 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D 5 PERSPECTIVESSCULPTURE INTERACTIONS PAINTING PAINTING 1rst metaphor: Shapes Movement parallel to the palm’s surface Curves Surfaces  1/2s or 1/10s sample 1  Surface between t and t-1 Shapes of the generated surface 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 12
    • 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D 5 PERSPECTIVESSCULPTURE INTERACTIONS PAINTING PAINTING 2nd metaphor: Volumes Movement perpendicularly to the palm’s surface  Volume section = user hand visual surface 2 2 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 13
    • 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D 5 PERSPECTIVESSCULPTURE INTERACTIONS PAINTING PAINTING 3rd metaphor: Curves Index/Middle finger conformation  More accurate  Same as a tinier brush 3  Automatic closed surfaces fullfilment 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 14
    • 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D 5 PERSPECTIVESSCULPTURE INTERACTIONS PAINTING PAINTING Thumb positions Pen-writing metaphor « Free To Move » « Draw » 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 15
    • 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D 5 PERSPECTIVESSCULPTURE INTERACTIONS PAINTING PAINTING Main interests 1 Simple Direct metaphor 2 Natural modes Hand orientation / Fingers tucked 3 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 16
    • 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D 5 PERSPECTIVES SCULPTURE INTERACTIONS PAINTING PAINTINGCAVE 3D PAINTING PROTOTYPE STAGE 3D PAINTING PROTOTYPE  Immersion in virtual space  3D Display in a Large Space  Co-located interactions  Multi viewers  Display Accuracy  Reduced cost  Development Tools  Two users simultaneously max  Dancer Blind From His Creations  Cost  Occlusion Problem  3mx3mx3m  One Projection Screen 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 17
    • 1 VIRTUAL 2 MANUAL 4 STAGE 3D 3 CAVE 3D PAINTING 5 PERSPECTIVESSCULPTURE INTERACTIONS PAINTING Instrumented Space Hand IR Trackers Virtual Platform with AR-DtrackGlasses with Positions Trackers 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 18
    • 1 VIRTUAL 2 MANUAL 4 STAGE 3D 3 CAVE 3D PAINTING 5 PERSPECTIVESSCULPTURE INTERACTIONS PAINTING isiVR Scalable and Interactive Virtual Reality Immersive 3D Visualization FrameWork OpenSceneGraph Ready-to-use Immersive Tools Render Engine Multi-display Cluster Position Tracking isiVR Features 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 19
    • 1 VIRTUAL 2 MANUAL 4 STAGE 3D 3 CAVE 3D PAINTING 5 PERSPECTIVESSCULPTURE INTERACTIONS PAINTING isiVR OpenSceneGraph Ready-to-use Immersive Tools Render Engine  Facilitate Additions/ModificationsInputs Behaviors Outputs  openSource license 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 20
    • 1 VIRTUAL 2 MANUAL 4 STAGE 3D 3 CAVE 3D PAINTING 5 PERSPECTIVESSCULPTURE INTERACTIONS PAINTING Schkölne Paradigm Application Flat Polygons Control Points Positions: Bézier Surfaces according to  Middle Finger Distal Phalanx Hand’s Movement  Wrist 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 21
    • 1 VIRTUAL 2 MANUAL 4 STAGE 3D 3 CAVE 3D PAINTING 5 PERSPECTIVESSCULPTURE INTERACTIONS PAINTING Surface Generation by Hand in a CAVE 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 22
    • 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D 5 PERSPECTIVES SCULPTURE INTERACTIONS PAINTING PAINTINGCAVE 3D PAINTING PROTOTYPE STAGE 3D PAINTING PROTOTYPE  Immersion in virtual space  3D Display in a Large Space  Co-located interactions  Multi viewers  Display Accuracy  Reduced cost  Development Tools  Two users simultaneously max  Dancer Blind From His Creations  Cost Effective  Occlusion Problem  3mx3mx3m  One Projection Screen 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 23
    • 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D PAINTING 5 PERSPECTIVESSCULPTURE INTERACTIONS PAINTING Prototype Scheme Instrumented User Signals received are different format 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 24
    • 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D PAINTING 5 PERSPECTIVESSCULPTURE INTERACTIONS PAINTING Motion Capture Suit XSens SIGNALS Mtx Trackers 17 Motion Trackers Gyroscope, magnetomet ers, and accelerometers signals Xbus Master Articulated and Rigid Kinematic Chain Master Slave Individual Body Segments Positions x,y,z(t) Orientations 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 25
    • 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D PAINTING 5 PERSPECTIVESSCULPTURE INTERACTIONS PAINTING Data Gloves 5DT SIGNALS 1 Thumb_Near 14 Flexion Trackers 2 Thumb_Far 3 Thumb_Index 4 7 10 Erase Objects 4 Index_Near 3 6 9 5 Index_Far 5 8 13 6 Index_Middle 11 12 Point an object 7 Index_Near 1 2 8 Index_Far 9 Index_Ring 0 Choose color 10 Ring_Near 11 Ring Far Display Palette 12 Ring_Little 13 Ring_Near 14 Ring_Far Choose color 0 < Flexion < 1 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 26
    • 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D PAINTING 5 PERSPECTIVESSCULPTURE INTERACTIONS PAINTING VIRTOOLS Behaviors Indirect Basis Render Engine Visual Programming Editor Procedural Approach SDK Inputs Building Blocks Management 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 27
    • 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D PAINTING 5 PERSPECTIVESSCULPTURE INTERACTIONS PAINTING VIRTOOLS Building Blocks Message Handler Actions Reactions Settings OBJECT Variable Configuration Loop Process Custom Dialog Box 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 28
    • 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D PAINTING 5 PERSPECTIVESSCULPTURE INTERACTIONS PAINTING VIRTOOLS COORDINATES Hands BUFFER x,y,z(t) vertices x,y,z(t+1) vertices ENTITY CREATION Object Mesh Material Texture Shader PROCESSING Lights Camera Main Object Sub Object 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 29
    • 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D PAINTING 5 PERSPECTIVESSCULPTURE INTERACTIONS PAINTING VIRTOOLS 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 30
    • 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D PAINTING 5 PERSPECTIVESSCULPTURE INTERACTIONS PAINTING 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 31
    • 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D PAINTING 5 PERSPECTIVESSCULPTURE INTERACTIONS PAINTING Tesselation Post-Process 1) 4.Tip 3.Distal 2) 2.Medial 3) 1.Proximal Tesselation More Details Save CPU Resources 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 32
    • 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D PAINTING 5 PERSPECTIVESSCULPTURE INTERACTIONS PAINTING Constraint Occlusion Projection Screen SCENE Semi-transparent projection screen AUDIENCE (between dancer and audience) 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 33
    • 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D PAINTING 5 PERSPECTIVESSCULPTURE INTERACTIONS PAINTING « La Mer », Debussy - 1905 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 34
    • 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3D PAINTING 5 PERSPECTIVESSCULPTURE INTERACTIONS PAINTING 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 35
    • 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3DSCULPTURE INTERACTIONS PAINTING PAINTING 5 PERSPECTIVES Current Work and Applications Dancer’s Movement Parameters:  Space  Time  Weight  Flow “Computational models of expressive gesture in multimedia systems”, G.Volpe et al, 2003 Current Work Credits: Proteigon - 2011 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 36
    • 1 VIRTUAL 2 MANUAL 3 CAVE 3D 4 STAGE 3DSCULPTURE INTERACTIONS PAINTING PAINTING 5 PERSPECTIVES Perspectives Creations Haptic Feedback Creations Visual Feedback 3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 37
    • Authors Alexis Clay Jean-Christophe Lombardo Nadine Couture Julien Conan3D Painting – A. Clay, J.C. Lombardo, N. Couture and J. CONAN / ESTIA - INRIA / 12-09-26 38
    • Alexis Clay - Jean-Christophe Lombardo - Nadine Couture - Julien Conan Bi-manual An interaction paradigm for augmented reality live performanceHuman-Computer Interaction, Tourism and Cultural Heritage (ESTIA - INRIA)