• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Ph.D. Defense: Expressive Sound Synthesis for Animation
 

Ph.D. Defense: Expressive Sound Synthesis for Animation

on

  • 2,143 views

The main objective of this thesis is to provide tools for an expressive and real-time synthesis of sounds resulting from physical interactions of various objects in a 3D virtual environment. Indeed, ...

The main objective of this thesis is to provide tools for an expressive and real-time synthesis of sounds resulting from physical interactions of various objects in a 3D virtual environment. Indeed, these sounds, such as collisions sounds or sounds from continuous interaction between surfaces, are
difficult to create in a pre-production process since they are highly dynamic and vary drastically depending on the interaction and objects. To achieve this goal, two approaches are proposed; the first one is based on simulation of physical phenomena responsible for sound production, the second one is based on the processing of a recordings database.

Statistics

Views

Total Views
2,143
Views on SlideShare
2,096
Embed Views
47

Actions

Likes
1
Downloads
18
Comments
0

4 Embeds 47

http://www-sop.inria.fr 33
http://www.linkedin.com 9
http://www.slideshare.net 4
http://www.lmodules.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Ph.D. Defense: Expressive Sound Synthesis for Animation Ph.D. Defense: Expressive Sound Synthesis for Animation Presentation Transcript

    • t Expressive Sound Synthesis For Animation Cécile Picard-Limpens University of Nice/Sophia-Antipolis École Doctorale STIC REVES INRIA Sophia-Antipolis, France Advisors: George Drettakis, INRIA Sophia Antipolis (Reves) François Faure, INRIA Rhône-Alpes (Evasion) Nicolas Tsingos, DOLBY Laboratories, CA, USA Defense for Ph.D. in Computer Science C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 1
    • t Outline 1 Sound and Virtuality 2 Physics-Based Sound Synthesis Contact Modeling Resonator Modeling 3 Example-Based Synthesis Flexible Sound Synthesis 4 Perspectives on a Hybrid Model Motivation and Application 5 Conclusion and Discussion Contributions Extensions and Applications C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 2
    • t Sound Rendering Sound and Virtuality General Background for Virtual Reality and Games Motivation Physics-Based Synthesis Example-Based Interactive Audio Rendering Synthesis Perspectives on a Hybrid Model Conclusion and Discussion (R. Vantielcke - WipeoutHD on Playstation 3) C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 3
    • t Sound Rendering Sound and Virtuality General Background for Virtual Reality and Games Motivation Physics-Based Synthesis Example-Based Interactive Audio Rendering Synthesis Perspectives on a Hybrid Model Conclusion and Discussion (R. Vantielcke - WipeoutHD on Playstation 3) Traditional Approach Pre-Recordings Triggered + : Easy to implement – : Repetitive audio, discrepancies, lack of flexibility C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 3
    • t From Playback of Samples Sound and Virtuality General Background to Synthesis Motivation Physics-Based Synthesis Digital Sound Synthesis Example-Based Synthesis Source modeling ← Perspectives on Sound propagation, Sound reception a Hybrid Model Conclusion and Techniques Discussion Rigid body simulation Finite Element Method (FEM) (ArtiSynth) C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 4
    • t From Playback of Samples Sound and Virtuality General Background to Synthesis Motivation Physics-Based Synthesis Digital Sound Synthesis Example-Based Synthesis Source modeling ← Perspectives on Sound propagation, Sound reception a Hybrid Model Conclusion and Techniques Discussion Rigid body simulation Finite Element Method (FEM) (ArtiSynth) Physical Sound Simulation + : Physical approach, easy parametrization, Low memory usage – : Preprocess computation, Interface between physics and sound system C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 4
    • t Controlling the Sound Simulation Sound and Virtuality Challenges General Background Motivation Physics-Based Synthesis Example-Based Synthesis Perspectives on Sound Coherent With Visuals a Hybrid Model Conclusion and Unpredictable character of sounds Discussion Real-time sound synthesis Parametrization and Expressiveness Control and interactivity Authoring C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 5
    • t Our Contribution Sound and Virtuality Three Research Axes General Background Motivation Physics-Based Synthesis Example-Based Synthesis Physics-Based Sound synthesis Perspectives on a Hybrid Model Contact modeling Conclusion and Resonator modeling Discussion C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 6
    • t Our Contribution Sound and Virtuality Three Research Axes General Background Motivation Physics-Based Synthesis Example-Based Synthesis Physics-Based Sound synthesis Perspectives on a Hybrid Model Contact modeling Conclusion and Resonator modeling Discussion Example-Based Sound Synthesis Automatic analysis of pre-recordings Flexible synthesis for physics-driven animation C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 6
    • t Our Contribution Sound and Virtuality Three Research Axes General Background Motivation Physics-Based Synthesis Example-Based Synthesis Physics-Based Sound synthesis Perspectives on a Hybrid Model Contact modeling Conclusion and Resonator modeling Discussion Example-Based Sound Synthesis Automatic analysis of pre-recordings Flexible synthesis for physics-driven animation Perspectives on a Hybrid Model C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 6
    • t Overview Sound and Virtuality Physics-Based Synthesis 1 Sound and Virtuality Contact Modeling Audio Texture Synthesis For Complex Contacts 2 Physics-Based Sound Synthesis Resonator Modeling Contact Modeling A Robust and Multi-Scale Modal Analysis Resonator Modeling Example-Based Synthesis 3 Example-Based Synthesis Perspectives on Flexible Sound Synthesis a Hybrid Model Conclusion and 4 Perspectives on a Hybrid Model Discussion Motivation and Application 5 Conclusion and Discussion Contributions Extensions and Applications C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 7
    • t Sound from Contacts Sound and Virtuality Physics-Based Synthesis Contact Modeling Audio Texture Synthesis For Complex Dichotomy Contacts Resonator Modeling Impacts A Robust and Multi-Scale Modal Analysis Continuous contacts Example-Based Synthesis Two Schemes for Contact Force Modelling Perspectives on a Hybrid Model Feed-forward scheme Conclusion and [van den Doel et al. 01] Discussion Additive synthesis Direct computation of contact forces [Avanzini et al. 02] Bristle model C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 8
    • t Contact Modeling Sound and Virtuality Physics-Based Synthesis Contact Modeling What Are The Current Limitations Audio Texture Synthesis For Complex for Continuous Contacts? Contacts Resonator Modeling Rate for physics engine report A Robust and Multi-Scale Modal Analysis No geometric details when using visual textures Example-Based Synthesis Authoring and control are challenging Perspectives on a Hybrid Model Conclusion and Discussion C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 9
    • t Contact Modeling Sound and Virtuality Physics-Based Synthesis Contact Modeling What Are The Current Limitations Audio Texture Synthesis For Complex for Continuous Contacts? Contacts Resonator Modeling Rate for physics engine report A Robust and Multi-Scale Modal Analysis No geometric details when using visual textures Example-Based Synthesis Authoring and control are challenging Perspectives on a Hybrid Model HOW Can We Solve Them? Conclusion and Discussion By extracting Excitation profiles from visual textures with Adaptive resolution [Picard et al., VRIPHYS 08] C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 9
    • t Method for Impact Sounds Sound and Virtuality Physics-Based Synthesis Contact Modeling Audio Texture Synthesis For Complex Contacts Resonator Modeling A Robust and Multi-Scale Modal Analysis Example-Based Synthesis Perspectives on a Hybrid Model Conclusion and Discussion C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 10
    • t Method for Continuous Contact Sounds Sound and Virtuality Extraction of Excitation Profiles Physics-Based Synthesis Contact Modeling Audio Texture Synthesis For Complex Contacts Resonator Modeling A Robust and Multi-Scale Modal Analysis Example-Based Synthesis Perspectives on a Hybrid Model Conclusion and Discussion C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 11
    • t Synthesis of Excitation Profiles Sound and Virtuality For the Audio Force Modelling Physics-Based Synthesis Contact Modeling Audio Texture Synthesis For Complex Contacts Technique Resonator Modeling A Robust and Multi-Scale Modal Analysis Extraction from the visual texture image Example-Based Re-sampling along the trajectory Synthesis of the contact interaction (60Hz vs 44kHz) Perspectives on a Hybrid Model Conclusion and Based on the Complexity of the Histogram Discussion Simple texture image: Gradient of the image intensity Complex texture image: Isocurves of constant brightness (isophotes) C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 12
    • t Complex Textures Sound and Virtuality Coding the Excitation Profiles Physics-Based Synthesis Contact Modeling Audio Texture Synthesis For Complex Isophotes = Large amount of data Contacts Resonator Modeling How Can We Lighten the Info? A Robust and Multi-Scale Modal Analysis Example-Based By Coding the Excitation Profiles Synthesis = Main Features + Noise Part Perspectives on a Hybrid Model Conclusion and Discussion = + Noise Part: Statistical approximation C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 13
    • t Real-Time Audio Management Sound and Virtuality A Flexible Audio Pipeline Physics-Based Synthesis Contact Modeling Audio Texture Synthesis For Complex Contacts Resonator Modeling Simulations Driven by Ageia’s PhysX (now NVIDIA) A Robust and Multi-Scale Modal Analysis Example-Based Synthesis Perspectives on a Hybrid Model Conclusion and Discussion C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 14
    • t Audio Texture Synthesis Sound and Virtuality A Solution for Interactive Simulations Physics-Based Synthesis Contact Modeling Audio Texture Synthesis For Complex Contacts Resonator Modeling A Robust and Multi-Scale Modal Analysis Example-Based A Sound in Coherence with Visuals Synthesis Perspectives on a Hybrid Model Flexible Resolution Conclusion and Discussion Adapted to Procedural Generation C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 15
    • t Overview Sound and Virtuality Physics-Based Synthesis 1 Sound and Virtuality Contact Modeling Audio Texture Synthesis For Complex Contacts 2 Physics-Based Sound Synthesis Resonator Modeling Contact Modeling A Robust and Multi-Scale Modal Analysis Resonator Modeling Example-Based Synthesis 3 Example-Based Synthesis Perspectives on Flexible Sound Synthesis a Hybrid Model Conclusion and 4 Perspectives on a Hybrid Model Discussion Motivation and Application 5 Conclusion and Discussion Contributions Extensions and Applications C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 16
    • t Vibration Models Sound and Virtuality Modal Analysis Physics-Based Synthesis Contact Modeling Audio Texture Synthesis For Complex Contacts Generating Sounds Based on Physics Simulation Resonator Modeling A Robust and Multi-Scale Modal In computer musics Analysis [Iovino et al. 97, Cook 02] Example-Based Synthesis In computer graphics Perspectives on [Van Den Doel 01, O Brien et al. 02] a Hybrid Model Conclusion and Discussion Improvements for Interactive Sound Rendering Modal parameter tracking [Maxwell et al. 07] Frequency content sparsity [Bonneel et al. 08] C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 17
    • t Vibration Models Sound and Virtuality Modal Analysis Physics-Based Synthesis Contact Modeling Audio Texture 1 Get a Sounding Object and its Geometry Synthesis For Complex Contacts Resonator Modeling A Robust and Multi-Scale Modal Analysis Example-Based Synthesis 2 Construct the FEM (ex: Tetrahedral Mesh) Perspectives on a Hybrid Model 3 Apply Newton Second Law to DOF Conclusion and Discussion ¨ ˙ M d + C d + Kd = f (1) 4 Eigendecomposition ⇒ Modal Parameters M = LL−T ; L−1 KL−T = V ΛV T (2) where V = matrix of eigenvectors Λ = diagonal matrix of eigenvalues C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 18
    • t Vibration Models Sound and Virtuality Modal Analysis Physics-Based Synthesis Contact Modeling Audio Texture Synthesis For Complex Contacts In Real-time: Resonator Modeling A Robust and Modal synthesis Multi-Scale Modal Analysis Example-Based Synthesis Perspectives on a Hybrid Model Conclusion and Discussion 1 s(t) = ai sin(wi t)e −di t (3) n Control for vibration models C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 19
    • t Vibration Models Sound and Virtuality Modal Analysis Physics-Based Synthesis Contact Modeling Audio Texture What Are Synthesis For Complex Contacts The Current Limitations? Resonator Modeling A Robust and Multi-Scale Modal Meshing is difficult Analysis Example-Based No real control on the FEM resolution Synthesis No clear interface between physics and audio Perspectives on a Hybrid Model Conclusion and Discussion C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 20
    • t Vibration Models Sound and Virtuality Modal Analysis Physics-Based Synthesis Contact Modeling Audio Texture What Are Synthesis For Complex Contacts The Current Limitations? Resonator Modeling A Robust and Multi-Scale Modal Meshing is difficult Analysis Example-Based No real control on the FEM resolution Synthesis No clear interface between physics and audio Perspectives on a Hybrid Model Conclusion and Discussion HOW Can We Solve Them? By proposing A robust and multi-scale modal analysis which is Coherent with the physics simulation [Picard et al., DAFx 09] C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 20
    • t Our Deformation Model Sound and Virtuality Physics-Based Synthesis Inspired from Work by Nesme et al. Contact Modeling Audio Texture [Nesme et al. 06] Synthesis For Complex Contacts Resonator Modeling A Robust and Technique Multi-Scale Modal Analysis Merged voxels used as Hexahedral Finite Elements Example-Based Synthesis Perspectives on a Hybrid Model Conclusion and Discussion Implementation with the Sofa Framework Validation of the Model Tests on a metal cube C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 21
    • t Robustness Sound and Virtuality Physics-Based Synthesis Contact Modeling Audio Texture Robust Even for Non-Manifold Geometries Synthesis For Complex Contacts Resonator Modeling A Robust and Multi-Scale Modal Analysis Example-Based Synthesis Perspectives on a Hybrid Model Conclusion and Discussion Material: Aluminium C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 22
    • t Multi-Scale for Efficient Memory Usage Sound and Virtuality Physics-Based A Squirrel in Pine Wood Synthesis Contact Modeling Audio Texture Synthesis For Complex Contacts Resonator Modeling A Robust and Multi-Scale Modal Analysis Example-Based Synthesis Perspectives on a Hybrid Model Conclusion and Discussion C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 23
    • t Multi-Scale for Efficient Memory Usage Sound and Virtuality Physics-Based A Squirrel in Pine Wood: Different FE resolutions Synthesis Contact Modeling Audio Texture 3x3x3 4x4x4 8x8x8 9x9x9 Synthesis For Complex Contacts Resonator Modeling A Robust and Multi-Scale Modal Analysis Example-Based Synthesis Perspectives on a Hybrid Model Frequency Content = f (Hexahedral FE Resolution) Conclusion and Discussion Higher resolution models Frequency centroid shift Convergence of the frequency content C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 24
    • t Comparison with Classical Approach Sound and Virtuality Physics-Based Sounding Bowl - Material: Aluminium Synthesis Contact Modeling Audio Texture Classical Approach Our Approach Synthesis For Complex Contacts (816 modes) (75 modes) Resonator Modeling A Robust and Multi-Scale Modal Analysis Example-Based Synthesis Perspectives on a Hybrid Model Conclusion and Discussion C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 25
    • t A Robust and Multi-Scale Modal Analysis Sound and Virtuality A Solution for Sound Synthesis Physics-Based Synthesis Contact Modeling Audio Texture Synthesis For Complex Contacts Resonator Modeling A Robust and Multi-Scale Modal Analysis Example-Based Realistic Synthesis Perspectives on a Hybrid Model Adapted to Non-Manifold Geometries Conclusion and Discussion Resources Flexibility C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 26
    • t Overview Sound and Virtuality Physics-Based Synthesis 1 Sound and Virtuality Example-Based Synthesis 2 Physics-Based Sound Synthesis Flexible Sound Synthesis Contact Modeling Retargetting Example Sounds Resonator Modeling Perspectives on a Hybrid Model 3 Example-Based Synthesis Conclusion and Discussion Flexible Sound Synthesis 4 Perspectives on a Hybrid Model Motivation and Application 5 Conclusion and Discussion Contributions Extensions and Applications C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 27
    • t Implementation of Signal-Based Models Sound and Virtuality Physics-Based Synthesis Example-Based Synthesis Flexible Sound Concatenative Synthesis Synthesis Retargetting Example [Roads 91, Schwarz 06] Sounds Perspectives on Sound Textures Based on Physics a Hybrid Model [Cook 99] Conclusion and Discussion [Dobashi et al. 03, Zheng et al. 09] Dobashi et al. 03 Authoring and Interactive Control [Cook 02] Cook 99 C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 28
    • t Implementation of Signal-Based Models Sound and Virtuality Physics-Based Synthesis Example-Based Synthesis What Are Flexible Sound Synthesis The Current Limitations? Retargetting Example Sounds Processing is not generic Perspectives on a Hybrid Model Parametrizing is difficult Conclusion and Discussion C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 29
    • t Implementation of Signal-Based Models Sound and Virtuality Physics-Based Synthesis Example-Based Synthesis What Are Flexible Sound Synthesis The Current Limitations? Retargetting Example Sounds Processing is not generic Perspectives on a Hybrid Model Parametrizing is difficult Conclusion and Discussion HOW Can We Solve Them? By Retargetting example sounds To physics-driven animation [Picard et al., AES 09] C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 29
    • t Our Approach Sound and Virtuality Physics-Based Synthesis SINUSOIDAL AUDIO RECORDING Example-Based TRANSIENT Synthesis OBJECT GEOMETRY VIRTUAL ENVIRONMENT Flexible Sound 1 DICTIONARY OF AUDIO GRAINS Synthesis Impulsive / Continuous Retargetting Example BUILD COLLISION Sounds STRUCTURES Perspectives on 2 CORRELATION PATTERNS a Hybrid Model DEFINE PROCEDURES PREPROCESSING INTERACTIVE Conclusion and RETARGETTING RIGID-BODY Discussion TO ANIMATION SIMULATION AUDIO RENDERER VIDEO RENDERER ANIMATION WITH AUDIO Amplitude Time Our Contributions C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 30
    • t Preprocess: A Generic Analysis Sound and Virtuality Physics-Based Synthesis Example-Based Synthesis Flexible Sound Synthesis Impulsive and Continuous Contacts Retargetting Example Sounds Spectral Modeling Synthesis (SMS) [Serra 97] Perspectives on a Hybrid Model Conclusion and Automatic Extraction of Audio Grains Discussion Dictionary: Impulsive/Continuous Generation of Correlation Patterns between original recordings and audio grains C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 31
    • t On-Line: Flexible Sound Synthesis Sound and Virtuality Physics-Based Synthesis Example-Based Synthesis Resynthesis of the Original Recordings Flexible Sound Synthesis Candidate grains: max. correlation amplitude Retargetting Example Sounds Perspectives on a Hybrid Model Conclusion and Discussion C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 32
    • t On-Line: Flexible Sound Synthesis Sound and Virtuality Physics-Based Synthesis Example-Based Synthesis Resynthesis of the Original Recordings Flexible Sound Synthesis Candidate grains: max. correlation amplitude Retargetting Example Sounds Perspectives on Interactive Physics-Driven Animations a Hybrid Model Physics Info for Retargetting Conclusion and Discussion Contact type: impulsive or continuous? Penetration force and relative velocity C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 32
    • t On-Line: Flexible Sound Synthesis Sound and Virtuality Physics-Based Synthesis Example-Based Synthesis Resynthesis of the Original Recordings Flexible Sound Synthesis Candidate grains: max. correlation amplitude Retargetting Example Sounds Perspectives on Interactive Physics-Driven Animations a Hybrid Model Physics Info for Retargetting Conclusion and Discussion Contact type: impulsive or continuous? Penetration force and relative velocity Flexible Audio Shading Approach Additional, User-defined Resynthesis Schemes Spectral domain adaptation/modification C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 32
    • t Resynthesis of the Original Recordings Sound and Virtuality Physics-Based Synthesis Example-Based Synthesis Flexible Sound 94 recordings (14.6Mb) Synthesis ≈ 5000 grains + 94 Correlation Patterns (20% Gain) Retargetting Example Sounds Perspectives on Breaking Glass a Hybrid Model Conclusion and Shooting Gun Discussion Rolling Additional Material: http://www-sop.inria.fr/members/Cecile.Picard/ "‘Supplemental AES"’ C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 33
    • t Flexible Audio Shading Approach Sound and Virtuality Physics-Based Synthesis Example-Based Synthesis Easy Implementation of Time-Scaling Flexible Sound Synthesis Retargetting Example Faster Rolling Sounds Perspectives on Slower Breaking a Hybrid Model Conclusion and Discussion Synthesis of An Infinity Similar Audio Events by varying the audio content Rythmic pattern from Breaking Stone New material content: stone and gun Rythmic pattern from Breaking Glass New material content: ceramic C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 34
    • t Interactive Physics-Driven Animations Sound and Virtuality Physics-Based Synthesis Example-Based Synthesis Flexible Sound Simulations Driven by Sofa Framework Synthesis Retargetting Example Sounds Perspectives on a Hybrid Model Conclusion and Discussion C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 35
    • t Retargetting Example Sounds Sound and Virtuality A Solution for Interactive Simulations Physics-Based Synthesis Example-Based Synthesis Flexible Sound Synthesis Retargetting Example Variety Sounds Perspectives on a Hybrid Model Adapted to Scenarios Conclusion and Discussion Small Memory Footprint Real-Time Rendering An attractive solution for industrial applications (Eden Games, an ATARI game studio) C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 36
    • t Overview Sound and Virtuality Physics-Based Synthesis 1 Sound and Virtuality Example-Based Synthesis 2 Physics-Based Sound Synthesis Perspectives on a Hybrid Model Contact Modeling Motivation Resonator Modeling A Hybrid Model for Fracture Events Conclusion and 3 Example-Based Synthesis Discussion Flexible Sound Synthesis 4 Perspectives on a Hybrid Model Motivation and Application 5 Conclusion and Discussion Contributions Extensions and Applications C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 37
    • t Sound Modeling Sound and Virtuality When Nonlinearity Occurs Physics-Based Synthesis Example-Based Synthesis Perspectives on a Hybrid Model Motivation Problems of Single Models A Hybrid Model for Fracture Events Vibration models assume linearity Conclusion and Discussion Example-based sounds are hard to parametrize C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 38
    • t Sound Modeling Sound and Virtuality When Nonlinearity Occurs Physics-Based Synthesis Example-Based Synthesis Perspectives on a Hybrid Model Motivation Problems of Single Models A Hybrid Model for Fracture Events Vibration models assume linearity Conclusion and Discussion Example-based sounds are hard to parametrize Previous Work Modeling nonlinearities [O Brien et al. 01, Chadwick et al. 09] [Cook 02] C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 38
    • t Fracture Events Sound and Virtuality Physics-Based Synthesis Example-Based Background Synthesis Frequently occur in virtual environments Perspectives on a Hybrid Model Motivation Visual rendering A Hybrid Model for Fracture Events [O Brien et al. 99, 02] Conclusion and [Parker and O Brien. 09] Discussion Sound rendering: Little research [Warren et al. 84] [Rath et al. 03] C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 39
    • t Fracture Events Sound and Virtuality Physics-Based Synthesis Example-Based Background Synthesis Frequently occur in virtual environments Perspectives on a Hybrid Model Motivation Visual rendering A Hybrid Model for Fracture Events [O Brien et al. 99, 02] Conclusion and [Parker and O Brien. 09] Discussion Sound rendering: Little research [Warren et al. 84] [Rath et al. 03] Challenges Event depends on the material involved Differents phases emerge from fracture event C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 39
    • t Parametrization of Our Hybrid Model Sound and Virtuality Physics-Based Synthesis Example-Based Synthesis Selection Criteria Perspectives on Hybrid model applied when nonlinearity occurs a Hybrid Model Motivation A Hybrid Model for Fracture Events Techniques Conclusion and Discussion FM synthesis Audio grains FM synthesis Parametrization Smooth transition with vibration model Coherence inside the hybrid model C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 40
    • t Discussion Sound and Virtuality Physics-Based Synthesis Example-Based Synthesis Perspectives on a Hybrid Model Motivation Prospective model A Hybrid Model for Fracture Events Conclusion and Possible problem: report from the physics engine Discussion Simplicity of the tools allows real-time rendering C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 41
    • t Overview Sound and Virtuality Physics-Based Synthesis 1 Sound and Virtuality Example-Based Synthesis 2 Physics-Based Sound Synthesis Perspectives on a Hybrid Model Contact Modeling Conclusion and Resonator Modeling Discussion Contributions 3 Example-Based Synthesis Extensions and Applications Flexible Sound Synthesis 4 Perspectives on a Hybrid Model Motivation and Application 5 Conclusion and Discussion Contributions Extensions and Applications C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 42
    • t Synthesis of Sounds for Animation Sound and Virtuality Difficulties Physics-Based Synthesis Example-Based Synthesis Perspectives on a Hybrid Model Conclusion and Audio-Visual Coherence Discussion Contributions Extensions and Applications Extremely Dynamic Character Precision of Synthesis Large Variety of Objects C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 43
    • t Contributions Sound and Virtuality An Overview Physics-Based Synthesis Example-Based Complex Contact Modeling Synthesis Perspectives on 2D visual textures used as roughness maps a Hybrid Model Audible and position-dependent variations Conclusion and Detail-layer mechanisms Discussion Contributions Extensions and Applications C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 44
    • t Contributions Sound and Virtuality An Overview Physics-Based Synthesis Example-Based Complex Contact Modeling Synthesis Perspectives on 2D visual textures used as roughness maps a Hybrid Model Audible and position-dependent variations Conclusion and Detail-layer mechanisms Discussion Contributions Extensions and Applications Improved Modal Analysis for Resonator Modeling Complex non-manifold geometries can be handled Multi-scale resolution Coherence between simulation and audio C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 44
    • t Contributions Sound and Virtuality An Overview Physics-Based Synthesis Example-Based Complex Contact Modeling Synthesis Perspectives on 2D visual textures used as roughness maps a Hybrid Model Audible and position-dependent variations Conclusion and Detail-layer mechanisms Discussion Contributions Extensions and Applications Improved Modal Analysis for Resonator Modeling Complex non-manifold geometries can be handled Multi-scale resolution Coherence between simulation and audio Flexibility of Sound Design Audio grains and correlation patterns Dynamic retargetting to events Extended sound synthesis capabilities C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 44
    • t Contributions Sound and Virtuality Perspectives Physics-Based Synthesis Example-Based Synthesis Perspectives on a Hybrid Model Conclusion and Discussion A Prospective Hybrid Model Contributions for Complex Physical Phenomena Extensions and Applications Focus on Nonlinearity Combination of physically based and example-based methods Application Case: Fracture Events C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 45
    • t Overview Sound and Virtuality Physics-Based Synthesis 1 Sound and Virtuality Example-Based Synthesis 2 Physics-Based Sound Synthesis Perspectives on a Hybrid Model Contact Modeling Conclusion and Resonator Modeling Discussion Contributions 3 Example-Based Synthesis Extensions and Applications Flexible Sound Synthesis 4 Perspectives on a Hybrid Model Motivation and Application 5 Conclusion and Discussion Contributions Extensions and Applications C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 46
    • t Promising Directions for Future Work Sound and Virtuality Physics-Based Synthesis Complex Contact Modeling Example-Based Synthesis Two interacting textures Perspectives on Surface-based interactions a Hybrid Model Adequate perceptual experiments Conclusion and Discussion Contributions Extensions and Applications C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 47
    • t Promising Directions for Future Work Sound and Virtuality Physics-Based Synthesis Complex Contact Modeling Example-Based Synthesis Two interacting textures Perspectives on Surface-based interactions a Hybrid Model Adequate perceptual experiments Conclusion and Discussion Contributions Improved Modal Analysis for Resonator Modeling Extensions and Applications Recent work from [Nesme et al. Siggraph 09] Investigations with GPU for in-line computation Complete integration in a virtual scene C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 47
    • t Promising Directions for Future Work Sound and Virtuality Physics-Based Synthesis Complex Contact Modeling Example-Based Synthesis Two interacting textures Perspectives on Surface-based interactions a Hybrid Model Adequate perceptual experiments Conclusion and Discussion Contributions Improved Modal Analysis for Resonator Modeling Extensions and Applications Recent work from [Nesme et al. Siggraph 09] Investigations with GPU for in-line computation Complete integration in a virtual scene Example-Based Technique Clustering of similar grains Statistical analysis of correlation patterns Physics engine design C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 47
    • t Promising Directions for Future Work Sound and Virtuality Physics-Based Synthesis Example-Based Synthesis Perspectives on a Hybrid Model Conclusion and Discussion Hybrid Model for Fracture Events Contributions Extensions and Fracture sound simulation framework Applications Tracking of relevant physical data C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 48
    • t Conclusion Sound and Virtuality Physics-Based Synthesis Example-Based Synthesis Perspectives on a Hybrid Model New Physically Based Algorithms Conclusion and Discussion for Sound Rendering Contributions Extensions and Flexibility of Sound Modeling Applications Ideas on an Adequate Hybrid Sound Model Additional info: http://www-sop.inria.fr/members/Cecile.Picard/ C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 49
    • t Acknowledgements Sound and Virtuality Physics-Based Synthesis Example-Based Synthesis Perspectives on a Hybrid Model George Drettakis, François Faure, Conclusion and and Nicolas Tsingos Discussion Contributions REVES Team Extensions and Applications Marie-Paule Cani and the Evasion Team Paul G. Kry at the McGill University, Montréal Eden Games, an ATARI game studio, Lyon C. Picard-Limpens December 4, 2009 Expressive Sound Synthesis For Animation 50