Interactive Physically-Based Sound Design of 3D Model using Material Optimization (SCA'16)

1. Interactive Physically-Based Sound Design of 3D Model using Material Optimization Kazuhiro Yamamoto Takeo Igarashi The University of Tokyo

2. “How can we design the physically-based sound with an intuitive way?” Motivation

3. Physically-Based Sound Rendering We focus on modal sound synthesis. It produces perfectly synchronized sounds to various visual events via physical simulation. Toward High-Quality Modal Contact Sound, Chanxi et al., SIGGRAPH 2011 Harmonic Shells: A Practical Nonlinear Sound Model for Near-Rigid Thin Shells, Jeffrey et al., SIGGRAPH 2009 Toward High-Quality Modal Contact Sound Changxi Zheng Doug L. James Cornell University Figure 1: A Rube-Goldberg contraption that demonstrates many challenging multibody contact sounds. A noisy block feeder (Left) with flexible tubes ejects marbles into a double helix of plastic chutes (Middle), which causes a cup to fill up, lifting a lever that drops a bunny into a runaway shopping cart (Right) producing familiar clattering and clanging sounds due to deformable micro-collisions. Our approach can accurately resolve modal vibrations and contact sounds using an asynchronous, adaptive, frictional contact solver. Abstract Contact sound models based on linear modal analysis are com- monly used with rigid body dynamics. Unfortunately, treating vi- brating objects as “rigid” during collision and contact processing fundamentally limits the range of sounds that can be computed, and contact solvers for rigid body animation can be ill-suited for modal contact sound synthesis, producing various sound artifacts. In this paper, we resolve modal vibrations in both collision and frictional contact processing stages, thereby enabling non-rigid sound phenomena such as micro-collisions, vibrational energy exchange, and chattering. We propose a frictional multibody contact formulation and modified Staggered Projections solver which is well-suited to sound rendering and avoids noise artifacts associated with spatial and temporal contact-force fluctuations which plague prior methods. To enable practical animation and sound synthesis of numer- ous bodies with many coupled modes, we propose a novel asynchronous integrator with model-level adaptivity built into the frictional contact solver. Vibrational contact damping is modeled to approximate contact-dependent sound dissipation. Results are pro- vided that demonstrate high-quality contact resolution with sound. CR Categories: I.3.5 [Computer Graphics]: Computational Geometry and Object Modeling—Physically based modeling; I.6.8 [Simulation and Mod- eling]: Types of Simulation—Animation; H.5.5 [Information Systems]: In- formation Interfaces and Presentation—Sound and Music Computing Keywords: Sound synthesis; contact sounds; modal analysis; asynchronous integration; frictional contact 1 Introduction Sound models based on linear modal vibrations are widely used to efficiently synthesize plausible contact sounds for so-called rigid bodies in computer animation and interactive virtual environments. Unfortunately, there still remain a number of significant contact- related deficiencies that limit the realism of modal contact sounds in practice. To begin with, for speed and simplicity, modal sound models are usually just excited by using contact force impulses from rigid body contact solvers. In reality, there is no such thing as a “rigid” object, and the same small vibrations that produce sound also play an important role in producing rich contact events: micro- collisions, chattering, squeaking, coupled vibrations, contact damping, etc. Ignoring contact-level vibrations is the source of many sound-related deficiencies, as these small vibrations can be visually inconsequential but aurally significant. For example, pounding on a seemingly “rigid” dinner table can shake dishes—and may also upset your friends (see Figure 2). Frictional contact and deformation coupling is also important for sound; for example, slip-stick phenomena is responsible for many familiar squeaking and scraping sounds, e.g., fingernails scraping on a chalkboard. Resolving these vibrational contact effects is challenging due to the need to resolve deformable collisions and contact at high temporal rates. Even in seemingly rigid scenarios, such as an object resting on a plane, current contact solver implementations can generate tem- porally incoherent contact impulses which lead to sound artifacts, such as resting objects that strangely humm or buzz when integrated at near-audio rates. These artifacts are a consequence of the fun- damental non-uniqueness of rigid body contact forces (e.g., static indeterminacy) which can lead to point-like and nonphysical contact force (traction) distributions. Additionally, rigid-body contact impulses can exhibit nonphysical temporal fluctuations, which lead to noise-related sound artifacts (especially with iterative contact solution techniques) that must be dissipated artificially. Moreover, the sound of a resting object should also depend on its contact state, and how contacts oppose surface vibrations. For example, a coffee mug exhibits distinctive vibrational damping when placed in different orientations on surfaces (see Figure 4). This con- Harmonic Shells: A Practical Nonlinear Sound Model for Near-Rigid Thin Shells Jeffrey N. Chadwick Steven S. An Cornell University Doug L. James Figure 1: Crash! Our physically based sound renderings of thin shells produce characteristic “crashing” and “rumbling” sounds when animated using rigid body dynamics. We synthesize nonlinear modal vibrations using an efficient reduced-order dynamics model that captures important nonlinear mode coupling. High-resolution sound field approximations are generated using far-field acoustic transfer (FFAT) maps, which are precomputed using efficient fast Helmholtz multipole methods, and provide cheap evaluation of detailed low- to high-frequency acoustic transfer functions for realistic sound rendering. Abstract We propose a procedural method for synthesizing realistic sounds due to nonlinear thin-shell vibrations. We use linear modal analysis to generate a small-deformation displacement basis, then couple the modes together using nonlinear thin-shell forces. To enable audio- rate time-stepping of mode amplitudes with mesh-independent cost, we propose a reduced-order dynamics model based on a thin-shell cubature scheme. Limitations such as mode locking and pitch glide are addressed. To support fast evaluation of mid-frequency mode- based sound radiation for detailed meshes, we propose far-field acoustic transfer maps (FFAT maps) which can be precomputed using state-of-the-art fast Helmholtz multipole methods. Familiar examples are presented including rumbling trash cans and plastic bottles, crashing cymbals, and noisy sheet metal objects, each with increased richness over linear modal sound models. CR Categories: I.3.5 [Computer Graphics]: Computa- tional Geometry and Object Modeling—Physically based modeling; I.6.8 [Simulation and Modeling]: Types of Simulation— Animation; H.5.5 [Information Systems]: Information Interfaces and Presentation—Sound and Music Computing can provide convincing physically based sound sources, especially for pure ringing tones such as chimes, bells, or “knocks.” Unfor- tunately, we lack effective sound models for a broad class of noisy virtual objects: thin shells (objects with thicknesses orders of mag- nitude smaller than their other dimensions). Thin shells are very common in real and virtual environments, and produce rich and easily recognizable impact sounds: sheet metal objects (trash cans, oil drums, tin roofs, machinery), plastic containers (water bottles), musical instruments (cymbals), etc. Their rich nonlinear vibrations produce proverbial “crashes” and “rumbles” that are poorly approximated by linear modal sound models which lack nonlinear mode coupling. To make matters worse, thin shells are often very loud and important sound sources due to their ability to vibrate and radi- ate sound so effectively, e.g., consider a metal roof pelted by hail. Alas, their expensive nonlinear dynamics have made thin shells computationally impractical for physically based sound synthesis. In this paper, we propose an efficient method for synthesizing realistic sounds from thin-shell structures undergoing small but nonlinear vibrations. Given a description of an object’s geometry and material properties, we compute linear vibration modes, then cou- Rigid-Body Fracture Sound with Precomputed Soundbanks Changxi Zheng Doug L. James Cornell University Figure 1: SMASH! We synthesize the violent fracture and impact sounds of a glass table setting smashed into over 300 pieces (see sound Rigid-Body Fracture Sound with Precomputed Soundbanks, Changxi et al., SIGGRAPH Asia 2010

4. Modal Sound Synthesis for quasi-rigid body sound (e.g., collision, bounce, and scratch) An Object Vibration Analysis (Modal Analysis) Sound Modal Parameters naliz- neral- itive- e the ere. n (1) (2) et the sym- e the may (3) Figure 4: The two rows show a side and top view of a bowl along with three of the bowl’s first vibrational modes. The modes selected for the illustration are the first three non-rigid modes with distinct eigenvalues that are excited by a transverse impulse to the bowl’s rim. the body’s six rigid-body modes. The rigid-body eigenvalues are zero because a rigid-body displacement will ditions, diagonaliz- o solving a general- mmetric, positive- Plesha describe the e end result here. mping equation (1) ¨d) = f , (2) oefficients. Let the e generalized sym- Figure 4: The two rows show a side and top view of a bowl along with three of the bowl’s first vibrational modes. The modes selected for the illustration are the applications. With these conditions, diagonaliz- tion (1) becomes equivalent to solving a general- mmetric eigenproblem with symmetric, positive- matrices. Cook, Malkus, and Plesha describe the n detail and we only repeat the end result here. he restriction of Rayleigh damping equation (1) ewritten as: K(d + ↵1 ˙d) + M(↵2 ˙d + ¨d) = f , (2) 1 and ↵2 are the Rayleigh coefficients. Let the of W be the solution to the generalized sym- igenproblem Kx + Mx = 0 and ⇤ be the matrix of eigenvalues1 , then equation (2) may ormed to: ⇤(z + ↵1 ˙z) + (↵2 ˙z + ¨z) = g , (3) = W 1 d is the vector of modal coordinates W T f is the external force vector in the modal te system. ow of equation (3) corresponds to a single scalar rder differential equation: izi + (↵1 i + ↵2) ˙zi + ¨zi = gi . (4) ytical solutions to each equation are zi = c1et!+ i + c2et!i (5) and c2 are arbitrary (complex) constants, and !i mplex frequency given by Figure 4: The two rows show a side and top view of a bowl along with three of the bowl’s first vibrational modes. The modes selected for the illustration are the first three non-rigid modes with distinct eigenvalues that are excited by a transverse impulse to the bowl’s rim. the body’s six rigid-body modes. The rigid-body eigenvalues are zero because a rigid-body displacement will not generate any elastic forces. The decoupled system of equations is not an approxi- mation of the original linear system, it will generate ex- actly the same results as the original linear system. Of course the linear system may have been an approxima- tion to some initial nonlinear one, but any problem that could be solved using equation (2) could also be solved with equation (3). Furthermore, simulation that would have required numerical time integration of equation (1) can now be solved without integration using the analyti- cal solutions in equations (5) or (7). h these conditions, diagonaliz- equivalent to solving a general- lem with symmetric, positive- Malkus, and Plesha describe the nly repeat the end result here. Rayleigh damping equation (1) M(↵2 ˙d + ¨d) = f , (2) Rayleigh coefficients. Let the lution to the generalized sym- + Mx = 0 and ⇤ be the alues1 , then equation (2) may + (↵2 ˙z + ¨z) = g , (3) e vector of modal coordinates ernal force vector in the modal Figure 4: The two rows show a side and top view of a bowl along with three of the bowl’s first vibrational modes. The modes selected for the illustration are the first three non-rigid modes with distinct eigenvalues that are excited by a transverse impulse to the bowl’s rim. the body’s six rigid-body modes. The rigid-body eigenvalues are zero because a rigid-body displacement will not generate any elastic forces. The decoupled system of equations is not an approxi- + modal frequency: f0 modal amplitude: a0 modal damping : d0 modal frequency: f1 modal amplitude: a1 modal damping : d1 modal frequency: f2 modal amplitude: a2 modal damping : d2

5. Modal Sound Synthesis for quasi-rigid body sound (e.g., collision, bounce, and scratch) An Object Vibration Analysis (Modal Analysis) Sound Modal Parameters naliz- neral- itive- e the ere. n (1) (2) et the sym- e the may (3) Figure 4: The two rows show a side and top view of a bowl along with three of the bowl’s first vibrational modes. The modes selected for the illustration are the first three non-rigid modes with distinct eigenvalues that are excited by a transverse impulse to the bowl’s rim. the body’s six rigid-body modes. The rigid-body eigenvalues are zero because a rigid-body displacement will ditions, diagonaliz- o solving a general- mmetric, positive- Plesha describe the e end result here. mping equation (1) ¨d) = f , (2) oefficients. Let the e generalized sym- Figure 4: The two rows show a side and top view of a bowl along with three of the bowl’s first vibrational modes. The modes selected for the illustration are the applications. With these conditions, diagonaliz- tion (1) becomes equivalent to solving a general- mmetric eigenproblem with symmetric, positive- matrices. Cook, Malkus, and Plesha describe the n detail and we only repeat the end result here. he restriction of Rayleigh damping equation (1) ewritten as: K(d + ↵1 ˙d) + M(↵2 ˙d + ¨d) = f , (2) 1 and ↵2 are the Rayleigh coefficients. Let the of W be the solution to the generalized sym- igenproblem Kx + Mx = 0 and ⇤ be the matrix of eigenvalues1 , then equation (2) may ormed to: ⇤(z + ↵1 ˙z) + (↵2 ˙z + ¨z) = g , (3) = W 1 d is the vector of modal coordinates W T f is the external force vector in the modal te system. ow of equation (3) corresponds to a single scalar rder differential equation: izi + (↵1 i + ↵2) ˙zi + ¨zi = gi . (4) ytical solutions to each equation are zi = c1et!+ i + c2et!i (5) and c2 are arbitrary (complex) constants, and !i mplex frequency given by Figure 4: The two rows show a side and top view of a bowl along with three of the bowl’s first vibrational modes. The modes selected for the illustration are the first three non-rigid modes with distinct eigenvalues that are excited by a transverse impulse to the bowl’s rim. the body’s six rigid-body modes. The rigid-body eigenvalues are zero because a rigid-body displacement will not generate any elastic forces. The decoupled system of equations is not an approxi- mation of the original linear system, it will generate ex- actly the same results as the original linear system. Of course the linear system may have been an approxima- tion to some initial nonlinear one, but any problem that could be solved using equation (2) could also be solved with equation (3). Furthermore, simulation that would have required numerical time integration of equation (1) can now be solved without integration using the analyti- cal solutions in equations (5) or (7). h these conditions, diagonaliz- equivalent to solving a general- lem with symmetric, positive- Malkus, and Plesha describe the nly repeat the end result here. Rayleigh damping equation (1) M(↵2 ˙d + ¨d) = f , (2) Rayleigh coefficients. Let the lution to the generalized sym- + Mx = 0 and ⇤ be the alues1 , then equation (2) may + (↵2 ˙z + ¨z) = g , (3) e vector of modal coordinates ernal force vector in the modal Figure 4: The two rows show a side and top view of a bowl along with three of the bowl’s first vibrational modes. The modes selected for the illustration are the first three non-rigid modes with distinct eigenvalues that are excited by a transverse impulse to the bowl’s rim. the body’s six rigid-body modes. The rigid-body eigenvalues are zero because a rigid-body displacement will not generate any elastic forces. The decoupled system of equations is not an approxi- + modal frequency: f0 modal amplitude: a0 modal damping : d0 modal frequency: f1 modal amplitude: a1 modal damping : d1 modal frequency: f2 modal amplitude: a2 modal damping : d2

6. Difﬁculty of Physically-Based Sound Design

7. Difﬁculty of Physically-Based Sound Design ？ Material Parameters

8. Difﬁculty of Physically-Based Sound Design ？ Material Parameters Physical Simulation ？

9. Our Approach Example-based interactive frameworks using material optimization. Input: 3D Model and Target Sound

10. Our Approach Example-based interactive frameworks using material optimization. Input: 3D Model and Target Sound Sound 4 Position 4 Sound 2 Position 2 Sound 3 Position 3 Sound 1 Position 1

11. Our Approach Example-based interactive frameworks using material optimization. Input: 3D Model and Target Sound Sound 4 Position 4 Sound 2 Position 2 Sound 3 Position 3 Sound 1 Position 1

12. Output: Material Distribution Our Approach Example-based interactive frameworks using material optimization. Input: 3D Model and Target Sound Sound 4 Position 4 Sound 2 Position 2 Sound 3 Position 3 Sound 1 Position 1

13. What is the “Material” ? We only optimize Young’s modulus. • Density • Young’s modulus • Poisson’s rate.

14. What is the “Material” ? We only optimize Young’s modulus. • Density • Young’s modulus • Poisson’s rate.

15. Related Work Foleyautomatic: physically-based sound effects for interactive simulation and animation, Pai et al., 2001 Example-guided physically based modal sound synthesis, Ren et al., 2013 TimbreFields: 3D Interactive Sound Models for Real-Time Audio., Corbett et al., 2007 Impulse responses were recorded digitally as 16 bit audio at a sampling rate of 44.1 KHz. The microphone was mounted on a programmable robotic arm, which automatically moved the microphone to a variety of lis- tening locations for a given contact location. The solenoid was synchronized with the robotic arm to expedite the recording process. All the microphone and excita- tion control programming was done in advance. The strategy was to gather all the impulse responses within a robotic environment so as to minimize errors intro- duced through human control. Before each impulse response recording, 0.5 s of am- bient sound was recorded to help identify spurious modes in the background noise. This precaution was 3.1 Estimation of the Modes The estimation of the modal model parameters from the measurements was achieved in the following three phases. First, we estimated the modal model parameters for each sound sample separately. This is done by computing the windowed discrete Fourier transform (Gabor transform) of the signal—extracting the frequencies, dampings, and amplitudes by fitting each frequency bin with a sum of a small number (e.g., four) of damped complex exponentials. The parameter fitting method is capable of very accurate frequency reconstructions and is able to resolve very close modes. Close modes are very common in artificial objects that have approximate sym- metries resulting in mode degeneracy. Manufacturing impurities break this symmetry, splitting the frequencies by a small amount. These nearby frequencies are dis- tinctly audible as beating, or “shimmering” sounds and significantly enhance the realism of the synthesized sound. Second, we determined which modes are actually audible, using a rough model of auditory masking (Doel, Pai, Adam, Kortchmar, & Pichora-Fuller, 2002; Doel, Knott, & Pai, 2004), and retained only the audible modes. This step is necessary because the parameter fitting algorithm produces many spurious modes. Third, we merged all the modal models, using a sim- ple model of human frequency discrimination, which results in a single frequency and damping model for the entire object, and a discrete sampling of the timbre field a on the 5D interaction space. In theory, all the models should share the same set of frequencies and dampings, but due to noise they will not be precisely the same, motivating this third step. We now describe the details of the parameter fitting. Figure 3. An automated measuring system was used to acquire the data. A robot arm moves the microphone to a preprogrammed set of locations. An impulse force was then automatically applied to the object with the solenoid and sounds were recorded for subsequent analysis. 648 PRESENCE: VOLUME 16, NUMBER 6 1:2 • Z. Ren et al. (e)(d)(c)(b)(a) Fig. 1. From the recording of a real-world object (a), our framework is able to find the material parameters and generates similar sound for a replicate object (b). The same set of parameters can be transfered to various virtual objects to produce sounds with the same material quality ((c), (d), (e)). (such as stiffness, damping coefficients, and mass density) that can be directly used in modal analysis. As a result, for objects with different geometries and runtime interactions, different sets of modes are generated or excited differ- ently, and different sounds are produced. However, if the material properties are the same, they should all sound like coming from the same material. For example, a plastic plate being hit, a plastic ball being dropped, and a plastic box sliding on the floor generate from many sound clips (measured) from a sound clip Example-based sound design for modal sound synthesis

16. Related Work Foleyautomatic: physically-based sound effects for interactive simulation and animation, Pai et al., 2001 Example-guided physically based modal sound synthesis, Ren et al., 2013 TimbreFields: 3D Interactive Sound Models for Real-Time Audio., Corbett et al., 2007 Impulse responses were recorded digitally as 16 bit audio at a sampling rate of 44.1 KHz. The microphone was mounted on a programmable robotic arm, which automatically moved the microphone to a variety of lis- tening locations for a given contact location. The solenoid was synchronized with the robotic arm to expedite the recording process. All the microphone and excita- tion control programming was done in advance. The strategy was to gather all the impulse responses within a robotic environment so as to minimize errors intro- duced through human control. Before each impulse response recording, 0.5 s of am- bient sound was recorded to help identify spurious modes in the background noise. This precaution was 3.1 Estimation of the Modes The estimation of the modal model parameters from the measurements was achieved in the following three phases. First, we estimated the modal model parameters for each sound sample separately. This is done by computing the windowed discrete Fourier transform (Gabor transform) of the signal—extracting the frequencies, dampings, and amplitudes by fitting each frequency bin with a sum of a small number (e.g., four) of damped complex exponentials. The parameter fitting method is capable of very accurate frequency reconstructions and is able to resolve very close modes. Close modes are very common in artificial objects that have approximate sym- metries resulting in mode degeneracy. Manufacturing impurities break this symmetry, splitting the frequencies by a small amount. These nearby frequencies are dis- tinctly audible as beating, or “shimmering” sounds and significantly enhance the realism of the synthesized sound. Second, we determined which modes are actually audible, using a rough model of auditory masking (Doel, Pai, Adam, Kortchmar, & Pichora-Fuller, 2002; Doel, Knott, & Pai, 2004), and retained only the audible modes. This step is necessary because the parameter fitting algorithm produces many spurious modes. Third, we merged all the modal models, using a sim- ple model of human frequency discrimination, which results in a single frequency and damping model for the entire object, and a discrete sampling of the timbre field a on the 5D interaction space. In theory, all the models should share the same set of frequencies and dampings, but due to noise they will not be precisely the same, motivating this third step. We now describe the details of the parameter fitting. Figure 3. An automated measuring system was used to acquire the data. A robot arm moves the microphone to a preprogrammed set of locations. An impulse force was then automatically applied to the object with the solenoid and sounds were recorded for subsequent analysis. 648 PRESENCE: VOLUME 16, NUMBER 6 1:2 • Z. Ren et al. (e)(d)(c)(b)(a) Fig. 1. From the recording of a real-world object (a), our framework is able to find the material parameters and generates similar sound for a replicate object (b). The same set of parameters can be transfered to various virtual objects to produce sounds with the same material quality ((c), (d), (e)). (such as stiffness, damping coefficients, and mass density) that can be directly used in modal analysis. As a result, for objects with different geometries and runtime interactions, different sets of modes are generated or excited differ- ently, and different sounds are produced. However, if the material properties are the same, they should all sound like coming from the same material. For example, a plastic plate being hit, a plastic ball being dropped, and a plastic box sliding on the floor generate from many sound clips (measured) from a sound clip Example-based sound design for modal sound synthesis It requires expensive measurement procedure.

17. Related Work Foleyautomatic: physically-based sound effects for interactive simulation and animation, Pai et al., 2001 Example-guided physically based modal sound synthesis, Ren et al., 2013 TimbreFields: 3D Interactive Sound Models for Real-Time Audio., Corbett et al., 2007 Impulse responses were recorded digitally as 16 bit audio at a sampling rate of 44.1 KHz. The microphone was mounted on a programmable robotic arm, which automatically moved the microphone to a variety of lis- tening locations for a given contact location. The solenoid was synchronized with the robotic arm to expedite the recording process. All the microphone and excita- tion control programming was done in advance. The strategy was to gather all the impulse responses within a robotic environment so as to minimize errors intro- duced through human control. Before each impulse response recording, 0.5 s of am- bient sound was recorded to help identify spurious modes in the background noise. This precaution was 3.1 Estimation of the Modes The estimation of the modal model parameters from the measurements was achieved in the following three phases. First, we estimated the modal model parameters for each sound sample separately. This is done by computing the windowed discrete Fourier transform (Gabor transform) of the signal—extracting the frequencies, dampings, and amplitudes by fitting each frequency bin with a sum of a small number (e.g., four) of damped complex exponentials. The parameter fitting method is capable of very accurate frequency reconstructions and is able to resolve very close modes. Close modes are very common in artificial objects that have approximate sym- metries resulting in mode degeneracy. Manufacturing impurities break this symmetry, splitting the frequencies by a small amount. These nearby frequencies are dis- tinctly audible as beating, or “shimmering” sounds and significantly enhance the realism of the synthesized sound. Second, we determined which modes are actually audible, using a rough model of auditory masking (Doel, Pai, Adam, Kortchmar, & Pichora-Fuller, 2002; Doel, Knott, & Pai, 2004), and retained only the audible modes. This step is necessary because the parameter fitting algorithm produces many spurious modes. Third, we merged all the modal models, using a sim- ple model of human frequency discrimination, which results in a single frequency and damping model for the entire object, and a discrete sampling of the timbre field a on the 5D interaction space. In theory, all the models should share the same set of frequencies and dampings, but due to noise they will not be precisely the same, motivating this third step. We now describe the details of the parameter fitting. Figure 3. An automated measuring system was used to acquire the data. A robot arm moves the microphone to a preprogrammed set of locations. An impulse force was then automatically applied to the object with the solenoid and sounds were recorded for subsequent analysis. 648 PRESENCE: VOLUME 16, NUMBER 6 1:2 • Z. Ren et al. (e)(d)(c)(b)(a) Fig. 1. From the recording of a real-world object (a), our framework is able to find the material parameters and generates similar sound for a replicate object (b). The same set of parameters can be transfered to various virtual objects to produce sounds with the same material quality ((c), (d), (e)). (such as stiffness, damping coefficients, and mass density) that can be directly used in modal analysis. As a result, for objects with different geometries and runtime interactions, different sets of modes are generated or excited differ- ently, and different sounds are produced. However, if the material properties are the same, they should all sound like coming from the same material. For example, a plastic plate being hit, a plastic ball being dropped, and a plastic box sliding on the floor generate from many sound clips (measured) from a sound clip Example-based sound design for modal sound synthesis It requires expensive measurement procedure. It requires same object in real world

18. Demo

19. Demo

20. Demo

21. Demo

22. Demo

23. Demo

24. Demo

25. Demo

26. Demo

27. Demo The system allows multiple assignments of example sounds.

31. Contributions • An example-based framework for designing physically-based sound • Fast approximate modal analysis for an interactive simulation • Extended data-driven FEM using regression forests • Hierarchical component mode synthesis with error correction • Handling a large range of continuous material settings. • Constant evaluation cost. • High generalization ability. • Parallel • Accurate

32. Modal Analysis Algorithm Overview Data-Driven Online Coarsening (§7.1) archical Component Mode Synthesis (§7.2) Runtime st Approximate Modal Analysis (§7) The evaluation of the objective function and Gradient Computation (§6) + Sample Position 1 User Input (Target Sound) Simulated Sound Material Optimization (§6) New Material Distribution Similarity Score Computation (§5) or Material Reduction (§6) Used for ain Decomposition (§7.2) Eigenpairs Sample Position 2 Sample Position 3 Timbre Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization Data-Driven Online Coarsening (§7.1) Hierarchical Component Mode Synthesis (§7.2) Runtime Fast Approximate Modal Analysis (§7) The evaluation of the objective function and Gradient Computation (§6) + Sample Position 1 User Input (Target Sound) Simulated Sound Material Optimization (§6) New Material Distribution Similarity Score Computation (§5) Used for Material Reduction (§6) Used for Domain Decomposition (§7.2) Eigenpairs Sample Position 2 Sample Position 3 Timbre The evaluation of th and Gradient C For Model (Independent of Materials) Input: Surface Mesh Eigenvectors of Volumetric Mesh Laplacian oxelization and Eigen Decomposition (§7.2) Material Opti New Material Distribution Used for Material Reduction (§6) Used for Domain Decomposition (§7.2) hm Overview. Our optimization algorithm consists of the precomputation and ru me consists of 3 steps. First, the system computes modal analysis to obtain the vibr arity score between the simulated sounds of the object and user speciﬁed target so s the material distribution inside the object to minimise the cost. A FEM Mesh Target Sounds → Target Parameters Kazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization Data-Driven Online Coarsening (§7.1) Hierarchical Component Mode Synthesis (§7.2) Runtime Fast Approximate Modal Analysis (§7) The evaluation of the objective function and Gradient Computation (§6) + … For Model (Independent of Materials) Input: Surface Mesh Eigenvectors of Volumetric Mesh Laplacian Training of Mapping (§7.1.3) For Material Set (Independent of Models) Voxelization and Eigen Decomposition (§7.2) Sample Position 1 User Input (Target Sound) Simulated Sound Material Optimization (§6) New Material Distribution Similarity Score Computation (§5) Used for Material Reduction (§6) Used for Domain Decomposition (§7.2) Eigenpairs Sample Position 2 Sample Position 3 Timbre Precomputation Regression Forests 2: Algorithm Overview. Our optimization algorithm consists of the precomputation and runtime. An iteration of our optimization e at runtime consists of 3 steps. First, the system computes modal analysis to obtain the vibrational property of the object. Second, it s the similarity score between the simulated sounds of the object and user speciﬁed target sounds. According to this similarity score, m updates the material distribution inside the object to minimise the cost. ed Work elset optimization, and controlled the several lowest eigenfrequen- Simulated Sound Material Optimization New Material Distribution Perceptual Differences and Gradients → (f1, a1, d1) (modal parameters) → (f2, a2, d2) → (f3, a3, d3)

38. Similar to [Bharaj et al. ’15]. Material Optimization Frequency Differences (Critical band rate) ses at an interactive rate. The distribution inside the model hen the sample positions are also can check the sound by ce during optimization at any ion procedure at an arbitrary ew sample point, and restart ay, the user can interactively r a 3D object as if it were a i sented as a pi k = uT k f pi n , where uk is the k-th eigenvector. Using the target frequencies F, amplitudes A and simulated parameters, our objective function for minimizing the perceptual difference of the mode frequencies is represented as E f = 1 2 N ∑ i=2 Bark(sf fi)−Bark(Fi) 2 (1) where Bark(f) is a function to transform the frequency to critical band rate [bark] [ZF99], and sf = F′ 1/f1 is the scaling factor. The ssociation. Amplitude Differences Kazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Ba objective function for amplitudes is also obtained using the bal- ances with other mode amplitudes at the position Ea = 1 2 T ∑ j=1 N ∑ i=2 a j i a j max − A j i A j max 2 . (2) where a j max and A j max denote the largest amplitude at the position j of the simulated and target’s modes respectively. These formu- lations are similar to [BLT∗ 15]; however, we use the perceptual metrics whereas they use square distances of frequencies and amplitudes. We minimize these functions by optimizing the Young’s modulus Ye ∈ RM at each finite element e, where M denotes the number of the elements. Finally, our design problem is formulated as objective function for amplitudes is also obtained using the bal- ances with other mode amplitudes at the position Ea = 1 2 T ∑ j=1 N ∑ i=2 a j i a j max − A j i A j max 2 . (2) where a j max and A j max denote the largest amplitude at the position j of the simulated and target’s modes respectively. These formu- lations are similar to [BLT∗ 15]; however, we use the perceptual metrics whereas they use square distances of frequencies and amplitudes. We minimize these functions by optimizing the Young’s modulus Ye ∈ RM at each finite element e, where M denotes the number of the elements. Finally, our design problem is formulated as argmin Ye : wf E f +waEa, sub ject to : Ye > 0 (3) where wf and wa denote the positive weights. Note that we do not optimize damping parameters. We instead × × × × e1 e3 16 Cubatu 4 fine mater × × × × Figure 4: Data-Driven tration). The function (e1,e2,e3,e4) of fine fo four coarse material pa points (right) to minim ReflectedVariations e1 e2 e4 e3 e1 e2 e4 e3 = Figure 5: Eight equiva We minimize: Y: Young’s modulus N: optimized modes, T: sample positions We minimize perceptual differences between simulated and target modal parameters.

42. Material Optimization We solve this problem by material reduction [Xu et al. ’15], and a hybrid optimization scheme of evolutional strategies (CMA-ES) and gradient descent (quasi-Newton). where amax and Amax denote the largest amplitude at the position j of the simulated and target’s modes respectively. These formu- lations are similar to [BLT∗ 15]; however, we use the perceptual metrics whereas they use square distances of frequencies and amplitudes. We minimize these functions by optimizing the Young’s modulus Ye ∈ RM at each finite element e, where M denotes the number of the elements. Finally, our design problem is formulated as argmin Ye : wf E f +waEa, sub ject to : Ye > 0 (3) where wf and wa denote the positive weights. Note that we do not optimize damping parameters. We instead reuse the estimated damping from the assigned sound clips as mode-dependent damping. This means that our damping is not spa- tially constant. This setting is physically incorrect, but it makes the problem simpler. tration). The functio (e1,e2,e3,e4) of fine four coarse material points (right) to mini ReflectedVariations e1 e e4 e e1 e e4 e = Figure 5: Eight equi top row represents fo resents four reflected elements) as input a cube element) as ou We minimize: Y: Young’s modulus

44. Modal Analysis Algorithm Overview Data-Driven Online Coarsening (§7.1) archical Component Mode Synthesis (§7.2) Runtime st Approximate Modal Analysis (§7) The evaluation of the objective function and Gradient Computation (§6) + Sample Position 1 User Input (Target Sound) Simulated Sound Material Optimization (§6) New Material Distribution Similarity Score Computation (§5) or Material Reduction (§6) Used for ain Decomposition (§7.2) Eigenpairs Sample Position 2 Sample Position 3 Timbre Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization Data-Driven Online Coarsening (§7.1) Hierarchical Component Mode Synthesis (§7.2) Runtime Fast Approximate Modal Analysis (§7) The evaluation of the objective function and Gradient Computation (§6) + Sample Position 1 User Input (Target Sound) Simulated Sound Material Optimization (§6) New Material Distribution Similarity Score Computation (§5) Used for Material Reduction (§6) Used for Domain Decomposition (§7.2) Eigenpairs Sample Position 2 Sample Position 3 Timbre The evaluation of th and Gradient C For Model (Independent of Materials) Input: Surface Mesh Eigenvectors of Volumetric Mesh Laplacian oxelization and Eigen Decomposition (§7.2) Material Opti New Material Distribution Used for Material Reduction (§6) Used for Domain Decomposition (§7.2) hm Overview. Our optimization algorithm consists of the precomputation and ru me consists of 3 steps. First, the system computes modal analysis to obtain the vibr arity score between the simulated sounds of the object and user speciﬁed target so s the material distribution inside the object to minimise the cost. A FEM Mesh Target Sounds → Target Parameters Kazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization Data-Driven Online Coarsening (§7.1) Hierarchical Component Mode Synthesis (§7.2) Runtime Fast Approximate Modal Analysis (§7) The evaluation of the objective function and Gradient Computation (§6) + … For Model (Independent of Materials) Input: Surface Mesh Eigenvectors of Volumetric Mesh Laplacian Training of Mapping (§7.1.3) For Material Set (Independent of Models) Voxelization and Eigen Decomposition (§7.2) Sample Position 1 User Input (Target Sound) Simulated Sound Material Optimization (§6) New Material Distribution Similarity Score Computation (§5) Used for Material Reduction (§6) Used for Domain Decomposition (§7.2) Eigenpairs Sample Position 2 Sample Position 3 Timbre Precomputation Regression Forests 2: Algorithm Overview. Our optimization algorithm consists of the precomputation and runtime. An iteration of our optimization e at runtime consists of 3 steps. First, the system computes modal analysis to obtain the vibrational property of the object. Second, it s the similarity score between the simulated sounds of the object and user speciﬁed target sounds. According to this similarity score, m updates the material distribution inside the object to minimise the cost. ed Work elset optimization, and controlled the several lowest eigenfrequen- Simulated Sound Material Optimization New Material Distribution Perceptual Differences and Gradients → (f1, a1, d1) (modal parameters) → (f2, a2, d2) → (f3, a3, d3) KU = ΛMU K: Stiffness Matrix M: Mass Matrix Expensive

45. Data-Driven Online Coarsening (§7.1) Hierarchical Component Mode Synthesis (§7.2) Fast Approximate Modal Analysis (§7) + … For Model (Independent of Materials) Input: Surface Mesh Training of Mapping (§7.1.3) For Material Set (Independent of Models) Voxelization and Eigen Decomposition (§7.2) Used for Domain Decomposition (§7.2) Regression Forests arashi / Interactive Physically-Based Sound Design of 3D Modelusing Data-Driven Online Coarsening (§7.1) Hierarchical Component Mode Synthesis (§7.2) Runtime Fast Approximate Modal Analysis (§7) + S Simulated Sound Similarity Score Compu Eigenpairs S S Mesh Coarsening with Data-Driven FEM Fast Approximate Modal Analysis Hierarchical Component Mode Synthesis Data-Driven Online Coarsening (§7.1) Hierarchical Component Mode Synthesis (§7.2) Fast Approximate Modal Analysis (§7) + … For Model (Independent of Materials) Input: Surface Mesh Training of Mapping (§7.1.3) For Material Set (Independent of Models) Voxelization and Eigen Decomposition (§7.2) Used for Domain Decomposition (§7.2) Eigen Regression Forests +

46. Data-Driven FEM using Regression Forests Data-Driven Online Coarsening (§7.1) Hierarchical Component Mode Synthesis (§7.2) Run Fast Approximate Modal Analysis (§7) + Eigenpairs Data-Driven Online Coarsening (§7.1) Hierarchical Component Mode Synthesis (§7.2) Fast Approximate Modal Analysis (§7) + … ing (§7.1.3) ial Set of Models) E utation Forests Machine Learning Detailed mesh Coarse mesh Based on “Data-Driven Finite Elements for Geometry and Material Design” , Chen et al. SIGGRAPH 2015

47. Data-Driven FEM (DDFEM) [Chen et al. 15]ound Design of 3D Modelusing Material Optimization 4 Cubature Points × × × × × × × × DDFEM(e1,…,e4) e1 e2 e3 e4 E1 E2 E3 E4 16 Cubature Points 4 fine materials (in 2D) 4 coarse materials × × × × × × × × × × × × ure 4: Data-Driven Coarsening [Chen et al. 2015] (in 2D on). The function DDFEM() takes four material param e2,e3,e4) of fine four elements (left) and returns correspon on meshes. evels of Coarsening 23x Faster (b) and figure. Notice that, after 1 level of coarsen ing neither compresses nor buckles as much as High-Res Simulation. After 2 levels of coarseni havior is lost. The Na¨ıve Coarsening fails to cap behavior of High-Res Simulation, whereas DDF 1 Level of Coarsening 2 Levels o 51x Faster 489x (a) “Data-Driven Finite Elements for Geometry and Material Design” , Chen et al. SIGGRAPH 2015

48. Data-Driven FEM (DDFEM) [Chen et al. 15] Given N materials, the number of combinations become N8 It is impractical to use for our problem, because it requires a large range of continuous material settings.

49. Extended Data-Driven FEM (DDFEM*) To Reduce the material space 1. Overlapping-Free Cell Ordering 2. Scaling Factor Separation 3. Regression Forests To handle large amount of dataset with a constant evaluation cost

50. 1. Overlapping-Free Cell Ordering(e1,e2,e3,e4) of ﬁne four elements (left) and returns corresponding four coarse material parameters (E1,E2,E3,E4) at the quadrature points (right) to minimize the error. = = = = = = 90° 90° 90°ReﬂectedVariations Rotated Variations e1 e2 e4 e3 e1 e2 e4 e3 e1 e2 e4 e3 e1 e2 e4 e3 e1 e2 e4 e3 e1 e2 e4 e3 e1 e2 e4 e3 e1 e2 e4 e3 = = = = Figure 5: Eight equivalent cell variations (in 2D illustration). The top row represents four rotated variations and the bottom row rep- Basically equivalent cell variations (in 2D) should be emitted from the dataset.

51. 1. Overlapping-Free Cell Ordering Kazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Based S The Origin Cell (i 1) e1 e’1 1: Initial ordering Reordering (i = 1) e1 e2 e4 e3 e’1 e2 e4 e3 2: Compare the neighbors e’1 e’2 e’3 e’4 e’1 e’3 e’2 e’4 e’1 e’2 e’3 e3 e’1 e’3 e’2 e3 3: If e2 e4 3: otherwise 4: Fill rests Figure 6: Overlapping Free Cell Ordering (in 2D illustration). 1: At the i-th cell evaluation (in this example, we assume i = 2), ei becomes the origin cell e′ 1. 2: we compare the material values of the adjacent cells. 3: the smaller cell becomes e′ 2 and the other becomes e′ 3. 4: The left cell becomes e′ 4. sK( of e is eq Thi DD cell DD coordinate. At the i-th evaluation within the eight eva define the i-th cell as the origin e′ 1. Then, we compare the Young’s modulus of the three adjacent cells of the o 2D, two cells), and define the index the cell who has value as e′ 2, the cell who has the secondary smallest val the other cell as e′ 4. Finally, we decide the ordering of cells by the following rule: The cell that is adjacent becomes e′ 5. The cell that is adjacent to e′ 3 and e′ 4 beco cell that is adjacent to e′ 2 and e′ 4 becomes e′ 7. The last o e′ 8. Then, using these reordered parameters, our DDFEM is redefined again as Ei = DDFEMi(e′ 1,e′ 2,...,e′ 8), e′ 1 = ei By using this representation, we can avoid explicit enu the eight rotated and eight reflected patterns of a mate sorted by the Young’s modulus. Please see the paper for the details of the sorting procedure. = = = ReflectedV e1 e2 e4 e3 e1 e2 e4 e3 e1 e2 e4 e3 e1 e2 e4 e3 = = = = Eight equivalent cell variations (in 2D illustration). The presents four rotated variations and the bottom row rep- r reflected variations. as input and generates a coarse approximated mesh (a ent) as output using the the material parameter map- ed from training data in the precomputation step (Fig- e concept of our data driven FEM coarsening is based 15]. The goal of their data-driven FEM is obtaining (E1,...,E8) = DDFEM(e1,..,e8) (5) FEM() is a function that takes eight material parame- ,e8) of a detailed mesh and returns the corresponding e material parameters (E ,...,E ) at the cubature points

52. Young’s modulus has a large range of the value (10-2 ~ 103 GPa) 2. Scaling Factor Separation It is difﬁcult to treat such a large range by DDFEM directly !!

53. 2. Scaling Factor Separation such a large range during training. To avoid this, we dramatically reduce the training size by separating the scale factor. Based on [CLSM15], our DDFEM() is constructed to minimize the square difference of the integral of the strain energy density functions between the detailed and coarse meshes. argmin Ei ∑ f∈F 8 ∑ i=1 wivc i ( f,Ei)− 8 ∑ j=1 8 ∑ i=1 wivd ji(f,e′ j) 2 (8) where w denotes the cubature weights, F denotes a set of randomly sampled external forces, and vc and vd represent the strain energy density function of the coarse and detailed mesh respectively. Here, the strain energy density function in linear elastic is represented F: force set SED of coarse mesh DDFEM() is designed to minimize the square diﬀerence of the integral of the strain energy density (SED). SED of ﬁne mesh

54. 2. Scaling Factor Separation

55. 2. Scaling Factor Separation In linear elastics: on of the coarse and detailed mesh respectively. Here, rgy density function in linear elastic is represented (e)u(e)2 = K(e)(K−1 (e) f)2 where K(e) and f are matrix and the external forces respectively. In addi- ng e by a scalar s, v( f,s · e) = K(s · e)u(s · e)2 = into seve into ﬁner the comp pensate f correctio s) ings c⃝ 2016 The Eurographics Association. cally-Based Sound Design of 3D Modelusing Material Optimization sK(e)((sK(e))−1 f)2 = v( f,e)/s because K() is the line of e. Then, the minimization problem argmin Ei ∑ f∈F 8 ∑ i=1 wivc i ( f,Ei)− 8 ∑ j=1 8 ∑ i=1 wivd ji( f,s·e is equivalent to 8 8 8 s: scholar

56. 2. Scaling Factor Separation In linear elastics: on of the coarse and detailed mesh respectively. Here, rgy density function in linear elastic is represented (e)u(e)2 = K(e)(K−1 (e) f)2 where K(e) and f are matrix and the external forces respectively. In addi- ng e by a scalar s, v( f,s · e) = K(s · e)u(s · e)2 = into seve into finer the comp pensate f correctio s) ings c⃝ 2016 The Eurographics Association. cally-Based Sound Design of 3D Modelusing Material Optimization sK(e)((sK(e))−1 f)2 = v( f,e)/s because K() is the line of e. Then, the minimization problem argmin Ei ∑ f∈F 8 ∑ i=1 wivc i ( f,Ei)− 8 ∑ j=1 8 ∑ i=1 wivd ji( f,s·e is equivalent to 8 8 8 s: scholar Then, e’1 e’3 e’2 e’4 e’1 e’3 e’2 e3 3: otherwise Ordering (in 2D illustration). 1: s example, we assume i = 2), ei e compare the material values of r cell becomes e′ 2 and the other mes e′ 4. turns a scalar while Eq.5 outputs this DDFEM() evaluation eight 2 × 2 element into a coarse ele- bering of the eight cells by each this operation as a 2D example in s are defined in a local R3 space Ei f∈F i=1 j=1i=1 is equivalent to argmin E′ i =Ei/s ∑ f∈F 8 ∑ i=1 wivc i (f,E′ i )− 8 ∑ j=1 8 ∑ i=1 wivd ji(f,e′ This means that we can separate the input paramete DDFEM() problem by the multiplication of the valu cell as a scale factor and their quotients. Finally, we c DDFEM() function as Ei = ei ·DDFEMi e′ 2 ei ,..., e′ 8 ei An advantage of this representation is that it reduce range of dataset but also the dimensions of the featur R8 to R7 . Note that we assume our model as linear ela the original DDFEM() treats nonlinearity because t is equivalent to ells), and define the index the cell who has the smallest , the cell who has the secondary smallest value as e′ 3, and ell as e′ 4. Finally, we decide the ordering of the rest four e following rule: The cell that is adjacent to e′ 2 and e′ 3 ′ 5. The cell that is adjacent to e′ 3 and e′ 4 becomes e′ 6. The adjacent to e′ 2 and e′ 4 becomes e′ 7. The last one becomes ing these reordered parameters, our DDFEM() function d again as Ei = DDFEMi(e′ 1,e′ 2,...,e′ 8), e′ 1 = ei (7) his representation, we can avoid explicit enumeration of otated and eight reflected patterns of a material pattern, e the input parameter space in 3D at both training and or dataset generation at the training, we first determine 7.1.3. Regression Fore In contract with Chen struct the database of sons. First, their databa are not included in the ization ability. Second, a rate proportional to th of the dataset should be terns. To address these tion using two regressio to [LJS∗ 15] which con tree structure constructi square solve for the reg all the dataset. The reg cost evaluation even if

57. 2. Scaling Factor Separation In linear elastics: on of the coarse and detailed mesh respectively. Here, rgy density function in linear elastic is represented (e)u(e)2 = K(e)(K−1 (e) f)2 where K(e) and f are matrix and the external forces respectively. In addi- ng e by a scalar s, v( f,s · e) = K(s · e)u(s · e)2 = into seve into finer the comp pensate f correctio s) ings c⃝ 2016 The Eurographics Association. cally-Based Sound Design of 3D Modelusing Material Optimization sK(e)((sK(e))−1 f)2 = v( f,e)/s because K() is the line of e. Then, the minimization problem argmin Ei ∑ f∈F 8 ∑ i=1 wivc i ( f,Ei)− 8 ∑ j=1 8 ∑ i=1 wivd ji( f,s·e is equivalent to 8 8 8 s: scholar Then, e’1 e’3 e’2 e’4 e’1 e’3 e’2 e3 3: otherwise Ordering (in 2D illustration). 1: s example, we assume i = 2), ei e compare the material values of r cell becomes e′ 2 and the other mes e′ 4. turns a scalar while Eq.5 outputs this DDFEM() evaluation eight 2 × 2 element into a coarse ele- bering of the eight cells by each this operation as a 2D example in s are defined in a local R3 space Ei f∈F i=1 j=1i=1 is equivalent to argmin E′ i =Ei/s ∑ f∈F 8 ∑ i=1 wivc i (f,E′ i )− 8 ∑ j=1 8 ∑ i=1 wivd ji(f,e′ This means that we can separate the input paramete DDFEM() problem by the multiplication of the valu cell as a scale factor and their quotients. Finally, we c DDFEM() function as Ei = ei ·DDFEMi e′ 2 ei ,..., e′ 8 ei An advantage of this representation is that it reduce range of dataset but also the dimensions of the featur R8 to R7 . Note that we assume our model as linear ela the original DDFEM() treats nonlinearity because t is equivalent to ells), and define the index the cell who has the smallest , the cell who has the secondary smallest value as e′ 3, and ell as e′ 4. Finally, we decide the ordering of the rest four e following rule: The cell that is adjacent to e′ 2 and e′ 3 ′ 5. The cell that is adjacent to e′ 3 and e′ 4 becomes e′ 6. The adjacent to e′ 2 and e′ 4 becomes e′ 7. The last one becomes ing these reordered parameters, our DDFEM() function d again as Ei = DDFEMi(e′ 1,e′ 2,...,e′ 8), e′ 1 = ei (7) his representation, we can avoid explicit enumeration of otated and eight reflected patterns of a material pattern, e the input parameter space in 3D at both training and or dataset generation at the training, we first determine 7.1.3. Regression Fore In contract with Chen struct the database of sons. First, their databa are not included in the ization ability. Second, a rate proportional to th of the dataset should be terns. To address these tion using two regressio to [LJS∗ 15] which con tree structure constructi square solve for the reg all the dataset. The reg cost evaluation even if

58. 2. Scaling Factor Separation In linear elastics: on of the coarse and detailed mesh respectively. Here, rgy density function in linear elastic is represented (e)u(e)2 = K(e)(K−1 (e) f)2 where K(e) and f are matrix and the external forces respectively. In addi- ng e by a scalar s, v( f,s · e) = K(s · e)u(s · e)2 = into seve into finer the comp pensate f correctio s) ings c⃝ 2016 The Eurographics Association. cally-Based Sound Design of 3D Modelusing Material Optimization sK(e)((sK(e))−1 f)2 = v( f,e)/s because K() is the line of e. Then, the minimization problem argmin Ei ∑ f∈F 8 ∑ i=1 wivc i ( f,Ei)− 8 ∑ j=1 8 ∑ i=1 wivd ji( f,s·e is equivalent to 8 8 8 s: scholar Then, e’1 e’3 e’2 e’4 e’1 e’3 e’2 e3 3: otherwise Ordering (in 2D illustration). 1: s example, we assume i = 2), ei e compare the material values of r cell becomes e′ 2 and the other mes e′ 4. turns a scalar while Eq.5 outputs this DDFEM() evaluation eight 2 × 2 element into a coarse ele- bering of the eight cells by each this operation as a 2D example in s are defined in a local R3 space Ei f∈F i=1 j=1i=1 is equivalent to argmin E′ i =Ei/s ∑ f∈F 8 ∑ i=1 wivc i (f,E′ i )− 8 ∑ j=1 8 ∑ i=1 wivd ji(f,e′ This means that we can separate the input paramete DDFEM() problem by the multiplication of the valu cell as a scale factor and their quotients. Finally, we c DDFEM() function as Ei = ei ·DDFEMi e′ 2 ei ,..., e′ 8 ei An advantage of this representation is that it reduce range of dataset but also the dimensions of the featur R8 to R7 . Note that we assume our model as linear ela the original DDFEM() treats nonlinearity because t is equivalent to ells), and define the index the cell who has the smallest , the cell who has the secondary smallest value as e′ 3, and ell as e′ 4. Finally, we decide the ordering of the rest four e following rule: The cell that is adjacent to e′ 2 and e′ 3 ′ 5. The cell that is adjacent to e′ 3 and e′ 4 becomes e′ 6. The adjacent to e′ 2 and e′ 4 becomes e′ 7. The last one becomes ing these reordered parameters, our DDFEM() function d again as Ei = DDFEMi(e′ 1,e′ 2,...,e′ 8), e′ 1 = ei (7) his representation, we can avoid explicit enumeration of otated and eight reflected patterns of a material pattern, e the input parameter space in 3D at both training and or dataset generation at the training, we first determine 7.1.3. Regression Fore In contract with Chen struct the database of sons. First, their databa are not included in the ization ability. Second, a rate proportional to th of the dataset should be terns. To address these tion using two regressio to [LJS∗ 15] which con tree structure constructi square solve for the reg all the dataset. The reg cost evaluation even if (i 1) e’1 e’3e’1 e’3 verlapping Free Cell Ordering (in 2D illustration). 1: ell evaluation (in this example, we assume i = 2), ei origin cell e′ 1. 2: we compare the material values of cells. 3: the smaller cell becomes e′ 2 and the other 4: The left cell becomes e′ 4. ven FEM function returns a scalar while Eq.5 outputs It means we repeat this DDFEM() evaluation eight nvert a detailed 2 × 2 × 2 element into a coarse ele- we reorder the numbering of the eight cells by each valuation. We show this operation as a 2D example in e indices of the cells are defined in a local R3 space argmin E′ i =Ei/s ∑ f∈F 8 ∑ i= This means that we DDFEM() problem cell as a scale factor DDFEM() function Ei An advantage of thi range of dataset but R8 to R7 . Note that w the original DDFEM Cell (i 1) e’1 e’1 e’3 e’2 e’4 e’1 e’3 e’2 e3 : Overlapping Free Cell Ordering (in 2D illustration). 1: h cell evaluation (in this example, we assume i = 2), ei the origin cell e′ 1. 2: we compare the material values of cent cells. 3: the smaller cell becomes e′ 2 and the other e′ 3. 4: The left cell becomes e′ 4. -driven FEM function returns a scalar while Eq.5 outputs tor. It means we repeat this DDFEM() evaluation eight convert a detailed 2 × 2 × 2 element into a coarse ele- xt, we reorder the numbering of the eight cells by each () evaluation. We show this operation as a 2D example in The indices of the cells are defined in a local R3 space is equivalent to argmin E′ i =Ei/s ∑ f∈F 8 ∑ i=1 w This means that we ca DDFEM() problem by cell as a scale factor an DDFEM() function as Ei = An advantage of this r range of dataset but als R8 to R7 . Note that we the original DDFEM(

59. … For Model (Independent of Materials) Training of Mapping (§7.1.3) For Material Set (Independent of Models) Regression Forests 3. Regression Forests ✔ A large dataset handling ✔ Constant evaluation cost ✔ High generalization ability Data-driven fluid simulations using regression forests, Ladický et al., SIGGRAPH Asia ‘15

60. 3. Regression ForestsLJS∗ 15] which construct each tree through two steps training: structure construction with a subset of learning data and least- are solve for the regression coefﬁcients at each leaf node with he dataset. The regression forest has an advantage of constant evaluation even if the amount of the dataset is increased. Fi- y, our DDFEM() becomes Ei = ¯e ·Reg1( e′ 2 ¯e ,..., e′ 8 ¯e ) (ei = 0) Ei = ei ·Reg2( e′ 2 ei ,..., e′ 8 ei ) (ei > 0) (12) re Reg() represents the regression function, and ¯e is the average he Young’s modulus in the target eight cells. Hierarchical Component Mode Synthesis Finally, our DDFEM becomes Reg(): regression function Data-driven fluid simulations using regression forests, Ladický et al., SIGGRAPH Asia ‘15

61. Data-Driven Online Coarsening (§7.1) Hierarchical Component Mode Synthesis (§7.2) Fast Approximate Modal Analysis (§7) + … For Model (Independent of Materials) Input: Surface Mesh Training of Mapping (§7.1.3) For Material Set (Independent of Models) Voxelization and Eigen Decomposition (§7.2) Used for Domain Decomposition (§7.2) Regression Forests arashi / Interactive Physically-Based Sound Design of 3D Modelusing Data-Driven Online Coarsening (§7.1) Hierarchical Component Mode Synthesis (§7.2) Runtime Fast Approximate Modal Analysis (§7) + S Simulated Sound Similarity Score Compu Eigenpairs S S Mesh Coarsening with Data-Driven FEM Fast Approximate Mode Analysis Hierarchical Component Mode Synthesis Data-Driven Online Coarsening (§7.1) Hierarchical Component Mode Synthesis (§7.2) Fast Approximate Modal Analysis (§7) + … For Model (Independent of Materials) Input: Surface Mesh Training of Mapping (§7.1.3) For Material Set (Independent of Models) Voxelization and Eigen Decomposition (§7.2) Used for Domain Decomposition (§7.2) Eigen Regression Forests +

62. Hierarchical Component Mode Synthesis (HCMS) Kazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization Error Correction (§7.2.1) Domain Decomposition (§7.2) Coarse Voxel Mesh (The Output of DDFEM*) Subdomains Local Eigen Problem Solves in Parallel (D: Local Eigenvalues, U: Local Eigenvectors) {D1,U1} {D2,U2} {D3,U3} {D4,U4} {D5,U5} {D6,U6} {D7,U7} {D8,U8} {D9,U9} Hierarchical Merge by Eq. (15) {D11,U11} {D12,U12} {D13,U13} {D14,U14} {D22,U22} {D21,U21} {D31,U31} {D41,U41} Merge adjacent domains in parallel Global Solution ure 7: Hierarchical Component Mode Synthesis. After coarsening the mesh, we decompose it into many subdomains and hierarch rges them with reducing their DoFs in parallel. Finally, we improve the accuracy using an error correction algorithm. To simplify the explanation for our HCMS, we ﬁrst begin h assuming that a model can be decomposed into two non- rlapping domains S1 and S2 as in conventional CMS, and the (15) to obtain the eigenvectors of the merged subdomain. Thi cedure also can be executed in parallel until all the subdomai merged. Finally, we merge all the subdomains and obtain th HCMS takes the coarse voxel mesh as input and solves a generalized eigenproblem via hierarchical merging.

63. Hierarchical Component Mode Synthesis (HCMS) Kazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization Error Correction (§7.2.1) Domain Decomposition (§7.2) Coarse Voxel Mesh (The Output of DDFEM*) Subdomains Local Eigen Problem Solves in Parallel (D: Local Eigenvalues, U: Local Eigenvectors) {D1,U1} {D2,U2} {D3,U3} {D4,U4} {D5,U5} {D6,U6} {D7,U7} {D8,U8} {D9,U9} Hierarchical Merge by Eq. (15) {D11,U11} {D12,U12} {D13,U13} {D14,U14} {D22,U22} {D21,U21} {D31,U31} {D41,U41} Merge adjacent domains in parallel Global Solution ure 7: Hierarchical Component Mode Synthesis. After coarsening the mesh, we decompose it into many subdomains and hierarch rges them with reducing their DoFs in parallel. Finally, we improve the accuracy using an error correction algorithm. To simplify the explanation for our HCMS, we ﬁrst begin h assuming that a model can be decomposed into two non- rlapping domains S1 and S2 as in conventional CMS, and the (15) to obtain the eigenvectors of the merged subdomain. Thi cedure also can be executed in parallel until all the subdomai merged. Finally, we merge all the subdomains and obtain th HCMS takes the coarse voxel mesh as input and solves a generalized eigenproblem via hierarchical merging.

64. Hierarchical Component Mode Synthesis (HCMS) subdomain 1 subdomain 2 Eigenvalues of A : Λ1 Eigenvectors of A: U1 Eigenvalues of B : Λ2 Eigenvectors of B: U2

65. Hierarchical Component Mode Synthesis (HCMS) subdomain 1 subdomain 2 Eigenvalues of A : Λ1 Eigenvectors of A: U1 Eigenvalues of B : Λ2 Eigenvectors of B: U2

66. Hierarchical Component Mode Synthesis (HCMS) subdomain 1 subdomain 2 Eigenvalues of A : Λ1 Eigenvectors of A: U1 Eigenvalues of B : Λ2 Eigenvectors of B: U2 Figure 7: Hierarchical Component Mode Synthesis. After coarsening the mesh, we decom merges them with reducing their DoFs in parallel. Finally, we improve the accuracy using To simplify the explanation for our HCMS, we ﬁrst begin with assuming that a model can be decomposed into two non- overlapping domains S1 and S2 as in conventional CMS, and the eigenpairs of each domain are already known. Under this assump- tion, the entire stiffness matrix Ktotal and the entire mass matrix Mtotal can be represented as Ktotal = K11 K12 K12 K22 , Mtotal = M1 0 0 M2 (13) where K11, K22 and M1, M2 denote the local stiffness and mass matrices of each sub-domain respectively. K12 and K21 are the inter- face matrices that connect the domains S1 and S2. If the eigenvectors of each domain U1 and U2 are already known, we can rewrite (15) to obtain th cedure also can merged. Finally, proximate eigen ing subdomains caused by subop tion (§7.2.1). W new. We implem main by a comb and QR method. 7.2.1. Error Co HCMS is just a Mtotal can be represented as Ktotal = K11 K12 K12 K22 , Mtotal = M1 0 0 M2 (13) where K11, K22 and M1, M2 denote the local stiffness and mass matrices of each sub-domain respectively. K12 and K21 are the inter- face matrices that connect the domains S1 and S2. If the eigenvectors of each domain U1 and U2 are already known, we can rewrite the Eq. (13) using the reduced matrices of each domain with re- maining the lower frequency modes as K′ total = D1 U1T K12U2 U2T KT 12U1 D2 (14) where D1 = U1T K11U1 and D2 = U2T K22U2 are diagonal matrices in which each diagonal entry is the eigenvalue of the respective subdomain. Note that the entire mass matrix also takes the same form for, UT 1 M1U1 = I, and UT 2 M2U2 = I, meaning that the en- caus tion new mai and 7.2. HCM racy duce Gra tors follo ≒ where Ktotal = K11 K12 K12 K22 , Mtotal = M1 0 0 M2 (13 where K11, K22 and M1, M2 denote the local stiffness and mass ma trices of each sub-domain respectively. K12 and K21 are the inter face matrices that connect the domains S1 and S2. If the eigenvec tors of each domain U1 and U2 are already known, we can rewrite the Eq. (13) using the reduced matrices of each domain with re maining the lower frequency modes as K′ total = D1 U1T K12U2 U2T KT 12U1 D2 (14 where D1 = U1T K11U1 and D2 = U2T K22U2 are diagonal matri ces in which each diagonal entry is the eigenvalue of the respective subdomain. Note that the entire mass matrix also takes the same form for, UT 1 M1U1 = I, and UT 2 M2U2 = I, meaning that the en Ktotal = K11 K12 K12 K22 , Mtotal = M1 0 0 M2 (13) where K11, K22 and M1, M2 denote the local stiffness and mass matrices of each sub-domain respectively. K12 and K21 are the inter- face matrices that connect the domains S1 and S2. If the eigenvectors of each domain U1 and U2 are already known, we can rewrite the Eq. (13) using the reduced matrices of each domain with re- maining the lower frequency modes as K′ total = D1 U1T K12U2 U2T KT 12U1 D2 (14) where D1 = U1T K11U1 and D2 = U2T K22U2 are diagonal matrices in which each diagonal entry is the eigenvalue of the respective subdomain. Note that the entire mass matrix also takes the same form for, UT M U = I, and UT M U = I, meaning that the en- Both D matrices become diagonal matrices in which each diagonal entry is the eigenvalue of the respective subdomain. Reduced SystemEntire System Ktotal = K12 K where K11, K22 and M1, M trices of each sub-domain face matrices that connect tors of each domain U1 an the Eq. (13) using the red maining the lower frequenc K′ total = U2 where D1 = U1T K11U1 an ces in which each diagonal subdomain. Note that the =

Interactive Physically-Based Sound Design of 3D Model using Material Optimization (SCA'16)

Recommended

Recommended

More Related Content

Similar to Interactive Physically-Based Sound Design of 3D Model using Material Optimization (SCA'16)

Similar to Interactive Physically-Based Sound Design of 3D Model using Material Optimization (SCA'16) (20)

Recently uploaded

Recently uploaded (20)

Interactive Physically-Based Sound Design of 3D Model using Material Optimization (SCA'16)