This document discusses advancements in interactive physically-based sound design for 3D models, emphasizing the use of modal sound synthesis to improve sound realism during collision and contact events. It critiques existing linear modal sound models that treat vibrating objects as rigid, leading to sound inaccuracies, and proposes a new frictional multibody contact formulation for better sound resolution. The authors present methods for synthesizing sounds of thin shells and other complex structures, aiming to enhance interactive virtual environments with more authentic audio outputs.

1. Interactive Physically-Based Sound Design of 3D Model
using Material Optimization
Kazuhiro Yamamoto Takeo Igarashi
The University of Tokyo

2. “How can we design the physically-based sound
with an intuitive way?”
Motivation

3. Physically-Based Sound Rendering
We focus on modal sound synthesis.
It produces perfectly synchronized sounds to various visual events
via physical simulation.
Toward High-Quality Modal Contact Sound,
Chanxi et al., SIGGRAPH 2011
Harmonic Shells: A Practical Nonlinear Sound Model
for Near-Rigid Thin Shells,
Jeffrey et al., SIGGRAPH 2009
Toward High-Quality Modal Contact Sound
Changxi Zheng Doug L. James
Cornell University
Figure 1: A Rube-Goldberg contraption that demonstrates many challenging multibody contact sounds. A noisy block feeder (Left) with
flexible tubes ejects marbles into a double helix of plastic chutes (Middle), which causes a cup to fill up, lifting a lever that drops a bunny into
a runaway shopping cart (Right) producing familiar clattering and clanging sounds due to deformable micro-collisions. Our approach can
accurately resolve modal vibrations and contact sounds using an asynchronous, adaptive, frictional contact solver.
Abstract
Contact sound models based on linear modal analysis are com-
monly used with rigid body dynamics. Unfortunately, treating vi-
brating objects as “rigid” during collision and contact processing
fundamentally limits the range of sounds that can be computed, and
contact solvers for rigid body animation can be ill-suited for modal
contact sound synthesis, producing various sound artifacts. In this
paper, we resolve modal vibrations in both collision and frictional
contact processing stages, thereby enabling non-rigid sound phe-
nomena such as micro-collisions, vibrational energy exchange, and
chattering. We propose a frictional multibody contact formulation
and modified Staggered Projections solver which is well-suited to
sound rendering and avoids noise artifacts associated with spatial
and temporal contact-force fluctuations which plague prior meth-
ods. To enable practical animation and sound synthesis of numer-
ous bodies with many coupled modes, we propose a novel asyn-
chronous integrator with model-level adaptivity built into the fric-
tional contact solver. Vibrational contact damping is modeled to
approximate contact-dependent sound dissipation. Results are pro-
vided that demonstrate high-quality contact resolution with sound.
CR Categories: I.3.5 [Computer Graphics]: Computational Geometry and
Object Modeling—Physically based modeling; I.6.8 [Simulation and Mod-
eling]: Types of Simulation—Animation; H.5.5 [Information Systems]: In-
formation Interfaces and Presentation—Sound and Music Computing
Keywords: Sound synthesis; contact sounds; modal analysis; asyn-
chronous integration; frictional contact
1 Introduction
Sound models based on linear modal vibrations are widely used to
efficiently synthesize plausible contact sounds for so-called rigid
bodies in computer animation and interactive virtual environments.
Unfortunately, there still remain a number of significant contact-
related deficiencies that limit the realism of modal contact sounds
in practice. To begin with, for speed and simplicity, modal sound
models are usually just excited by using contact force impulses
from rigid body contact solvers. In reality, there is no such thing as
a “rigid” object, and the same small vibrations that produce sound
also play an important role in producing rich contact events: micro-
collisions, chattering, squeaking, coupled vibrations, contact damp-
ing, etc. Ignoring contact-level vibrations is the source of many
sound-related deficiencies, as these small vibrations can be visually
inconsequential but aurally significant. For example, pounding on
a seemingly “rigid” dinner table can shake dishes—and may also
upset your friends (see Figure 2). Frictional contact and deforma-
tion coupling is also important for sound; for example, slip-stick
phenomena is responsible for many familiar squeaking and scrap-
ing sounds, e.g., fingernails scraping on a chalkboard. Resolving
these vibrational contact effects is challenging due to the need to
resolve deformable collisions and contact at high temporal rates.
Even in seemingly rigid scenarios, such as an object resting on a
plane, current contact solver implementations can generate tem-
porally incoherent contact impulses which lead to sound artifacts,
such as resting objects that strangely humm or buzz when integrated
at near-audio rates. These artifacts are a consequence of the fun-
damental non-uniqueness of rigid body contact forces (e.g., static
indeterminacy) which can lead to point-like and nonphysical con-
tact force (traction) distributions. Additionally, rigid-body contact
impulses can exhibit nonphysical temporal fluctuations, which lead
to noise-related sound artifacts (especially with iterative contact so-
lution techniques) that must be dissipated artificially.
Moreover, the sound of a resting object should also depend on its
contact state, and how contacts oppose surface vibrations. For ex-
ample, a coffee mug exhibits distinctive vibrational damping when
placed in different orientations on surfaces (see Figure 4). This con-
Harmonic Shells: A Practical Nonlinear Sound Model for Near-Rigid Thin Shells
Jeffrey N. Chadwick Steven S. An
Cornell University
Doug L. James
Figure 1: Crash! Our physically based sound renderings of thin shells produce characteristic “crashing” and “rumbling” sounds when animated using rigid
body dynamics. We synthesize nonlinear modal vibrations using an efficient reduced-order dynamics model that captures important nonlinear mode coupling.
High-resolution sound field approximations are generated using far-field acoustic transfer (FFAT) maps, which are precomputed using efficient fast Helmholtz
multipole methods, and provide cheap evaluation of detailed low- to high-frequency acoustic transfer functions for realistic sound rendering.
Abstract
We propose a procedural method for synthesizing realistic sounds
due to nonlinear thin-shell vibrations. We use linear modal analysis
to generate a small-deformation displacement basis, then couple the
modes together using nonlinear thin-shell forces. To enable audio-
rate time-stepping of mode amplitudes with mesh-independent cost,
we propose a reduced-order dynamics model based on a thin-shell
cubature scheme. Limitations such as mode locking and pitch glide
are addressed. To support fast evaluation of mid-frequency mode-
based sound radiation for detailed meshes, we propose far-field
acoustic transfer maps (FFAT maps) which can be precomputed
using state-of-the-art fast Helmholtz multipole methods. Familiar
examples are presented including rumbling trash cans and plastic
bottles, crashing cymbals, and noisy sheet metal objects, each with
increased richness over linear modal sound models.
CR Categories: I.3.5 [Computer Graphics]: Computa-
tional Geometry and Object Modeling—Physically based mod-
eling; I.6.8 [Simulation and Modeling]: Types of Simulation—
Animation; H.5.5 [Information Systems]: Information Interfaces
and Presentation—Sound and Music Computing
can provide convincing physically based sound sources, especially
for pure ringing tones such as chimes, bells, or “knocks.” Unfor-
tunately, we lack effective sound models for a broad class of noisy
virtual objects: thin shells (objects with thicknesses orders of mag-
nitude smaller than their other dimensions). Thin shells are very
common in real and virtual environments, and produce rich and
easily recognizable impact sounds: sheet metal objects (trash cans,
oil drums, tin roofs, machinery), plastic containers (water bottles),
musical instruments (cymbals), etc. Their rich nonlinear vibrations
produce proverbial “crashes” and “rumbles” that are poorly approx-
imated by linear modal sound models which lack nonlinear mode
coupling. To make matters worse, thin shells are often very loud
and important sound sources due to their ability to vibrate and radi-
ate sound so effectively, e.g., consider a metal roof pelted by hail.
Alas, their expensive nonlinear dynamics have made thin shells
computationally impractical for physically based sound synthesis.
In this paper, we propose an efficient method for synthesizing re-
alistic sounds from thin-shell structures undergoing small but non-
linear vibrations. Given a description of an object’s geometry and
material properties, we compute linear vibration modes, then cou-
Rigid-Body Fracture Sound with Precomputed Soundbanks
Changxi Zheng Doug L. James
Cornell University
Figure 1: SMASH! We synthesize the violent fracture and impact sounds of a glass table setting smashed into over 300 pieces (see sound
Rigid-Body Fracture Sound with Precomputed Soundbanks,
Changxi et al., SIGGRAPH Asia 2010

4. Modal Sound Synthesis
for quasi-rigid body sound (e.g., collision, bounce, and scratch)
An Object
Vibration Analysis
(Modal Analysis)
Sound
Modal Parameters
naliz-
neral-
itive-
e the
ere.
n (1)
(2)
et the
sym-
e the
may
(3)
Figure 4: The two rows show a side and top view of
a bowl along with three of the bowl’s first vibrational
modes. The modes selected for the illustration are the
first three non-rigid modes with distinct eigenvalues that
are excited by a transverse impulse to the bowl’s rim.
the body’s six rigid-body modes. The rigid-body eigen-
values are zero because a rigid-body displacement will
ditions, diagonaliz-
o solving a general-
mmetric, positive-
Plesha describe the
e end result here.
mping equation (1)
¨d) = f , (2)
oefficients. Let the
e generalized sym-
Figure 4: The two rows show a side and top view of
a bowl along with three of the bowl’s first vibrational
modes. The modes selected for the illustration are the
applications. With these conditions, diagonaliz-
tion (1) becomes equivalent to solving a general-
mmetric eigenproblem with symmetric, positive-
matrices. Cook, Malkus, and Plesha describe the
n detail and we only repeat the end result here.
he restriction of Rayleigh damping equation (1)
ewritten as:
K(d + ↵1
˙d) + M(↵2
˙d + ¨d) = f , (2)
1 and ↵2 are the Rayleigh coefficients. Let the
of W be the solution to the generalized sym-
igenproblem Kx + Mx = 0 and ⇤ be the
matrix of eigenvalues1
, then equation (2) may
ormed to:
⇤(z + ↵1 ˙z) + (↵2 ˙z + ¨z) = g , (3)
= W 1
d is the vector of modal coordinates
W T
f is the external force vector in the modal
te system.
ow of equation (3) corresponds to a single scalar
rder differential equation:
izi + (↵1 i + ↵2) ˙zi + ¨zi = gi . (4)
ytical solutions to each equation are
zi = c1et!+
i + c2et!i (5)
and c2 are arbitrary (complex) constants, and !i
mplex frequency given by
Figure 4: The two rows show a side and top view of
a bowl along with three of the bowl’s first vibrational
modes. The modes selected for the illustration are the
first three non-rigid modes with distinct eigenvalues that
are excited by a transverse impulse to the bowl’s rim.
the body’s six rigid-body modes. The rigid-body eigen-
values are zero because a rigid-body displacement will
not generate any elastic forces.
The decoupled system of equations is not an approxi-
mation of the original linear system, it will generate ex-
actly the same results as the original linear system. Of
course the linear system may have been an approxima-
tion to some initial nonlinear one, but any problem that
could be solved using equation (2) could also be solved
with equation (3). Furthermore, simulation that would
have required numerical time integration of equation (1)
can now be solved without integration using the analyti-
cal solutions in equations (5) or (7).
h these conditions, diagonaliz-
equivalent to solving a general-
lem with symmetric, positive-
Malkus, and Plesha describe the
nly repeat the end result here.
Rayleigh damping equation (1)
M(↵2
˙d + ¨d) = f , (2)
Rayleigh coefficients. Let the
lution to the generalized sym-
+ Mx = 0 and ⇤ be the
alues1
, then equation (2) may
+ (↵2 ˙z + ¨z) = g , (3)
e vector of modal coordinates
ernal force vector in the modal
Figure 4: The two rows show a side and top view of
a bowl along with three of the bowl’s first vibrational
modes. The modes selected for the illustration are the
first three non-rigid modes with distinct eigenvalues that
are excited by a transverse impulse to the bowl’s rim.
the body’s six rigid-body modes. The rigid-body eigen-
values are zero because a rigid-body displacement will
not generate any elastic forces.
The decoupled system of equations is not an approxi-
+
modal frequency: f0
modal amplitude: a0
modal damping : d0
modal frequency: f1
modal amplitude: a1
modal damping : d1
modal frequency: f2
modal amplitude: a2
modal damping : d2

5. Modal Sound Synthesis
for quasi-rigid body sound (e.g., collision, bounce, and scratch)
An Object
Vibration Analysis
(Modal Analysis)
Sound
Modal Parameters
naliz-
neral-
itive-
e the
ere.
n (1)
(2)
et the
sym-
e the
may
(3)
Figure 4: The two rows show a side and top view of
a bowl along with three of the bowl’s first vibrational
modes. The modes selected for the illustration are the
first three non-rigid modes with distinct eigenvalues that
are excited by a transverse impulse to the bowl’s rim.
the body’s six rigid-body modes. The rigid-body eigen-
values are zero because a rigid-body displacement will
ditions, diagonaliz-
o solving a general-
mmetric, positive-
Plesha describe the
e end result here.
mping equation (1)
¨d) = f , (2)
oefficients. Let the
e generalized sym-
Figure 4: The two rows show a side and top view of
a bowl along with three of the bowl’s first vibrational
modes. The modes selected for the illustration are the
applications. With these conditions, diagonaliz-
tion (1) becomes equivalent to solving a general-
mmetric eigenproblem with symmetric, positive-
matrices. Cook, Malkus, and Plesha describe the
n detail and we only repeat the end result here.
he restriction of Rayleigh damping equation (1)
ewritten as:
K(d + ↵1
˙d) + M(↵2
˙d + ¨d) = f , (2)
1 and ↵2 are the Rayleigh coefficients. Let the
of W be the solution to the generalized sym-
igenproblem Kx + Mx = 0 and ⇤ be the
matrix of eigenvalues1
, then equation (2) may
ormed to:
⇤(z + ↵1 ˙z) + (↵2 ˙z + ¨z) = g , (3)
= W 1
d is the vector of modal coordinates
W T
f is the external force vector in the modal
te system.
ow of equation (3) corresponds to a single scalar
rder differential equation:
izi + (↵1 i + ↵2) ˙zi + ¨zi = gi . (4)
ytical solutions to each equation are
zi = c1et!+
i + c2et!i (5)
and c2 are arbitrary (complex) constants, and !i
mplex frequency given by
Figure 4: The two rows show a side and top view of
a bowl along with three of the bowl’s first vibrational
modes. The modes selected for the illustration are the
first three non-rigid modes with distinct eigenvalues that
are excited by a transverse impulse to the bowl’s rim.
the body’s six rigid-body modes. The rigid-body eigen-
values are zero because a rigid-body displacement will
not generate any elastic forces.
The decoupled system of equations is not an approxi-
mation of the original linear system, it will generate ex-
actly the same results as the original linear system. Of
course the linear system may have been an approxima-
tion to some initial nonlinear one, but any problem that
could be solved using equation (2) could also be solved
with equation (3). Furthermore, simulation that would
have required numerical time integration of equation (1)
can now be solved without integration using the analyti-
cal solutions in equations (5) or (7).
h these conditions, diagonaliz-
equivalent to solving a general-
lem with symmetric, positive-
Malkus, and Plesha describe the
nly repeat the end result here.
Rayleigh damping equation (1)
M(↵2
˙d + ¨d) = f , (2)
Rayleigh coefficients. Let the
lution to the generalized sym-
+ Mx = 0 and ⇤ be the
alues1
, then equation (2) may
+ (↵2 ˙z + ¨z) = g , (3)
e vector of modal coordinates
ernal force vector in the modal
Figure 4: The two rows show a side and top view of
a bowl along with three of the bowl’s first vibrational
modes. The modes selected for the illustration are the
first three non-rigid modes with distinct eigenvalues that
are excited by a transverse impulse to the bowl’s rim.
the body’s six rigid-body modes. The rigid-body eigen-
values are zero because a rigid-body displacement will
not generate any elastic forces.
The decoupled system of equations is not an approxi-
+
modal frequency: f0
modal amplitude: a0
modal damping : d0
modal frequency: f1
modal amplitude: a1
modal damping : d1
modal frequency: f2
modal amplitude: a2
modal damping : d2

6. Difficulty of Physically-Based Sound Design

7. Difficulty of Physically-Based Sound Design
？
Material Parameters

8. Difficulty of Physically-Based Sound Design
？
Material Parameters Physical Simulation
？

9. Our Approach
Example-based interactive frameworks using material optimization.
Input:
3D Model and Target Sound

10. Our Approach
Example-based interactive frameworks using material optimization.
Input:
3D Model and Target Sound
Sound 4
Position 4
Sound 2
Position 2
Sound 3
Position 3
Sound 1
Position 1

11. Our Approach
Example-based interactive frameworks using material optimization.
Input:
3D Model and Target Sound
Sound 4
Position 4
Sound 2
Position 2
Sound 3
Position 3
Sound 1
Position 1

12. Output:
Material Distribution
Our Approach
Example-based interactive frameworks using material optimization.
Input:
3D Model and Target Sound
Sound 4
Position 4
Sound 2
Position 2
Sound 3
Position 3
Sound 1
Position 1

13. What is the “Material” ?
We only optimize Young’s modulus.
• Density
• Young’s modulus
• Poisson’s rate.

14. What is the “Material” ?
We only optimize Young’s modulus.
• Density
• Young’s modulus
• Poisson’s rate.

15. Related Work
Foleyautomatic: physically-based sound effects
for interactive simulation and animation,
Pai et al., 2001
Example-guided physically based modal sound synthesis,
Ren et al., 2013
TimbreFields: 3D Interactive Sound Models for Real-Time Audio.,
Corbett et al., 2007
Impulse responses were recorded digitally as 16 bit
audio at a sampling rate of 44.1 KHz. The microphone
was mounted on a programmable robotic arm, which
automatically moved the microphone to a variety of lis-
tening locations for a given contact location. The sole-
noid was synchronized with the robotic arm to expedite
the recording process. All the microphone and excita-
tion control programming was done in advance. The
strategy was to gather all the impulse responses within a
robotic environment so as to minimize errors intro-
duced through human control.
Before each impulse response recording, 0.5 s of am-
bient sound was recorded to help identify spurious
modes in the background noise. This precaution was
3.1 Estimation of the Modes
The estimation of the modal model parameters
from the measurements was achieved in the following
three phases.
First, we estimated the modal model parameters for
each sound sample separately. This is done by comput-
ing the windowed discrete Fourier transform (Gabor
transform) of the signal—extracting the frequencies,
dampings, and amplitudes by fitting each frequency bin
with a sum of a small number (e.g., four) of damped
complex exponentials. The parameter fitting method is
capable of very accurate frequency reconstructions and
is able to resolve very close modes. Close modes are very
common in artificial objects that have approximate sym-
metries resulting in mode degeneracy. Manufacturing
impurities break this symmetry, splitting the frequencies
by a small amount. These nearby frequencies are dis-
tinctly audible as beating, or “shimmering” sounds and
significantly enhance the realism of the synthesized
sound.
Second, we determined which modes are actually au-
dible, using a rough model of auditory masking (Doel,
Pai, Adam, Kortchmar, & Pichora-Fuller, 2002; Doel,
Knott, & Pai, 2004), and retained only the audible
modes. This step is necessary because the parameter fit-
ting algorithm produces many spurious modes.
Third, we merged all the modal models, using a sim-
ple model of human frequency discrimination, which
results in a single frequency and damping model for the
entire object, and a discrete sampling of the timbre field
a on the 5D interaction space. In theory, all the models
should share the same set of frequencies and dampings,
but due to noise they will not be precisely the same,
motivating this third step.
We now describe the details of the parameter fitting.
Figure 3. An automated measuring system was used to acquire the
data. A robot arm moves the microphone to a preprogrammed set of
locations. An impulse force was then automatically applied to the
object with the solenoid and sounds were recorded for subsequent
analysis.
648 PRESENCE: VOLUME 16, NUMBER 6
1:2 • Z. Ren et al.
(e)(d)(c)(b)(a)
Fig. 1. From the recording of a real-world object (a), our framework is able to find the material parameters and generates similar sound for a replicate object
(b). The same set of parameters can be transfered to various virtual objects to produce sounds with the same material quality ((c), (d), (e)).
(such as stiffness, damping coefficients, and mass density) that can
be directly used in modal analysis.
As a result, for objects with different geometries and runtime
interactions, different sets of modes are generated or excited differ-
ently, and different sounds are produced. However, if the material
properties are the same, they should all sound like coming from
the same material. For example, a plastic plate being hit, a plastic
ball being dropped, and a plastic box sliding on the floor generate
from many sound clips (measured)
from a sound clip
Example-based sound design for modal sound synthesis

16. Related Work
Foleyautomatic: physically-based sound effects
for interactive simulation and animation,
Pai et al., 2001
Example-guided physically based modal sound synthesis,
Ren et al., 2013
TimbreFields: 3D Interactive Sound Models for Real-Time Audio.,
Corbett et al., 2007
Impulse responses were recorded digitally as 16 bit
audio at a sampling rate of 44.1 KHz. The microphone
was mounted on a programmable robotic arm, which
automatically moved the microphone to a variety of lis-
tening locations for a given contact location. The sole-
noid was synchronized with the robotic arm to expedite
the recording process. All the microphone and excita-
tion control programming was done in advance. The
strategy was to gather all the impulse responses within a
robotic environment so as to minimize errors intro-
duced through human control.
Before each impulse response recording, 0.5 s of am-
bient sound was recorded to help identify spurious
modes in the background noise. This precaution was
3.1 Estimation of the Modes
The estimation of the modal model parameters
from the measurements was achieved in the following
three phases.
First, we estimated the modal model parameters for
each sound sample separately. This is done by comput-
ing the windowed discrete Fourier transform (Gabor
transform) of the signal—extracting the frequencies,
dampings, and amplitudes by fitting each frequency bin
with a sum of a small number (e.g., four) of damped
complex exponentials. The parameter fitting method is
capable of very accurate frequency reconstructions and
is able to resolve very close modes. Close modes are very
common in artificial objects that have approximate sym-
metries resulting in mode degeneracy. Manufacturing
impurities break this symmetry, splitting the frequencies
by a small amount. These nearby frequencies are dis-
tinctly audible as beating, or “shimmering” sounds and
significantly enhance the realism of the synthesized
sound.
Second, we determined which modes are actually au-
dible, using a rough model of auditory masking (Doel,
Pai, Adam, Kortchmar, & Pichora-Fuller, 2002; Doel,
Knott, & Pai, 2004), and retained only the audible
modes. This step is necessary because the parameter fit-
ting algorithm produces many spurious modes.
Third, we merged all the modal models, using a sim-
ple model of human frequency discrimination, which
results in a single frequency and damping model for the
entire object, and a discrete sampling of the timbre field
a on the 5D interaction space. In theory, all the models
should share the same set of frequencies and dampings,
but due to noise they will not be precisely the same,
motivating this third step.
We now describe the details of the parameter fitting.
Figure 3. An automated measuring system was used to acquire the
data. A robot arm moves the microphone to a preprogrammed set of
locations. An impulse force was then automatically applied to the
object with the solenoid and sounds were recorded for subsequent
analysis.
648 PRESENCE: VOLUME 16, NUMBER 6
1:2 • Z. Ren et al.
(e)(d)(c)(b)(a)
Fig. 1. From the recording of a real-world object (a), our framework is able to find the material parameters and generates similar sound for a replicate object
(b). The same set of parameters can be transfered to various virtual objects to produce sounds with the same material quality ((c), (d), (e)).
(such as stiffness, damping coefficients, and mass density) that can
be directly used in modal analysis.
As a result, for objects with different geometries and runtime
interactions, different sets of modes are generated or excited differ-
ently, and different sounds are produced. However, if the material
properties are the same, they should all sound like coming from
the same material. For example, a plastic plate being hit, a plastic
ball being dropped, and a plastic box sliding on the floor generate
from many sound clips (measured)
from a sound clip
Example-based sound design for modal sound synthesis
It requires expensive measurement procedure.

17. Related Work
Foleyautomatic: physically-based sound effects
for interactive simulation and animation,
Pai et al., 2001
Example-guided physically based modal sound synthesis,
Ren et al., 2013
TimbreFields: 3D Interactive Sound Models for Real-Time Audio.,
Corbett et al., 2007
Impulse responses were recorded digitally as 16 bit
audio at a sampling rate of 44.1 KHz. The microphone
was mounted on a programmable robotic arm, which
automatically moved the microphone to a variety of lis-
tening locations for a given contact location. The sole-
noid was synchronized with the robotic arm to expedite
the recording process. All the microphone and excita-
tion control programming was done in advance. The
strategy was to gather all the impulse responses within a
robotic environment so as to minimize errors intro-
duced through human control.
Before each impulse response recording, 0.5 s of am-
bient sound was recorded to help identify spurious
modes in the background noise. This precaution was
3.1 Estimation of the Modes
The estimation of the modal model parameters
from the measurements was achieved in the following
three phases.
First, we estimated the modal model parameters for
each sound sample separately. This is done by comput-
ing the windowed discrete Fourier transform (Gabor
transform) of the signal—extracting the frequencies,
dampings, and amplitudes by fitting each frequency bin
with a sum of a small number (e.g., four) of damped
complex exponentials. The parameter fitting method is
capable of very accurate frequency reconstructions and
is able to resolve very close modes. Close modes are very
common in artificial objects that have approximate sym-
metries resulting in mode degeneracy. Manufacturing
impurities break this symmetry, splitting the frequencies
by a small amount. These nearby frequencies are dis-
tinctly audible as beating, or “shimmering” sounds and
significantly enhance the realism of the synthesized
sound.
Second, we determined which modes are actually au-
dible, using a rough model of auditory masking (Doel,
Pai, Adam, Kortchmar, & Pichora-Fuller, 2002; Doel,
Knott, & Pai, 2004), and retained only the audible
modes. This step is necessary because the parameter fit-
ting algorithm produces many spurious modes.
Third, we merged all the modal models, using a sim-
ple model of human frequency discrimination, which
results in a single frequency and damping model for the
entire object, and a discrete sampling of the timbre field
a on the 5D interaction space. In theory, all the models
should share the same set of frequencies and dampings,
but due to noise they will not be precisely the same,
motivating this third step.
We now describe the details of the parameter fitting.
Figure 3. An automated measuring system was used to acquire the
data. A robot arm moves the microphone to a preprogrammed set of
locations. An impulse force was then automatically applied to the
object with the solenoid and sounds were recorded for subsequent
analysis.
648 PRESENCE: VOLUME 16, NUMBER 6
1:2 • Z. Ren et al.
(e)(d)(c)(b)(a)
Fig. 1. From the recording of a real-world object (a), our framework is able to find the material parameters and generates similar sound for a replicate object
(b). The same set of parameters can be transfered to various virtual objects to produce sounds with the same material quality ((c), (d), (e)).
(such as stiffness, damping coefficients, and mass density) that can
be directly used in modal analysis.
As a result, for objects with different geometries and runtime
interactions, different sets of modes are generated or excited differ-
ently, and different sounds are produced. However, if the material
properties are the same, they should all sound like coming from
the same material. For example, a plastic plate being hit, a plastic
ball being dropped, and a plastic box sliding on the floor generate
from many sound clips (measured)
from a sound clip
Example-based sound design for modal sound synthesis
It requires expensive measurement procedure.
It requires same object in real world

18. Demo

19. Demo

20. Demo

21. Demo

22. Demo

23. Demo

24. Demo

25. Demo

26. Demo

27. Demo
The system allows multiple assignments of example sounds.

28. Demo
The system allows multiple assignments of example sounds.

29. Demo
The system allows multiple assignments of example sounds.

30. Demo
The system allows multiple assignments of example sounds.

31. Contributions
• An example-based framework for designing physically-based sound
• Fast approximate modal analysis for an interactive simulation
• Extended data-driven FEM using regression forests
• Hierarchical component mode synthesis with error correction
• Handling a large range of continuous material settings.
• Constant evaluation cost.
• High generalization ability.
• Parallel
• Accurate

32. Modal Analysis
Algorithm Overview
Data-Driven Online Coarsening (§7.1)
archical Component Mode Synthesis (§7.2)
Runtime
st Approximate Modal Analysis (§7)
The evaluation of the objective function
and Gradient Computation (§6)
+
Sample Position 1
User Input
(Target Sound)
Simulated Sound
Material Optimization (§6)
New Material Distribution
Similarity Score Computation (§5)
or Material Reduction (§6)
Used for
ain Decomposition (§7.2)
Eigenpairs
Sample Position 2
Sample Position 3
Timbre
Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization
Data-Driven Online Coarsening (§7.1)
Hierarchical Component Mode Synthesis (§7.2)
Runtime
Fast Approximate Modal Analysis (§7)
The evaluation of the objective function
and Gradient Computation (§6)
+
Sample Position 1
User Input
(Target Sound)
Simulated Sound
Material Optimization (§6)
New Material Distribution
Similarity Score Computation (§5)
Used for Material Reduction (§6)
Used for
Domain Decomposition (§7.2)
Eigenpairs
Sample Position 2
Sample Position 3
Timbre
The evaluation of th
and Gradient C
For Model
(Independent of Materials)
Input: Surface Mesh
Eigenvectors of Volumetric Mesh Laplacian
oxelization and Eigen Decomposition (§7.2)
Material Opti
New Material Distribution
Used for Material Reduction (§6)
Used for
Domain Decomposition (§7.2)
hm Overview. Our optimization algorithm consists of the precomputation and ru
me consists of 3 steps. First, the system computes modal analysis to obtain the vibr
arity score between the simulated sounds of the object and user specified target so
s the material distribution inside the object to minimise the cost.
A FEM Mesh
Target Sounds → Target Parameters
Kazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization
Data-Driven Online Coarsening (§7.1)
Hierarchical Component Mode Synthesis (§7.2)
Runtime
Fast Approximate Modal Analysis (§7)
The evaluation of the objective function
and Gradient Computation (§6)
+
…
For Model
(Independent of Materials)
Input: Surface Mesh
Eigenvectors of Volumetric Mesh Laplacian
Training of Mapping (§7.1.3)
For Material Set
(Independent of Models)
Voxelization and Eigen Decomposition (§7.2)
Sample Position 1
User Input
(Target Sound)
Simulated Sound
Material Optimization (§6)
New Material Distribution
Similarity Score Computation (§5)
Used for Material Reduction (§6)
Used for
Domain Decomposition (§7.2)
Eigenpairs
Sample Position 2
Sample Position 3
Timbre
Precomputation
Regression Forests
2: Algorithm Overview. Our optimization algorithm consists of the precomputation and runtime. An iteration of our optimization
e at runtime consists of 3 steps. First, the system computes modal analysis to obtain the vibrational property of the object. Second, it
s the similarity score between the simulated sounds of the object and user specified target sounds. According to this similarity score,
m updates the material distribution inside the object to minimise the cost.
ed Work elset optimization, and controlled the several lowest eigenfrequen-
Simulated Sound
Material Optimization
New Material Distribution
Perceptual Differences
and Gradients
→ (f1, a1, d1)
(modal parameters)
→ (f2, a2, d2)
→ (f3, a3, d3)

33. Modal Analysis
Algorithm Overview
Data-Driven Online Coarsening (§7.1)
archical Component Mode Synthesis (§7.2)
Runtime
st Approximate Modal Analysis (§7)
The evaluation of the objective function
and Gradient Computation (§6)
+
Sample Position 1
User Input
(Target Sound)
Simulated Sound
Material Optimization (§6)
New Material Distribution
Similarity Score Computation (§5)
or Material Reduction (§6)
Used for
ain Decomposition (§7.2)
Eigenpairs
Sample Position 2
Sample Position 3
Timbre
Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization
Data-Driven Online Coarsening (§7.1)
Hierarchical Component Mode Synthesis (§7.2)
Runtime
Fast Approximate Modal Analysis (§7)
The evaluation of the objective function
and Gradient Computation (§6)
+
Sample Position 1
User Input
(Target Sound)
Simulated Sound
Material Optimization (§6)
New Material Distribution
Similarity Score Computation (§5)
Used for Material Reduction (§6)
Used for
Domain Decomposition (§7.2)
Eigenpairs
Sample Position 2
Sample Position 3
Timbre
The evaluation of th
and Gradient C
For Model
(Independent of Materials)
Input: Surface Mesh
Eigenvectors of Volumetric Mesh Laplacian
oxelization and Eigen Decomposition (§7.2)
Material Opti
New Material Distribution
Used for Material Reduction (§6)
Used for
Domain Decomposition (§7.2)
hm Overview. Our optimization algorithm consists of the precomputation and ru
me consists of 3 steps. First, the system computes modal analysis to obtain the vibr
arity score between the simulated sounds of the object and user specified target so
s the material distribution inside the object to minimise the cost.
A FEM Mesh
Target Sounds → Target Parameters
Kazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization
Data-Driven Online Coarsening (§7.1)
Hierarchical Component Mode Synthesis (§7.2)
Runtime
Fast Approximate Modal Analysis (§7)
The evaluation of the objective function
and Gradient Computation (§6)
+
…
For Model
(Independent of Materials)
Input: Surface Mesh
Eigenvectors of Volumetric Mesh Laplacian
Training of Mapping (§7.1.3)
For Material Set
(Independent of Models)
Voxelization and Eigen Decomposition (§7.2)
Sample Position 1
User Input
(Target Sound)
Simulated Sound
Material Optimization (§6)
New Material Distribution
Similarity Score Computation (§5)
Used for Material Reduction (§6)
Used for
Domain Decomposition (§7.2)
Eigenpairs
Sample Position 2
Sample Position 3
Timbre
Precomputation
Regression Forests
2: Algorithm Overview. Our optimization algorithm consists of the precomputation and runtime. An iteration of our optimization
e at runtime consists of 3 steps. First, the system computes modal analysis to obtain the vibrational property of the object. Second, it
s the similarity score between the simulated sounds of the object and user specified target sounds. According to this similarity score,
m updates the material distribution inside the object to minimise the cost.
ed Work elset optimization, and controlled the several lowest eigenfrequen-
Simulated Sound
Material Optimization
New Material Distribution
Perceptual Differences
and Gradients
→ (f1, a1, d1)
(modal parameters)
→ (f2, a2, d2)
→ (f3, a3, d3)

34. Modal Analysis
Algorithm Overview
Data-Driven Online Coarsening (§7.1)
archical Component Mode Synthesis (§7.2)
Runtime
st Approximate Modal Analysis (§7)
The evaluation of the objective function
and Gradient Computation (§6)
+
Sample Position 1
User Input
(Target Sound)
Simulated Sound
Material Optimization (§6)
New Material Distribution
Similarity Score Computation (§5)
or Material Reduction (§6)
Used for
ain Decomposition (§7.2)
Eigenpairs
Sample Position 2
Sample Position 3
Timbre
Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization
Data-Driven Online Coarsening (§7.1)
Hierarchical Component Mode Synthesis (§7.2)
Runtime
Fast Approximate Modal Analysis (§7)
The evaluation of the objective function
and Gradient Computation (§6)
+
Sample Position 1
User Input
(Target Sound)
Simulated Sound
Material Optimization (§6)
New Material Distribution
Similarity Score Computation (§5)
Used for Material Reduction (§6)
Used for
Domain Decomposition (§7.2)
Eigenpairs
Sample Position 2
Sample Position 3
Timbre
The evaluation of th
and Gradient C
For Model
(Independent of Materials)
Input: Surface Mesh
Eigenvectors of Volumetric Mesh Laplacian
oxelization and Eigen Decomposition (§7.2)
Material Opti
New Material Distribution
Used for Material Reduction (§6)
Used for
Domain Decomposition (§7.2)
hm Overview. Our optimization algorithm consists of the precomputation and ru
me consists of 3 steps. First, the system computes modal analysis to obtain the vibr
arity score between the simulated sounds of the object and user specified target so
s the material distribution inside the object to minimise the cost.
A FEM Mesh
Target Sounds → Target Parameters
Kazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization
Data-Driven Online Coarsening (§7.1)
Hierarchical Component Mode Synthesis (§7.2)
Runtime
Fast Approximate Modal Analysis (§7)
The evaluation of the objective function
and Gradient Computation (§6)
+
…
For Model
(Independent of Materials)
Input: Surface Mesh
Eigenvectors of Volumetric Mesh Laplacian
Training of Mapping (§7.1.3)
For Material Set
(Independent of Models)
Voxelization and Eigen Decomposition (§7.2)
Sample Position 1
User Input
(Target Sound)
Simulated Sound
Material Optimization (§6)
New Material Distribution
Similarity Score Computation (§5)
Used for Material Reduction (§6)
Used for
Domain Decomposition (§7.2)
Eigenpairs
Sample Position 2
Sample Position 3
Timbre
Precomputation
Regression Forests
2: Algorithm Overview. Our optimization algorithm consists of the precomputation and runtime. An iteration of our optimization
e at runtime consists of 3 steps. First, the system computes modal analysis to obtain the vibrational property of the object. Second, it
s the similarity score between the simulated sounds of the object and user specified target sounds. According to this similarity score,
m updates the material distribution inside the object to minimise the cost.
ed Work elset optimization, and controlled the several lowest eigenfrequen-
Simulated Sound
Material Optimization
New Material Distribution
Perceptual Differences
and Gradients
→ (f1, a1, d1)
(modal parameters)
→ (f2, a2, d2)
→ (f3, a3, d3)

35. Modal Analysis
Algorithm Overview
Data-Driven Online Coarsening (§7.1)
archical Component Mode Synthesis (§7.2)
Runtime
st Approximate Modal Analysis (§7)
The evaluation of the objective function
and Gradient Computation (§6)
+
Sample Position 1
User Input
(Target Sound)
Simulated Sound
Material Optimization (§6)
New Material Distribution
Similarity Score Computation (§5)
or Material Reduction (§6)
Used for
ain Decomposition (§7.2)
Eigenpairs
Sample Position 2
Sample Position 3
Timbre
Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization
Data-Driven Online Coarsening (§7.1)
Hierarchical Component Mode Synthesis (§7.2)
Runtime
Fast Approximate Modal Analysis (§7)
The evaluation of the objective function
and Gradient Computation (§6)
+
Sample Position 1
User Input
(Target Sound)
Simulated Sound
Material Optimization (§6)
New Material Distribution
Similarity Score Computation (§5)
Used for Material Reduction (§6)
Used for
Domain Decomposition (§7.2)
Eigenpairs
Sample Position 2
Sample Position 3
Timbre
The evaluation of th
and Gradient C
For Model
(Independent of Materials)
Input: Surface Mesh
Eigenvectors of Volumetric Mesh Laplacian
oxelization and Eigen Decomposition (§7.2)
Material Opti
New Material Distribution
Used for Material Reduction (§6)
Used for
Domain Decomposition (§7.2)
hm Overview. Our optimization algorithm consists of the precomputation and ru
me consists of 3 steps. First, the system computes modal analysis to obtain the vibr
arity score between the simulated sounds of the object and user specified target so
s the material distribution inside the object to minimise the cost.
A FEM Mesh
Target Sounds → Target Parameters
Kazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization
Data-Driven Online Coarsening (§7.1)
Hierarchical Component Mode Synthesis (§7.2)
Runtime
Fast Approximate Modal Analysis (§7)
The evaluation of the objective function
and Gradient Computation (§6)
+
…
For Model
(Independent of Materials)
Input: Surface Mesh
Eigenvectors of Volumetric Mesh Laplacian
Training of Mapping (§7.1.3)
For Material Set
(Independent of Models)
Voxelization and Eigen Decomposition (§7.2)
Sample Position 1
User Input
(Target Sound)
Simulated Sound
Material Optimization (§6)
New Material Distribution
Similarity Score Computation (§5)
Used for Material Reduction (§6)
Used for
Domain Decomposition (§7.2)
Eigenpairs
Sample Position 2
Sample Position 3
Timbre
Precomputation
Regression Forests
2: Algorithm Overview. Our optimization algorithm consists of the precomputation and runtime. An iteration of our optimization
e at runtime consists of 3 steps. First, the system computes modal analysis to obtain the vibrational property of the object. Second, it
s the similarity score between the simulated sounds of the object and user specified target sounds. According to this similarity score,
m updates the material distribution inside the object to minimise the cost.
ed Work elset optimization, and controlled the several lowest eigenfrequen-
Simulated Sound
Material Optimization
New Material Distribution
Perceptual Differences
and Gradients
→ (f1, a1, d1)
(modal parameters)
→ (f2, a2, d2)
→ (f3, a3, d3)

36. Modal Analysis
Algorithm Overview
Data-Driven Online Coarsening (§7.1)
archical Component Mode Synthesis (§7.2)
Runtime
st Approximate Modal Analysis (§7)
The evaluation of the objective function
and Gradient Computation (§6)
+
Sample Position 1
User Input
(Target Sound)
Simulated Sound
Material Optimization (§6)
New Material Distribution
Similarity Score Computation (§5)
or Material Reduction (§6)
Used for
ain Decomposition (§7.2)
Eigenpairs
Sample Position 2
Sample Position 3
Timbre
Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization
Data-Driven Online Coarsening (§7.1)
Hierarchical Component Mode Synthesis (§7.2)
Runtime
Fast Approximate Modal Analysis (§7)
The evaluation of the objective function
and Gradient Computation (§6)
+
Sample Position 1
User Input
(Target Sound)
Simulated Sound
Material Optimization (§6)
New Material Distribution
Similarity Score Computation (§5)
Used for Material Reduction (§6)
Used for
Domain Decomposition (§7.2)
Eigenpairs
Sample Position 2
Sample Position 3
Timbre
The evaluation of th
and Gradient C
For Model
(Independent of Materials)
Input: Surface Mesh
Eigenvectors of Volumetric Mesh Laplacian
oxelization and Eigen Decomposition (§7.2)
Material Opti
New Material Distribution
Used for Material Reduction (§6)
Used for
Domain Decomposition (§7.2)
hm Overview. Our optimization algorithm consists of the precomputation and ru
me consists of 3 steps. First, the system computes modal analysis to obtain the vibr
arity score between the simulated sounds of the object and user specified target so
s the material distribution inside the object to minimise the cost.
A FEM Mesh
Target Sounds → Target Parameters
Kazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization
Data-Driven Online Coarsening (§7.1)
Hierarchical Component Mode Synthesis (§7.2)
Runtime
Fast Approximate Modal Analysis (§7)
The evaluation of the objective function
and Gradient Computation (§6)
+
…
For Model
(Independent of Materials)
Input: Surface Mesh
Eigenvectors of Volumetric Mesh Laplacian
Training of Mapping (§7.1.3)
For Material Set
(Independent of Models)
Voxelization and Eigen Decomposition (§7.2)
Sample Position 1
User Input
(Target Sound)
Simulated Sound
Material Optimization (§6)
New Material Distribution
Similarity Score Computation (§5)
Used for Material Reduction (§6)
Used for
Domain Decomposition (§7.2)
Eigenpairs
Sample Position 2
Sample Position 3
Timbre
Precomputation
Regression Forests
2: Algorithm Overview. Our optimization algorithm consists of the precomputation and runtime. An iteration of our optimization
e at runtime consists of 3 steps. First, the system computes modal analysis to obtain the vibrational property of the object. Second, it
s the similarity score between the simulated sounds of the object and user specified target sounds. According to this similarity score,
m updates the material distribution inside the object to minimise the cost.
ed Work elset optimization, and controlled the several lowest eigenfrequen-
Simulated Sound
Material Optimization
New Material Distribution
Perceptual Differences
and Gradients
→ (f1, a1, d1)
(modal parameters)
→ (f2, a2, d2)
→ (f3, a3, d3)

37. Modal Analysis
Algorithm Overview
Data-Driven Online Coarsening (§7.1)
archical Component Mode Synthesis (§7.2)
Runtime
st Approximate Modal Analysis (§7)
The evaluation of the objective function
and Gradient Computation (§6)
+
Sample Position 1
User Input
(Target Sound)
Simulated Sound
Material Optimization (§6)
New Material Distribution
Similarity Score Computation (§5)
or Material Reduction (§6)
Used for
ain Decomposition (§7.2)
Eigenpairs
Sample Position 2
Sample Position 3
Timbre
Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization
Data-Driven Online Coarsening (§7.1)
Hierarchical Component Mode Synthesis (§7.2)
Runtime
Fast Approximate Modal Analysis (§7)
The evaluation of the objective function
and Gradient Computation (§6)
+
Sample Position 1
User Input
(Target Sound)
Simulated Sound
Material Optimization (§6)
New Material Distribution
Similarity Score Computation (§5)
Used for Material Reduction (§6)
Used for
Domain Decomposition (§7.2)
Eigenpairs
Sample Position 2
Sample Position 3
Timbre
The evaluation of th
and Gradient C
For Model
(Independent of Materials)
Input: Surface Mesh
Eigenvectors of Volumetric Mesh Laplacian
oxelization and Eigen Decomposition (§7.2)
Material Opti
New Material Distribution
Used for Material Reduction (§6)
Used for
Domain Decomposition (§7.2)
hm Overview. Our optimization algorithm consists of the precomputation and ru
me consists of 3 steps. First, the system computes modal analysis to obtain the vibr
arity score between the simulated sounds of the object and user specified target so
s the material distribution inside the object to minimise the cost.
A FEM Mesh
Target Sounds → Target Parameters
Kazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization
Data-Driven Online Coarsening (§7.1)
Hierarchical Component Mode Synthesis (§7.2)
Runtime
Fast Approximate Modal Analysis (§7)
The evaluation of the objective function
and Gradient Computation (§6)
+
…
For Model
(Independent of Materials)
Input: Surface Mesh
Eigenvectors of Volumetric Mesh Laplacian
Training of Mapping (§7.1.3)
For Material Set
(Independent of Models)
Voxelization and Eigen Decomposition (§7.2)
Sample Position 1
User Input
(Target Sound)
Simulated Sound
Material Optimization (§6)
New Material Distribution
Similarity Score Computation (§5)
Used for Material Reduction (§6)
Used for
Domain Decomposition (§7.2)
Eigenpairs
Sample Position 2
Sample Position 3
Timbre
Precomputation
Regression Forests
2: Algorithm Overview. Our optimization algorithm consists of the precomputation and runtime. An iteration of our optimization
e at runtime consists of 3 steps. First, the system computes modal analysis to obtain the vibrational property of the object. Second, it
s the similarity score between the simulated sounds of the object and user specified target sounds. According to this similarity score,
m updates the material distribution inside the object to minimise the cost.
ed Work elset optimization, and controlled the several lowest eigenfrequen-
Simulated Sound
Material Optimization
New Material Distribution
Perceptual Differences
and Gradients
→ (f1, a1, d1)
(modal parameters)
→ (f2, a2, d2)
→ (f3, a3, d3)

38. Similar to [Bharaj et al. ’15].
Material Optimization
Frequency Differences
(Critical band rate)
ses at an interactive rate. The
distribution inside the model
hen the sample positions are
also can check the sound by
ce during optimization at any
ion procedure at an arbitrary
ew sample point, and restart
ay, the user can interactively
r a 3D object as if it were a
i
sented as a
pi
k = uT
k f
pi
n , where uk is the k-th eigenvector.
Using the target frequencies F, amplitudes A and simulated pa-
rameters, our objective function for minimizing the perceptual dif-
ference of the mode frequencies is represented as
E f =
1
2
N
∑
i=2
Bark(sf fi)−Bark(Fi)
2
(1)
where Bark(f) is a function to transform the frequency to critical
band rate [bark] [ZF99], and sf = F′
1/f1 is the scaling factor. The
ssociation.
Amplitude Differences
Kazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Ba
objective function for amplitudes is also obtained using the bal-
ances with other mode amplitudes at the position
Ea =
1
2
T
∑
j=1
N
∑
i=2
a
j
i
a
j
max
−
A
j
i
A
j
max
2
. (2)
where a
j
max and A
j
max denote the largest amplitude at the position
j of the simulated and target’s modes respectively. These formu-
lations are similar to [BLT∗
15]; however, we use the perceptual
metrics whereas they use square distances of frequencies and am-
plitudes. We minimize these functions by optimizing the Young’s
modulus Ye ∈ RM
at each finite element e, where M denotes the
number of the elements. Finally, our design problem is formulated
as
objective function for amplitudes is also obtained using the bal-
ances with other mode amplitudes at the position
Ea =
1
2
T
∑
j=1
N
∑
i=2
a
j
i
a
j
max
−
A
j
i
A
j
max
2
. (2)
where a
j
max and A
j
max denote the largest amplitude at the position
j of the simulated and target’s modes respectively. These formu-
lations are similar to [BLT∗
15]; however, we use the perceptual
metrics whereas they use square distances of frequencies and am-
plitudes. We minimize these functions by optimizing the Young’s
modulus Ye ∈ RM
at each finite element e, where M denotes the
number of the elements. Finally, our design problem is formulated
as
argmin
Ye
: wf E f +waEa, sub ject to : Ye > 0 (3)
where wf and wa denote the positive weights.
Note that we do not optimize damping parameters. We instead
× ×
× ×
e1
e3
16 Cubatu
4 fine mater
× ×
× ×
Figure 4: Data-Driven
tration). The function
(e1,e2,e3,e4) of fine fo
four coarse material pa
points (right) to minim
ReflectedVariations
e1 e2
e4 e3
e1 e2
e4 e3
=
Figure 5: Eight equiva
We minimize:
Y: Young’s modulus
N: optimized modes, T: sample positions
We minimize perceptual differences
between simulated and target modal parameters.

39. Similar to [Bharaj et al. ’15].
Material Optimization
Frequency Differences
(Critical band rate)
ses at an interactive rate. The
distribution inside the model
hen the sample positions are
also can check the sound by
ce during optimization at any
ion procedure at an arbitrary
ew sample point, and restart
ay, the user can interactively
r a 3D object as if it were a
i
sented as a
pi
k = uT
k f
pi
n , where uk is the k-th eigenvector.
Using the target frequencies F, amplitudes A and simulated pa-
rameters, our objective function for minimizing the perceptual dif-
ference of the mode frequencies is represented as
E f =
1
2
N
∑
i=2
Bark(sf fi)−Bark(Fi)
2
(1)
where Bark(f) is a function to transform the frequency to critical
band rate [bark] [ZF99], and sf = F′
1/f1 is the scaling factor. The
ssociation.
Amplitude Differences
Kazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Ba
objective function for amplitudes is also obtained using the bal-
ances with other mode amplitudes at the position
Ea =
1
2
T
∑
j=1
N
∑
i=2
a
j
i
a
j
max
−
A
j
i
A
j
max
2
. (2)
where a
j
max and A
j
max denote the largest amplitude at the position
j of the simulated and target’s modes respectively. These formu-
lations are similar to [BLT∗
15]; however, we use the perceptual
metrics whereas they use square distances of frequencies and am-
plitudes. We minimize these functions by optimizing the Young’s
modulus Ye ∈ RM
at each finite element e, where M denotes the
number of the elements. Finally, our design problem is formulated
as
objective function for amplitudes is also obtained using the bal-
ances with other mode amplitudes at the position
Ea =
1
2
T
∑
j=1
N
∑
i=2
a
j
i
a
j
max
−
A
j
i
A
j
max
2
. (2)
where a
j
max and A
j
max denote the largest amplitude at the position
j of the simulated and target’s modes respectively. These formu-
lations are similar to [BLT∗
15]; however, we use the perceptual
metrics whereas they use square distances of frequencies and am-
plitudes. We minimize these functions by optimizing the Young’s
modulus Ye ∈ RM
at each finite element e, where M denotes the
number of the elements. Finally, our design problem is formulated
as
argmin
Ye
: wf E f +waEa, sub ject to : Ye > 0 (3)
where wf and wa denote the positive weights.
Note that we do not optimize damping parameters. We instead
× ×
× ×
e1
e3
16 Cubatu
4 fine mater
× ×
× ×
Figure 4: Data-Driven
tration). The function
(e1,e2,e3,e4) of fine fo
four coarse material pa
points (right) to minim
ReflectedVariations
e1 e2
e4 e3
e1 e2
e4 e3
=
Figure 5: Eight equiva
We minimize:
Y: Young’s modulus
N: optimized modes, T: sample positions
We minimize perceptual differences
between simulated and target modal parameters.

40. Similar to [Bharaj et al. ’15].
Material Optimization
Frequency Differences
(Critical band rate)
ses at an interactive rate. The
distribution inside the model
hen the sample positions are
also can check the sound by
ce during optimization at any
ion procedure at an arbitrary
ew sample point, and restart
ay, the user can interactively
r a 3D object as if it were a
i
sented as a
pi
k = uT
k f
pi
n , where uk is the k-th eigenvector.
Using the target frequencies F, amplitudes A and simulated pa-
rameters, our objective function for minimizing the perceptual dif-
ference of the mode frequencies is represented as
E f =
1
2
N
∑
i=2
Bark(sf fi)−Bark(Fi)
2
(1)
where Bark(f) is a function to transform the frequency to critical
band rate [bark] [ZF99], and sf = F′
1/f1 is the scaling factor. The
ssociation.
Amplitude Differences
Kazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Ba
objective function for amplitudes is also obtained using the bal-
ances with other mode amplitudes at the position
Ea =
1
2
T
∑
j=1
N
∑
i=2
a
j
i
a
j
max
−
A
j
i
A
j
max
2
. (2)
where a
j
max and A
j
max denote the largest amplitude at the position
j of the simulated and target’s modes respectively. These formu-
lations are similar to [BLT∗
15]; however, we use the perceptual
metrics whereas they use square distances of frequencies and am-
plitudes. We minimize these functions by optimizing the Young’s
modulus Ye ∈ RM
at each finite element e, where M denotes the
number of the elements. Finally, our design problem is formulated
as
objective function for amplitudes is also obtained using the bal-
ances with other mode amplitudes at the position
Ea =
1
2
T
∑
j=1
N
∑
i=2
a
j
i
a
j
max
−
A
j
i
A
j
max
2
. (2)
where a
j
max and A
j
max denote the largest amplitude at the position
j of the simulated and target’s modes respectively. These formu-
lations are similar to [BLT∗
15]; however, we use the perceptual
metrics whereas they use square distances of frequencies and am-
plitudes. We minimize these functions by optimizing the Young’s
modulus Ye ∈ RM
at each finite element e, where M denotes the
number of the elements. Finally, our design problem is formulated
as
argmin
Ye
: wf E f +waEa, sub ject to : Ye > 0 (3)
where wf and wa denote the positive weights.
Note that we do not optimize damping parameters. We instead
× ×
× ×
e1
e3
16 Cubatu
4 fine mater
× ×
× ×
Figure 4: Data-Driven
tration). The function
(e1,e2,e3,e4) of fine fo
four coarse material pa
points (right) to minim
ReflectedVariations
e1 e2
e4 e3
e1 e2
e4 e3
=
Figure 5: Eight equiva
We minimize:
Y: Young’s modulus
N: optimized modes, T: sample positions
We minimize perceptual differences
between simulated and target modal parameters.

41. Similar to [Bharaj et al. ’15].
Material Optimization
Frequency Differences
(Critical band rate)
ses at an interactive rate. The
distribution inside the model
hen the sample positions are
also can check the sound by
ce during optimization at any
ion procedure at an arbitrary
ew sample point, and restart
ay, the user can interactively
r a 3D object as if it were a
i
sented as a
pi
k = uT
k f
pi
n , where uk is the k-th eigenvector.
Using the target frequencies F, amplitudes A and simulated pa-
rameters, our objective function for minimizing the perceptual dif-
ference of the mode frequencies is represented as
E f =
1
2
N
∑
i=2
Bark(sf fi)−Bark(Fi)
2
(1)
where Bark(f) is a function to transform the frequency to critical
band rate [bark] [ZF99], and sf = F′
1/f1 is the scaling factor. The
ssociation.
Amplitude Differences
Kazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Ba
objective function for amplitudes is also obtained using the bal-
ances with other mode amplitudes at the position
Ea =
1
2
T
∑
j=1
N
∑
i=2
a
j
i
a
j
max
−
A
j
i
A
j
max
2
. (2)
where a
j
max and A
j
max denote the largest amplitude at the position
j of the simulated and target’s modes respectively. These formu-
lations are similar to [BLT∗
15]; however, we use the perceptual
metrics whereas they use square distances of frequencies and am-
plitudes. We minimize these functions by optimizing the Young’s
modulus Ye ∈ RM
at each finite element e, where M denotes the
number of the elements. Finally, our design problem is formulated
as
objective function for amplitudes is also obtained using the bal-
ances with other mode amplitudes at the position
Ea =
1
2
T
∑
j=1
N
∑
i=2
a
j
i
a
j
max
−
A
j
i
A
j
max
2
. (2)
where a
j
max and A
j
max denote the largest amplitude at the position
j of the simulated and target’s modes respectively. These formu-
lations are similar to [BLT∗
15]; however, we use the perceptual
metrics whereas they use square distances of frequencies and am-
plitudes. We minimize these functions by optimizing the Young’s
modulus Ye ∈ RM
at each finite element e, where M denotes the
number of the elements. Finally, our design problem is formulated
as
argmin
Ye
: wf E f +waEa, sub ject to : Ye > 0 (3)
where wf and wa denote the positive weights.
Note that we do not optimize damping parameters. We instead
× ×
× ×
e1
e3
16 Cubatu
4 fine mater
× ×
× ×
Figure 4: Data-Driven
tration). The function
(e1,e2,e3,e4) of fine fo
four coarse material pa
points (right) to minim
ReflectedVariations
e1 e2
e4 e3
e1 e2
e4 e3
=
Figure 5: Eight equiva
We minimize:
Y: Young’s modulus
N: optimized modes, T: sample positions
We minimize perceptual differences
between simulated and target modal parameters.

42. Material Optimization
We solve this problem by material reduction [Xu et al. ’15],
and a hybrid optimization scheme of evolutional strategies (CMA-ES)
and gradient descent (quasi-Newton).
where amax and Amax denote the largest amplitude at the position
j of the simulated and target’s modes respectively. These formu-
lations are similar to [BLT∗
15]; however, we use the perceptual
metrics whereas they use square distances of frequencies and am-
plitudes. We minimize these functions by optimizing the Young’s
modulus Ye ∈ RM
at each finite element e, where M denotes the
number of the elements. Finally, our design problem is formulated
as
argmin
Ye
: wf E f +waEa, sub ject to : Ye > 0 (3)
where wf and wa denote the positive weights.
Note that we do not optimize damping parameters. We instead
reuse the estimated damping from the assigned sound clips as
mode-dependent damping. This means that our damping is not spa-
tially constant. This setting is physically incorrect, but it makes the
problem simpler.
tration). The functio
(e1,e2,e3,e4) of fine
four coarse material
points (right) to mini
ReflectedVariations
e1 e
e4 e
e1 e
e4 e
=
Figure 5: Eight equi
top row represents fo
resents four reflected
elements) as input a
cube element) as ou
We minimize:
Y: Young’s modulus

43. Modal Analysis
Algorithm Overview
Data-Driven Online Coarsening (§7.1)
archical Component Mode Synthesis (§7.2)
Runtime
st Approximate Modal Analysis (§7)
The evaluation of the objective function
and Gradient Computation (§6)
+
Sample Position 1
User Input
(Target Sound)
Simulated Sound
Material Optimization (§6)
New Material Distribution
Similarity Score Computation (§5)
or Material Reduction (§6)
Used for
ain Decomposition (§7.2)
Eigenpairs
Sample Position 2
Sample Position 3
Timbre
Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization
Data-Driven Online Coarsening (§7.1)
Hierarchical Component Mode Synthesis (§7.2)
Runtime
Fast Approximate Modal Analysis (§7)
The evaluation of the objective function
and Gradient Computation (§6)
+
Sample Position 1
User Input
(Target Sound)
Simulated Sound
Material Optimization (§6)
New Material Distribution
Similarity Score Computation (§5)
Used for Material Reduction (§6)
Used for
Domain Decomposition (§7.2)
Eigenpairs
Sample Position 2
Sample Position 3
Timbre
The evaluation of th
and Gradient C
For Model
(Independent of Materials)
Input: Surface Mesh
Eigenvectors of Volumetric Mesh Laplacian
oxelization and Eigen Decomposition (§7.2)
Material Opti
New Material Distribution
Used for Material Reduction (§6)
Used for
Domain Decomposition (§7.2)
hm Overview. Our optimization algorithm consists of the precomputation and ru
me consists of 3 steps. First, the system computes modal analysis to obtain the vibr
arity score between the simulated sounds of the object and user specified target so
s the material distribution inside the object to minimise the cost.
A FEM Mesh
Target Sounds → Target Parameters
Kazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization
Data-Driven Online Coarsening (§7.1)
Hierarchical Component Mode Synthesis (§7.2)
Runtime
Fast Approximate Modal Analysis (§7)
The evaluation of the objective function
and Gradient Computation (§6)
+
…
For Model
(Independent of Materials)
Input: Surface Mesh
Eigenvectors of Volumetric Mesh Laplacian
Training of Mapping (§7.1.3)
For Material Set
(Independent of Models)
Voxelization and Eigen Decomposition (§7.2)
Sample Position 1
User Input
(Target Sound)
Simulated Sound
Material Optimization (§6)
New Material Distribution
Similarity Score Computation (§5)
Used for Material Reduction (§6)
Used for
Domain Decomposition (§7.2)
Eigenpairs
Sample Position 2
Sample Position 3
Timbre
Precomputation
Regression Forests
2: Algorithm Overview. Our optimization algorithm consists of the precomputation and runtime. An iteration of our optimization
e at runtime consists of 3 steps. First, the system computes modal analysis to obtain the vibrational property of the object. Second, it
s the similarity score between the simulated sounds of the object and user specified target sounds. According to this similarity score,
m updates the material distribution inside the object to minimise the cost.
ed Work elset optimization, and controlled the several lowest eigenfrequen-
Simulated Sound
Material Optimization
New Material Distribution
Perceptual Differences
and Gradients
→ (f1, a1, d1)
(modal parameters)
→ (f2, a2, d2)
→ (f3, a3, d3)

44. Modal Analysis
Algorithm Overview
Data-Driven Online Coarsening (§7.1)
archical Component Mode Synthesis (§7.2)
Runtime
st Approximate Modal Analysis (§7)
The evaluation of the objective function
and Gradient Computation (§6)
+
Sample Position 1
User Input
(Target Sound)
Simulated Sound
Material Optimization (§6)
New Material Distribution
Similarity Score Computation (§5)
or Material Reduction (§6)
Used for
ain Decomposition (§7.2)
Eigenpairs
Sample Position 2
Sample Position 3
Timbre
Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization
Data-Driven Online Coarsening (§7.1)
Hierarchical Component Mode Synthesis (§7.2)
Runtime
Fast Approximate Modal Analysis (§7)
The evaluation of the objective function
and Gradient Computation (§6)
+
Sample Position 1
User Input
(Target Sound)
Simulated Sound
Material Optimization (§6)
New Material Distribution
Similarity Score Computation (§5)
Used for Material Reduction (§6)
Used for
Domain Decomposition (§7.2)
Eigenpairs
Sample Position 2
Sample Position 3
Timbre
The evaluation of th
and Gradient C
For Model
(Independent of Materials)
Input: Surface Mesh
Eigenvectors of Volumetric Mesh Laplacian
oxelization and Eigen Decomposition (§7.2)
Material Opti
New Material Distribution
Used for Material Reduction (§6)
Used for
Domain Decomposition (§7.2)
hm Overview. Our optimization algorithm consists of the precomputation and ru
me consists of 3 steps. First, the system computes modal analysis to obtain the vibr
arity score between the simulated sounds of the object and user specified target so
s the material distribution inside the object to minimise the cost.
A FEM Mesh
Target Sounds → Target Parameters
Kazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization
Data-Driven Online Coarsening (§7.1)
Hierarchical Component Mode Synthesis (§7.2)
Runtime
Fast Approximate Modal Analysis (§7)
The evaluation of the objective function
and Gradient Computation (§6)
+
…
For Model
(Independent of Materials)
Input: Surface Mesh
Eigenvectors of Volumetric Mesh Laplacian
Training of Mapping (§7.1.3)
For Material Set
(Independent of Models)
Voxelization and Eigen Decomposition (§7.2)
Sample Position 1
User Input
(Target Sound)
Simulated Sound
Material Optimization (§6)
New Material Distribution
Similarity Score Computation (§5)
Used for Material Reduction (§6)
Used for
Domain Decomposition (§7.2)
Eigenpairs
Sample Position 2
Sample Position 3
Timbre
Precomputation
Regression Forests
2: Algorithm Overview. Our optimization algorithm consists of the precomputation and runtime. An iteration of our optimization
e at runtime consists of 3 steps. First, the system computes modal analysis to obtain the vibrational property of the object. Second, it
s the similarity score between the simulated sounds of the object and user specified target sounds. According to this similarity score,
m updates the material distribution inside the object to minimise the cost.
ed Work elset optimization, and controlled the several lowest eigenfrequen-
Simulated Sound
Material Optimization
New Material Distribution
Perceptual Differences
and Gradients
→ (f1, a1, d1)
(modal parameters)
→ (f2, a2, d2)
→ (f3, a3, d3)
KU = ΛMU
K: Stiffness Matrix
M: Mass Matrix
Expensive

45. Data-Driven Online Coarsening (§7.1)
Hierarchical Component Mode Synthesis (§7.2)
Fast Approximate Modal Analysis (§7)
+
…
For Model
(Independent of Materials)
Input: Surface Mesh
Training of Mapping (§7.1.3)
For Material Set
(Independent of Models)
Voxelization and Eigen Decomposition (§7.2)
Used for
Domain Decomposition (§7.2)
Regression Forests
arashi / Interactive Physically-Based Sound Design of 3D Modelusing
Data-Driven Online Coarsening (§7.1)
Hierarchical Component Mode Synthesis (§7.2)
Runtime
Fast Approximate Modal Analysis (§7)
+
S
Simulated Sound
Similarity Score Compu
Eigenpairs
S
S
Mesh Coarsening with Data-Driven FEM
Fast Approximate Modal Analysis
Hierarchical Component Mode Synthesis
Data-Driven Online Coarsening (§7.1)
Hierarchical Component Mode Synthesis (§7.2)
Fast Approximate Modal Analysis (§7)
+
…
For Model
(Independent of Materials)
Input: Surface Mesh
Training of Mapping (§7.1.3)
For Material Set
(Independent of Models)
Voxelization and Eigen Decomposition (§7.2)
Used for
Domain Decomposition (§7.2)
Eigen
Regression Forests
+

46. Data-Driven FEM using Regression Forests
Data-Driven Online Coarsening (§7.1)
Hierarchical Component Mode Synthesis (§7.2)
Run
Fast Approximate Modal Analysis (§7)
+
Eigenpairs
Data-Driven Online Coarsening (§7.1)
Hierarchical Component Mode Synthesis (§7.2)
Fast Approximate Modal Analysis (§7)
+
…
ing (§7.1.3)
ial Set
of Models)
E
utation
Forests
Machine Learning
Detailed mesh Coarse mesh
Based on “Data-Driven Finite Elements for Geometry and Material Design” ,
Chen et al. SIGGRAPH 2015

47. Data-Driven FEM (DDFEM) [Chen et al. 15]ound Design of 3D Modelusing Material Optimization
4 Cubature Points
× ×
× ×
× ×
× ×
DDFEM(e1,…,e4)
e1 e2
e3 e4
E1 E2
E3 E4
16 Cubature Points
4 fine materials (in 2D) 4 coarse materials
× ×
× ×
× ×
× ×
× ×
× ×
ure 4: Data-Driven Coarsening [Chen et al. 2015] (in 2D
on). The function DDFEM() takes four material param
e2,e3,e4) of fine four elements (left) and returns correspon
on meshes.
evels of Coarsening
23x Faster
(b)
and figure. Notice that, after 1 level of coarsen
ing neither compresses nor buckles as much as
High-Res Simulation. After 2 levels of coarseni
havior is lost. The Na¨ıve Coarsening fails to cap
behavior of High-Res Simulation, whereas DDF
1 Level of Coarsening 2 Levels o
51x Faster 489x
(a)
“Data-Driven Finite Elements for Geometry and Material Design” ,
Chen et al. SIGGRAPH 2015

48. Data-Driven FEM (DDFEM) [Chen et al. 15]
Given N materials, the number of combinations become N8
It is impractical to use for our problem,
because it requires a large range of continuous material settings.

49. Extended Data-Driven FEM (DDFEM*)
To Reduce the material space
1. Overlapping-Free Cell Ordering
2. Scaling Factor Separation
3. Regression Forests To handle large amount of dataset
with a constant evaluation cost

50. 1. Overlapping-Free Cell Ordering(e1,e2,e3,e4) of fine four elements (left) and returns corresponding
four coarse material parameters (E1,E2,E3,E4) at the quadrature
points (right) to minimize the error.
= = =
= = =
90° 90° 90°ReflectedVariations
Rotated Variations
e1 e2
e4 e3
e1 e2
e4 e3
e1 e2
e4 e3
e1 e2
e4 e3
e1 e2
e4 e3
e1 e2
e4 e3
e1 e2
e4 e3
e1 e2
e4 e3
=
=
=
=
Figure 5: Eight equivalent cell variations (in 2D illustration). The
top row represents four rotated variations and the bottom row rep-
Basically equivalent cell variations (in 2D)
should be emitted from the dataset.

51. 1. Overlapping-Free Cell Ordering
Kazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Based S
The Origin Cell (i 1)
e1 e’1
1: Initial ordering
Reordering (i = 1)
e1 e2
e4 e3
e’1 e2
e4 e3
2: Compare the neighbors
e’1 e’2
e’3 e’4
e’1 e’3
e’2 e’4
e’1 e’2
e’3 e3
e’1 e’3
e’2 e3
3: If e2 e4
3: otherwise
4: Fill rests
Figure 6: Overlapping Free Cell Ordering (in 2D illustration). 1:
At the i-th cell evaluation (in this example, we assume i = 2), ei
becomes the origin cell e′
1. 2: we compare the material values of
the adjacent cells. 3: the smaller cell becomes e′
2 and the other
becomes e′
3. 4: The left cell becomes e′
4.
sK(
of e
is eq
Thi
DD
cell
DD
coordinate. At the i-th evaluation within the eight eva
define the i-th cell as the origin e′
1. Then, we compare
the Young’s modulus of the three adjacent cells of the o
2D, two cells), and define the index the cell who has
value as e′
2, the cell who has the secondary smallest val
the other cell as e′
4. Finally, we decide the ordering of
cells by the following rule: The cell that is adjacent
becomes e′
5. The cell that is adjacent to e′
3 and e′
4 beco
cell that is adjacent to e′
2 and e′
4 becomes e′
7. The last o
e′
8.
Then, using these reordered parameters, our DDFEM
is redefined again as
Ei = DDFEMi(e′
1,e′
2,...,e′
8), e′
1 = ei
By using this representation, we can avoid explicit enu
the eight rotated and eight reflected patterns of a mate
sorted by the Young’s modulus.
Please see the paper for the details of the sorting procedure.
= = =
ReflectedV
e1 e2
e4 e3
e1 e2
e4 e3
e1 e2
e4 e3
e1 e2
e4 e3
=
=
=
=
Eight equivalent cell variations (in 2D illustration). The
presents four rotated variations and the bottom row rep-
r reflected variations.
as input and generates a coarse approximated mesh (a
ent) as output using the the material parameter map-
ed from training data in the precomputation step (Fig-
e concept of our data driven FEM coarsening is based
15]. The goal of their data-driven FEM is obtaining
(E1,...,E8) = DDFEM(e1,..,e8) (5)
FEM() is a function that takes eight material parame-
,e8) of a detailed mesh and returns the corresponding
e material parameters (E ,...,E ) at the cubature points

52. Young’s modulus has a large range of the value (10-2 ~ 103 GPa)
2. Scaling Factor Separation
It is difficult to treat such a large
range by DDFEM directly !!

53. 2. Scaling Factor Separation
such a large range during training. To avoid this, we dramatically
reduce the training size by separating the scale factor.
Based on [CLSM15], our DDFEM() is constructed to minimize
the square difference of the integral of the strain energy density
functions between the detailed and coarse meshes.
argmin
Ei
∑
f∈F
8
∑
i=1
wivc
i ( f,Ei)−
8
∑
j=1
8
∑
i=1
wivd
ji(f,e′
j)
2
(8)
where w denotes the cubature weights, F denotes a set of randomly
sampled external forces, and vc
and vd
represent the strain energy
density function of the coarse and detailed mesh respectively. Here,
the strain energy density function in linear elastic is represented
F: force set SED of coarse mesh
DDFEM() is designed to minimize the square difference of the integral
of the strain energy density (SED).
SED of fine mesh

54. 2. Scaling Factor Separation

55. 2. Scaling Factor Separation
In linear elastics:
on of the coarse and detailed mesh respectively. Here,
rgy density function in linear elastic is represented
(e)u(e)2
= K(e)(K−1
(e) f)2
where K(e) and f are
matrix and the external forces respectively. In addi-
ng e by a scalar s, v( f,s · e) = K(s · e)u(s · e)2
=
into seve
into finer
the comp
pensate f
correctio
s)
ings c⃝ 2016 The Eurographics Association.
cally-Based Sound Design of 3D Modelusing Material Optimization
sK(e)((sK(e))−1
f)2
= v( f,e)/s because K() is the line
of e. Then, the minimization problem
argmin
Ei
∑
f∈F
8
∑
i=1
wivc
i ( f,Ei)−
8
∑
j=1
8
∑
i=1
wivd
ji( f,s·e
is equivalent to
8 8 8
s: scholar

56. 2. Scaling Factor Separation
In linear elastics:
on of the coarse and detailed mesh respectively. Here,
rgy density function in linear elastic is represented
(e)u(e)2
= K(e)(K−1
(e) f)2
where K(e) and f are
matrix and the external forces respectively. In addi-
ng e by a scalar s, v( f,s · e) = K(s · e)u(s · e)2
=
into seve
into finer
the comp
pensate f
correctio
s)
ings c⃝ 2016 The Eurographics Association.
cally-Based Sound Design of 3D Modelusing Material Optimization
sK(e)((sK(e))−1
f)2
= v( f,e)/s because K() is the line
of e. Then, the minimization problem
argmin
Ei
∑
f∈F
8
∑
i=1
wivc
i ( f,Ei)−
8
∑
j=1
8
∑
i=1
wivd
ji( f,s·e
is equivalent to
8 8 8
s: scholar
Then,
e’1 e’3
e’2 e’4
e’1 e’3
e’2 e3
3: otherwise
Ordering (in 2D illustration). 1:
s example, we assume i = 2), ei
e compare the material values of
r cell becomes e′
2 and the other
mes e′
4.
turns a scalar while Eq.5 outputs
this DDFEM() evaluation eight
2 × 2 element into a coarse ele-
bering of the eight cells by each
this operation as a 2D example in
s are defined in a local R3
space
Ei f∈F i=1 j=1i=1
is equivalent to
argmin
E′
i =Ei/s
∑
f∈F
8
∑
i=1
wivc
i (f,E′
i )−
8
∑
j=1
8
∑
i=1
wivd
ji(f,e′
This means that we can separate the input paramete
DDFEM() problem by the multiplication of the valu
cell as a scale factor and their quotients. Finally, we c
DDFEM() function as
Ei = ei ·DDFEMi
e′
2
ei
,...,
e′
8
ei
An advantage of this representation is that it reduce
range of dataset but also the dimensions of the featur
R8
to R7
. Note that we assume our model as linear ela
the original DDFEM() treats nonlinearity because t
is equivalent to
ells), and define the index the cell who has the smallest
, the cell who has the secondary smallest value as e′
3, and
ell as e′
4. Finally, we decide the ordering of the rest four
e following rule: The cell that is adjacent to e′
2 and e′
3
′
5. The cell that is adjacent to e′
3 and e′
4 becomes e′
6. The
adjacent to e′
2 and e′
4 becomes e′
7. The last one becomes
ing these reordered parameters, our DDFEM() function
d again as
Ei = DDFEMi(e′
1,e′
2,...,e′
8), e′
1 = ei (7)
his representation, we can avoid explicit enumeration of
otated and eight reflected patterns of a material pattern,
e the input parameter space in 3D at both training and
or dataset generation at the training, we first determine
7.1.3. Regression Fore
In contract with Chen
struct the database of
sons. First, their databa
are not included in the
ization ability. Second,
a rate proportional to th
of the dataset should be
terns. To address these
tion using two regressio
to [LJS∗
15] which con
tree structure constructi
square solve for the reg
all the dataset. The reg
cost evaluation even if

57. 2. Scaling Factor Separation
In linear elastics:
on of the coarse and detailed mesh respectively. Here,
rgy density function in linear elastic is represented
(e)u(e)2
= K(e)(K−1
(e) f)2
where K(e) and f are
matrix and the external forces respectively. In addi-
ng e by a scalar s, v( f,s · e) = K(s · e)u(s · e)2
=
into seve
into finer
the comp
pensate f
correctio
s)
ings c⃝ 2016 The Eurographics Association.
cally-Based Sound Design of 3D Modelusing Material Optimization
sK(e)((sK(e))−1
f)2
= v( f,e)/s because K() is the line
of e. Then, the minimization problem
argmin
Ei
∑
f∈F
8
∑
i=1
wivc
i ( f,Ei)−
8
∑
j=1
8
∑
i=1
wivd
ji( f,s·e
is equivalent to
8 8 8
s: scholar
Then,
e’1 e’3
e’2 e’4
e’1 e’3
e’2 e3
3: otherwise
Ordering (in 2D illustration). 1:
s example, we assume i = 2), ei
e compare the material values of
r cell becomes e′
2 and the other
mes e′
4.
turns a scalar while Eq.5 outputs
this DDFEM() evaluation eight
2 × 2 element into a coarse ele-
bering of the eight cells by each
this operation as a 2D example in
s are defined in a local R3
space
Ei f∈F i=1 j=1i=1
is equivalent to
argmin
E′
i =Ei/s
∑
f∈F
8
∑
i=1
wivc
i (f,E′
i )−
8
∑
j=1
8
∑
i=1
wivd
ji(f,e′
This means that we can separate the input paramete
DDFEM() problem by the multiplication of the valu
cell as a scale factor and their quotients. Finally, we c
DDFEM() function as
Ei = ei ·DDFEMi
e′
2
ei
,...,
e′
8
ei
An advantage of this representation is that it reduce
range of dataset but also the dimensions of the featur
R8
to R7
. Note that we assume our model as linear ela
the original DDFEM() treats nonlinearity because t
is equivalent to
ells), and define the index the cell who has the smallest
, the cell who has the secondary smallest value as e′
3, and
ell as e′
4. Finally, we decide the ordering of the rest four
e following rule: The cell that is adjacent to e′
2 and e′
3
′
5. The cell that is adjacent to e′
3 and e′
4 becomes e′
6. The
adjacent to e′
2 and e′
4 becomes e′
7. The last one becomes
ing these reordered parameters, our DDFEM() function
d again as
Ei = DDFEMi(e′
1,e′
2,...,e′
8), e′
1 = ei (7)
his representation, we can avoid explicit enumeration of
otated and eight reflected patterns of a material pattern,
e the input parameter space in 3D at both training and
or dataset generation at the training, we first determine
7.1.3. Regression Fore
In contract with Chen
struct the database of
sons. First, their databa
are not included in the
ization ability. Second,
a rate proportional to th
of the dataset should be
terns. To address these
tion using two regressio
to [LJS∗
15] which con
tree structure constructi
square solve for the reg
all the dataset. The reg
cost evaluation even if

58. 2. Scaling Factor Separation
In linear elastics:
on of the coarse and detailed mesh respectively. Here,
rgy density function in linear elastic is represented
(e)u(e)2
= K(e)(K−1
(e) f)2
where K(e) and f are
matrix and the external forces respectively. In addi-
ng e by a scalar s, v( f,s · e) = K(s · e)u(s · e)2
=
into seve
into finer
the comp
pensate f
correctio
s)
ings c⃝ 2016 The Eurographics Association.
cally-Based Sound Design of 3D Modelusing Material Optimization
sK(e)((sK(e))−1
f)2
= v( f,e)/s because K() is the line
of e. Then, the minimization problem
argmin
Ei
∑
f∈F
8
∑
i=1
wivc
i ( f,Ei)−
8
∑
j=1
8
∑
i=1
wivd
ji( f,s·e
is equivalent to
8 8 8
s: scholar
Then,
e’1 e’3
e’2 e’4
e’1 e’3
e’2 e3
3: otherwise
Ordering (in 2D illustration). 1:
s example, we assume i = 2), ei
e compare the material values of
r cell becomes e′
2 and the other
mes e′
4.
turns a scalar while Eq.5 outputs
this DDFEM() evaluation eight
2 × 2 element into a coarse ele-
bering of the eight cells by each
this operation as a 2D example in
s are defined in a local R3
space
Ei f∈F i=1 j=1i=1
is equivalent to
argmin
E′
i =Ei/s
∑
f∈F
8
∑
i=1
wivc
i (f,E′
i )−
8
∑
j=1
8
∑
i=1
wivd
ji(f,e′
This means that we can separate the input paramete
DDFEM() problem by the multiplication of the valu
cell as a scale factor and their quotients. Finally, we c
DDFEM() function as
Ei = ei ·DDFEMi
e′
2
ei
,...,
e′
8
ei
An advantage of this representation is that it reduce
range of dataset but also the dimensions of the featur
R8
to R7
. Note that we assume our model as linear ela
the original DDFEM() treats nonlinearity because t
is equivalent to
ells), and define the index the cell who has the smallest
, the cell who has the secondary smallest value as e′
3, and
ell as e′
4. Finally, we decide the ordering of the rest four
e following rule: The cell that is adjacent to e′
2 and e′
3
′
5. The cell that is adjacent to e′
3 and e′
4 becomes e′
6. The
adjacent to e′
2 and e′
4 becomes e′
7. The last one becomes
ing these reordered parameters, our DDFEM() function
d again as
Ei = DDFEMi(e′
1,e′
2,...,e′
8), e′
1 = ei (7)
his representation, we can avoid explicit enumeration of
otated and eight reflected patterns of a material pattern,
e the input parameter space in 3D at both training and
or dataset generation at the training, we first determine
7.1.3. Regression Fore
In contract with Chen
struct the database of
sons. First, their databa
are not included in the
ization ability. Second,
a rate proportional to th
of the dataset should be
terns. To address these
tion using two regressio
to [LJS∗
15] which con
tree structure constructi
square solve for the reg
all the dataset. The reg
cost evaluation even if
(i 1)
e’1 e’3e’1 e’3
verlapping Free Cell Ordering (in 2D illustration). 1:
ell evaluation (in this example, we assume i = 2), ei
origin cell e′
1. 2: we compare the material values of
cells. 3: the smaller cell becomes e′
2 and the other
4: The left cell becomes e′
4.
ven FEM function returns a scalar while Eq.5 outputs
It means we repeat this DDFEM() evaluation eight
nvert a detailed 2 × 2 × 2 element into a coarse ele-
we reorder the numbering of the eight cells by each
valuation. We show this operation as a 2D example in
e indices of the cells are defined in a local R3
space
argmin
E′
i =Ei/s
∑
f∈F
8
∑
i=
This means that we
DDFEM() problem
cell as a scale factor
DDFEM() function
Ei
An advantage of thi
range of dataset but
R8
to R7
. Note that w
the original DDFEM
Cell (i 1)
e’1
e’1 e’3
e’2 e’4
e’1 e’3
e’2 e3
: Overlapping Free Cell Ordering (in 2D illustration). 1:
h cell evaluation (in this example, we assume i = 2), ei
the origin cell e′
1. 2: we compare the material values of
cent cells. 3: the smaller cell becomes e′
2 and the other
e′
3. 4: The left cell becomes e′
4.
-driven FEM function returns a scalar while Eq.5 outputs
tor. It means we repeat this DDFEM() evaluation eight
convert a detailed 2 × 2 × 2 element into a coarse ele-
xt, we reorder the numbering of the eight cells by each
() evaluation. We show this operation as a 2D example in
The indices of the cells are defined in a local R3
space
is equivalent to
argmin
E′
i =Ei/s
∑
f∈F
8
∑
i=1
w
This means that we ca
DDFEM() problem by
cell as a scale factor an
DDFEM() function as
Ei =
An advantage of this r
range of dataset but als
R8
to R7
. Note that we
the original DDFEM(

59. …
For Model
(Independent of Materials)
Training of Mapping (§7.1.3)
For Material Set
(Independent of Models)
Regression Forests
3. Regression Forests
✔ A large dataset handling
✔ Constant evaluation cost
✔ High generalization ability
Data-driven fluid simulations using regression forests,
Ladický et al., SIGGRAPH Asia ‘15

60. 3. Regression ForestsLJS∗
15] which construct each tree through two steps training:
structure construction with a subset of learning data and least-
are solve for the regression coefficients at each leaf node with
he dataset. The regression forest has an advantage of constant
evaluation even if the amount of the dataset is increased. Fi-
y, our DDFEM() becomes
Ei = ¯e ·Reg1(
e′
2
¯e ,...,
e′
8
¯e ) (ei = 0)
Ei = ei ·Reg2(
e′
2
ei
,...,
e′
8
ei
) (ei > 0)
(12)
re Reg() represents the regression function, and ¯e is the average
he Young’s modulus in the target eight cells.
Hierarchical Component Mode Synthesis
Finally, our DDFEM becomes
Reg(): regression function
Data-driven fluid simulations using regression forests,
Ladický et al., SIGGRAPH Asia ‘15

61. Data-Driven Online Coarsening (§7.1)
Hierarchical Component Mode Synthesis (§7.2)
Fast Approximate Modal Analysis (§7)
+
…
For Model
(Independent of Materials)
Input: Surface Mesh
Training of Mapping (§7.1.3)
For Material Set
(Independent of Models)
Voxelization and Eigen Decomposition (§7.2)
Used for
Domain Decomposition (§7.2)
Regression Forests
arashi / Interactive Physically-Based Sound Design of 3D Modelusing
Data-Driven Online Coarsening (§7.1)
Hierarchical Component Mode Synthesis (§7.2)
Runtime
Fast Approximate Modal Analysis (§7)
+
S
Simulated Sound
Similarity Score Compu
Eigenpairs
S
S
Mesh Coarsening with Data-Driven FEM
Fast Approximate Mode Analysis
Hierarchical Component Mode Synthesis
Data-Driven Online Coarsening (§7.1)
Hierarchical Component Mode Synthesis (§7.2)
Fast Approximate Modal Analysis (§7)
+
…
For Model
(Independent of Materials)
Input: Surface Mesh
Training of Mapping (§7.1.3)
For Material Set
(Independent of Models)
Voxelization and Eigen Decomposition (§7.2)
Used for
Domain Decomposition (§7.2)
Eigen
Regression Forests
+

62. Hierarchical Component Mode Synthesis (HCMS)
Kazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization
Error Correction (§7.2.1)
Domain Decomposition (§7.2)
Coarse Voxel Mesh
(The Output of DDFEM*)
Subdomains
Local Eigen Problem Solves in Parallel
(D: Local Eigenvalues, U: Local Eigenvectors)
{D1,U1}
{D2,U2}
{D3,U3}
{D4,U4}
{D5,U5}
{D6,U6}
{D7,U7}
{D8,U8}
{D9,U9}
Hierarchical Merge by Eq. (15)
{D11,U11}
{D12,U12}
{D13,U13}
{D14,U14}
{D22,U22}
{D21,U21}
{D31,U31}
{D41,U41}
Merge adjacent domains in parallel
Global Solution
ure 7: Hierarchical Component Mode Synthesis. After coarsening the mesh, we decompose it into many subdomains and hierarch
rges them with reducing their DoFs in parallel. Finally, we improve the accuracy using an error correction algorithm.
To simplify the explanation for our HCMS, we first begin
h assuming that a model can be decomposed into two non-
rlapping domains S1 and S2 as in conventional CMS, and the
(15) to obtain the eigenvectors of the merged subdomain. Thi
cedure also can be executed in parallel until all the subdomai
merged. Finally, we merge all the subdomains and obtain th
HCMS takes the coarse voxel mesh as input
and solves a generalized eigenproblem via hierarchical merging.

63. Hierarchical Component Mode Synthesis (HCMS)
Kazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization
Error Correction (§7.2.1)
Domain Decomposition (§7.2)
Coarse Voxel Mesh
(The Output of DDFEM*)
Subdomains
Local Eigen Problem Solves in Parallel
(D: Local Eigenvalues, U: Local Eigenvectors)
{D1,U1}
{D2,U2}
{D3,U3}
{D4,U4}
{D5,U5}
{D6,U6}
{D7,U7}
{D8,U8}
{D9,U9}
Hierarchical Merge by Eq. (15)
{D11,U11}
{D12,U12}
{D13,U13}
{D14,U14}
{D22,U22}
{D21,U21}
{D31,U31}
{D41,U41}
Merge adjacent domains in parallel
Global Solution
ure 7: Hierarchical Component Mode Synthesis. After coarsening the mesh, we decompose it into many subdomains and hierarch
rges them with reducing their DoFs in parallel. Finally, we improve the accuracy using an error correction algorithm.
To simplify the explanation for our HCMS, we first begin
h assuming that a model can be decomposed into two non-
rlapping domains S1 and S2 as in conventional CMS, and the
(15) to obtain the eigenvectors of the merged subdomain. Thi
cedure also can be executed in parallel until all the subdomai
merged. Finally, we merge all the subdomains and obtain th
HCMS takes the coarse voxel mesh as input
and solves a generalized eigenproblem via hierarchical merging.

64. Hierarchical Component Mode Synthesis (HCMS)
subdomain 1 subdomain 2
Eigenvalues of A : Λ1
Eigenvectors of A: U1
Eigenvalues of B : Λ2
Eigenvectors of B: U2

65. Hierarchical Component Mode Synthesis (HCMS)
subdomain 1 subdomain 2
Eigenvalues of A : Λ1
Eigenvectors of A: U1
Eigenvalues of B : Λ2
Eigenvectors of B: U2

66. Hierarchical Component Mode Synthesis (HCMS)
subdomain 1 subdomain 2
Eigenvalues of A : Λ1
Eigenvectors of A: U1
Eigenvalues of B : Λ2
Eigenvectors of B: U2
Figure 7: Hierarchical Component Mode Synthesis. After coarsening the mesh, we decom
merges them with reducing their DoFs in parallel. Finally, we improve the accuracy using
To simplify the explanation for our HCMS, we first begin
with assuming that a model can be decomposed into two non-
overlapping domains S1 and S2 as in conventional CMS, and the
eigenpairs of each domain are already known. Under this assump-
tion, the entire stiffness matrix Ktotal and the entire mass matrix
Mtotal can be represented as
Ktotal =
K11 K12
K12 K22
, Mtotal =
M1 0
0 M2
(13)
where K11, K22 and M1, M2 denote the local stiffness and mass ma-
trices of each sub-domain respectively. K12 and K21 are the inter-
face matrices that connect the domains S1 and S2. If the eigenvec-
tors of each domain U1 and U2 are already known, we can rewrite
(15) to obtain th
cedure also can
merged. Finally,
proximate eigen
ing subdomains
caused by subop
tion (§7.2.1). W
new. We implem
main by a comb
and QR method.
7.2.1. Error Co
HCMS is just a
Mtotal can be represented as
Ktotal =
K11 K12
K12 K22
, Mtotal =
M1 0
0 M2
(13)
where K11, K22 and M1, M2 denote the local stiffness and mass ma-
trices of each sub-domain respectively. K12 and K21 are the inter-
face matrices that connect the domains S1 and S2. If the eigenvec-
tors of each domain U1 and U2 are already known, we can rewrite
the Eq. (13) using the reduced matrices of each domain with re-
maining the lower frequency modes as
K′
total =
D1 U1T
K12U2
U2T
KT
12U1 D2
(14)
where D1 = U1T
K11U1 and D2 = U2T
K22U2 are diagonal matri-
ces in which each diagonal entry is the eigenvalue of the respective
subdomain. Note that the entire mass matrix also takes the same
form for, UT
1 M1U1 = I, and UT
2 M2U2 = I, meaning that the en-
caus
tion
new
mai
and
7.2.
HCM
racy
duce
Gra
tors
follo
≒
where
Ktotal =
K11 K12
K12 K22
, Mtotal =
M1 0
0 M2
(13
where K11, K22 and M1, M2 denote the local stiffness and mass ma
trices of each sub-domain respectively. K12 and K21 are the inter
face matrices that connect the domains S1 and S2. If the eigenvec
tors of each domain U1 and U2 are already known, we can rewrite
the Eq. (13) using the reduced matrices of each domain with re
maining the lower frequency modes as
K′
total =
D1 U1T
K12U2
U2T
KT
12U1 D2
(14
where D1 = U1T
K11U1 and D2 = U2T
K22U2 are diagonal matri
ces in which each diagonal entry is the eigenvalue of the respective
subdomain. Note that the entire mass matrix also takes the same
form for, UT
1 M1U1 = I, and UT
2 M2U2 = I, meaning that the en
Ktotal =
K11 K12
K12 K22
, Mtotal =
M1 0
0 M2
(13)
where K11, K22 and M1, M2 denote the local stiffness and mass ma-
trices of each sub-domain respectively. K12 and K21 are the inter-
face matrices that connect the domains S1 and S2. If the eigenvec-
tors of each domain U1 and U2 are already known, we can rewrite
the Eq. (13) using the reduced matrices of each domain with re-
maining the lower frequency modes as
K′
total =
D1 U1T
K12U2
U2T
KT
12U1 D2
(14)
where D1 = U1T
K11U1 and D2 = U2T
K22U2 are diagonal matri-
ces in which each diagonal entry is the eigenvalue of the respective
subdomain. Note that the entire mass matrix also takes the same
form for, UT
M U = I, and UT
M U = I, meaning that the en-
Both D matrices become diagonal matrices in which each
diagonal entry is the eigenvalue of the respective subdomain.
Reduced SystemEntire System
Ktotal =
K12 K
where K11, K22 and M1, M
trices of each sub-domain
face matrices that connect
tors of each domain U1 an
the Eq. (13) using the red
maining the lower frequenc
K′
total =
U2
where D1 = U1T
K11U1 an
ces in which each diagonal
subdomain. Note that the
=

67. Hierarchical Component Mode Synthesis (HCMS)
subdomain 1 subdomain 2
Eigenvalues of A : Λ1
Eigenvectors of A: U1
Eigenvalues of B : Λ2
Eigenvectors of B: U2
Figure 7: Hierarchical Component Mode Synthesis. After coarsening the mesh, we decom
merges them with reducing their DoFs in parallel. Finally, we improve the accuracy using
To simplify the explanation for our HCMS, we first begin
with assuming that a model can be decomposed into two non-
overlapping domains S1 and S2 as in conventional CMS, and the
eigenpairs of each domain are already known. Under this assump-
tion, the entire stiffness matrix Ktotal and the entire mass matrix
Mtotal can be represented as
Ktotal =
K11 K12
K12 K22
, Mtotal =
M1 0
0 M2
(13)
where K11, K22 and M1, M2 denote the local stiffness and mass ma-
trices of each sub-domain respectively. K12 and K21 are the inter-
face matrices that connect the domains S1 and S2. If the eigenvec-
tors of each domain U1 and U2 are already known, we can rewrite
(15) to obtain th
cedure also can
merged. Finally,
proximate eigen
ing subdomains
caused by subop
tion (§7.2.1). W
new. We implem
main by a comb
and QR method.
7.2.1. Error Co
HCMS is just a
Mtotal can be represented as
Ktotal =
K11 K12
K12 K22
, Mtotal =
M1 0
0 M2
(13)
where K11, K22 and M1, M2 denote the local stiffness and mass ma-
trices of each sub-domain respectively. K12 and K21 are the inter-
face matrices that connect the domains S1 and S2. If the eigenvec-
tors of each domain U1 and U2 are already known, we can rewrite
the Eq. (13) using the reduced matrices of each domain with re-
maining the lower frequency modes as
K′
total =
D1 U1T
K12U2
U2T
KT
12U1 D2
(14)
where D1 = U1T
K11U1 and D2 = U2T
K22U2 are diagonal matri-
ces in which each diagonal entry is the eigenvalue of the respective
subdomain. Note that the entire mass matrix also takes the same
form for, UT
1 M1U1 = I, and UT
2 M2U2 = I, meaning that the en-
caus
tion
new
mai
and
7.2.
HCM
racy
duce
Gra
tors
follo
≒
where
Ktotal =
K11 K12
K12 K22
, Mtotal =
M1 0
0 M2
(13
where K11, K22 and M1, M2 denote the local stiffness and mass ma
trices of each sub-domain respectively. K12 and K21 are the inter
face matrices that connect the domains S1 and S2. If the eigenvec
tors of each domain U1 and U2 are already known, we can rewrite
the Eq. (13) using the reduced matrices of each domain with re
maining the lower frequency modes as
K′
total =
D1 U1T
K12U2
U2T
KT
12U1 D2
(14
where D1 = U1T
K11U1 and D2 = U2T
K22U2 are diagonal matri
ces in which each diagonal entry is the eigenvalue of the respective
subdomain. Note that the entire mass matrix also takes the same
form for, UT
1 M1U1 = I, and UT
2 M2U2 = I, meaning that the en
Ktotal =
K11 K12
K12 K22
, Mtotal =
M1 0
0 M2
(13)
where K11, K22 and M1, M2 denote the local stiffness and mass ma-
trices of each sub-domain respectively. K12 and K21 are the inter-
face matrices that connect the domains S1 and S2. If the eigenvec-
tors of each domain U1 and U2 are already known, we can rewrite
the Eq. (13) using the reduced matrices of each domain with re-
maining the lower frequency modes as
K′
total =
D1 U1T
K12U2
U2T
KT
12U1 D2
(14)
where D1 = U1T
K11U1 and D2 = U2T
K22U2 are diagonal matri-
ces in which each diagonal entry is the eigenvalue of the respective
subdomain. Note that the entire mass matrix also takes the same
form for, UT
M U = I, and UT
M U = I, meaning that the en-
Both D matrices become diagonal matrices in which each
diagonal entry is the eigenvalue of the respective subdomain.
Reduced SystemEntire System
Ktotal =
K12 K
where K11, K22 and M1, M
trices of each sub-domain
face matrices that connect
tors of each domain U1 an
the Eq. (13) using the red
maining the lower frequenc
K′
total =
U2
where D1 = U1T
K11U1 an
ces in which each diagonal
subdomain. Note that the
=

68. Hierarchical Component Mode Synthesis (HCMS)
CMS distinguishes the interface of adjacent subd
domains, and assumes the interface as fixed [BC
the boundary modes [YXG∗
13], our approach ne
them nor fix the interface, and does not treat the
itly. We can obtain a reduced eigenproblem of th
as K′
totalU′
total = ΛtotalU′
total, where Λtotal is a
in which each diagonal entry is the eigenvalues
main. We solve this reduced eigenproblem, and fi
global eigenvectors by
Utotal =
U1
U2
U′
total.
CMS distinguishes the interface of adjacent subdomains and sub-
domains, and assumes the interface as fixed [BC68] or considers
the boundary modes [YXG∗
13], our approach neither distinguish
them nor fix the interface, and does not treat the interface explic-
itly. We can obtain a reduced eigenproblem of the entire structure
as K′
totalU′
total = ΛtotalU′
total, where Λtotal is a diagonal matrix
in which each diagonal entry is the eigenvalues of the entire do-
main. We solve this reduced eigenproblem, and finally recover the
global eigenvectors by
Utotal =
U1
U2
U′
total. (15)
We apply the pair wise merger explained above in a hierarchical
Reduced Generalized EigenProblem:
Approximated EigenVectors of The Entire System:

69. Hierarchical Component Mode Synthesis (HCMS)
A coarse mesh

70. Hierarchical Component Mode Synthesis (HCMS)
A coarse mesh
SubDomain 1 SubDomain 2 SubDomain 3 SubDomain 4
Domain Decomposition

71. Hierarchical Component Mode Synthesis (HCMS)
A coarse mesh
SubDomain 1 SubDomain 2 SubDomain 3 SubDomain 4
Domain Decomposition
MergedDomain 1 MergedDomain 2

72. Hierarchical Component Mode Synthesis (HCMS)
A coarse mesh
SubDomain 1 SubDomain 2 SubDomain 3 SubDomain 4
Domain Decomposition
MergedDomain 1 MergedDomain 2
EntireMesh

73. Hierarchical Component Mode Synthesis (HCMS)
A coarse mesh
In parallel:
In parallel:
SubDomain 1 SubDomain 2 SubDomain 3 SubDomain 4
Domain Decomposition
MergedDomain 1 MergedDomain 2
EntireMesh

74. Hierarchical Component Mode Synthesis (HCMS)
Kazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization
Error Correction (§7.2.1)
Domain Decomposition (§7.2)
Coarse Voxel Mesh
(The Output of DDFEM*)
Subdomains
Local Eigen Problem Solves in Parallel
(D: Local Eigenvalues, U: Local Eigenvectors)
{D1,U1}
{D2,U2}
{D3,U3}
{D4,U4}
{D5,U5}
{D6,U6}
{D7,U7}
{D8,U8}
{D9,U9}
Hierarchical Merge by Eq. (15)
{D11,U11}
{D12,U12}
{D13,U13}
{D14,U14}
{D22,U22}
{D21,U21}
{D31,U31}
{D41,U41}
Merge adjacent domains in parallel
Global Solution
ure 7: Hierarchical Component Mode Synthesis. After coarsening the mesh, we decompose it into many subdomains and hierarch
rges them with reducing their DoFs in parallel. Finally, we improve the accuracy using an error correction algorithm.
To simplify the explanation for our HCMS, we first begin
h assuming that a model can be decomposed into two non-
rlapping domains S1 and S2 as in conventional CMS, and the
(15) to obtain the eigenvectors of the merged subdomain. Thi
cedure also can be executed in parallel until all the subdomai
merged. Finally, we merge all the subdomains and obtain th
After the merging of HCMS, we reduce the error by an error correction.

75. Hierarchical Component Mode Synthesis (HCMS)
Kazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Based Sound Design of 3D Modelusing Material Optimization
Error Correction (§7.2.1)
Domain Decomposition (§7.2)
Coarse Voxel Mesh
(The Output of DDFEM*)
Subdomains
Local Eigen Problem Solves in Parallel
(D: Local Eigenvalues, U: Local Eigenvectors)
{D1,U1}
{D2,U2}
{D3,U3}
{D4,U4}
{D5,U5}
{D6,U6}
{D7,U7}
{D8,U8}
{D9,U9}
Hierarchical Merge by Eq. (15)
{D11,U11}
{D12,U12}
{D13,U13}
{D14,U14}
{D22,U22}
{D21,U21}
{D31,U31}
{D41,U41}
Merge adjacent domains in parallel
Global Solution
ure 7: Hierarchical Component Mode Synthesis. After coarsening the mesh, we decompose it into many subdomains and hierarch
rges them with reducing their DoFs in parallel. Finally, we improve the accuracy using an error correction algorithm.
To simplify the explanation for our HCMS, we first begin
h assuming that a model can be decomposed into two non-
rlapping domains S1 and S2 as in conventional CMS, and the
(15) to obtain the eigenvectors of the merged subdomain. Thi
cedure also can be executed in parallel until all the subdomai
merged. Finally, we merge all the subdomains and obtain th
After the merging of HCMS, we reduce the error by an error correction.

76. Σ: over relaxation weights
Error Correction
the eigenvec-
we can rewrite
main with re-
(14)
iagonal matri-
the respective
akes the same
g that the en-
h conventional
ains and sub-
] or considers
er distinguish
erface explic-
ntire structure
agonal matrix
the entire do-
7.2.1. Error Correction
HCMS is just an approximation method and sacrifices the accu-
racy for computational efficiency. To correct this error, we intro-
duce the subspace iteration method [Bat13] using reduced mass
Gram-Shmidzt process [YLX∗
15]. We set approximated eigenvec-
tors of HCMS as the starting iteration vectors X0 and execute the
following iteration k = 1,2,3... until it converges.
Solve PCG : KU = MXk−1 (16)
U ← ReducedMGS(U) (17)
K′
= UT
KU, M′
= UT
MU (18)
Solve QR : K′
Q = ΛM′
Q (19)
Xk = U +Σ(UQ−U) (20)
Xk ← ReducedMGS(Xk) (21)
where ReducedMGS() is the reduced mass Modified Gram-
Schmidt process to orthogonalize the eigenvectors [YLX∗
15], Σ
denotes a diagonal matrix in which each diagonal corresponds
to the overrelaxation weight of the i-th eigenvalue to accelerate
X0 ← Approximate EigenVectors of HCMS
and iterate…
The subspace iteration method - revisited, Bathe et al., Computers and Structure ’13

77. Σ: over relaxation weights
Error Correction
the eigenvec-
we can rewrite
main with re-
(14)
iagonal matri-
the respective
akes the same
g that the en-
h conventional
ains and sub-
] or considers
er distinguish
erface explic-
ntire structure
agonal matrix
the entire do-
7.2.1. Error Correction
HCMS is just an approximation method and sacrifices the accu-
racy for computational efficiency. To correct this error, we intro-
duce the subspace iteration method [Bat13] using reduced mass
Gram-Shmidzt process [YLX∗
15]. We set approximated eigenvec-
tors of HCMS as the starting iteration vectors X0 and execute the
following iteration k = 1,2,3... until it converges.
Solve PCG : KU = MXk−1 (16)
U ← ReducedMGS(U) (17)
K′
= UT
KU, M′
= UT
MU (18)
Solve QR : K′
Q = ΛM′
Q (19)
Xk = U +Σ(UQ−U) (20)
Xk ← ReducedMGS(Xk) (21)
where ReducedMGS() is the reduced mass Modified Gram-
Schmidt process to orthogonalize the eigenvectors [YLX∗
15], Σ
denotes a diagonal matrix in which each diagonal corresponds
to the overrelaxation weight of the i-th eigenvalue to accelerate
X0 ← Approximate EigenVectors of HCMS
and iterate…
Full-DoF stiffness and mass matrices
The subspace iteration method - revisited, Bathe et al., Computers and Structure ’13

78. Σ: over relaxation weights
Error Correction
the eigenvec-
we can rewrite
main with re-
(14)
iagonal matri-
the respective
akes the same
g that the en-
h conventional
ains and sub-
] or considers
er distinguish
erface explic-
ntire structure
agonal matrix
the entire do-
7.2.1. Error Correction
HCMS is just an approximation method and sacrifices the accu-
racy for computational efficiency. To correct this error, we intro-
duce the subspace iteration method [Bat13] using reduced mass
Gram-Shmidzt process [YLX∗
15]. We set approximated eigenvec-
tors of HCMS as the starting iteration vectors X0 and execute the
following iteration k = 1,2,3... until it converges.
Solve PCG : KU = MXk−1 (16)
U ← ReducedMGS(U) (17)
K′
= UT
KU, M′
= UT
MU (18)
Solve QR : K′
Q = ΛM′
Q (19)
Xk = U +Σ(UQ−U) (20)
Xk ← ReducedMGS(Xk) (21)
where ReducedMGS() is the reduced mass Modified Gram-
Schmidt process to orthogonalize the eigenvectors [YLX∗
15], Σ
denotes a diagonal matrix in which each diagonal corresponds
to the overrelaxation weight of the i-th eigenvalue to accelerate
X0 ← Approximate EigenVectors of HCMS
and iterate…
Expediting Precomputation for Reduced Deformable Simulation,
Yang et al. SIGGRAPH Asia ’15.
The subspace iteration method - revisited, Bathe et al., Computers and Structure ’13

79. Ground truth
CMS without EC
CMS with EC (4 iterations)
CMS with EC (8 iterations)
CMS with EC (16 iterations)
Eigenvalue
Mode Number
0 50 100 150 200
0
0.5
1
1.5
2
2.5
x 10
4
Eigenvalue
Mode Number
Ground truth
CMS without EC
CMS with EC (4 iterations)
CMS with EC (8 iterations)
CMS with EC (16 iterations)
Random starting vectors (16 iterations)
The Result of Error Correction
For Model
(Independent of Materials)
Input: Surface Mesh
Eigenvectors of Volumetric Mesh Laplacian
Voxelization and Eigen Decomposition (§7.2)
U
Figure 2: Algorithm Overview. Our optimiza
procedure at runtime consists of 3 steps. First
computes the similarity score between the sim
the system updates the material distribution i
It converges very quickly
in only a few iterations,
and efficiently reduces the error. Ground truth

80. Ground truth
CMS without EC
CMS with EC (4 iterations)
CMS with EC (8 iterations)
CMS with EC (16 iterations)
Eigenvalue
Mode Number
0 50 100 150 200
0
0.5
1
1.5
2
2.5
x 10
4
Eigenvalue
Mode Number
Ground truth
CMS without EC
CMS with EC (4 iterations)
CMS with EC (8 iterations)
CMS with EC (16 iterations)
Random starting vectors (16 iterations)
The Result of Error Correction
For Model
(Independent of Materials)
Input: Surface Mesh
Eigenvectors of Volumetric Mesh Laplacian
Voxelization and Eigen Decomposition (§7.2)
U
Figure 2: Algorithm Overview. Our optimiza
procedure at runtime consists of 3 steps. First
computes the similarity score between the sim
the system updates the material distribution i
It converges very quickly
in only a few iterations,
and efficiently reduces the error. Ground truth

81. Results

82. Fast Approximate Modal Analysis

83. ically-Based Sound Design of 3D Modelusing Material Optimization
DDFEM HCMS HCMS + EC DDFEM + HCMS + EC
15.3m 9.8m 28.4m 2.6s
2.1m 1.1m 4.3m 0.86s
5.7m 4.9m 17.2m 1.1s
3.2m 3.3m 12.5m 1.4s
6.4m 3.4m 17.8m 2.3s
12.3m 9.7m 25.6m 2.4s
* *
al analysis. We computed the first 256 modes for all model.
o Yamamoto & Takeo Igarashi / Interactive Physically-Based Sound Design
el DoF Precomputation ARPack DDFEM HCMS
Bunny 31419 18.8m 4.1h 15.3m 9.8m
agon 6009 23s 14.8m 2.1m 1.1m
apot 12057 1.5m 52.8m 5.7m 4.9m
Dragon 11394 50s 21.9m 3.2m 3.3m
er 15927 2.3m 37,6m 6.4m 3.4m
Drum 35484 21.5m 3.9h 12.3m 9.7m
*
11: Computation time comparison of modal analysis. We compute
Kazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Ba
Model DoF Precomputation ARPack DDFEM
Stanford Bunny 31419 18.8m 4.1h 15.3m
Asian Dragon 6009 23s 14.8m 2.1m
Utah Teapot 12057 1.5m 52.8m 5.7m
Chinese Dragon 11394 50s 21.9m 3.2m
Pitcher 15927 2.3m 37,6m 6.4m
Snare Drum 35484 21.5m 3.9h 12.3m
Figure 11: Computation time comparison of modal analy
Computation Time (ARPack vs Ours)
Ours
CPU: Intel Core i7 2.6 GHz, RAM: 16GB, GPU: NVIDIA GeForce GT 750M

84. ically-Based Sound Design of 3D Modelusing Material Optimization
DDFEM HCMS HCMS + EC DDFEM + HCMS + EC
15.3m 9.8m 28.4m 2.6s
2.1m 1.1m 4.3m 0.86s
5.7m 4.9m 17.2m 1.1s
3.2m 3.3m 12.5m 1.4s
6.4m 3.4m 17.8m 2.3s
12.3m 9.7m 25.6m 2.4s
* *
al analysis. We computed the first 256 modes for all model.
o Yamamoto & Takeo Igarashi / Interactive Physically-Based Sound Design
el DoF Precomputation ARPack DDFEM HCMS
Bunny 31419 18.8m 4.1h 15.3m 9.8m
agon 6009 23s 14.8m 2.1m 1.1m
apot 12057 1.5m 52.8m 5.7m 4.9m
Dragon 11394 50s 21.9m 3.2m 3.3m
er 15927 2.3m 37,6m 6.4m 3.4m
Drum 35484 21.5m 3.9h 12.3m 9.7m
*
11: Computation time comparison of modal analysis. We compute
Kazuhiko Yamamoto & Takeo Igarashi / Interactive Physically-Ba
Model DoF Precomputation ARPack DDFEM
Stanford Bunny 31419 18.8m 4.1h 15.3m
Asian Dragon 6009 23s 14.8m 2.1m
Utah Teapot 12057 1.5m 52.8m 5.7m
Chinese Dragon 11394 50s 21.9m 3.2m
Pitcher 15927 2.3m 37,6m 6.4m
Snare Drum 35484 21.5m 3.9h 12.3m
Figure 11: Computation time comparison of modal analy
Computation Time (ARPack vs Ours)
Three orders of magnitude acceleration
Ours
CPU: Intel Core i7 2.6 GHz, RAM: 16GB, GPU: NVIDIA GeForce GT 750M

85. Comparison (Deformation)
Ground Truth (21.9 minutes) Ours (1.4 seconds)

86. Comparison (Deformation)
Ground Truth (21.9 minutes) Ours (1.4 seconds)

87. Comparison (Sound Simulation)
Ground Truth (4.1 hours) Ours (2.6 seconds)

88. Comparison (Sound Simulation)
Ground Truth (4.1 hours) Ours (2.6 seconds)

89. Comparison (Sound Simulation)
Ground Truth (4.1 hours) Ours (2.6 seconds)

90. Comparison (Sound Simulation)
Ground Truth (4.1 hours) Ours (2.6 seconds)

91. Comparison (Sound Simulation)
Ground Truth (4.1 hours) Ours (2.6 seconds)

92. Comparison (Sound Simulation)
Ground Truth (4.1 hours) Ours (2.6 seconds)

93. Physically-Based Sound Design

94. Physically-Based Sound Design
Interactive Editing

95. Physically-Based Sound Design
Interactive Editing

96. Physically-Based Sound Design
Interactive Editing

97. Physically-Based Sound Design
Imaginary Sound Assignment

98. Physically-Based Sound Design
Imaginary Sound Assignment

99. Physically-Based Sound Design
Complicated Scenario Example

100. Physically-Based Sound Design
Complicated Scenario Example

101. • Our optimization is not suited for fabrication.
• We only optimized Young’s modulus.
• We did not consider the radiation and propagation.
Limitation

102. Summery
• An example-based framework for designing physically-based sound
• Fast approximate modal analysis for an interactive simulation
• Extended data-driven FEM using regression forests
• Hierarchical component mode synthesis with error correction
• Handling a large range of continuous material settings.
• Constant evaluation cost.
• High generalization ability.
• Parallel
• Accurate
Thanks