Depth estimation of sound images using directional clustering and activation-shared nonnegative matrix factorization
Report
Share
•1 like•919 views
![Related study: wave field synthesis
WFS allows us to create sound
images at the front of loudspeakers.
Wave Field Synthesis (WFS)
Sound field reproduction
Representation "depth“
of sound images
[A. J. Berkhout, et al., 1993]
…… …
Listener
4
Drawback of WFS×
Source separation
Localization estimation of
sound images
1
2
These information have been lost in
existing contents by down-mix.
Up-mixing method are required.
↓
Sound image
Mixed signal → individual source
WFS requires the primary source
information of sound images.
1. Individual sound source
2. Localization information](https://image.slidesharecdn.com/ncspmiyauchi-150318233952-conversion-gate01/75/depth-estimation-of-sound-images-using-directional-clustering-and-activationshared-nonnegative-matrix-factorization-4-2048.jpg?cb=1668886518)
![Related study: directional clustering [Araki, et al., 2007]
6：Source component ：Spatial representative vector
L-chinputsignal
R-ch input signal
L-chinputsignal
R-ch input signal
Normalization Clustering
Mixed stereo signal
L-chinputsignal
R-ch input signal
Individual sources of each cluster
: Fourier transform : Inverse Fourier transform
1](https://image.slidesharecdn.com/ncspmiyauchi-150318233952-conversion-gate01/75/depth-estimation-of-sound-images-using-directional-clustering-and-activationshared-nonnegative-matrix-factorization-6-2048.jpg?cb=1668886518)
![Problem and purpose
8
Depth estimation method using
direction of arrival (DOA) distribution
Proposed method
Establishing new depth estimation method
How can we get depth information?
Purpose
Problem WFS requires specific localization information of
individual sound sources to reproduce a sound field.
Up-mixer
Directional estimation method have been developed.
Directional estimation based on VBAP [Hirata, et al., 2011]](https://image.slidesharecdn.com/ncspmiyauchi-150318233952-conversion-gate01/75/depth-estimation-of-sound-images-using-directional-clustering-and-activationshared-nonnegative-matrix-factorization-8-2048.jpg?cb=1668886518)
![12
Generalized Gaussian distribution: GGD [Box, et al., 1973]
Proposed method 1: modeling of DOA distribution
βshape = 2: Gaussian
distribution PDF
βshape = 1: Laplacian
distribution PDF
Definition of GGD
Flexible family of probability
density function (PDF)
To model DOA, we propose a new modeling method using GGD.
Shape of GGD changes
depending on βshape.](https://image.slidesharecdn.com/ncspmiyauchi-150318233952-conversion-gate01/75/depth-estimation-of-sound-images-using-directional-clustering-and-activationshared-nonnegative-matrix-factorization-12-2048.jpg?cb=1668886518)
![Proposed method 2: activation-shared multichannel NMF
16
Time
Frequency
AmplitudeFrequency
Amplitude
Time
Ω: Number of frequency bins
𝑇: Number of time frames
𝐾: Number of bases
Nonnegative matrix factorization: NMF [Lee, et al., 2001]
Activation matrix
(Time-varying gain)
Basis matrix
(Spectral patterns)
Observed matrix
(Spectrogram)
— is a sparse representation.
— can extract significant features from the observed matrix.
The sparse representation provides high performance
for noise reduction, compression, and feature extraction.
We eliminate background noise and artificial distortion.](https://image.slidesharecdn.com/ncspmiyauchi-150318233952-conversion-gate01/75/depth-estimation-of-sound-images-using-directional-clustering-and-activationshared-nonnegative-matrix-factorization-16-2048.jpg?cb=1668886518)
![- divergence [Eguchi, et al., 2001]
: Euclidean distance
: Generalized Kullback-Leibler divergence
: Itakura–Saito divergence
Generalized divergence of variable corresponding to .
19
Proposed method 2: activation-shared multichannel NMF](https://image.slidesharecdn.com/ncspmiyauchi-150318233952-conversion-gate01/75/depth-estimation-of-sound-images-using-directional-clustering-and-activationshared-nonnegative-matrix-factorization-19-2048.jpg?cb=1668886518)
![Real source Image source
Geometry of image method
Time index
Amplitude
Example of room impulse response
Experimental conditions
Technique of simulating
room impulse response
Volume of room
Source location
Microphone location
Absorption coefficient
– can be set arbitrarily
Reference sound sources
were generated using
image method.
Image method
[Allen, et al., 1979]
27](https://image.slidesharecdn.com/ncspmiyauchi-150318233952-conversion-gate01/75/depth-estimation-of-sound-images-using-directional-clustering-and-activationshared-nonnegative-matrix-factorization-27-2048.jpg?cb=1668886518)
![35
L-ch
NMF
R-ch
NMF
Preliminary experiment
Fluctuation are
generated in DOA Direction of arrival [degree]
L-ch
NMF
R-ch
NMF
(Individually applied)
conventional NMF
(Activation-shared)
proposed NMF
Weighted
DOA histogram
Center cluster DOA
of mixed source
(3 instrument)Direction of arrival [degree]
Direction of arrival [degree]
Feature extraction
while maintaining
directional information
Proposed method 2: activation-shared multichannel NMF
Example of
DOA histogram](https://image.slidesharecdn.com/ncspmiyauchi-150318233952-conversion-gate01/75/depth-estimation-of-sound-images-using-directional-clustering-and-activationshared-nonnegative-matrix-factorization-35-2048.jpg?cb=1668886518)
Depth estimation of sound images using directional clustering and activation-shared nonnegative matrix factorization
•1 like•919 views
Report
Share
Download to read offline
Presented at 2014 RISP International Workshop on Nonlinear Circuits, Communications and Signal Processing (NCSP 2014) (international conference) Tomo Miyauchi, Daichi Kitamura, Hiroshi Saruwatari, Satoshi Nakamura, "Depth estimation of sound images using directional clustering and activation-shared nonnegative matrix factorization," Proceedings of 2014 RISP International Workshop on Nonlinear Circuits, Communications and Signal Processing (NCSP 2014), pp.437-440, Hawaii, USA, March 2014 (Student Paper Award).
Recommended
More Related Content
What's hot
What's hot(20)
Viewers also liked
Viewers also liked(14)
Similar to Depth estimation of sound images using directional clustering and activation-shared nonnegative matrix factorization
Similar to Depth estimation of sound images using directional clustering and activation-shared nonnegative matrix factorization(20)
More from Daichi Kitamura
More from Daichi Kitamura(10)
Recently uploaded
Recently uploaded(20)
Depth estimation of sound images using directional clustering and activation-shared nonnegative matrix factorization
- 1. Depth Estimation of Sound Images Using Directional Clustering and Activation-Shared Nonnegative Matrix Factorization Tomo Miyauchi, Daichi Kitamura, Hiroshi Saruwatari, Satoshi Nakamura (Nara Institute of Science and Technology, Japan)
- 2. Outline Background and related study Problem and purpose Proposed method 1 - Depth estimation based on DOA distribution Proposed method 2 - Activation shared nonnegative matrix factorization Experiments Conclusions 2
- 3. Background With the advent of 3D TV, the reproduction of 3D image is realized. Viewer feels uncomfortable due to mismatch of images. Problem Picture image Sound image : Sound image 3D TV 3 To solve this problem, sound field reproduction technique have been studied actively. can present the “direction” and “depth” of the sound images to the listener. 3D sound reproduction system has not been established yet.
- 4. Related study: wave field synthesis WFS allows us to create sound images at the front of loudspeakers. Wave Field Synthesis (WFS) Sound field reproduction Representation "depth“ of sound images [A. J. Berkhout, et al., 1993] …… … Listener 4 Drawback of WFS× Source separation Localization estimation of sound images 1 2 These information have been lost in existing contents by down-mix. Up-mixing method are required. ↓ Sound image Mixed signal → individual source WFS requires the primary source information of sound images. 1. Individual sound source 2. Localization information
- 5. Mixed multi- channel signal Wave field Synthesis Stereo contents Spatial sound reproduction Spatial sound system using existing contents Flow of proposed up-mixer Depth estimation New depth estimation Sound source separation 1 Directional estimation Depth estimation of sound images has not been proposed Conventional method 2 This study 5
- 6. Related study: directional clustering [Araki, et al., 2007] 6：Source component ：Spatial representative vector L-chinputsignal R-ch input signal L-chinputsignal R-ch input signal Normalization Clustering Mixed stereo signal L-chinputsignal R-ch input signal Individual sources of each cluster : Fourier transform : Inverse Fourier transform 1
- 7. Outline Background and related study Problem and purpose Proposed method 1 - Depth estimation based on DOA distribution Proposed method 2 - Activation-shared multichannel NMF Experiments Conclusions 7
- 8. Problem and purpose 8 Depth estimation method using direction of arrival (DOA) distribution Proposed method Establishing new depth estimation method How can we get depth information? Purpose Problem WFS requires specific localization information of individual sound sources to reproduce a sound field. Up-mixer Directional estimation method have been developed. Directional estimation based on VBAP [Hirata, et al., 2011]
- 9. Outline Background and related study Problem and purpose Proposed method 1 - Depth estimation based on DOA distribution Proposed method 2 - Activation-shared multichannel NMF Experiments Conclusions 9
- 10. → “Direction of arrival” of sound waves We estimate the depth using the DOA distribution. Center RightLeft Frequencyof sourcecomponents Direction of arrival Directional clustering Weighted DOA histogram DOA Amplitude ratio of 10 Directional information Weighting term Proposed method 1: depth estimation based on DOA Mixed signal Individual sources Magnitude of each vector
- 11. Proposed method 1: depth estimation based on DOA 11 sourcecomponent Frequencyof sourcecomponent Frequencyof Direction of arrival Close Far Observed DOA histogram becomes smooth shape Difference of DOA shape corresponding to source distance Observed DOA distribution of the target source can be used as a cue for depth estimation. Observed DOA histogram becomes spiky shape Close source Direction of arrival Far source In sound fields, when a sound source is far from the listener, sound waves arrive from various directions owing to sound diffusion.
- 12. 12 Generalized Gaussian distribution: GGD [Box, et al., 1973] Proposed method 1: modeling of DOA distribution βshape = 2: Gaussian distribution PDF βshape = 1: Laplacian distribution PDF Definition of GGD Flexible family of probability density function (PDF) To model DOA, we propose a new modeling method using GGD. Shape of GGD changes depending on βshape.
- 13. 13 Modeling of DOA distribution based on GGD parameter Proposed method 1: modeling of DOA distribution Close Direction of arrival sourcecomponents Frequencyof Far Source is close ⇔ βshape is small Source is Far ⇔ βshape is large We propose a new depth estimation based on GGD. Shape parameter βshape is utilized as metric.
- 14. Proposed method 2： problem in proposed method 1 Problem of signal processing L-ch R-ch Small noise components are enhanced. L-chinputsignal R-ch input signalBinaural – recorded Normalization problem 14 DOA Frequencyof sourcecomponents Center RightLeft Background noise and artificial distortion generated by signal processing interfere with DOA histogram. Activation-shared multichannel NMFFeature extraction Noise ×
- 15. Outline Background and related study Problem and purpose Proposed method 1 - Depth estimation based on DOA distribution Proposed method 2 - Activation-shared multichannel NMF Experiments Conclusions 15
- 16. Proposed method 2: activation-shared multichannel NMF 16 Time Frequency AmplitudeFrequency Amplitude Time Ω: Number of frequency bins 𝑇: Number of time frames 𝐾: Number of bases Nonnegative matrix factorization: NMF [Lee, et al., 2001] Activation matrix (Time-varying gain) Basis matrix (Spectral patterns) Observed matrix (Spectrogram) — is a sparse representation. — can extract significant features from the observed matrix. The sparse representation provides high performance for noise reduction, compression, and feature extraction. We eliminate background noise and artificial distortion.
- 17. 17 L-ch NMF R-ch NMF Conventional NMFs generate an artificial fluctuation. Directional information DOA information is disturbed. Conventional NMF Proposed method 2: problem of conventional NMF NMFs are applied in parallel Amplitude ratioBases are trained uncorrelated.
- 18. 18 This reduces dimensionality of input signal while maintaining directional information. Cost function Activation matrix is shared through all channels Activation-shared multichannel NMFProposed method : cost function, : β-divergence, : entries of matrices L-ch NMF R-ch NMF Proposed method 2: activation-shared multichannel NMF
- 19. - divergence [Eguchi, et al., 2001] : Euclidean distance : Generalized Kullback-Leibler divergence : Itakura–Saito divergence Generalized divergence of variable corresponding to . 19 Proposed method 2: activation-shared multichannel NMF
- 20. 20 Using -divergence Proposed method 2: activation-shared multichannel NMF Auxiliary function method is an optimization scheme that uses the upper bound function. 1. Design the auxiliary function for as . 2. Minimize the original cost functions indirectly by minimizing the auxiliary functions. Derivation of optimal variables
- 21. The first and second terms become convex or concave functions with respect to value. concave convex convex concave convex concave 21 Proposed method 2: activation-shared multichannel NMF Cost function
- 22. Convex: Jensen’s inequality Concave: tangent line inequality : Convex function : Concave function 22 Proposed method 2: activation-shared multichannel NMF Cost function Upper bound function of each term is defined by applying
- 23. The update rules for optimization are obtained from the derivative of auxiliary function w.r.t. each objective variable. 23 are entries of matrices . Proposed method 2: activation-shared multichannel NMF Update rules
- 24. Flow of proposed depth estimation method Input stereo signal L-ch R-ch STFT Cluster RCluster CCluster L Weighted DOA histogram estimation Depth estimation Depth estimation Depth shared NMF Activation- Direction of arrivalWe can estimate depth information by calculate shape parameter of DOA histogram. Frequencyof sourcecomponents Direction of arrival Direction of arrival shared NMF Activation- shared NMF Activation- 24 Frequencyof sourcecomponents Frequencyof sourcecomponents
- 25. Outline Background and related study Problem and purpose Proposed method 1 - Depth estimation based on DOA distribution Proposed method 2 - Activation-shared multichannel NMF Experiments Conclusions 25
- 26. Experimental conditions 26 Conditions Mixed stereo signals consist of 3 instruments. Target source is located center with 7 distances. Combination related to direction is 6 patterns. Mixing source parameter Test source 1 Test source 2 Test source 3 Reverberation time NMF beta NMF basis: Interference source : Target source at intervals Conventional method 2 Conventional method 1 Proposed method Weighted DOA histogram （Not processed by NMF） Processed by conventional NMF Processed by proposed NMF
- 27. Real source Image source Geometry of image method Time index Amplitude Example of room impulse response Experimental conditions Technique of simulating room impulse response Volume of room Source location Microphone location Absorption coefficient – can be set arbitrarily Reference sound sources were generated using image method. Image method [Allen, et al., 1979] 27
- 28. 28 Experimental results Results 1 ・ Results of conventional methods have no agreement with the oracle (image method). ・ Results of proposed method correctly estimates distance of the target source. : Interference source : Target source Target source: Vocal Interference source (left): Piano Interference source (right): Guitar Data set 1
- 29. 29 Data set 1 2 3 4 5 6 Target source Interference source (left) Interference source (right) Vocal Piano Guitar Vocal Guitar Piano Guitar Piano Vocal Guitar Vocal Piano Piano Vocal Guitar Piano Guitar Vocal Conventional method 1 0.350 0.532 0.154 0.277 0.602 0.496 Conventional method 2 0.189 0.165 0.044 -0.037 0.426 0.157 Proposed method 0.986 0.925 0.777 0.651 0.791 0.856 Experimental results: correlation coefficient Correlation coefficient between reference value and estimated value • Strong relation between the estimated value of proposed method and the distance of the target source is indicated. • The efficacy of the proposed method is confirmed. Table Correlation coefficient of each method Results 2
- 30. Conclusions 30 We proposed a new depth estimation method of sound source in mixed signal using the shape of DOA distribution. The shape of DOA distribution is modeling by GGD. We also proposed a new feature extraction method for the multichannel signal, activation-shared multichannel NMF. The result of the experiment indicated the efficacy of the proposed method.
- 31. 31
- 32. Derivation of parameter βshape Kurtosis of DOA histogram we propose a closed-form parameter estimation algorithm based on some approximation and kurtosis. th moment of GGD : Observed DOA histogram : Gamma function × 32 Relation equation of kurtosis and shape parameter The maximum-likelihood based shape parameter estimation has no closed-form solution in GGD.
- 33. Modified Stirling's formula There is no exact closed-form solution of the inverse function.× Approximation of gamma function Take a logarithm 33 Derivation of parameter βshape Introduce Modified String’s formula
- 34. This results in the following quadratic equation of to be solved closed-form estimate of shape parameter Preparation of depth estimation method is completed. we can derive the closed-form estimation 34 Derivation of parameter βshape
- 35. 35 L-ch NMF R-ch NMF Preliminary experiment Fluctuation are generated in DOA Direction of arrival [degree] L-ch NMF R-ch NMF (Individually applied) conventional NMF (Activation-shared) proposed NMF Weighted DOA histogram Center cluster DOA of mixed source (3 instrument)Direction of arrival [degree] Direction of arrival [degree] Feature extraction while maintaining directional information Proposed method 2: activation-shared multichannel NMF Example of DOA histogram
Editor's Notes
- Hello, everyone. I’m Tomo Miyauchi from Nara institute of science and technology, Japan. Today / I’d like to talk about Depth estimation of sound images using directional clustering and activation-shared nonnegative matrix factorization.
- Here is the outline of today’s presentation. My presentation is divided into five parts. First, I talk about research background and related study.
- Recently, with the advent of 3D TV, the reproduction of 3D image is realized. On the other hand, 3D sound reproduction system has not been established yet. Therefore, Viewers feel uncomfortable due to mismatch of images. To solve this problem, sound field reproduction technique have been studied actively. Thanks to this technique, / we can present the direction and depth of the sound images to the listener.
- Wave field synthesis, WFS in short, is one of the sound field reproduction technique. WFS allows (アラウズ) us to create sound images at the front of loudspeakers. WFS requires the primary source information of sound images, where primary source information means the individual sound source and the localization information. However, these information have been lost in existing contents by down-mix. This is the drawback of WFS. Therefore, up-mixing method are required. Up-mix process can be divided into 2 steps. 1st step is the source separation, and 2nd step is the localization estimation of sound images.
- This is the flow of the proposed up-mixer. The signal of stereo contents is processed by sound source separation method and the localization estimation method. The processed signal are used in WFS finally. The localization method consists of directional estimation and depth estimation. In previous research, sound source separation and directional estimation have been proposed in conventional method. On the other hand, depth estimation has not been proposed yet. Therefore, in this study, we proposed a new depth estimation method.
- Now, I explain about the directional clustering, / which is used as source separation method in proposed up-mixer. This is the procedure of clustering. First, the mixed stereo signal is processed by Fourier transform. Next, the time-frequency components of signal are represented into the two-dimensional space, where XL and XR are the amplitude of each channel. Then, these components are normalized and separated by k-means clustering. Finally, the individual sources of each cluster is obtained by inverse Fourier transform.
- Next, I explain about problem and purpose of this study.
- WFS requires specific localization information of individual sound sources to reproduce a sound field. This is the problem of the spatial (スペィシアル) sound system using WFS. As mentioned above (アバブ), the directional estimation method have been developed. Therefore, the purpose of this study is establishing a new estimation method. In this study, we propose depth estimation method using direction of arrival distribution.
- Next, I explain about proposed method 1, depth estimation based on DOA distribution.
- DOA means direction of arrival of sound waves. We estimate the depth information using the DOA distribution. In the directional clustering, we using a amplitude ratio of signal / as the directional information. This parameter is reused as DOA. Now, we calculate a weighted DOA histogram. In this process, DOAs are calculated as θ. Then, DOAs are weighted by the magnitude of each vector w.
- In sound fields, when a sound source is far from the listener, / sound waves arrive from various directions owing to sound diffusion. If the source is close, observed DOA histogram becomes spiky shape. On the other hand, if the sound source is far, observed DOA histogram becomes smooth shape. Therefore, the shape of an observed DOA distribution of the target source / can be used as a cue for depth estimation.
- To model of DOA, we propose a new modeling method using GGD. Generalized Gaussian distribution, GGD in short, is a flexible family of probability density function. As can be seen, the shape of GGD changes depending on βshape. β of 2 corresponds to Gaussian PDF / and that β of 1 corresponds to Laplacian PDF.
- If β is small, GGD becomes a spiky shape, and if β is large, GGD becomes a smooth shape. Based on this property, we propose a new depth estimation based on GGD. In our method, shape parameter is utilized as metric. Then, we define the target source is close when β is small, and the target source is far when β is large.
- In the actual calculating process, back ground noise and artificial distortion generated by signal processing / interfere with DOA histogram. These noise have a negative effect in the depth estimation. Therefore, we proposed a feature (フィーチャー) extraction method, activation-shared multichannel NMF.
- Next, I mention about proposed method 2, activation-shared multichannel NMF.
- Nonnegative matrix factorization, NMF in short, has been proposed. NMF is a sparse representation, and can extract the significant features from the observed matrix. NMF decomposes the observed matrix, spectrogram Y, into two nonnegative matrices F and G. Here, F has frequently-appearing spectral patterns. And G has time-varying (ベリン) gains. So, F is called ‘basis matrix,’ and G is called ‘activation matrix.’ The aim of sparse representations is / to reveal basis structures, / and to represent these structures in a compact. Also, the sparse representation provides high performance for noise reduction, compression and feature extraction. Using this property, we eliminate background noise and artificial distortion.
- However, if the conventional NMFs are applied in parallel, artificial fluctuation is generated. This is due to the fact that bases are trained uncorrelated. As a result, DOA information is disturbed.
- Therefore, we propose activation-shared multichannel NMF. In this method, the activation matrix is shared through all channels. Thus (ザス), we can reduce dimensionality of the input signal / while maintaining directional information. This is the cost function of the proposed NMF.
- β-divergence is a generalized divergence of variable (ベリアブル) x corresponding to y. Dβ indicates the generalized divergence function, / which includes Euclidean distance, Kullback-Leibler divergence, and Itakura-Saito divergence.
- And we derive the optimal variables F, G, which minimize these cost functions. Auxiliary function method is an optimization scheme that uses the upper bound function, as the auxiliary function. In this method, we design the auxiliary functions for the cost functions J, as J plus. Then, we can minimize the original cost functions indirectly by minimizing the auxiliary functions. To design the auxiliary function, we have to derive the upper bounds. Using β-divergence, the cost function is redefined like this.
- The 1st and 2nd terms become convex or concave function with respect to β value, like this.
- For the convex function, Jensen’s inequality (インイクアリティ) can be used to derive the upper bound. On the other hand, for the concave function, we can use the tangent line inequality for making upper bound.
- The update rules for optimization are obtained from the derivative of auxiliary function with respect to each objective variable. These are the update rules of proposed NMF.
- This is the flow of the proposed depth estimation method. First, input stereo signal is processed by Fourier transform. Next, weighted DOA histogram is calculated. Then, the signal is separated by directional clustering. Activation-shared NMF is applied as the feature extraction method. Finally, we can estimate the depth of sound images by calculate shape parameter of DOA histogram.
- Next, I explain about experiments.
- In the experiment, we prepared mixed stereo signals, which consist of three instruments, vocal, piano, and guitar. The target source was located in the center with seven distances. In addition, combination related to direction is six patterns. As for β, we conducted a preliminary experiment. Then, we decided β equals 1 / corresponds to KL divergence. In this experiments, the signal not processed by NMF was evaluated as conventional method 1. Also, the signal processed by conventional NMF was evaluated as conventional method 2.
- We used the image method as a reference for this experiment, which is a technique of simulating the room impulse response. In this method, volume of room, source location, microphone location, and absorption coefficient (コエフィシェント) can be set arbitrarily (アービタラリリー). The reference sound sources were generated using image method.
- Here is the experimental result. In this graph, the gray line is reference values of the image method. As can be seen, shape parameter is increased corresponding to distance between source and listener. In addition, triangle of green is conventional method 1, circle of blue is conventional method 2, and diamond of red is proposed method. From this graph, the results of the conventional methods have no agreement with the oracle. On the other hand, the results of the proposed method correctly estimates distance of the target source.
- In addition, this is the correlation coefficient between the reference value and the estimated value. As can be seen, results of proposed method are highest value in all conditions. This result indicates strong relation between the estimated value of proposed method and the distance of the target source. Thus (ザス!), the efficacy of the proposed method as the depth estimation is confirmed.
- This is my conclusions. Thank you for your attention.
- 31