Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Audio MorphingProposed AlgorithmResultsConclusionAudio Morphing for Percussive Hybrid SoundGenerationAndrea Primavera1, Fr...
Audio MorphingProposed AlgorithmResultsConclusionIntroductionState of the ArtAudio Morphing is typically employed to accom...
Audio MorphingProposed AlgorithmResultsConclusionIntroductionState of the ArtAudio Morphing is typically employed to accom...
Audio MorphingProposed AlgorithmResultsConclusionIntroductionState of the ArtAudio Morphing is typically employed to accom...
Audio MorphingProposed AlgorithmResultsConclusionIntroductionState of the ArtDuring the last decades several algorithms ha...
Audio MorphingProposed AlgorithmResultsConclusionIntroductionState of the ArtDuring the last decades several algorithms ha...
Audio MorphingProposed AlgorithmResultsConclusionIntroductionState of the ArtDuring the last decades several algorithms ha...
Audio MorphingProposed AlgorithmResultsConclusionProposed AlgorithmPreProcessingProcessingAn automatic audio morphing algo...
Audio MorphingProposed AlgorithmResultsConclusionProposed AlgorithmPreProcessingProcessingThe PreProcessing operation is p...
Audio MorphingProposed AlgorithmResultsConclusionProposed AlgorithmPreProcessingProcessingThe PreProcessing operation is p...
Audio MorphingProposed AlgorithmResultsConclusionProposed AlgorithmPreProcessingProcessingThe PreProcessing operation is p...
Audio MorphingProposed AlgorithmResultsConclusionProposed AlgorithmPreProcessingProcessingThe PreProcessing operation is p...
Audio MorphingProposed AlgorithmResultsConclusionProposed AlgorithmPreProcessingProcessingThe Processing operation is exec...
Audio MorphingProposed AlgorithmResultsConclusionProposed AlgorithmPreProcessingProcessingThe Processing operation is exec...
Audio MorphingProposed AlgorithmResultsConclusionResultsAnalysis of Time and Spectral Features1 Listening Test (MOS)2 List...
Audio MorphingProposed AlgorithmResultsConclusionResultsAnalysis of Time and Spectral Features1 Listening Test (MOS)2 List...
Audio MorphingProposed AlgorithmResultsConclusionResultsAnalysis of Time and Spectral Features1 Listening Test (MOS)2 List...
Audio MorphingProposed AlgorithmResultsConclusionResultsAnalysis of Time and Spectral Features1 Listening Test (MOS)2 List...
Audio MorphingProposed AlgorithmResultsConclusionResultsAnalysis of Time and Spectral Features1 Listening Test (MOS)2 List...
Audio MorphingProposed AlgorithmResultsConclusionResultsAnalysis of Time and Spectral Features1 Listening Test (MOS)2 List...
Audio MorphingProposed AlgorithmResultsConclusionResultsAnalysis of Time and Spectral Features1 Listening Test (MOS)2 List...
Audio MorphingProposed AlgorithmResultsConclusionConclusionFuture WorksQuestionsBibliographyIn conclusion:• An automatic a...
Audio MorphingProposed AlgorithmResultsConclusionConclusionFuture WorksQuestionsBibliographyFuture work will be oriented t...
Audio MorphingProposed AlgorithmResultsConclusionConclusionFuture WorksQuestionsBibliographyQUESTIONS?Andrea Primavera Aud...
Audio MorphingProposed AlgorithmResultsConclusionConclusionFuture WorksQuestionsBibliographyM. H. Serra, D Rubine, and R. ...
Audio MorphingProposed AlgorithmResultsConclusionConclusionFuture WorksQuestionsBibliographyF. Boccardi and C. Drioli,“Sou...
Audio MorphingProposed AlgorithmResultsConclusionConclusionFuture WorksQuestionsBibliography“BFD2 version 2.0.1 Acoustic D...
Upcoming SlideShare
Loading in …5
×

Audio Morphing for Percussive Sound Generation

651 views

Published on

The aim of audio morphing algorithms is to combine two or more sounds to create a new sound with intermediate timbre and duration. During the last two decades several efforts have been made to improve morphing algorithms in order to obtain more realistic and perceptually relevant sounds. In this paper we present an automatic audio morphing technique applied to percussive musical instruments. Based on preprocessing of the sound references in frequency domain and linear interpolation in time domain, the presented approach allows one to generate high quality hybrid sounds at a low computational cost. Several results are reported in order to show the effectiveness of the proposed approach in terms of audio quality and acoustic perception of the generated hybrid sounds, taking into consideration different percussive samples. Mean opinion score and multidimensional scaling were used to compare the presented approach with existing state of the art techniques.

Published in: Education
  • Be the first to comment

  • Be the first to like this

Audio Morphing for Percussive Sound Generation

  1. 1. Audio MorphingProposed AlgorithmResultsConclusionAudio Morphing for Percussive Hybrid SoundGenerationAndrea Primavera1, Francesco Piazza1and Joshua D. Reiss21A3lab - DIBET - Universita’ Politecnica delleMarche - Ancona - ITALY2C4DM - Queen Mary University of London -London - UKAndrea Primavera Audio Morphing for Percussive Hybrid Sound Generation 1/21
  2. 2. Audio MorphingProposed AlgorithmResultsConclusionIntroductionState of the ArtAudio Morphing is typically employed to accomplish two differenttasks:• Obtaining a smooth transition between two sounds;• Generating a new intermediate sound which has thecharacteristics of the two given samples (e.g., timbre andduration).TopicAn automatic audio morphing technique applied to percussive musicalinstruments has been developed in this work.Proposed InnovationCreating new hybrid percussive sounds perceptually interesting (e.g.,morphing among different brands of the same instrument).AIMAndrea Primavera Audio Morphing for Percussive Hybrid Sound Generation 2/21
  3. 3. Audio MorphingProposed AlgorithmResultsConclusionIntroductionState of the ArtAudio Morphing is typically employed to accomplish two differenttasks:• Obtaining a smooth transition between two sounds;• Generating a new intermediate sound which has thecharacteristics of the two given samples (e.g., timbre andduration).TopicAn automatic audio morphing technique applied to percussive musicalinstruments has been developed in this work.Proposed InnovationCreating new hybrid percussive sounds perceptually interesting (e.g.,morphing among different brands of the same instrument).AIMAndrea Primavera Audio Morphing for Percussive Hybrid Sound Generation 2/21
  4. 4. Audio MorphingProposed AlgorithmResultsConclusionIntroductionState of the ArtAudio Morphing is typically employed to accomplish two differenttasks:• Obtaining a smooth transition between two sounds;• Generating a new intermediate sound which has thecharacteristics of the two given samples (e.g., timbre andduration).TopicAn automatic audio morphing technique applied to percussive musicalinstruments has been developed in this work.Proposed InnovationCreating new hybrid percussive sounds perceptually interesting (e.g.,morphing among different brands of the same instrument).AIMAndrea Primavera Audio Morphing for Percussive Hybrid Sound Generation 2/21
  5. 5. Audio MorphingProposed AlgorithmResultsConclusionIntroductionState of the ArtDuring the last decades several algorithms have been presented to morphaudio signals: linear interpolation, sinusoidal plus residual [1] [2] andspectral envelope using various features [3].One of the first approaches used toimplement audio morphing.y(t) = αs1(t) + (1 − α)s2(t)PRO: Low computational cost;CONS: When the timbre of the in-put signals is slightly different, boththe references could be individuallyperceivable.Linear Interpolation (LI) Based on harmonic modeling [4] [5]:s(t) =Nn=1An(t)cos[θn(t)] + e(t)The morphed sound is obtained in-terpolating the fundamental frequen-cy, the harmonics timbre, and the re-sidual envelope.PRO: Capability of morphing soundswith different timbre;CONS: Higher computational co-st than LI, some problems withpercussive sounds.Sinusoidal Plus Noise (SMS)Andrea Primavera Audio Morphing for Percussive Hybrid Sound Generation 3/21
  6. 6. Audio MorphingProposed AlgorithmResultsConclusionIntroductionState of the ArtDuring the last decades several algorithms have been presented to morphaudio signals: linear interpolation, sinusoidal plus residual [1] [2] andspectral envelope using various features [3].One of the first approaches used toimplement audio morphing.y(t) = αs1(t) + (1 − α)s2(t)PRO: Low computational cost;CONS: When the timbre of the in-put signals is slightly different, boththe references could be individuallyperceivable.Linear Interpolation (LI) Based on harmonic modeling [4] [5]:s(t) =Nn=1An(t)cos[θn(t)] + e(t)The morphed sound is obtained in-terpolating the fundamental frequen-cy, the harmonics timbre, and the re-sidual envelope.PRO: Capability of morphing soundswith different timbre;CONS: Higher computational co-st than LI, some problems withpercussive sounds.Sinusoidal Plus Noise (SMS)Andrea Primavera Audio Morphing for Percussive Hybrid Sound Generation 3/21
  7. 7. Audio MorphingProposed AlgorithmResultsConclusionIntroductionState of the ArtDuring the last decades several algorithms have been presented to morphaudio signals: linear interpolation, sinusoidal plus residual [1] [2] andspectral envelope using various features [3].One of the first approaches used toimplement audio morphing.y(t) = αs1(t) + (1 − α)s2(t)PRO: Low computational cost;CONS: When the timbre of the in-put signals is slightly different, boththe references could be individuallyperceivable.Linear Interpolation (LI) Based on harmonic modeling [4] [5]:s(t) =Nn=1An(t)cos[θn(t)] + e(t)The morphed sound is obtained in-terpolating the fundamental frequen-cy, the harmonics timbre, and the re-sidual envelope.PRO: Capability of morphing soundswith different timbre;CONS: Higher computational co-st than LI, some problems withpercussive sounds.Sinusoidal Plus Noise (SMS)Andrea Primavera Audio Morphing for Percussive Hybrid Sound Generation 3/21
  8. 8. Audio MorphingProposed AlgorithmResultsConclusionProposed AlgorithmPreProcessingProcessingAn automatic audio morphing algorithm employed to generate newhybrid percussive sounds is presented here. It is based on:• PreProcessing (frequency domain);• Processing exploiting linear interpolation (time domain).Proposed AlgorithmFig.1 Block diagram of the proposed morphing algorithm: yellow indicates thefrequency domain while blue represents the time domain.Andrea Primavera Audio Morphing for Percussive Hybrid Sound Generation 4/21
  9. 9. Audio MorphingProposed AlgorithmResultsConclusionProposed AlgorithmPreProcessingProcessingThe PreProcessing operation is performed on both the signal referencesin order to obtain hybrid sounds that change in a linear fashion when theinterpolation factor varies linearly.It allows to align the samples before executing the morphing exploitingcross-correlation.1. Time AlignAndrea Primavera Audio Morphing for Percussive Hybrid Sound Generation 5/21
  10. 10. Audio MorphingProposed AlgorithmResultsConclusionProposed AlgorithmPreProcessingProcessingThe PreProcessing operation is performed on both the signal referencesin order to obtain hybrid sounds that change in a linear fashion when theinterpolation factor varies linearly.It allows to align the samples before executing the morphing exploitingcross-correlation.1. Time AlignAndrea Primavera Audio Morphing for Percussive Hybrid Sound Generation 5/21
  11. 11. Audio MorphingProposed AlgorithmResultsConclusionProposed AlgorithmPreProcessingProcessingThe PreProcessing operation is performed on both the signal referencesin order to obtain hybrid sounds that change in a linear fashion when theinterpolation factor varies linearly.• The AR model is employed instead of the ADSR;• Attack and release are determined using the Amplitude CentroidTrajectory [6] [7] approach and evaluating the T60 [8] for eachaudio sample.Fig. 2: Time evolution of the spectralcentroid for a percussive sample.SpectralCentroid[t] =Mb=1 fb[t]ab[t]Mb=1 ab[t]where:f is the frequency and a is the amplitude ofthe b up to M bands, evaluated using FFT.2. AR Model AnalysisAndrea Primavera Audio Morphing for Percussive Hybrid Sound Generation 6/21
  12. 12. Audio MorphingProposed AlgorithmResultsConclusionProposed AlgorithmPreProcessingProcessingThe PreProcessing operation is performed on both the signal referencesin order to obtain hybrid sounds that change in a linear fashion when theinterpolation factor varies linearly.The release portions are time stretched or compressed as a functionof the interpolation factor value:tr = αtr1 + (1 − α)tr23. Time StretchingAndrea Primavera Audio Morphing for Percussive Hybrid Sound Generation 7/21
  13. 13. Audio MorphingProposed AlgorithmResultsConclusionProposed AlgorithmPreProcessingProcessingThe Processing operation is executed on the preprocessed signalreferences using the linear interpolation approach.Since percussive sounds are typically characterized by a noisy spectrumand “similar” timbre this approach is preferable to additive synthesis.Linear InterpolationAndrea Primavera Audio Morphing for Percussive Hybrid Sound Generation 8/21
  14. 14. Audio MorphingProposed AlgorithmResultsConclusionProposed AlgorithmPreProcessingProcessingThe Processing operation is executed on the preprocessed signalreferences using the linear interpolation approach.Since percussive sounds are typically characterized by a noisy spectrumand “similar” timbre this approach is preferable to additive synthesis.Linear InterpolationAndrea Primavera Audio Morphing for Percussive Hybrid Sound Generation 8/21
  15. 15. Audio MorphingProposed AlgorithmResultsConclusionResultsAnalysis of Time and Spectral Features1 Listening Test (MOS)2 Listening Test (MDS)Several tests have been carried out to evaluate the effectiveness of theproposed approach through objective and subjective comparisons.Real recordings of drum kitshave been employed using theBFD2 sound library [9].Andrea Primavera Audio Morphing for Percussive Hybrid Sound Generation 9/21
  16. 16. Audio MorphingProposed AlgorithmResultsConclusionResultsAnalysis of Time and Spectral Features1 Listening Test (MOS)2 Listening Test (MDS)Several tests have been carried out to evaluate the effectiveness of theproposed approach through objective and subjective comparisons.Spectral centroid evolution and release time have been evaluated as afunction of the interpolation factor.Analyzed algorithms:• Linear Interpolation (LI);• Linear Interpolation with Preprocessing (LIP);• Sinusoidal plus Residual (SMS).Morphing examples:• Different kind of snare drums (i.e, electronic and classic snare);• Toms of different size and brand;• Hi-hats of different brand.Analysis of Time and Spectral Features (Objective Measure)Andrea Primavera Audio Morphing for Percussive Hybrid Sound Generation 10/21
  17. 17. Audio MorphingProposed AlgorithmResultsConclusionResultsAnalysis of Time and Spectral Features1 Listening Test (MOS)2 Listening Test (MDS)Several tests have been carried out to evaluate the effectiveness of theproposed approach through objective and subjective comparisons.In order to evaluate the audio perception of the produced hybrid soundstwo different listening tests [10] have been performed:• Mean Opinion Score (MOS) −→ Audio quality, estimatedinterpolation factor;• MultiDimensional Scaling (MDS) −→ Perceptual space.Analyzed algorithms:• Linear Interpolation (LI);• Linear Interpolation with Preprocessing (LIP);• Sinusoidal plus Residual (SMS).Morphing examples:• Different kind of snare drums (i.e, electronic and classic snare);Listening test (Subjective Measure)Andrea Primavera Audio Morphing for Percussive Hybrid Sound Generation 11/21
  18. 18. Audio MorphingProposed AlgorithmResultsConclusionResultsAnalysis of Time and Spectral Features1 Listening Test (MOS)2 Listening Test (MDS)SNARE TOM HI-HAT0 0.2 0.4 0.6 0.8 10.811.21.41.61.8Interpolation FactorReleaseTime(sec)LILIPSMS0 0.2 0.4 0.6 0.8 12.833.23.43.63.84Interpolation FactorReleaseTime(sec)LILIPSMS0 0.2 0.4 0.6 0.8 11.41.61.822.22.42.62.8Interpolation FactorReleaseTime(sec)LILIPSMSFig. 3 Release time as a function of different interpolation factor.0 0.2 0.4 0.6 0.8 12000250030003500Interpolation FactorSpectralcentroid(Hz)LILIPSMS0 0.2 0.4 0.6 0.8 1820840860880900920940960980Interpolation FactorSpectralcentroid(Hz)LILIPSMS0 0.2 0.4 0.6 0.8 18000850090009500Interpolation FactorSpectralcentroid(Hz)LILIPSMSFig.4 Spectral centroid evolution as a function of different interpolation factor.• Release time changes linearly with the change of the interp. factor;• The spectral centroid does not seem affected by the preprocessing.ConsiderationsAndrea Primavera Audio Morphing for Percussive Hybrid Sound Generation 12/21
  19. 19. Audio MorphingProposed AlgorithmResultsConclusionResultsAnalysis of Time and Spectral Features1 Listening Test (MOS)2 Listening Test (MDS)Based on MOS, the estimated interpolation factor and the perceivedaudio quality for LI, LIP and SMS are evaluated in the morphing betweena classic and electronic snare.0 0.2 0.4 0.6 0.8 100.20.40.60.81Interpolation FactorEst.Interp.FactorLILIPSMSFig. 5 Estimated interpolation factor.LI LIP SMSQuality 0.90 0.88 0.10STD 0.06 0.07 0.08Tab. 1 Percived audio quality expressed ina scale from 0 to 1 (0 bad - 1 excellent).• For the proposed approach (LIP) the estimated interpolation factorchanges roughly linearly with the change of the interpolation factor.• For linear interpolation (LI) the factor 0.5 was perceived as near 0.8.• Samples generated using SMS sound unnatural, this confirm theharmonic models limitation in the reproduction of percussive sound.ConsiderationsAndrea Primavera Audio Morphing for Percussive Hybrid Sound Generation 13/21
  20. 20. Audio MorphingProposed AlgorithmResultsConclusionResultsAnalysis of Time and Spectral Features1 Listening Test (MOS)2 Listening Test (MDS)Based on MDS [11] [12], the test aims to recreate the correspondingperceptual spaces of the hybrid sound generated in morphing between aclassic and electronic snare with LI and LIP.−0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6−0.8−0.6−0.4−0.200.20.40.6123456781 Dimension2DimensionFig.6 MDS analysis obtained with LI.−0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6−0.8−0.6−0.4−0.200.20.40.612 3 456781 Dimension2DimensionFig.7 MDS analysis obtained with LIP.LI LIPCORR 0.986 0.980P-VAL 7.2*10−6 1.9*10−5Tab. 2 Correlation between firstdimension and release time.LI LIPCORR 0.66 0.77P-VAL 0.07 0.02Tab. 3 Correlation between seconddimension and audio quality.Andrea Primavera Audio Morphing for Percussive Hybrid Sound Generation 14/21
  21. 21. Audio MorphingProposed AlgorithmResultsConclusionResultsAnalysis of Time and Spectral Features1 Listening Test (MOS)2 Listening Test (MDS)Fig.8 Correlation between firstdimension and release time for LI andLIP.The curves have been shifted onthe y-axis in order to enhance thereadability.• High correlation between therelease time and themovement along the firstdimension.• Release time is an importantfactor in the evaluation ofpercussive audio morphingalgorithms.• Although the correlationbetween the seconddimension and the audioquality is not zero, the valuesobtained are not sufficient toestablish a strong relationshipbetween them.ConsiderationsAndrea Primavera Audio Morphing for Percussive Hybrid Sound Generation 15/21
  22. 22. Audio MorphingProposed AlgorithmResultsConclusionConclusionFuture WorksQuestionsBibliographyIn conclusion:• An automatic audio morphing technique applied to percussivemusical instruments has been presented in this work.• The presented technique is based on preprocessing of the sound andlinear interpolation time domain.• Different tests have been carried out in order to evaluate theproposed approach, in terms of subjective evaluation and objectivemeasures comparing it with the previous state of the art.• Objective analysis confirms that release time changes linearly withthe change of the interpolation factor while spectral centroid doesnot seem affected by the preprocessing.• Listening tests confirm a linear evolution in the perception of thehybrid sounds using the presented approach.• MDS multidimensional scaling suggests that release time is the mostimportant feature in the perception of hybrid percussive sounds.Andrea Primavera Audio Morphing for Percussive Hybrid Sound Generation 16/21
  23. 23. Audio MorphingProposed AlgorithmResultsConclusionConclusionFuture WorksQuestionsBibliographyFuture work will be oriented toward:• Further investigation of the proposed approach analyzing theobtained performance for more dissimilar sounds.• Real time implementation of the approach, considering an embeddedplatform.• Refinement of the preprocessing phase, extending it with the ADSRmodel.Andrea Primavera Audio Morphing for Percussive Hybrid Sound Generation 17/21
  24. 24. Audio MorphingProposed AlgorithmResultsConclusionConclusionFuture WorksQuestionsBibliographyQUESTIONS?Andrea Primavera Audio Morphing for Percussive Hybrid Sound Generation 18/21
  25. 25. Audio MorphingProposed AlgorithmResultsConclusionConclusionFuture WorksQuestionsBibliographyM. H. Serra, D Rubine, and R. Dannenberg,“Analysis and synthesis of tones by spectral interpolation,”J. Audio Eng. Soc., vol. 38, no. 3, pp. 111–128, March. 1990.N Osaka,“Timbre interpolation of sounds using a sinusoidal model,”in Proc. Int. Computer Music Conference, Banff, Canada, Sep 1995,vol. 34.M. Slaney, M. Covell, and B. Lassiter,“Automatic Audio Morphing,”in Proc. IEEE International Conference on Acoustics, Speech andSignal Processing, Atlanta, May. 1996.X. Serra and J. Smith,“Spectral Modeling Synthesis: A Sound Analysis/Synthesis SystemBased on a Deterministic Plus Stochastic Decomposition,”J. Computer Music, vol. 14, no. 4, pp. 12–24, 1990.Andrea Primavera Audio Morphing for Percussive Hybrid Sound Generation 19/21
  26. 26. Audio MorphingProposed AlgorithmResultsConclusionConclusionFuture WorksQuestionsBibliographyF. Boccardi and C. Drioli,“Sound morphing with Gaussian mixture models,”in Proc. International Conference on Digital Audio Effects),Limerick, Ireland, December 2001.John Hajda,“A New Model for Segmenting the Envelope of Musical Signals: TheRelative Salience of Steady State versus Attack, Revisited,”in Proc. 101th Audio Engineering Society Convention (AES’96), LosAngeles, USA, Nov. 1996.J.J. Burred, X. Rodet, and M. Caetano,“Automatic Segmentation of the Temporal Evolution of IsolatedAcoustic Musical Instrument Sounds Using Spectro-Temporal Cues,”in Proc. International Conference on Digital Audio Effects(DAFX’10), Graz, AU, Sep. 2010.Sabine, W.C.,“Collected Papers on Acoustics,” 1964, reprinted by Dover, NewYork.Andrea Primavera Audio Morphing for Percussive Hybrid Sound Generation 20/21
  27. 27. Audio MorphingProposed AlgorithmResultsConclusionConclusionFuture WorksQuestionsBibliography“BFD2 version 2.0.1 Acoustic Drum Production Environment,fxpansion,” .ITU-R BS. 1534,“Method for subjective listening tests of intermediate audio quality,”2001.D. Williams and T. Brookes,“Perceptually motivated audio morphing: warmth,”in Proc. of 128th Audio Engineering Society Convention (AES’10),London, UK, May. 2010.A. Zacharakis and J. Reiss,“An additive synthesis technique for independent modification of theauditory perceptions of brightness and warmth,”in Proc. of 130th Audio Engineering Society Convention (AES’11),London, UK, May. 2011.Andrea Primavera Audio Morphing for Percussive Hybrid Sound Generation 21/21

×