Your SlideShare is downloading. ×
BPM JS
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

BPM JS

1,524
views

Published on

Onset Detection in the Browser …

Onset Detection in the Browser
JSConf 2013

Published in: Technology

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,524
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
11
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. jsbpmonset detection in the BrowserJanessa Det - WEB eng @TWITTERNYC
  • 2. hello.bpmjsjanessa det (@jandet)twitter nycSoftware engineerWeb Teamcolumbia universityM.S. Computer scienceGraphics & Visionduke universityB.S.E. biomedical engineering
  • 3. IRON MANonset detectionbpm. what?bpmjsjanessa det (@jandet)onset detectionfundamental to:sonar, speech, EKGs,movement detectionthisthingIRON MAN
  • 4. IRON MANonset detectionbpm. what?bpmjsjanessa det (@jandet)onset detectionfundamental to:sonar, speech, EKGs,movement detectionthisthingIRON MAN
  • 5. AUTOMATICTRANSCRIPTIONOF MUSIConset detectionbpm. what?bpmjsjanessa det (@jandet)AUTOMATICTRANSCRIPTIONOF MUSICthisthing1. instrument identification2. instrument isolation3. pitch detection4. dynamics
  • 6. accessibleapplicablesharablecross-platformfastdigital signals processingin the browser. why?bpmjsjanessa det (@jandet)accessibleapplicablesharablecross-platformfastdigital signals processing
  • 7. javascriptin the browser. why?bpmjsjanessa det (@jandet)becausejavascriptand it would be coolis funto play with
  • 8. diversediversity is powerful.bpmjsjanessa det (@jandet)the browserisdiversethe community
  • 9. diversediversity is powerful.bpmjsjanessa det (@jandet)the browserisdiversethe community
  • 10. diversity is powerful.bpmjsjanessa det (@jandet)yeah,that’s merealitythe dream"finding ways to breedengineering research, music, and artwith the trusty steed of javascriptto make the magical unicornsof my imagination a reality"
  • 11. diversity is powerful.bpmjsjanessa det (@jandet)yeah,that’s merealitythe dream"finding ways to breedengineering research, music, and artwith the trusty steed of javascriptto make the magical unicornsof my imagination a reality"ok, let’sbreed someunicorns
  • 12. onset detection?how do we doonset detection?
  • 13. GENREStemposbeatsonsetswhat is bpm?bpmjsjanessa det (@jandet)1. pitch2. notes3. percussive beatsDRUM AND BASS 160-180 BPMHouse 118-135 BPMhip-hop 115 BPMConcert marches 120 BPMScreamers 130-150 BPMbeatsGENRESwhat is a beat?1. pitched non-percussive2. pitched percussive3. non-pitched percussive4. complex mixedonsetstemposLargo 40-60 BPMLarghetto 60-66 BPMAdagio 66-76 BPMAndante 76-108 BPMModerato 108-120 BPMAllegro 120-168 BPMPresto 168-200 BPMPrestissimo 200+ BPM
  • 14. bpmjsjanessa det (@jandet)detection method.DETECTIONFUNCTIONstftDETECTIONFUNCTIONSTFT PEAK PICKINGPEAK PICKING BPMBPMPrep the signalfor DetectionFunctionsSelectively maximizefor onset propertiesPick out peak times fromDetection FunctionCalculateBPMಠ_ಠ
  • 15. bpmjsjanessa det (@jandet)detection method.DETECTIONFUNCTIONstftDETECTIONFUNCTIONSTFT PEAK PICKINGPEAK PICKING BPMBPMPrep the signalfor DetectionFunctionsSelectively maximizefor onset propertiesPick out peak times fromDetection FunctionCalculateBPM
  • 16. stft. short-time fourier transform.bpmjsjanessa det (@jandet)4. zero paddingZero pad to nearest power of 22. hamming windowApply windowing function -- Hamming3. normalizationNormalize magnitude5. fast fourier transformConvert to frequency domain1. window slicingSlice signal into overlapping windows
  • 17. detection functions.bpmjsjanessa det (@jandet)frequency based methods3. phase deviation1. High frequency content2. spectral difference4. euclidian distance
  • 18. peak pickingbuild thresholdpeak picking.bpmjsjanessa det (@jandet)build threshold1. median filterSmooth out the signal for thresholding2. adaptive thresholdingThreshold by median-filtered signal3. initial peak findingIdentify local maxima4. thresholding peaksApply adaptive threshold to peak findings5. refractory periodApply minimum time between peaks as thresholdpeak picking
  • 19. 120 BPMbpm calculation.bpmjsjanessa det (@jandet)0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0peak times (in seconds)0.51 beat / 0.5 sec * 60 sec / 1 min= 120 BPM
  • 20. in javascript?how do we do itin javascript?
  • 21. color profile ofmusic characterization.bpmjsjanessa det (@jandet)color profile ofso insane by discoveryaverage ~ 115 bpm0:30 1:000:15 0:45 1:15 1:45 2:151:30 2:000:00 2:30 3:002:45 3:15
  • 22. dsp.jsweb audiocomplex arithmetica lot of loops & mathbrowserifytools used.bpmjsjanessa det (@jandet)dsp.jsweb audiocomplex arithmetica lot of loops & mathbrowserifyi
  • 23. modules.bpmjsjanessa det (@jandet)peak pickingdetectionbpmweb audioAudioBuffer nodehfc sd pd e. dist.dsp.jsfft windowingcomplexstft
  • 24. how to not break chromejs challenges.bpmjsjanessa det (@jandet)float32arrayhow to not break chromecompatibilitycomplex arithmeticmatrices
  • 25. how to not break chromejs challenges.bpmjsjanessa det (@jandet)float32arrayhow to not break chromecompatibilitycomplex arithmeticmatricesNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
  • 26. 0. signal winndowstft. short-time fourier transform.bpmjsjanessa det (@jandet)3. ZERO PADDING1. HAMMING WINDOW2. NORMALIZATION4. FFT -> Complex objects
  • 27. bpmjsjanessa det (@jandet)detection functions.hfcsdpde. dist.not bad...
  • 28. bpmjsjanessa det (@jandet)peak picking.detectionfunction (ed)median filteredthresholdedetc
  • 29. THE DREAM:matlab!!benchmarksoptimizations& performancebpmjsjanessa det (@jandet)1. web workers for stft2. SLICE WELL-SELECTED WINDOWSbenchmarksvs. matlab!!THE DREAM: “REAL-TIME”3. asm.js, enscripten, LLjs
  • 30. weaknesses.future work.weaknesses.bpmjsjanessa det (@jandet)1. sensitive tuning parameters2. a priori knowledge helpful to select detection functions3. median filtering or aggressive thresholding can cripplethe detection functionfuture work.1. only scratching the surface of research2. many detection functions3. onset detection for different applications
  • 31. is the unicorn beautiful?ok so how did i do?is the unicorn beautiful?
  • 32. bpmjsjanessa det (@jandet)ok, it feels alittle forcedbut we’re gettingthere!!
  • 33. the web is changing.bpmjsjanessa det (@jandet)web audio apidsp.jsGod Blessnpm
  • 34. rethinkingweb engineering.bpmjsjanessa det (@jandet)what does it mean to be aweb engineer?how about more engineering?this is not thefinal frontier
  • 35. bpmjsjanessa det (@jandet)embrace diversityopen community of learningknowledge builds upon itselfcombining fields is very powerfulthink outside the boxbreed those unicornsrethinkingweb engineering.
  • 36. bpmjsjanessa det (@jandet)thanks.go forth and breed unicorns
  • 37. main referencesBeat Detection for Automated Music Transcription:An exploration of Onset Detection Algorithms.http://bingweb.binghamton.edu/~ahess2/Onset_Detection_Nov302011.pdfThe Scientist and Engineers Guide toDigital Signal ProcessingBy Steven W. Smith, Ph.D.http://www.dspguide.com/pdfbook.htmadditional resources:Gamedev: beat detection algorithmshttp://archive.gamedev.net/archive/reference/programming/features/beatdetection/index.htmlBPM Databasehttp://www.bpmdatabase.com/