BPM JS

2,460 views

Published on

Onset Detection in the Browser
JSConf 2013

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,460
On SlideShare
0
From Embeds
0
Number of Embeds
335
Actions
Shares
0
Downloads
18
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

BPM JS

  1. 1. jsbpmonset detection in the BrowserJanessa Det - WEB eng @TWITTERNYC
  2. 2. hello.bpmjsjanessa det (@jandet)twitter nycSoftware engineerWeb Teamcolumbia universityM.S. Computer scienceGraphics & Visionduke universityB.S.E. biomedical engineering
  3. 3. IRON MANonset detectionbpm. what?bpmjsjanessa det (@jandet)onset detectionfundamental to:sonar, speech, EKGs,movement detectionthisthingIRON MAN
  4. 4. IRON MANonset detectionbpm. what?bpmjsjanessa det (@jandet)onset detectionfundamental to:sonar, speech, EKGs,movement detectionthisthingIRON MAN
  5. 5. AUTOMATICTRANSCRIPTIONOF MUSIConset detectionbpm. what?bpmjsjanessa det (@jandet)AUTOMATICTRANSCRIPTIONOF MUSICthisthing1. instrument identification2. instrument isolation3. pitch detection4. dynamics
  6. 6. accessibleapplicablesharablecross-platformfastdigital signals processingin the browser. why?bpmjsjanessa det (@jandet)accessibleapplicablesharablecross-platformfastdigital signals processing
  7. 7. javascriptin the browser. why?bpmjsjanessa det (@jandet)becausejavascriptand it would be coolis funto play with
  8. 8. diversediversity is powerful.bpmjsjanessa det (@jandet)the browserisdiversethe community
  9. 9. diversediversity is powerful.bpmjsjanessa det (@jandet)the browserisdiversethe community
  10. 10. diversity is powerful.bpmjsjanessa det (@jandet)yeah,that’s merealitythe dream"finding ways to breedengineering research, music, and artwith the trusty steed of javascriptto make the magical unicornsof my imagination a reality"
  11. 11. diversity is powerful.bpmjsjanessa det (@jandet)yeah,that’s merealitythe dream"finding ways to breedengineering research, music, and artwith the trusty steed of javascriptto make the magical unicornsof my imagination a reality"ok, let’sbreed someunicorns
  12. 12. onset detection?how do we doonset detection?
  13. 13. GENREStemposbeatsonsetswhat is bpm?bpmjsjanessa det (@jandet)1. pitch2. notes3. percussive beatsDRUM AND BASS 160-180 BPMHouse 118-135 BPMhip-hop 115 BPMConcert marches 120 BPMScreamers 130-150 BPMbeatsGENRESwhat is a beat?1. pitched non-percussive2. pitched percussive3. non-pitched percussive4. complex mixedonsetstemposLargo 40-60 BPMLarghetto 60-66 BPMAdagio 66-76 BPMAndante 76-108 BPMModerato 108-120 BPMAllegro 120-168 BPMPresto 168-200 BPMPrestissimo 200+ BPM
  14. 14. bpmjsjanessa det (@jandet)detection method.DETECTIONFUNCTIONstftDETECTIONFUNCTIONSTFT PEAK PICKINGPEAK PICKING BPMBPMPrep the signalfor DetectionFunctionsSelectively maximizefor onset propertiesPick out peak times fromDetection FunctionCalculateBPMಠ_ಠ
  15. 15. bpmjsjanessa det (@jandet)detection method.DETECTIONFUNCTIONstftDETECTIONFUNCTIONSTFT PEAK PICKINGPEAK PICKING BPMBPMPrep the signalfor DetectionFunctionsSelectively maximizefor onset propertiesPick out peak times fromDetection FunctionCalculateBPM
  16. 16. stft. short-time fourier transform.bpmjsjanessa det (@jandet)4. zero paddingZero pad to nearest power of 22. hamming windowApply windowing function -- Hamming3. normalizationNormalize magnitude5. fast fourier transformConvert to frequency domain1. window slicingSlice signal into overlapping windows
  17. 17. detection functions.bpmjsjanessa det (@jandet)frequency based methods3. phase deviation1. High frequency content2. spectral difference4. euclidian distance
  18. 18. peak pickingbuild thresholdpeak picking.bpmjsjanessa det (@jandet)build threshold1. median filterSmooth out the signal for thresholding2. adaptive thresholdingThreshold by median-filtered signal3. initial peak findingIdentify local maxima4. thresholding peaksApply adaptive threshold to peak findings5. refractory periodApply minimum time between peaks as thresholdpeak picking
  19. 19. 120 BPMbpm calculation.bpmjsjanessa det (@jandet)0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0peak times (in seconds)0.51 beat / 0.5 sec * 60 sec / 1 min= 120 BPM
  20. 20. in javascript?how do we do itin javascript?
  21. 21. color profile ofmusic characterization.bpmjsjanessa det (@jandet)color profile ofso insane by discoveryaverage ~ 115 bpm0:30 1:000:15 0:45 1:15 1:45 2:151:30 2:000:00 2:30 3:002:45 3:15
  22. 22. dsp.jsweb audiocomplex arithmetica lot of loops & mathbrowserifytools used.bpmjsjanessa det (@jandet)dsp.jsweb audiocomplex arithmetica lot of loops & mathbrowserifyi
  23. 23. modules.bpmjsjanessa det (@jandet)peak pickingdetectionbpmweb audioAudioBuffer nodehfc sd pd e. dist.dsp.jsfft windowingcomplexstft
  24. 24. how to not break chromejs challenges.bpmjsjanessa det (@jandet)float32arrayhow to not break chromecompatibilitycomplex arithmeticmatrices
  25. 25. how to not break chromejs challenges.bpmjsjanessa det (@jandet)float32arrayhow to not break chromecompatibilitycomplex arithmeticmatricesNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN...NaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaNNaN
  26. 26. 0. signal winndowstft. short-time fourier transform.bpmjsjanessa det (@jandet)3. ZERO PADDING1. HAMMING WINDOW2. NORMALIZATION4. FFT -> Complex objects
  27. 27. bpmjsjanessa det (@jandet)detection functions.hfcsdpde. dist.not bad...
  28. 28. bpmjsjanessa det (@jandet)peak picking.detectionfunction (ed)median filteredthresholdedetc
  29. 29. THE DREAM:matlab!!benchmarksoptimizations& performancebpmjsjanessa det (@jandet)1. web workers for stft2. SLICE WELL-SELECTED WINDOWSbenchmarksvs. matlab!!THE DREAM: “REAL-TIME”3. asm.js, enscripten, LLjs
  30. 30. weaknesses.future work.weaknesses.bpmjsjanessa det (@jandet)1. sensitive tuning parameters2. a priori knowledge helpful to select detection functions3. median filtering or aggressive thresholding can cripplethe detection functionfuture work.1. only scratching the surface of research2. many detection functions3. onset detection for different applications
  31. 31. is the unicorn beautiful?ok so how did i do?is the unicorn beautiful?
  32. 32. bpmjsjanessa det (@jandet)ok, it feels alittle forcedbut we’re gettingthere!!
  33. 33. the web is changing.bpmjsjanessa det (@jandet)web audio apidsp.jsGod Blessnpm
  34. 34. rethinkingweb engineering.bpmjsjanessa det (@jandet)what does it mean to be aweb engineer?how about more engineering?this is not thefinal frontier
  35. 35. bpmjsjanessa det (@jandet)embrace diversityopen community of learningknowledge builds upon itselfcombining fields is very powerfulthink outside the boxbreed those unicornsrethinkingweb engineering.
  36. 36. bpmjsjanessa det (@jandet)thanks.go forth and breed unicorns
  37. 37. main referencesBeat Detection for Automated Music Transcription:An exploration of Onset Detection Algorithms.http://bingweb.binghamton.edu/~ahess2/Onset_Detection_Nov302011.pdfThe Scientist and Engineers Guide toDigital Signal ProcessingBy Steven W. Smith, Ph.D.http://www.dspguide.com/pdfbook.htmadditional resources:Gamedev: beat detection algorithmshttp://archive.gamedev.net/archive/reference/programming/features/beatdetection/index.htmlBPM Databasehttp://www.bpmdatabase.com/

×