Marsyas

  • 4,781 views
Uploaded on

Presentation of the Marsyas opensource project. …

Presentation of the Marsyas opensource project.
More info at http://marsyas.sf.net

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
4,781
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
92
Comments
0
Likes
3

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. An open source platform for multimedia processing http://marsyas.sourceforge.net Nov 2010Saturday, 20 November, 2010
  • 2. Notice This work is licensed under the Creative Commons Attribution-Share Alike 2.5 Portugal License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/2.5/pt/ or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA.Saturday, 20 November, 2010
  • 3. Summary Marsyas Overview Users & Applications Usage Scenarios Architecture Interoperability http://marsyas.sourceforge.net MarsyasX Future WorkSaturday, 20 November, 2010
  • 4. Marsyas Overview Software framework for media analysis, synthesis and retrieval Open source (GPL license) http://www.fsf.org Efficient and extensible framework design original emphasis on Music Information Retrieval (MIR) ; now Multimedia! C++, OOP Multiplatform (Linux, MacOSX®, MS Windows®, …) Provides a variety of building blocks for performing common audio tasks: sound/video file IO, audio/video IO, signal processing and machine learning modules blocks can be combined into data flow networks that can be modified and controlled dynamically while they process data in soft real-time. GM M WAV WAV source Hanning FFT FFT source KNN LPCSaturday, 20 November, 2010
  • 5. Marsyas Overview Marsyas Brief History 1998 ~2000 Created by George Tzanetakis during his PhD activities at Princeton 2000 ~2002 Marsyas 0.1 First stable revisions of the toolkit Distributions hosted at SourceForge Creation of a developer community User and Developer Mailing lists 2002 ~ … Marsyas 0.2 Major framework revision SourceForge SubVersion http://marsyas.sourceforge.net/ http://www.cs.princeton.edu/~gtzanSaturday, 20 November, 2010
  • 6. Marsyas Overview Marsyas Core Developers (Present and past...) http://marsyas.info/community/people George Tzanetakis Main culprit (also main designer and developer) Luís Gustavo Martins Refactoring freak Graham Percieval Documentation overlord Luís Filipe Teixeira Marsyas X motor-head Mathieu Lagrange Biggest network award Steven Ness Ruby and Web guy Check updated list at the website!Saturday, 20 November, 2010
  • 7. Marsyas Community https://sourceforge.net/project/stats/graph/detail-graph.php? group_id=84982&ugn=marsyas&type=prdownload&mode=alltime&file_id=0&graph=1Saturday, 20 November, 2010
  • 8. Marsyas Community Google AnalyticsSaturday, 20 November, 2010
  • 9. Marsyas Community Google Analytics (Nov 2010)Saturday, 20 November, 2010
  • 10. Marsyas Community Mailing lists Developer https://lists.sourceforge.net/lists/listinfo/marsyas-developers User https://lists.sourceforge.net/lists/listinfo/marsyas-users Tracker Feature RequestsSaturday, 20 November, 2010
  • 11. Marsyas Community SourceForge Community Choice Awards 2009 Finalist http://community.sourceforge.net/cca_winners/bestacademia.htmlSaturday, 20 November, 2010
  • 12. Marsyas Project Analysis Marsyas value? Ohloh Analysis http://www.ohloh.net/p/marsyasSaturday, 20 November, 2010
  • 13. Users & Applications Musicream (Masataka Goto) http://staff.aist.go.jp/m.goto/Musicream/ Music playback system with similarity capabilities Uses Marsyas as its music similarity engineSaturday, 20 November, 2010
  • 14. Users & Applications SndPeek (Princeton) http://www.cs.princeton.edu/sound/software/sndpeek/ Uses Marsyas 0.1 for: FFT magnitude spectrum real-time spectral feature extraction centroid rms flux rolloffSaturday, 20 November, 2010
  • 15. Users & Applications Marsyas @ INESC Porto http://www.inescporto.pt Audio Analysis Software prototypes: Feature Extraction Audio segmentation/classification Audio fingerprinting Speaker Segmentation Music and Auditory Scene Analysis Video Analysis Software prototypes: Background modeling and subtraction Local feature extraction for object matching http://www.inescporto.pt/~lfpt http://www.inescporto.pt/~lmartinsSaturday, 20 November, 2010
  • 16. Users & Applications Desert Island Undergraduate work at the Univ. Missouri Kansas Jared Hoberock Dan Kelly Ben TietgenSaturday, 20 November, 2010
  • 17. Related Work Open Source frameworks CLAM (http://clam.iua.upf.edu/) STK (http://ccrma.stanford.edu/software/stk/) Chuck (http://chuck.cs.princeton.edu/) PureData (Pd) (http://crca.ucsd.edu/~msp/software.html) Open Sound Control (OSC) (http://cnmat.berkeley.edu/OpenSoundControl/) FAUST (http://faudiostream.sourceforge.net/) EyesWeb (http://www.infomus.dist.unige.it/EywMain.html) ... Commercial toolkits MAX/MSP® (http://www.cycling74.com/) MATLAB® Simulink® (http://www.mathworks.com/products/simulink/) LabView® (http://www.ni.com/labview/) DirectShow® GraphEdit (http://www.microsoft.com) ...Saturday, 20 November, 2010
  • 18. Usage Scenarios Marsyas command line tools Demonstrate key capabilities of the framework Some are actually research tools Efficient and can execute in real-time ANSI C++ only core several optional libraries MP3 reading/writing (libMad) Tools and examples: sfplay bextract phasevocoder sfplugin …Saturday, 20 November, 2010
  • 19. Usage Scenarios Playing audio files sfplay > sfplay foo.wav > sfplay –s 10.0 –l 3.2 –r 2.5 –g 0.5 foo1.wav foo2.au –f output.wav > sfplay –l 3.0 foo.wav > sfplay foo.wav –p playback.mpl > -s : start time for playback -l : length of playback -r : repeat times -g : volume (gain) value -p : playback.mpl (save playback network as a .mpl plugin file)Saturday, 20 November, 2010
  • 20. Usage Scenarios Machine Learning: feature extraction and training of classifiers bextract > bextract -e STFTMFCC music.mf speech.mf -p ms.mpl -w myweka.arff > -e STFTMFCC extracts spectral and MFCC features music.mf, speech.mf lists of sound files (collections) -w myWeka.arff WEKA file with extracted features -p ms.mpl “trained” Marsyas plug-in for realtime music/speech classificationSaturday, 20 November, 2010
  • 21. Usage Scenarios Marsyas plugins (.mpl files) Allow to dynamically recreate a processing network in runtime sfplugin e.g. Audio playback > sfplugin –p playback.mpl foo.wav > e.g. Realtime audio classification > sfplugin –p ms.mpl unknownAudioSignal.wav >Saturday, 20 November, 2010
  • 22. Usage Scenarios Digital Signal Processing e.g. phasevocoder > phasevocoder –p 1.4 -s 100 > http://eceserv0.ece.wisc.edu/~sethares/vocoders/phasevocoder.html e.g. onset detector > mudbox -t onsets myMusic.wav >Saturday, 20 November, 2010
  • 23. Architecture Marsyas 0.2 New dataflow model of audio computation hierarchical messaging system used to control the dataflow network inspired on Open Sound Control (OSC) general matrices instead of 1-D arrays as dataSaturday, 20 November, 2010
  • 24. Architecture MarSystem Slices Separating things that happen at the same time from things that happen in different times Correct semantics for spectral processingSaturday, 20 November, 2010
  • 25. Architecture Implicit Patching VS Explicit Patching # EXPLICIT PATCHING source, F1, F2, F3, destination # connect the appropriate in/out ports connect(source, F1); connect(source, F2); connect(source, F3); connect(F1, destination); connect(F2, destination); connect(F3, destination); # IMPLICIT PATCHING source, F1, F2, F3, destination Fanout(F1, F2, F3) Series(source, Fanout, destination);Saturday, 20 November, 2010
  • 26. Architecture MarSystem Composites Series Fanout Fanin Parallel Accumulator …Saturday, 20 November, 2010
  • 27. Architecture Implicit VS Explicit Patching :: Neural Network #IMPLICIT PATCHING fanoutLayer1(ANN_Node11, …, ANN_Node1N) … fanoutLayerM(ANN_NodeM1, …,ANN_NodeMN) ANN_Series(fanoutLayer1, …, fanoutLayerM)Saturday, 20 November, 2010
  • 28. Architecture Typical feature extraction Time-frequency representation Frequency summarization (MFCC) Time summarization (Texture features)Saturday, 20 November, 2010
  • 29. Architecture Feature Extraction using Implicit Patching MarSystemManager mng; MarSystem* Series1 = mng.create("Series", “Series1"); MarSystem* Fanout1 = mng.create(“Fanout", “Fanout1"); MarSystem* Series2 = mng.create("Series", “Series2"); MarSystem* Fanout2 = mng.create(“Fanout”, “Fanout2”); MarSystem* Fanout3 = mng.create(“Fanout”, “Fanout3”); Fanout3->addMarSystem(mng.create(“Mean”, “Mean”)); Fanout3->addMarSystem(mng.create(“Variance”, “Variance”)); Fanout2->addMarSystem(mng.create(“Centroid”, “Centroid”)); Fanout2->addMarSystem(mng.create(“RollOff”, “Rolloff”)); Fanout2->addMarSystem(mng.create(“Flux”, “Flux”); Series2->addMarSystem(mng.create(“Spectrum”, “Spectrum”); Series2->addMarSystem(Fanout2); Fanout1->addMarSystem(mng.create(“ZeroCrossings”, “ZeroCrossings”); Fanout1->addMarSystem(Series2); Series1->addMarSystem(mng.create("SoundFileSource",“Source")); Series1->addMarSystem(Fanout1); Series1->addMarSystem(mng.create(“Memory”, “TextureMemory”)); Series1->addMarSystem(Fanout3); Series1->addMarSystem(mng.create(“classifier”, “Classifier”));Saturday, 20 November, 2010
  • 30. Interoperability Marsyas Audio and MIDI I/0 http://www.music.mcgill.ca/~gary/rtaudio/ http://www.music.mcgill.ca/~gary/rtmidi/ RtAudio Multiplatform C++ API for realtime audio input/output Linux (native ALSA, JACK, and OSS) MacOSX® Windows® (DirectSound® and ASIO®) SGI® RtMIDI Multiplatform C++ API for realtime MIDI input/output Linux (ALSA) MacOSX® Windows® (Multimedia Library) SGI®Saturday, 20 November, 2010
  • 31. Interoperability Marsyas & WEKA http://www.cs.waikato.ac.nz/ml/weka/ WEKA: Data Mining Software in Java Marsyas already includes some machine learning blocks Marsyas outputs extracted features as .arff files (WEKA) features can be opened in WEKA for further evaluation and data modelingSaturday, 20 November, 2010
  • 32. Interoperability Calling MATLAB® from C++ Marsyas code: MATLAB® engine API http://www.mathworks.com exchange data (i.e. matrices) in run-time between C++ and MATLAB® remotely execute commands in MATLAB® from a C++ routine Access to all MATLAB® toolboxes, algorithms and available routines Algorithmic validation of C++ routines Quick and easy evaluation of proof of concepts May not allow real-time operation… Not such a big problem when evaluating or developing algorithmsSaturday, 20 November, 2010
  • 33. Interoperability Calling MATLAB® from C++ Marsyas code: Marsyas::MATLABengine class // create a std::vector of real numbers Utility class std::vector<double> vector_real(4); vector_real[0] = 1.123456789; Wraps MATLAB® engine calls for most vector_real[1] = 2.123456789; POD types and Marsyas data types vector_real[2] = 3.123456789; vector_real[3] = 4.123456789; Easy to send/receive data to/from MATLAB® from anywhere in the code // send a std::vector<double> to MATLAB PUTVAR(vector_real, "vector_real"); // do some dummy math in MATLAB EVALUATE("mu = mean(vector_real);"); EVALUATE("sigma = std(vector_real);"); EVALUATE("vector_real = vector_real/max(vector_real);"); // get values from MATLAB double m, s; GETVAR(m, "mu"); GETVAR(s, "sigma"); GETVAR(vector_real, "vector_real");Saturday, 20 November, 2010
  • 34. Interoperability Python™ Bindings http://www.python.org easily create scripts for rapid testing and prototyping of data flow networks would require much more development effort in C++ bonus: no compiling overheads can also be embedded in C++ code, similarly to MATLAB® (TBD) less tools for signal processing in general, but can be used for many other purposes (“batteries included”) less licensing headachesSaturday, 20 November, 2010
  • 35. Interoperability Marsyas and Trolltech Qt4® http://www.trolltech.com Qt® Core features optionally used by Marsyas Multi-platform signal/slot architecture Multi-platform threads (multithreaded processing) Multi-platform database access Multi-platform XML I/O Qt® GUI Features optionally used by Marsyas Multi-platform Widgets Multi-platform OpenGL Qt4® is available as GPL open source code for all platformsSaturday, 20 November, 2010
  • 36. MarsyasX What is MarsyasX? Next step for Marsyas Evolution rather than an alternative implementation Generalization of the framework to support processing different media Key points: crossmodal processing expandable support interoperability GM AV MPEG M Contour source Sink AV MP WAV source S EG Hanning FFT ink source Motion ur to ConSaturday, 20 November, 2010
  • 37. MarsyasX MarsyasX brief history: MarsyasX started as a patch to Marsyas... ...add visual processing support However, the previous architecture has some restrictions: Some design options are audio-oriented No in-place processing - for visual processing this can be too much of a burden Representing complex data in a single buffer requires inefficient hacks The slice-based processing paradigm is an elegant solution, but can be very difficult to adapt to a more generic frameworkSaturday, 20 November, 2010
  • 38. MarsyasX - architecture First step was the creation of a generic data handling between modules: Data carried in payloads Flows define a coherent stream of data and are identified by type and name Modules are explicitly associated to flows and can handle multiple flows with different behaviours (in, inout, out) source source filter filter sink sinkSaturday, 20 November, 2010
  • 39. MarsyasX - architecture Payload Mechanism: Payloads are created in factories, associated to a source; Payloads are carried between modules through channels (created automatically by implicit patching); When payloads are no longer necessary, they are sent back to its source to be reutilized -> memory is allocated only once.Saturday, 20 November, 2010
  • 40. MarsyasX - architecture Controls are formally related to the flows and changes are propagated automatically e.g. visual flow is characterized by width, height and colour spaceSaturday, 20 November, 2010
  • 41. MarsyasX - architecture Timing? each “tick” corresponds to an elapsed time interval synchronization is assured by the timing information associated to each payload How is a module created? define how many and what type of flows it supports factories for outputs and flow-related controls are added implicitly add specific controls, if needed create a process function input and output data structures are automatically passed as argumentsSaturday, 20 November, 2010
  • 42. MarsyasX - architecture Modules are now loaded from extensions implemented separately dynamically loaded application marsyas other visual audio learning av specific coreSaturday, 20 November, 2010
  • 43. MarsyasX - interoperability Python Scripting language plays a more central role Complete bindings to create networks and use core functionalities Marsyas 0.2 Legacy layer was created to support modules from Marsyas 0.2 transparently A network can be an hybrid, containing MarsyasX and Marsyas 0.2 modules MATLABSaturday, 20 November, 2010
  • 44. MarsyasX - status Status of development: pre-alpha release 0.1 in mid-2011 (tentative) feature status: payload mechanism [done] flow management [done] definition of API [partly done] python bindings [partly done] legacy layer [partly done] processing modules [needs a lot more] event management (expressions, GUI) [not fully defined]Saturday, 20 November, 2010
  • 45. MarsyasX - future Main development trunk of the Marsyas project Larger community: associating the current users and developers of Marsyas with many others coming from different areas Distributed MarsyasX Data and events exchanged transparentlySaturday, 20 November, 2010
  • 46. Thank you! (Lifetime) Future work… http://marsyas.sourceforge.netSaturday, 20 November, 2010