Published on

Presentation of the Marsyas opensource project.
More info at

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1. An open source platform for multimedia processing Nov 2010Saturday, 20 November, 2010
  2. 2. Notice This work is licensed under the Creative Commons Attribution-Share Alike 2.5 Portugal License. To view a copy of this license, visit or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA.Saturday, 20 November, 2010
  3. 3. Summary Marsyas Overview Users & Applications Usage Scenarios Architecture Interoperability MarsyasX Future WorkSaturday, 20 November, 2010
  4. 4. Marsyas Overview Software framework for media analysis, synthesis and retrieval Open source (GPL license) Efficient and extensible framework design original emphasis on Music Information Retrieval (MIR) ; now Multimedia! C++, OOP Multiplatform (Linux, MacOSX®, MS Windows®, …) Provides a variety of building blocks for performing common audio tasks: sound/video file IO, audio/video IO, signal processing and machine learning modules blocks can be combined into data flow networks that can be modified and controlled dynamically while they process data in soft real-time. GM M WAV WAV source Hanning FFT FFT source KNN LPCSaturday, 20 November, 2010
  5. 5. Marsyas Overview Marsyas Brief History 1998 ~2000 Created by George Tzanetakis during his PhD activities at Princeton 2000 ~2002 Marsyas 0.1 First stable revisions of the toolkit Distributions hosted at SourceForge Creation of a developer community User and Developer Mailing lists 2002 ~ … Marsyas 0.2 Major framework revision SourceForge SubVersion, 20 November, 2010
  6. 6. Marsyas Overview Marsyas Core Developers (Present and past...) George Tzanetakis Main culprit (also main designer and developer) Luís Gustavo Martins Refactoring freak Graham Percieval Documentation overlord Luís Filipe Teixeira Marsyas X motor-head Mathieu Lagrange Biggest network award Steven Ness Ruby and Web guy Check updated list at the website!Saturday, 20 November, 2010
  7. 7. Marsyas Community group_id=84982&ugn=marsyas&type=prdownload&mode=alltime&file_id=0&graph=1Saturday, 20 November, 2010
  8. 8. Marsyas Community Google AnalyticsSaturday, 20 November, 2010
  9. 9. Marsyas Community Google Analytics (Nov 2010)Saturday, 20 November, 2010
  10. 10. Marsyas Community Mailing lists Developer User Tracker Feature RequestsSaturday, 20 November, 2010
  11. 11. Marsyas Community SourceForge Community Choice Awards 2009 Finalist, 20 November, 2010
  12. 12. Marsyas Project Analysis Marsyas value? Ohloh Analysis, 20 November, 2010
  13. 13. Users & Applications Musicream (Masataka Goto) Music playback system with similarity capabilities Uses Marsyas as its music similarity engineSaturday, 20 November, 2010
  14. 14. Users & Applications SndPeek (Princeton) Uses Marsyas 0.1 for: FFT magnitude spectrum real-time spectral feature extraction centroid rms flux rolloffSaturday, 20 November, 2010
  15. 15. Users & Applications Marsyas @ INESC Porto Audio Analysis Software prototypes: Feature Extraction Audio segmentation/classification Audio fingerprinting Speaker Segmentation Music and Auditory Scene Analysis Video Analysis Software prototypes: Background modeling and subtraction Local feature extraction for object matching, 20 November, 2010
  16. 16. Users & Applications Desert Island Undergraduate work at the Univ. Missouri Kansas Jared Hoberock Dan Kelly Ben TietgenSaturday, 20 November, 2010
  17. 17. Related Work Open Source frameworks CLAM ( STK ( Chuck ( PureData (Pd) ( Open Sound Control (OSC) ( FAUST ( EyesWeb ( ... Commercial toolkits MAX/MSP® ( MATLAB® Simulink® ( LabView® ( DirectShow® GraphEdit ( ...Saturday, 20 November, 2010
  18. 18. Usage Scenarios Marsyas command line tools Demonstrate key capabilities of the framework Some are actually research tools Efficient and can execute in real-time ANSI C++ only core several optional libraries MP3 reading/writing (libMad) Tools and examples: sfplay bextract phasevocoder sfplugin …Saturday, 20 November, 2010
  19. 19. Usage Scenarios Playing audio files sfplay > sfplay foo.wav > sfplay –s 10.0 –l 3.2 –r 2.5 –g 0.5 foo1.wav –f output.wav > sfplay –l 3.0 foo.wav > sfplay foo.wav –p playback.mpl > -s : start time for playback -l : length of playback -r : repeat times -g : volume (gain) value -p : playback.mpl (save playback network as a .mpl plugin file)Saturday, 20 November, 2010
  20. 20. Usage Scenarios Machine Learning: feature extraction and training of classifiers bextract > bextract -e STFTMFCC -p ms.mpl -w myweka.arff > -e STFTMFCC extracts spectral and MFCC features, lists of sound files (collections) -w myWeka.arff WEKA file with extracted features -p ms.mpl “trained” Marsyas plug-in for realtime music/speech classificationSaturday, 20 November, 2010
  21. 21. Usage Scenarios Marsyas plugins (.mpl files) Allow to dynamically recreate a processing network in runtime sfplugin e.g. Audio playback > sfplugin –p playback.mpl foo.wav > e.g. Realtime audio classification > sfplugin –p ms.mpl unknownAudioSignal.wav >Saturday, 20 November, 2010
  22. 22. Usage Scenarios Digital Signal Processing e.g. phasevocoder > phasevocoder –p 1.4 -s 100 > e.g. onset detector > mudbox -t onsets myMusic.wav >Saturday, 20 November, 2010
  23. 23. Architecture Marsyas 0.2 New dataflow model of audio computation hierarchical messaging system used to control the dataflow network inspired on Open Sound Control (OSC) general matrices instead of 1-D arrays as dataSaturday, 20 November, 2010
  24. 24. Architecture MarSystem Slices Separating things that happen at the same time from things that happen in different times Correct semantics for spectral processingSaturday, 20 November, 2010
  25. 25. Architecture Implicit Patching VS Explicit Patching # EXPLICIT PATCHING source, F1, F2, F3, destination # connect the appropriate in/out ports connect(source, F1); connect(source, F2); connect(source, F3); connect(F1, destination); connect(F2, destination); connect(F3, destination); # IMPLICIT PATCHING source, F1, F2, F3, destination Fanout(F1, F2, F3) Series(source, Fanout, destination);Saturday, 20 November, 2010
  26. 26. Architecture MarSystem Composites Series Fanout Fanin Parallel Accumulator …Saturday, 20 November, 2010
  27. 27. Architecture Implicit VS Explicit Patching :: Neural Network #IMPLICIT PATCHING fanoutLayer1(ANN_Node11, …, ANN_Node1N) … fanoutLayerM(ANN_NodeM1, …,ANN_NodeMN) ANN_Series(fanoutLayer1, …, fanoutLayerM)Saturday, 20 November, 2010
  28. 28. Architecture Typical feature extraction Time-frequency representation Frequency summarization (MFCC) Time summarization (Texture features)Saturday, 20 November, 2010
  29. 29. Architecture Feature Extraction using Implicit Patching MarSystemManager mng; MarSystem* Series1 = mng.create("Series", “Series1"); MarSystem* Fanout1 = mng.create(“Fanout", “Fanout1"); MarSystem* Series2 = mng.create("Series", “Series2"); MarSystem* Fanout2 = mng.create(“Fanout”, “Fanout2”); MarSystem* Fanout3 = mng.create(“Fanout”, “Fanout3”); Fanout3->addMarSystem(mng.create(“Mean”, “Mean”)); Fanout3->addMarSystem(mng.create(“Variance”, “Variance”)); Fanout2->addMarSystem(mng.create(“Centroid”, “Centroid”)); Fanout2->addMarSystem(mng.create(“RollOff”, “Rolloff”)); Fanout2->addMarSystem(mng.create(“Flux”, “Flux”); Series2->addMarSystem(mng.create(“Spectrum”, “Spectrum”); Series2->addMarSystem(Fanout2); Fanout1->addMarSystem(mng.create(“ZeroCrossings”, “ZeroCrossings”); Fanout1->addMarSystem(Series2); Series1->addMarSystem(mng.create("SoundFileSource",“Source")); Series1->addMarSystem(Fanout1); Series1->addMarSystem(mng.create(“Memory”, “TextureMemory”)); Series1->addMarSystem(Fanout3); Series1->addMarSystem(mng.create(“classifier”, “Classifier”));Saturday, 20 November, 2010
  30. 30. Interoperability Marsyas Audio and MIDI I/0 RtAudio Multiplatform C++ API for realtime audio input/output Linux (native ALSA, JACK, and OSS) MacOSX® Windows® (DirectSound® and ASIO®) SGI® RtMIDI Multiplatform C++ API for realtime MIDI input/output Linux (ALSA) MacOSX® Windows® (Multimedia Library) SGI®Saturday, 20 November, 2010
  31. 31. Interoperability Marsyas & WEKA WEKA: Data Mining Software in Java Marsyas already includes some machine learning blocks Marsyas outputs extracted features as .arff files (WEKA) features can be opened in WEKA for further evaluation and data modelingSaturday, 20 November, 2010
  32. 32. Interoperability Calling MATLAB® from C++ Marsyas code: MATLAB® engine API exchange data (i.e. matrices) in run-time between C++ and MATLAB® remotely execute commands in MATLAB® from a C++ routine Access to all MATLAB® toolboxes, algorithms and available routines Algorithmic validation of C++ routines Quick and easy evaluation of proof of concepts May not allow real-time operation… Not such a big problem when evaluating or developing algorithmsSaturday, 20 November, 2010
  33. 33. Interoperability Calling MATLAB® from C++ Marsyas code: Marsyas::MATLABengine class // create a std::vector of real numbers Utility class std::vector<double> vector_real(4); vector_real[0] = 1.123456789; Wraps MATLAB® engine calls for most vector_real[1] = 2.123456789; POD types and Marsyas data types vector_real[2] = 3.123456789; vector_real[3] = 4.123456789; Easy to send/receive data to/from MATLAB® from anywhere in the code // send a std::vector<double> to MATLAB PUTVAR(vector_real, "vector_real"); // do some dummy math in MATLAB EVALUATE("mu = mean(vector_real);"); EVALUATE("sigma = std(vector_real);"); EVALUATE("vector_real = vector_real/max(vector_real);"); // get values from MATLAB double m, s; GETVAR(m, "mu"); GETVAR(s, "sigma"); GETVAR(vector_real, "vector_real");Saturday, 20 November, 2010
  34. 34. Interoperability Python™ Bindings easily create scripts for rapid testing and prototyping of data flow networks would require much more development effort in C++ bonus: no compiling overheads can also be embedded in C++ code, similarly to MATLAB® (TBD) less tools for signal processing in general, but can be used for many other purposes (“batteries included”) less licensing headachesSaturday, 20 November, 2010
  35. 35. Interoperability Marsyas and Trolltech Qt4® Qt® Core features optionally used by Marsyas Multi-platform signal/slot architecture Multi-platform threads (multithreaded processing) Multi-platform database access Multi-platform XML I/O Qt® GUI Features optionally used by Marsyas Multi-platform Widgets Multi-platform OpenGL Qt4® is available as GPL open source code for all platformsSaturday, 20 November, 2010
  36. 36. MarsyasX What is MarsyasX? Next step for Marsyas Evolution rather than an alternative implementation Generalization of the framework to support processing different media Key points: crossmodal processing expandable support interoperability GM AV MPEG M Contour source Sink AV MP WAV source S EG Hanning FFT ink source Motion ur to ConSaturday, 20 November, 2010
  37. 37. MarsyasX MarsyasX brief history: MarsyasX started as a patch to Marsyas... ...add visual processing support However, the previous architecture has some restrictions: Some design options are audio-oriented No in-place processing - for visual processing this can be too much of a burden Representing complex data in a single buffer requires inefficient hacks The slice-based processing paradigm is an elegant solution, but can be very difficult to adapt to a more generic frameworkSaturday, 20 November, 2010
  38. 38. MarsyasX - architecture First step was the creation of a generic data handling between modules: Data carried in payloads Flows define a coherent stream of data and are identified by type and name Modules are explicitly associated to flows and can handle multiple flows with different behaviours (in, inout, out) source source filter filter sink sinkSaturday, 20 November, 2010
  39. 39. MarsyasX - architecture Payload Mechanism: Payloads are created in factories, associated to a source; Payloads are carried between modules through channels (created automatically by implicit patching); When payloads are no longer necessary, they are sent back to its source to be reutilized -> memory is allocated only once.Saturday, 20 November, 2010
  40. 40. MarsyasX - architecture Controls are formally related to the flows and changes are propagated automatically e.g. visual flow is characterized by width, height and colour spaceSaturday, 20 November, 2010
  41. 41. MarsyasX - architecture Timing? each “tick” corresponds to an elapsed time interval synchronization is assured by the timing information associated to each payload How is a module created? define how many and what type of flows it supports factories for outputs and flow-related controls are added implicitly add specific controls, if needed create a process function input and output data structures are automatically passed as argumentsSaturday, 20 November, 2010
  42. 42. MarsyasX - architecture Modules are now loaded from extensions implemented separately dynamically loaded application marsyas other visual audio learning av specific coreSaturday, 20 November, 2010
  43. 43. MarsyasX - interoperability Python Scripting language plays a more central role Complete bindings to create networks and use core functionalities Marsyas 0.2 Legacy layer was created to support modules from Marsyas 0.2 transparently A network can be an hybrid, containing MarsyasX and Marsyas 0.2 modules MATLABSaturday, 20 November, 2010
  44. 44. MarsyasX - status Status of development: pre-alpha release 0.1 in mid-2011 (tentative) feature status: payload mechanism [done] flow management [done] definition of API [partly done] python bindings [partly done] legacy layer [partly done] processing modules [needs a lot more] event management (expressions, GUI) [not fully defined]Saturday, 20 November, 2010
  45. 45. MarsyasX - future Main development trunk of the Marsyas project Larger community: associating the current users and developers of Marsyas with many others coming from different areas Distributed MarsyasX Data and events exchanged transparentlySaturday, 20 November, 2010
  46. 46. Thank you! (Lifetime) Future work… http://marsyas.sourceforge.netSaturday, 20 November, 2010