Music Objects to Social Machines


Published on

Seminar for University of Manchester School of Computer Science, Wednesday 30th April 2014 at 14:00 in Lecture Theatre 1.4.

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Thanks to Simon Hettrick for additional input to this slide.
  • Music Objects to Social Machines

    1. 1. Digital Music Research: from Music Objects to Social Machines David De Roure e-Research Centre, University of Oxford @dder
    2. 2. The nature of multidisciplinary research Structural Analysis of Music Music as an exemplar of end-to-end digital Social Objects and Social Machines
    3. 3. YES
    4. 4. Richard Klavans and Kevin W. Boyack. 2009. Toward a consensus map of science. J. Am. Soc. Inf. Sci. Technol. 60, 3 (March 2009), 455-476. DOI=10.1002/asi.v60:3
    5. 5. Pip WillcoxPip Willcox From data to signal to understanding
    6. 6. INT. VERSE VERSE VERSE VERSEBRIDGEBRIDGE OUT.  The Problem signal understanding
    7. 7. Community Software Supercomputer Digital Music Collections Student-sourced ground truth Community Software Linked Data Repositories Supercomputer 23,000 hours of recorded music Music Information Retrieval Community SALAMI
    8. 8. Ashley Burgoyne
    9. 9. Jordan B. L. Smith, J. Ashley Burgoyne, Ichiro Fujinaga, David De Roure, and J. Stephen Downie. 2011. Design and creation of a large-scale database of structural annotations. In Proceedings of the International Society for Music Information Retrieval Conference, Miami, FL, 555–60
    10. 10. class structure Ontology models properties from musicological domain • Independent of Music Information Retrieval research and signal processing foundations • Maintains an accurate and complete description of relationships that link them Segment Ontology Ben Fields, Kevin Page, David De Roure and Tim Crawford (2011) "The Segment Ontology: Bridging Music-Generic and Domain-Specific" in 3rd International Workshop on Advances in Music Information Research (AdMIRe 2011) held in conjunction with IEEE International Conference on Multimedia and Expo (ICME), Barcelona, July 2011
    11. 11. MIREX TASKS Audio Artist Identification Audio Onset Detection Audio Beat Tracking Audio Tag Classification Audio Chord Detection Audio Tempo Extraction Audio Classical Composer ID Multiple F0 Estimation Audio Cover Song Identification Multiple F0 Note Detection Audio Drum Detection Query-by-Singing/Humming Audio Genre Classification Query-by-Tapping Audio Key Finding Score Following Audio Melody Extraction Symbolic Genre Classification Audio Mood Classification Symbolic Key Finding Audio Music Similarity Symbolic Melodic Similarity Downie, J. Stephen, Andreas F. Ehmann, Mert Bay and M. Cameron Jones. (2010). The Music Information Retrieval Evaluation eXchange: Some Observations and Insights. Advances in Music Information Retrieval Vol. 274, pp. 93-115 Music Information Retrieval Evaluation eXchange
    12. 12.
    13. 13. chromogram Representations symbolic
    14. 14. Structural analysis
    15. 15. Autocorrelation
    16. 16. Bach
    17. 17. Hard Day’s Night: Self-Similarity Map
    18. 18. Stephen Downie
    19. 19. SALAMI results: a living experiment
    20. 20. DavidBainbridge
    21. 21. ABABCB… where A is bars 1-2, B is 3-4, C is 9-10 • This is like dictionary-based compression • Or genetic programming (see also Schenkerian Analysis) Symbolic algorithms
    22. 22. “Signal” Digital Audio “Ground Truth” Community It’s web-like! Structural Analysis De Roure, D. Page, K.R., Fields, B., Crawford, T.,Downie, J.S. and Fujinaga, I. (2011) “An e-Research Approach to Web-Scale Music Analysis”, Philosophical Transactions of the Royal Society Series A
    23. 23. Sean Bechhofer
    24. 24. How country is my country? Kevin Page
    25. 25. Sean Bechhofer, Kevin Page and David De Roure. Hello Cleveland! Linked Data Publication Of Live Music Archives. 14th International Workshop on Image and Audio Analysis for Multimedia Interactive services Sean Bechhofer
    26. 26. ElEPHãT from a distance EEBO -TCP Hathi Trust • Smaller collection • Well understood and described • Managed metadata • Focussed corpus • Manual transcriptions • Extremely large collection • Incomplete understanding of content • Variable metadata • Broad corpus • Variable quality OCR Strengths of each informs understanding of the other Scholarly investigations through Worksets bridging both collections Technical challenges • Necessary “anchors” at each “end” • Tools for dynamic alignment • Linked Data “bridging” between the collections • Creation and viewing of Worksets using this linked data Informing future integration of external collections KevinPageandPipWillcox
    27. 27. • Transforming Musicology is funded under the AHRC Digital Transformations in the Arts and Humanities scheme. It seeks to explore how emerging technologies for working with music as sound and score can transform musicology, both as an academic discipline and as a practice outside the university. • The work is being carried out collaboratively between Goldsmiths College, Queen Mary College, Oxford University, the Oxford e-Research Centre, and Lancaster University with an international partner at Utrecht University.
    28. 28. • The world of music has changed for good in the digital age. This revolution must be matched by a transformation of the means by which music is studied. • While preserving the best traditional values and practices of musicology we must take advantage of the immense opportunities offered by music information retrieval • Three parallel musicological investigations 1. 16th-century vocal and lute music 2. Wagner's leitmotifs 3. Musicology of the social media • Ensure sustainability and repeatability by embedding the above research activities in a framework enabling data, methods and results to be shared permanently as Linked Data • Enhance Semantic Web workflow description methods for musicology
    30. 30. “Gold Standard” Music Metadata Enhancements for musical enjoyment by home consumers In-song browsing • learn how songs and symphonies are structured • e.g. find (and repeat) the guitar solo • e.g. find vocals and enhance them • e.g. create/locate guitar tablature In-collection browsing • build great playlists easily: by mood or emotion. e.g. for jogging, driving, relaxing; containing only pieces in G Major; containing Rock & Roll with orchestral strings; with a synth sound like Stevie Wonder • discover and purchase new music, whether using Spotify or iTunes • discover shared musical tastes
    31. 31. “Gold Standard” Music Metadata Enhancements for professionals Content owners • get instantaneous information on trends, etc., from social media feeds • enhance their product with exclusive artist information, locked to purchase • distributers provide Digital Music Objects with the right bandwidth for the context and ease congestion Recording studio workflow • engineers intelligently navigate complex mixes • producers can apply new sound effects to isolated elements of the music Broadcast studio workflow • producers select content for the radio or TV show by mood, by example or by intelligent navigation
    32. 32. consume produce compose perform capture distribute Mark Sandler(plus curation, preservation, …)
    33. 33. Now • No production or content metadata capture – c.f. still and video cameras • Clear audio standards (e.g. 192 kHz/24 bit) but incompatible product-specific project files • No intelligent, content-semantic automation or assistance Goals • Capture/ compute of GSMM to drive all down-stream processes • Improved interoperability across system vendors Challenges • Develop equipment and instruments that capture metadata (e.g mic with time-code and GPS) • Standardised semantic, linked metadata capture produce distribute consume
    34. 34. Now • Convergence in function of pro- and consumer products • No/little metadata kept • No standards, particularly in describing processes (audio effects) • Mostly PC/Mac software solutions for Digital Audio Workstation Goals • low cost equipment, including software and tablets • assist/semi-automate (post) production • capture post-production metadata for re-engineering content, user- customisation. Challenges • Using cloud • Standardised semantic, linked metadata • Tools & kit for automated metadata processing, capture, logging capture produce distribute consume
    35. 35. Now • Different platforms & formats. Piracy. • Increasing use of IP for distribution. • Transcoding within channels, quality loss, managing multiple copies Goals • Simpler transcoding (e.g. embedded scalability • Distribute content linked to metadata • Encrypted metadata: supports consumer while defying piracy • Digital Music Object Challenges • Encryption standards for metadata • Linking semantic, standardised metadata. • Aggregate metadata from up/down stream capture produce distribute consume
    36. 36. Now • No context awareness, no customisation. • Some transcoding of bit-rates, #channels. • Little immersion, both intellectual and audio. • Unfulfilled desires to share, re-purpose, integrate with social media Goals • Modify experience to suit context • Re-balance between instruments • Seamlessly switch #channels as user context changes • Navigate collections; songs • Edutainment Challenges • Repurposing content to match device and context capture produce distribute consume
    37. 37. NeilChueHong An exemplar for software practice • Global distributed system: software, data and processor allocation by bandwidth but also rights, copyright, … • Realtime, streaming (cf big data) • Digital Rights Management and provenance • Algorithm IPR • Heavily app based • MIR open source community and MIREX • Non-consumptive research
    38. 38. Digital Music Object Mark Sandler, Geraint Wiggins
    39. 39. Edwards, P. N., et al. (2013) Knowledge Infrastructures: Intellectual Frameworks and Research Challenges. Ann Arbor: Deep Blue.
    40. 40. Research Objects Computational Research Objects The Evolution of Research Objects Workflows Packs OAI ORE W3CPROV Social Objects
    41. 41. Join the W3C Community Group Jun Zhao
    42. 42. The R Dimensions Research Objects facilitate research that is reproducible, repeatable, replicable, reusable, referenceable, retrievable, reviewable, replayable, re-interpretable, reprocessable, recomposable, reconstructable, repurposable, reliable, respectful, reputable, revealable, recoverable, restorable, reparable, refreshable?” @dder 14 April 2014 sci method access understand new use social curation Research Object Principles
    43. 43. The Big Picture More people Moremachines Big Data Big Compute Conventional Computation “Big Social” Social Networks e-infrastructure online R&D Social Machines deeply about society
    44. 44. Real life is and must be full of all kinds of social constraint – the very processes from which society arises. Computers can help if we use them to create abstract social machines on the Web: processes in which the people do the creative work and the machine does the administration... The stage is set for an evolutionary growth of new social engines. The ability to create new forms of social process would be given to the world at large, and development would be rapid. Berners-Lee, Weaving the Web, 1999 (pp. 172–175) Social Machines
    45. 45. SOCIAM: The Theory and Practice of Social Machines is funded by the UK Engineering and Physical Sciences Research Council (EPSRC) under grant number EPJ017728/1 and comprises the Universities of Southampton, Oxford and Edinburgh. See
    46. 46. Mark d’Inverno PRAISE: Performance and pRactice Agents Inspiring Social Education
    47. 47. The Web Observatory Tiropanis, T., Hall, W., Shadbolt, N., De Roure, D., Contractor, N., and Hendler, J. The web science observatory. IEEE Intelligent Systems 28, 2 (2013), 100–104.
    48. 48. Nigel Shadbolt et al
    49. 49. STORYTELLING AS A STETHOSCOPE FOR SOCIAL MACHINES 1. Sociality through storytelling potential and realization 2. Sustainability through reactivity and interactivity 3. Emergence through collaborative authorship and mixed authority Zooniverse is a highly storified Social Machine Facebook doesn’t allow for improvisation Wikipedia assigns authority rights rigidly Tarte, S. M., De Roure, D., and Willcox, P. Working out the plot: the role of stories in social machines. In Proceedings of the companion publication of the 23rd international conference on World wide web companion (2014), International World Wide Web Conferences Steering Committee, pp. 909–914.
    50. 50. Big data elephant versus sense-making network? The challenge is to foster the co-constituted socio-technical system on the right i.e. a computationally-enabled sense-making network of expertise, data, models, software, visualisations and narratives Iain Buchan
    51. 51. • Digital doesn’t respect disciplinary boundaries – don’t just retrofit digital inside the barriers of historic academic structures, think forward instead: – End to end digital systems – End to end semantics • Try applying the lenses of – Social Objects – Social Machines • Music as an exemplar for science, informing ICT strategy and future of scholarly communications • Always ask hard questions, especially given the disruptions of increasing empowerment and automation Take home messages
    52. 52. @dder SOCIAM: The Theory and Practice of Social Machines is funded by the UK Engineering and Physical Sciences Research Council (EPSRC) under grant number EPJ017728/1 and comprises the Universities of Southampton, Oxford and Edinburgh. See Slide and image credits: Sean Bechhofer, Iain Buchan, Neil Chue Hong, Tim Crawford, Stephen Downie, Ben Fields, Ichinaro Fujinaga, Carole Goble, Mark d’Inverno, Kevin Page, Mark Sandler, Pip Willcox, Jun Zhao. Thanks to NEMA, SALAMI, Wf4Ever, Transforming Musicology, FAST, SOCIAM, PRAISE and all our colleagues in the ISMIR community.
    53. 53. Bechhofer, S., Page, K., and De Roure, D. Hello Cleveland! linked data publication of live music archives. In Image Analysis for Multimedia Interactive Services (WIAMIS), 2013 14th International Workshop on (2013), IEEE, pp. 1–4. De Roure, D. Towards computational research objects. In Proceedings of the 1st International Workshop on Digital Preservation of Research Methods and Artefacts (2013), ACM, pp. 16–19. De Roure, D., Page, K. R., Fields, B., Crawford, T., Downie, J. S., and Fujinaga, I. An e-research approach to web-scale music analysis. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 369, 1949 (2011), 3300–3317. Fields, B., Page, K., De Roure, D., and Crawford, T. The segment ontology: Bridging music- generic and domain-specific. In Multimedia and Expo (ICME), 2011 IEEE International Conference on (2011), IEEE, pp. 1–6. Page, K. R., Fields, B., De Roure, D., Crawford, T., and Downie, J. S. Capturing the workflows of music information retrieval for repeatability and reuse. Journal of Intelligent Information Systems 41, 3 (2013), 435–459. (Also Reuse, remix, repeat: the workflows of mir. In ISMIR (2012), pp. 409–414.) Page, K. R., Fields, B., Nagel, B. J., O’Neill, G., De Roure, D. C., and Crawford, T. Semantics for music analysis through linked data: How country is my country? In e-Science (e-Science), 2010 IEEE Sixth International Conference on (2010), IEEE, pp. 41–48. Tarte, S. M., De Roure, D., and Willcox, P. Working out the plot: the role of stories in social machines. In Proceedings of the companion publication of the 23rd international conference on World Wide Web companion (2014), pp. 909–914. Tiropanis, T., Hall, W., Shadbolt, N., De Roure, D., Contractor, N., and Hendler, J. The web science observatory. IEEE Intelligent Systems 28, 2 (2013), 100–104. De Roure, D. Machines, methods and music: On the evolution of e-research. In High Performance Computing and Simulation (HPCS), 2011 International Conference on (2011), IEEE, pp. 8–13.
    54. 54. @dder