Social Tags and Music Information Retrieval (Part II)

Social Tags and Music Information Retrieval Part II

ISMIR 2008

Paul Lamere – Sun Microsystems Inc.

Elias Pampalk – Last.fm

Outline

What are social tags?

Why do people tag?

Issues with social tags

Other sources of tags (continued)

Search, Discovery & Recommendation

Data & Tools

Future Research

Conclusion

Discussion

Other sources of tags Autotagging

Uses content analysis to automatically apply tags

Tags acquired from other sources (social tags, games, web crawling) can be 'learned'

New music or unpopular music can be autotagged with the 'learned' tags.

Can scale to the long tail

Time per million songs:

Manual: with 100 people = 3 Years

Automatic: with 100 CPUs = 8 Hours

Cost per million songs

Manual: ~ $10,000,000

Automatic: ~ $100

Other sources of tags: Autotagging How it works Labeled Examples Unknown Examples Machine Learning Model Labeled Examples Feature Extraction Training Tagging

Other sources of tags: Autotagging Ground Truth

Standard (genre) classification

each item: 1 category/class/tag

annotated by a trusted expert

Social tags

each item: unlimited (weighted) categories/class/tag

annotated by an anonymous crowd

Other sources of tags: Autotagging Ground Truth for Classifiers [Craft, Wiggens, Crawford, ISMIR 2007]

Other sources of tags: Autotagging Ground Truth for Classifiers [Paul Lamere, JNMR 2008]

Autotagging: Ground truth & evaluation MIREX 2008: Tag Track

Data

MajorMiner game (Mandel & Ellis)

2300 audio clips (10 second) from 1400 tracks from 500 artists (artist-filtering used to ensure training and test don't have different sets of artists)

43 tags verified at least 35 times each (drums, guitar, male, rock, ...)

Autotagging: Ground truth & evaluation MIREX 2008: Tag Track

Tasks

Clip -> tag (yes/no)

Clip -> list of tags (ranked)

Tag -> list of clips (ranked)

Standardized evaluation procedures

Statistical significance etc

http://www.music-ir.org/mirex/2008/index.php/Audio_Tag_Classification

Autotagging LabROSA

Features: MFCCs + temporal features

Learning: Support Vector Machine with a radial basis function kernel

Compared ability to learn 'game' tags vs. social tags

M. I. Mandel and D. P. W. Ellis. A Web-Based Game for Collecting Music Metadata. In Journal of New Music Research, 2008 (to appear).

http://majorminer.com/search Other sources of tags: Autotagging LabROSA

T. Bertin-Mahiex, D. Eck, F. Maillet, and P. Lamere. Autotagger: A model for predicting social tags from acoustic features on large music databases. In Journal of New Music Research, 2008 (to appear). Other sources of tags: Autotagging BRAMS, Sun Labs

[Bertin-Mahiex et al. JNMR 2008]

Features: MFCCs + Temporal

Learning: AdaBoost and FilterBoost

Medium scale: 100,000 tracks

"Large and Noise" vs. "Small and Clean" There's no data like more data

Other sources of tags: Autotagging BRAMS, Sun Labs

D. Turnbull, L. Barrington, D. Torres, and G. Lanckriet. Towards musical query-by-semantic description using the CAL500 data set. SIGIR 2007. Other sources of tags: Autotagging CAL UCSD

Other sources of tags: Autotagging Challenges

Not all tags are can be easily derived from audio

Examples: seen live, great lyrics, a wesome, crap, Boston, Montreal, UK

Can we identify words that are musically meaningful?

Some tags may be too subtle for current systems to distinguish

Power Metal vs. Speed Metal

Grunge vs. Post-Grunge

Dealing with co-occurring tags

Dealing with weak labeling – not every song with piano is labeled piano

Scale

Outline

Why do people tag?

Other sources of tags

Data & Tools

Future Research

Conclusion

Discussion

Search & Discovery

Search & Discovery The Vocabulary Problem

When searching – people will often use the wrong words.

Should I search for "rnb", "r and b", "r&b", or "rnb" ?

With well tagged items – it doesn't matter

http://www.last.fm/music/Rihanna/+tags

Search & Discovery The Vocabulary Problem

What about infrequently tagged items?

We can use overall tag overlap to infer synonymy

Cluster tags

via overlap, tf-idf or other similarity metric

Augment query with synonyms

Not just for synonyms

can deal with spelling errors (“rithm and blues”)

can help with multi-lingual tags

The Vocabulary Problem Using tag clustering to combat synonymy The 'female' cluster of a 2,000 node tag hierarchy

The Vocabulary Problem Multiple languages

Similar tags to deutscher hiphop

German hip-hop

Deutscher Hip Hop

Hamburg

Eimbush

Deutschrap

Deutsch

Deutschsprachig

Hiphop

German rap

German

The Vocabulary Problem Using item clustering to combat polysemy

The Vocabulary Problem Using Latent Semantic Analysis Female Singing Unstructured/Unreliable representation Latent Semantic Model

Can address:

Synonym, Polysemy

Noise

Dimensionality reduction

Latent Semantic Analysis Query Latent Semantic Space Search

Project query into semantic space

Best match may not have any tags that match the query!

See: Levy & Sandler, Learning Latent Semantic Models for Music from Social Tags . JNMR 2008. Girl, GirlBand, Grrl, Girly Female Singer Female Female Vocalists Diva Chicks Woman Pop American Idol Pop RnB Rhythm & Blues Girl Pop LSA

Similarity of text

Weight the terms

Some terms are more meaningful than others: rock vs. shoegaze

TF x IDF

Distance between term vectors

Cosine distance

the cosine of the angle between the two vectors

Independent of length of vectors

Search, Discovery & Recommendation Item Similarity

Artist similarity based on tags for Weezer

Top Tags

Alternative

Rock

Indie

Punk

Pop

Power pop

Geek Rock

90s

Metal

Indie pop

Distinctive Tags

Geek Rock

Punk-pop

College Rock

Not Emo

Overrated

Los Angeles

Modern Rock

Pop punk

California

Pop Rock

Similar Artists via Tags

Green Day

Phantom Planet

The Offspring

Sugarcult

Foo Fighters

blink-182

Ozma

Jimmy Eat World

Nada Surf

The Ataris

Similar Artists via CF

The White Stripes

Foo Fighters

Death Cab For Cutie

Beck

Radiohead

Green Day

Coldplay

The Beatles

The Killers

The Smashing Pumpkins

Web survey: Tag-based artist similarity scores better than CF-based similarity Search, Discovery & Recommendation Artist Similarity

Tag similarity based on artists

Metal

Metallica

System of a down

Iron Maiden

Rammstein

Slipknot

In Flames

Korn

Pantera

Judas Priest

Heavy Metal

Iron Maiden

Judas Priest

Black Sabbath

Manowar

Motorhead

Pantera

Megadeth

Ozzy Osbourne

Dio

Pop

Madonna

The Beatles

Black Eyed Peas

Beach Boys

Kelly Clarkson

Michael Jackson

Gwen Stefani

Coldplay

U2

Search, Discovery & Recommendation Tag Similarity

Tag similarity examples

Similar tags to relax

Relaxing

Calm

Chill

Meditation

Spiritual

Chill out

Soft

Dreamy

New age

Mellow

Search, Discovery & Recommendation Tag Similarity

Similar tags to deutscher hiphop

German hip-hop

Deutscher Hip Hop

Hamburg

Eimbush

Deutschrap

Deutsch

Deutschsprachig

Hiphop

German rap

German

Similar tags to turntablism

dj

abstract hip-hop

scratch

instrumental hip-hop

beats

ninja tune

ambient breakbeat

hip hop

instrumental hip hop

underground hip hop

Similar tags to metal

heavy metal

death metal

thrash metal

hard rock

progressive metal

metalcore

power metal

melodic death metal

gothic metal

hardcore

Search, Discovery & Recommendation User Similarity

Paul (green) vs Elias (red)

http://anthony.liekens.net/

Discovery & Recommendation Goal: Recreate Ishkur's Guide to EDM

Discovery & Recommendation From Folksonomy to Taxonomy

Metal Discovery & Recommendation From Folksonomy to Taxonomy

Discovery & Recommendation Creating an Artist Hierarchy Attachment order: Artist Popularity

Attachment order: Artist Date Discovery & Recommendation Creating an Artist Hierarchy

Discovery & Recommendation Creating a Personal Artist Hierarchy

Discovery & Recommendation Transparency - Explaura

2) Drag a tag to make it bigger or smaller. Discovery & Recommendation Steerable recommendations - Explaura 1) Click to add a tag or artist 3) Receive recommendations that match the tag cloud 4) Get an explanation for each recommendation

Discovery & Recommendation Transparency - Pandora

http://flickr.com/photos/libraryman/1225285863/

Discovery & Recommendation Transparency - Pandora

Technology loosing it's "cold"

Pandora feels like a smart friend to me. This friend can articulate the reasons I love some of the things I love most (songs) better than I can, but only because I have told it what I like. This is one of my very favorite Prince songs and Pandora knows just why I like it so much. And I didn't know how to say it so well. Makes the technology seem very warm and reflective of my feelings an identity. It's an extension of the user, not a cold, isolating technology. I feel a part of Pandora some times.

http://flickr.com/photos/libraryman/1225285863/

Discovery & Recommendation Transparency – Musical MadLibs

Norah Jones - Don’t Know Why

Summary generated automatically using SML model:

This is soft rock , jazz song that is mellow and sad . It features piano , synthesizer , ambient sounds , and monotone , breathy vocals . It is a song with a slow tempo and with low energy that you might like to listen to while studying .

[Turnbull, Liu, Barrington & Lanckriet, ISMIR 2007]

Discovery & Recommendation Cold start: new users

Need 'taste data' for new users

Results:

Awkward user enrollment

Poor recommendations

Solution

Represent taste 'portably'

Taste data can move with the user

Several efforts

Attention Profile Markup Language - APML

OpenTaste

Attention Profile Markup Language <APML> <Profile name="music"> <ImplicitData> <Concepts> <Concept key=" piano " value =".81"/> <Concept key=" female vocalists " value =".72"/> <Concept key=" indie pop " value =".65"/> <Concept key=" art rock " value =".60"/> <Concept key=" experimental " value =".57"/> <Concept key=" great lyricist " value =".55"/> <Concept key=" noise " value =".35"/> </Concepts> </ImplicitData> </Profile> </APML> Turn this Into this Then this APML

Discovery & Recommendation Last.fm Products using Tag Data

Search and Discovery Last.fm Playground http://playground.last.fm/multitag

Multi Tag Search

Categories

Popular

Up-and-coming

Free downloads

Example queries

pop british 90s sad piano

catchy funny soundtrack

hilarious cover

"one hit wonder" 90s

american "guilty pleasure"

happy sad

relaxing “speed metal”

More information : Late breaking/demo session Thursday 11.00 - 13.00 Klaas Bosteels

Last.fm Playground: Islands of Music Clustering listeners by tags

13k randomly sampled Last.fm listeners

Each listener represented by a tag cloud

2000 dimensions (tags) -> 120 dimensions (SVD)

k-means clustering to extract 400 prototypical listeners

Self-organizing map (20x40)

http://playground.last.fm/iom

Search, Discovery & Recommendations

Vocabulary problem

Item similarity

Hierarchical clustering

Transparency

User cold start and APML

Last.fm

Outline

Why do people tag?

Data & Tools

Future Research

Conclusion

Discussion

Data and Tools

Pointers to all data at SocialMusicResearch.org/data

Expert/Survey data

CAL -500

http://cosmal.ucsd.edu/cal/projects/AnnRet/AnnRet.php

1700 human generated musical annotations

500 popular western tracks

All Music

Available through commercial license

Genre, Styles and Moods for thousands of artists

Data and Tools

Game Data

ListenGame

26,000 annotations

250 songs

120 words

440 unique players

Available upon request from Doug Turnbull

MajorMiner

Available for browsing at:

http://majorminer.com/search

human tags

autotags

MIREX 2008

Data and Tools

Social tags

LastFM-ArtistTags2007

Tag data for over 20,000 artists

labs.strands.com

http://labs.strands.com/music/affinity/

Playlist co-occurrence data

Artist-Tag data for 4,000 artists

Data and Tools TagWorks

Loads tag data from LastFM-ArtistTags2007

Simple-overlap similarity

tf-idf / cosine distance similarity

Agglomerative clustering of

tags, artists, users

Folksonomy -> Taxonomy algorithm

tags, artists, users

Last.fm crawler for user-tags

Written in the Java programming language

Uses the Minion search engine: https://minion.dev.java.net/

Available at: SocialMusicResearch.org/code

Data and Tools Last.fm API 1.0

Tags applied by a user

Tags a user applied to specific artist/album/track

Tag clouds for artist/track or overall

Top artists/album/tracks for given tag

Creative Commons Attribution-Non Commercial-Share Alike License

http://www.audioscrobbler.net/data/webservices/

Data and Tools Last.fm API 1.0 Examples

Social Tags for any MP3 (~ 70 lines of Python code)

Given MP3 file (without ID3)

Use Last.fm Fingerprinter to obtain ID3 info

Use ID3 info to obtain tags applied by Last.fm community

Tagger age vs vocabulary size (~ 85 lines of Python code)

Start with one or more seed users

Get friends, of friends, of friends (-> large list of users)

For each user get tag applied, and demographic information (age)

Analyze data (tagger's vocabulary vs age)

http://SocialM usicR esearch.org / code

Data and Tools Last.fm API 2.0

user: add , remove or get tags for an artist/a lbum/track

get similar tags for given tag

get top albums/artists/tracks for given tag

tag clouds for artist/track/overall

search for a tag

tags applied by a user

http://www.last.fm/api

Outline

Why do people tag?

Data & Tools

Future Research

Conclusion

Discussion

Future Research Librarians, musicologists, anthropologists

What are tags? Who tags? Why do people tag? How do people use tags?

Is tagging behaviour and tag usage different for different demographics?

Can music tags be mapped to taxonomies?

Impact on

Digital libraries

Organizing and discovering music?

Musicology

Perceiving, describing, categorizing and talking about music?

Anthropology

Emergence of new subcultures?

Future Research Can machines play tag? Signal processing & statistical learning

Preliminary results look very promising but ...

What are the limitations and how far can we push them?

Quality in general?

What types of tags that cannot be learned?

Integration of machine learned tags in human interfaces?

Plenty of training data available!

And new fun challenges (data sparsity, noise, ...)

Future Research User experience and UI design

Tagging

How do suggestions impact quality of tags?

Can suggestions be improved?

Social interaction through tags

Can how people communicate with tags be improved?

Discovery/browsing/steering recommendations

Beyond tag clouds?

Are tags a better way to organize playlists?

Future Research More Open Questions

What can we learn about a user from her tagging behaviour?

Tag games

How do game tags differ from social tags?

How can you make games even more fun?

Computation of similarity

Combination of tags with other data sources?

Transparent recommendations

Using tags to explain recommendations?

Future Research More Open Questions (2)

Semantic web

Connecting information and tags across the web?

E.g. Flickr & Last.fm

MOAT?

Internationalization?

Different communities tagging the same item with different languages? (Japanese vs Russian vs English)

“World” music?

Social tagging of other musical entities?

Detecting tag abuse?

Future Research Tag Gardening

Tools for improving tags

Weeding

Seeding

Landscaping

Fertilizing

TagCare

MOAT

Future Research Learn from Japanese Style Tagging? http://joilab.ito.com/2008/02/nico-nico-douga.html http://www.nicovideo.jp/watch/sm9

User generated content

Mash-ups

Community

Tags as art form

Business model

Poster Session 2c: Mon 16.00 – 18.00

Ternary Semantic Analysis of Social Tags for Personalized Music Recommendation

P. Symeonidis, M. Ruxanda, A. Nanopoulos and Y. Manolopoulos

Five Approaches to Collecting Tags for Music

D. Turnbull, L. Barrington and G. Lanckriet

MoodSwings: A Collaborative Game for Music Mood Label Collection

Y. Kim, E. Schmidt and L. Emelle

ISMIR 2008

Poster Session 2c: Mon 16.00 – 18.00

Collective Annotation of Music From Multiple Semantic Categories

Z. Duan, L. Lu and C. Zhang

Connecting the Dots: Music Metadata Generation, Schemas and Applications (*)

N. Corthaut, S. Govaerts, K. Verbert and E. Duval

The Quest for Musical Genres: Do the Experts and the Wisdom of Crowds Agree? (*)

M. Sordo, O. Celma, M. Blech and E. Guaus

ISMIR 2008

Poster Session 2d: Mon 16.00 – 18.00

A Web of Musical Information

Y. Raimond and M. Sandler

Uncovering Affinity of Artists to Multiple Genres From Social Behaviour Data

C. Baccigalupo, J. Donaldson and E. Plaza

Oh Oh Oh Whoah! Towards Automatic Topic Detection in Song Lyrics

F. Kleedorfer, P. Knees and T. Pohle

ISMIR 2008

Poster Session 3a: Tue 11.00 – 13.00

Multi-Label Classification of Music Into Emotions

K. Trohidis, G. Tsoumakas, G. Kalliris and I. Vlahavas

Poster Session 5a: Wed 11.00 – 13.00

Multiple-Instance Learning for Music Information Retrieval

M. Mandel and D. Ellis

MIREX Poster Session: Wed 16.00 – 18.00

Audio Tag Classification

M. Mandel, T. Bertin-Mahieux, G. Tsoumakas, D. Turnbull & L. Barrington, G. Peeters

ISMIR 2008

Late-Breaking / Demo Session: Thu 11.00 – 13.00

Music Retrieval Based on Social Tags: A Case Study

K. Bosteels, E. Kerre, and E. Pampalk

Herd the Music - A Social Music Annotation Game

L. Barrington, D. O’Malley, D. Turnbull, and G. Lanckriet

Music and Lyrics: Can Lyrics Improve Emotion Estimation for Music?

Daniel C. Wu Jr et al.

MOODY: A Web-Based Music Mood Classification and Recommendation System

Xiao Hu et al.

Creating Transparent, Steerable Recommendations

P. Lamere and F. Maillet

ISMIR 2008

From genres to tags: Music information retrieval in the age of social tagging

Editors: J.-J. Aucouturier and E. Pampalk

Scanning the Dial: The Rapid Recognition of Music Genres

R. O. Gjerdingen and D. Perrott

Social Tagging and Music Information Retrieval

P. Lamere

Autotagger a Model for Predicting Social Tags from Acoustic Features

T. Bertin- Mahieux, D. Eck, F. Maillet and P. Lamere

Learning Latent Semantic Models for Music

M. Levy and M. Sandler

A Web-Based Game for Collecting Music Metadata,

M. Mandel and D. Ellis

Journal of New Music Research Special issue to appear in 2008

Outline

Why do people tag?

Data & Tools

Future Research

Conclusion

Discussion

Conclusions SocialMusicResearch.org

Wiki

Slides

Data

Code

Bibliography

You can participate

Participate

The Fundamental Theorem of Music Informatics

Music is created by humans for other humans, and humans can bring a tremendous amount of contextual knowledge to bear on anything they do; in fact, they can't avoid it, and they're rarely conscious of it. But computers can never bring much contextual knowledge to bear, often none at all, and never without being specifically programmed to do so. Therefore doing almost anything with music by computers is very difficult ;many problems are essentially intractable . -- Don Byrd, January 2008

(I wrote the above statement, which I described with tongue firmly in cheek as a "theorem" -- a more accurate term would be "axiom" or "dogma" -- for my Spring 2008 graduate seminar, Organization and Searching of Musical Information . It's probably a bit too strongly worded, but I think it really comes close to describing a very basic problem that most music-informatics research has to deal with.) --Don Byrd, September 2008

Acknowledgments

Outline

Why do people tag?

Data & Tools

Future Research

Conclusion

Discussion

Bibliography http://SocialMusicResearch.org

C. Baccigalupo, E. Plaza, J. Donaldson. Uncovering affinity of artists to multiple genres from social behaviour data. ISMIR 2008.

L. Barrington, D. O’Malley, D. Turnbull, and G. Lanckriet. Herd the Music - A Social Music Annotation Game. ISMIR 2008.

T. Bertin-Mahieux, D. Eck, F. Maillet, and P. Lamere. Autotagger: A model for predicting social tags from acoustic features on large music databases. In Journal of New Music Research, 2008 (to appear).

K. Bosteels, E. Kerre, and E. Pampalk. Music Retrieval Based on Social Tags: A Case Study. ISMIR 2008.

A. Craft, G. Wiggins, T. Crawford. How Many Beans Make Five? The Consensus Problem in Music-Genre Classification and a New Evaluation Method for Single-Genre Categorisation Systems. ISMIR 2007.

N. Corthaut, S. Govaerts, K. Verbert, and E. Duval. Connecting the Dots: Music Metadata Generation, Schemas and Applications. ISMIR 2008.

Z. Duan, L. Lu and C. Zhang. Collective Annotation Of Music From Multiple Semantic Categories. ISMIR 2008.

R. Gjerdingen and D. Perrott. Scanning the dial: The Rapid Recognition of Music Genre. In Journal of New Music Research, 2008 (to appear).

M. Guy & E. Tonkin. Folksonomies: Tidying up Tags? D-Lib Magazine, January 2006:12(1).

X. Hu, M. Bert, & J. S. Downie. Creating a simplified music mood classification ground-truth set. ISMIR 2007.

Xiao Hu et al. MOODY: A Web-Based Music Mood Classification and Recommendation System. ISMIR 2008.

Y. Kim, E. Schmidt and L. Emelle. MoodSwings: A Collaborative Game For Music Mood Label Collection. ISMIR 2008.

F. Kleedorfer, P. Knees and T. Pohle. Oh Oh Oh Whoah! Towards Automatic Topic Detection in Song Lyrics. ISMIR 2008.

P. Knees, M. Schedl, T. Pohle, and G. Widmer. An Innovative Three-Dimensional User Interface for Exploring Music Collections Enriched with Meta-Information from the Web. ACM MM 2006.

P. Lamere. Social Tagging and Music Information Retrieval. Journal of New Music Research 2008 (to appear).

P. Lamere and F. Maillet. Creating Transparent, Steerable Recommendations. ISMIR 2008.

E. L. M. Law, L. von Ahn, R. B. Dannenberg, and M. Crawford. TagATune: A game for music and sound annotation. ISMIR 2007.

M. Levy & M. Sandler. Learning Latent Semantic Models for Music from Social Tags. In Journal of New Music Research, 2008 (to appear).

M. I. Mandel and D. P. W. Ellis. A Web-Based Game for Collecting Music Metadata. In Journal of New Music Research, 2008 (to appear).

M. I. Mandel and D. P. W. Ellis. Multiple-instance learning for music information retrieval. ISMIR 2008.

Nielsen Report: Nielsen soundscan state of the industry. 2008 Convention of the National Association of Recording Merchandisers, 2008. http://www.narm.com/2008Conv/StateoftheIndustry.pdf.

E. Peterson. Beneath the metadata: Some philosophical problems with folksonomy. D-Lib Magazine 2006:12(11). http://www.dlib.org/dlib/november06/peterson/11peterson.html

E. Pampalk, A. Flexer, and G. Widmer. Hierarchical Organization and Description of Music Collections at the Artist Level. ECDL 2005.

E. Pampalk and M. Goto. MusicSun: A New Approach to Artist Recommendation. ISMIR 2007.

E. Pampalk and M. Goto. MusicRainbow: A New User Interface to Discover Artists Using Audio-based Similarity and Web-based Labeling. ISMIR 2006.

Y. Raimond and M. Sandler. A Web of Musical Information. ISMIR 2008.

R. Sinha. Tagging – From Personal to Social. SXSW 2006. http://rashmisinha.com/2006/03/12/my-slides-for-tagging-20-panel-at-sxsw/

M. Sordo, O. Celma and M. Blech. The Quest For Musical Genres: Do the Experts and the Wisdom of Crowds Agree? ISMIR 2008.

P. Symeonidis, M. Ruxanda, A. Nanopoulos and Y. Manolopoulos. Ternary Semantic Analysis Of Social Tags For Personalized Music Recommendation. ISMIR 2008.

A. E. Thompson. Playing Tag: An Analysis of Vocabulary Patterns and Relationships Within a Popular Music Folksonomy. A Master’s Paper for the M.S. in L.S. degree. April, 2008. http://etd.ils.unc.edu/dspace/bitstream/1901/535/1/abbeythompson.pdf

K. Trohidis, G. Tsoumakas, G. Kalliris and I. Vlahavas, Multi-Label Classification of Music Into Emotions. ISMIR 2008.

D. Turnbull, L. Barrington, G. Lanckriet. Five Approaches to Collecting Tags for Music. ISMIR 2008.

D. Turnbull, L. Barrington, D. Torres, and G. Lanckriet. Towards musical query-by-semantic description using the CAL500 data set. SIGIR 2007.

D. Turnbull, R. Liu, L. Barringon, and G. Lanckriet. A game-based approach for collecting semantic annotations of music. ISMIR 2007.

L. von Ahn and L. Dabbish. Labeling images with a computer game. SIGCHI 2004.

http://www.cs.cmu.edu/~biglou/ESP.pdf

D. Weinberger. How tagging changes peoples relationship to information and each other. Pew Internet & American Life Project 2007. http://www.pewinternet.org/pdfs/PIP_Tagging.pdf

K. Weller. Folksonomies and Ontologies. Two New Players in Indexing and Knowledge Representation. 2007. http://www.phil-fak.uni-duesseldorf.de/infowiss/admin/public_dateien/files/35/1204288118weller009_.htm