Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
By Benjamin Timmermans
In colla...
Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
PERCEPTIONS
Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
CONTRADICTIONS EXIST
Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
CONTRADICTIONS EXIST
Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
THERE IS NO RIGHT OR WRONG
Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
WHAT DO YOU HEAR?
Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
“This is the sound of a piece o...
Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
PROBLEMS
Incomplete descriptions
Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
Experts describe different thin...
Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
PROBLEMS:
POOR SEARCH RESULTS
Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
WHAT IS SEARCHED FOR
• Top 5000...
Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
HYPOTHESIS:
THE CROWD CAN PROVI...
Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
• 2148 sounds from freesound.or...
Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
ANNOTATION TASK
Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
• CrowdTruth + CrowdFlower
• US...
Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
Three versions:
• 38 x 3 sounds...
Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
• No phonetic words
• No senten...
Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
• Remove spammers using CrowdTr...
Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
• 573 workers
• 30.289 crowd ta...
Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
Experts
EXPERTS VS. CROWD
Crowd
Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
VOCABULARY OVERLAP
Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
TERMS ONLY USED BY CROWD
Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
WHAT IS SEARCHED BUT NOT FOUND
Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
Expert:
Scissors Stereo Cutting...
Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
Avg terms per sound in search d...
Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
FROM MOST “CAR” TO LEAST “CAR”
Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
• Expert sound tags often do no...
Vrije Universiteit Amsterdam
Faculty of Sciences / Computer Sciences / Benjamin Timmermans
• Investigate the type of conte...
Upcoming SlideShare
Loading in …5
×

Defining spacial representations for the meaning of sounds

2,501 views

Published on

An artificial intelligence talk at the VU Amsterdam about defining spacial representations for the meaning of sounds

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Defining spacial representations for the meaning of sounds

  1. 1. Vrije Universiteit Amsterdam Faculty of Sciences / Computer Sciences / Benjamin Timmermans By Benjamin Timmermans In collaboration with Emiel van Miltenburg
  2. 2. Vrije Universiteit Amsterdam Faculty of Sciences / Computer Sciences / Benjamin Timmermans PERCEPTIONS
  3. 3. Vrije Universiteit Amsterdam Faculty of Sciences / Computer Sciences / Benjamin Timmermans CONTRADICTIONS EXIST
  4. 4. Vrije Universiteit Amsterdam Faculty of Sciences / Computer Sciences / Benjamin Timmermans CONTRADICTIONS EXIST
  5. 5. Vrije Universiteit Amsterdam Faculty of Sciences / Computer Sciences / Benjamin Timmermans THERE IS NO RIGHT OR WRONG
  6. 6. Vrije Universiteit Amsterdam Faculty of Sciences / Computer Sciences / Benjamin Timmermans WHAT DO YOU HEAR?
  7. 7. Vrije Universiteit Amsterdam Faculty of Sciences / Computer Sciences / Benjamin Timmermans “This is the sound of a piece of A4 paper being cut with scissors. The paper was wedged between the capsules of a Rode NT-4 stereo microphone and so was in contact with them.” Author tags: Scissors Stereo Cutting Foley Paper AMBIGUOUS SOUND
  8. 8. Vrije Universiteit Amsterdam Faculty of Sciences / Computer Sciences / Benjamin Timmermans PROBLEMS Incomplete descriptions
  9. 9. Vrije Universiteit Amsterdam Faculty of Sciences / Computer Sciences / Benjamin Timmermans Experts describe different things then what people are interested in PROBLEMS
  10. 10. Vrije Universiteit Amsterdam Faculty of Sciences / Computer Sciences / Benjamin Timmermans PROBLEMS: POOR SEARCH RESULTS
  11. 11. Vrije Universiteit Amsterdam Faculty of Sciences / Computer Sciences / Benjamin Timmermans WHAT IS SEARCHED FOR • Top 5000 search queries of freesound.org • Representing 5,6 million searches during 5 months
  12. 12. Vrije Universiteit Amsterdam Faculty of Sciences / Computer Sciences / Benjamin Timmermans HYPOTHESIS: THE CROWD CAN PROVIDE BETTER PERCEPTUAL DESCRIPTIONS THAN EXPERTS
  13. 13. Vrije Universiteit Amsterdam Faculty of Sciences / Computer Sciences / Benjamin Timmermans • 2148 sounds from freesound.org • 19.098 tags by experts • 2.008 unique terms • Avg 8,9 terms per sound DATASET
  14. 14. Vrije Universiteit Amsterdam Faculty of Sciences / Computer Sciences / Benjamin Timmermans ANNOTATION TASK
  15. 15. Vrije Universiteit Amsterdam Faculty of Sciences / Computer Sciences / Benjamin Timmermans • CrowdTruth + CrowdFlower • US/UK/AUS/CA crowd workers • 10 annotations for each sound • 3 sounds in one task • $ 0.02 reward EXPERIMENTAL SETUP
  16. 16. Vrije Universiteit Amsterdam Faculty of Sciences / Computer Sciences / Benjamin Timmermans Three versions: • 38 x 3 sounds < 30 sec. • 300 x 3 sounds • short: < 1 sec. • med: 5 to 6 sec. • long: 17 to 21 sec. • 378 x 3 sounds < 1 sec. DATASET
  17. 17. Vrije Universiteit Amsterdam Faculty of Sciences / Computer Sciences / Benjamin Timmermans • No phonetic words • No sentences • Correct spelling • Comma separated tags • Infinite number of tags TASK INSTRUCTIONS
  18. 18. Vrije Universiteit Amsterdam Faculty of Sciences / Computer Sciences / Benjamin Timmermans • Remove spammers using CrowdTruth metrics • Clean up non-alphanumeric characters • Loud! -> loud • Spell correction • Barkking -> barking • Cluster on morphology, substrings, word order • Walk, walks, walking -> walk • Dog, Fat dog -> Fat dog • Running dog = Dog running POST-PROCESSING
  19. 19. Vrije Universiteit Amsterdam Faculty of Sciences / Computer Sciences / Benjamin Timmermans • 573 workers • 30.289 crowd tags • 7.327 unique terms • Avg 15 terms per sound • 5% spam removed RESULTS
  20. 20. Vrije Universiteit Amsterdam Faculty of Sciences / Computer Sciences / Benjamin Timmermans Experts EXPERTS VS. CROWD Crowd
  21. 21. Vrije Universiteit Amsterdam Faculty of Sciences / Computer Sciences / Benjamin Timmermans VOCABULARY OVERLAP
  22. 22. Vrije Universiteit Amsterdam Faculty of Sciences / Computer Sciences / Benjamin Timmermans TERMS ONLY USED BY CROWD
  23. 23. Vrije Universiteit Amsterdam Faculty of Sciences / Computer Sciences / Benjamin Timmermans WHAT IS SEARCHED BUT NOT FOUND
  24. 24. Vrije Universiteit Amsterdam Faculty of Sciences / Computer Sciences / Benjamin Timmermans Expert: Scissors Stereo Cutting Foley Paper Crowd: WHAT THE CROWD HEARD Knocking (3x) Stamping Stomping Hammering Clattering Drumming (3x) Drum (2x) Tapping Drum beat Beat Bang (2x) Music Irregular No fixed rhythm Hollow Loud Thud
  25. 25. Vrije Universiteit Amsterdam Faculty of Sciences / Computer Sciences / Benjamin Timmermans Avg terms per sound in search data: • Crowd: 10.38 • Experts: 7.01 • Crowd + Expert: 16.63 SEARCH IMPROVEMENT
  26. 26. Vrije Universiteit Amsterdam Faculty of Sciences / Computer Sciences / Benjamin Timmermans FROM MOST “CAR” TO LEAST “CAR”
  27. 27. Vrije Universiteit Amsterdam Faculty of Sciences / Computer Sciences / Benjamin Timmermans • Expert sound tags often do not describe what is heard • Crowd can efficiently generate useful sound tags • Crowd provided better perceptual descriptions • Perceptual descriptions improve search CONCLUSION
  28. 28. Vrije Universiteit Amsterdam Faculty of Sciences / Computer Sciences / Benjamin Timmermans • Investigate the type of content described with sound tags • Specificity • Subjectivity • Further investigate influence of sound length • Improve search for sounds FUTURE WORK

×