Presentation slides for the "Catching Waves" Panel Discussion on Sustainable Digital Audio Delivery held at Lamont Library, Harvard University, Friday May 8th, 2015

1. Decoding Digital Audio
Visualizing and Annotating
Linear Time-Based Media
Phil Desenne
Center
for
Hellenic
Studies,
Harvard
University
May 8, 2015

2. To decode or interpret audio is to
explain the meaning
or
understanding
of something about it
Relevant for Research, Teaching and Learning
across all disciplines
Decode –> Interpret

3. Curiosity - Discovery - Interpretation - Research
Amateurs - Learners - Educators - Scholars
Each one decodes audio in their own realm
the process transcends realms and roles
}
}

4. Songs
Music
Voice recordings
Field recordings
Lyrics / Transcription / Translation
Notes / Musicology / Ethnomusicology
Oral History / Languages / Speech Therapy
Bioacoustics Research / Anthropology
Audio Visualize and Annotate
{}
Listen Decode

5. In our day-to-day we are constantly, filtering, decoding
and attaching meaning in our brain to daily sound
bites that hit our ears
!
digital audio opens a broader spectrum of
decoding possibilities for
research, teaching and learning

6. Visualization and Annotation of Audio
Lorem ipsum dolor sit amet,
consectetur adipiscing elit.
Nam in augue sodales nisl
pharetra efficitur in in felis. Ut
malesuada justo nec libero
finibus placerat. Donec vitae
enim risus. Nunc eget purus
eget nunc bibendum tempus
hendrerit vel eros. Praesent
mollis diam augue, vel
convallis quam interdum eu.
Transcript
we now have digital tools that facilitate and
enhance the process of decoding, attaching
meaning and understanding

7. Audio Annotation & Visualizations in
Research
Acoustics / Bioacoustics / Hydroacoustics
Anthropology / Musicology / Ethnomusicology
Speech Therapy
Education / Teaching

8. Importance of Visualizing Sound:
Human ear does not hear or discern all sounds
www.cochlea.org739 × 252Search by image
Frequency hearing range in man and some common animal
very high-
frequency
sounds
very low-
frequency
sounds
}
visualization –> accessibility

9. Recording Natural Sounds
www.leaps.ms900 × 396Search by image
You can also use Raven Lite to slow down natural recordings so that the full complexity of a song may be heard. Listen to the complexity in the ending trill ...
Bioacoustics:
detection and interpretation of sounds in animals
Annotating the visual wave form of audio:
Amplitude and Frequency

10. BIRD SONGS AND CALLS WITH SPECTROGRAMS ( SONOGRAMS ) OF SOUTHERN ...
www.birdsongs.it489 × 396Search by image
Fig . 2 shows a 20 second fragment over 1-minute song sequence of a Cirl Bunting (Emberiza cirlus). Fig. 3
blows up one phrase of the same song and shows ...
Sonograms and Spectrograms of Bird Songs

11. The Mind's Machine - Chapter 15 A Step Further
www.mindsmachine.com800 × 619Search by image
(a) These sonograms show the typical adult song patterns of two sparrow
species. The songs illustrated in part (b) were produced by males reared ...
Species identification and animal behavior

12. WarblerWatch: Warbler Guy, where do I learn about "reading ...
warblerwatch.blogspot.com650 × 578Search by image
Ergo, you'll quickly have no problems identifying a song sparrow classic
song via its sonogram in comparison to a common yellowthroat's, and
so on.

13. Teaching and Learning with audio annotations
Examples at Harvard

14. Prof. Tom Kelly, First Nights course, Harvard College
Music Courses: Annotated Interactive Play-throughs
Learners explore music

15. Foreign Culture Courses: Listening Guides
Prof. Richard Wolfe, Foreign Cultures Course, Harvard College

16. Foreign Culture Courses: Listening Guides
Prof. Richard Wolfe, Foreign Cultures Course, Harvard College

17. Table text
XML text
Listening guide player

18. Harvard iSites CAT tool
Prof. Richard Wolfe, Harvard College

19. Research Teachingdrives

20. Research Teachingdrives
Prof. Richard Wolfe
Ethnomusicology research in
South, Central and West Asia
Courses in Ethnomusicology at
Harvard College
Audio
annotations
richardkwolf.com www.music.fas.harvard.
edu/faculty/rwolf.html

21. Repositories
Media and Annotation Metadata
How does it all connect ?

22. Annotation
Meta layer
Visualization layer
Audio layer (URI)
Annotation
DB
Media
Repositories
ideally under same repository entities
Media Search
Open Annotation
Data Model
Client tool !

23. Interoperability: Tying it all together

24. Persistent Annotation Meta-layer
Open API Access
Stable Digital Repositories
URNs resolving to URLs
Ephemeral Tools / Content / Learning
Management Systems
Open Annotation Model

25. Public
Archives
Other
Institutions
Archives
Open
Source
Museums
HUL
DRS
Research
Database
Catalyst,
LibraryLab
HILT
...
Course
& Student
Content
Peer
Researchers
Personal
Research &
Archives
Subject
Experts
Incubator
Projects
Persistent
Annotation
Repositories
Individual
Repositories
External
Repositories
Internal Repositories
Ann
An
Open Annotation Federated Systems
across all media

26. Future of audio annotation
• Searching: faceted, specific range target searches
• Semantic tagging: machine learning
• Automatic Annotations:
• transcription / translation
• acoustic detection,
• individual voice recognition
• bioacoustics species id
• AI -> Pairing crowdsourced data and automatic
annotation using semantic annotated data (OA)
• High definition audio and detailed audio analysis
• Collaboration and crowdsourcing tools
• Cross-referencing media annotations

27. ThankYou !
Questions?
!
desenne[ at ]fas[ dot ]harvard[ dot ]edu