Download to read offline
![HiPSTAS team
• Tanya Clement, [PI] Assistant Professor, University of
Texas at Austin
• Loretta Auvil [Co-PI] Senior Project Coordinator at the
Illinois Informatics Institute (I3) at the University of
Illinois at Urbana-Champaign
• David Tcheng [Co-PI] Research Scientist at I3; ARLO
developer
• Tony Borries, Research Programmer working as a
consultant with I3; ARLO programmer
• David Enstrom, Biologist, University of Illinois at
Urbana-Champaign; consultant](https://image.slidesharecdn.com/fiat2016ced-181212181347/85/Let-the-Computer-Do-the-Work-13-320.jpg)
Uploaded byWGBH Media Library and Archives
Let the Computer Do the Work
This document discusses using computational tools to analyze and provide access to large audiovisual collections. It describes a project called HiPSTAS that developed the open-source ARLO tool to analyze 68,000 hours of audio content using speech-to-text transcription, audio waveform analysis, and machine learning. The goal was to assess how these tools could help humanities scholars better access spoken word collections. Key challenges discussed include balancing accuracy with efficiency and ensuring tools are accessible and usable across disciplines.
More Related Content
Similar to Let the Computer Do the Work
More from WGBH Media Library and Archives
Let the Computer Do the Work
- 1. LET THE COMPUTER DOTHE WORK Karen Cariani & Casey Davis WGBH | Boston, MA, USA
- 4. the situation ■ 68,000digitized television and radio programs ■ incomplete, inaccurate metadata records ■ limited staff resources ■ we need to know what we have in the collection ■ we have a responsibility to users to provide access to the collection ■ continued growth of the collection (content and sparse metadata)
- 6.
- 7.
- 8.
- 9. The State ofRecorded Sound Preservation in the United States: A National Legacy at Risk in the Digital Age (2010) Suggested that if scholars and students do not use sound archives, cultural heritage institutions will be less inclined to preserve them. Archives and libraries must collaborate with patrons and scholars to understand how recordings are and might be used. Scholars need to know what kinds of analysis are possible in an age of large, freely available collections and advanced computational analysis.
- 10. State of theart
- 11. A vision “ .. . the sound file would become . . . a text for study, much like the visual document. The acoustic experience of listening to the poem would begin to compete with the visual experience of reading the poem.” Bernstein, Charles. Attack of the Difficult Poems: Essays and Inventions. University Of Chicago Press, 2011, 114.
- 12.
- 13. HiPSTAS team • TanyaClement, [PI] Assistant Professor, University of Texas at Austin • Loretta Auvil [Co-PI] Senior Project Coordinator at the Illinois Informatics Institute (I3) at the University of Illinois at Urbana-Champaign • David Tcheng [Co-PI] Research Scientist at I3; ARLO developer • Tony Borries, Research Programmer working as a consultant with I3; ARLO programmer • David Enstrom, Biologist, University of Illinois at Urbana-Champaign; consultant
- 14. Participants, Hipstas Institute,2013- 2014 • 8 librarians and archivists • 9 humanities scholars • 3 advanced graduate students in humanities and information science
- 15. Participating collections • poetryfrom PennSound at the University of Pennsylvania 30,000 audio files • folklore at the Dolph the Briscoe Center for American History at UT Austin, 57 feet of tapes (reels and audiocassettes) • storytelling traditions at the Native American Projects (NAP) at the American Philosophical Society in Philadelphia , 50 tribes, 3,000 hours
- 16. • Field recordings(200,000 recordings) American Folklife Center, Library of Congress • 30, 000 hours, Oral histories, Storycorps • Speeches in the Southern Christian Leadership Conference recordings, Emory University • 700 recordings in the Elliston Poetry Collection at the University of Cincinnati • 36 interviews in the Dust, Drought and Dreams Gone Dry: Oklahoma Women and the Dust Bowl (WDB) oral history project out of the Oklahoma State Libraries OTHER COLLECTIONS OF INTEREST TO PARTICIPANTS
- 17. To develop avirtual research environment in which users can better access and analyze spoken word collections of interest to humanists through: 1. an assessment of scholarly requirements for analyzing sound 2. an assessment of technological infrastructures needed to support discovery 3. preliminary tests that demonstrate the efficacy of using such tools in humanities scholarship 4. A freely available, open-source, API-driven version for general use HIPSTAS: PRIMARY GOALS
- 18. ARLO (Adaptive Recognitionwith Layered Optimization) HZ, a unit of frequency Time a heat based color scheme. White – hottest, most intense Yellow Red Green Blue Black – coolest, least intense Energy represented by
- 19. OpenMary LaynorStein Searching for Soundwith Sound Supervised Classification
- 20.
- 21.
- 22.
- 23.
- 24.
- 25. Blue = sung;green = spoken; red = instrumental 55 John Alan Lomax recordings 1926-1941 Visualize results
- 26. Visualize results 55 JohnAlan Lomax recordings 1926-1941
- 27. Takeaways: ■ What doscholars talk about when they talk about sound? • Language dynamics: tempo, pitch, tone/timbre, volume, pace, laughter, silence, applause, moans, screams, dialects, changing speakers, gender, age, changing genres • Environment: fan hums, car horns, chickens, train whistles, bird calls, frogs mating • Materiality: recording noises, needle drops, feedback, the electronic grid, changing tracks
- 28. ■What do engineerstalk about they talk about audio? • Resolution: Bit depth, Bit rate, sample rate • Signal processing: Fast Fourier Transform (FFT) and filter banks • Dynamics: Damping ratios, gain, frequencies, spectra, energy, and pitch energy TAKEAWAYS
- 29. ■ What docomputer scientists talk about when they talk about ML? • Features: What are we measuring? • Ground Truth: What’s the answer? How do we know when we’re accurate? • Optimization: Accuracy vs. Efficiency – how do you balance the accuracy of your results against the computational resources you need to achieve that level of accuracy? Takeaways
- 30. Takeaways • Literacy: Howmuch do we need to know about the technology of audio, of computational methods, and of humanist inquiry to do new kinds of research in this area? • Usability: What kinds of interfaces and tools facilitate AV analysis in a diverse range of disciplines and communities? Who gets access to these tools and for what kinds of questions? • Accuracy: Is good enough, good enough? • Scalability: How much storage and processing power do users need to conduct local and large-scale AV analyses? A Laptop? A Supercomputer? • Sustainability: What are local, national, and global scale issues? How does this work fit back into the access infrastructure already in place in archives, libraries, classrooms? Is data enough to get us over the hump of our limited means for discovery?
- 31.
- 32. Computational tools ■ Language ■Speech to text ■ Image recognition ■ Sound
- 33.
- 34. We will wantto show sample files ■ Popup archive ■ Speech to text
- 35. ■ Games tocorrect
- 36.
- 37. LAPPS grid ■ Toolslisted
- 44.
Editor's Notes
- #20 Perform machine learning with the instance based algorithm (with distance weighting power = 15 and threshold at .4 being optimal) using inductive biased optimization to come up with configuration; all machine learning algorithms have control parameters -- you need to try out different ones to find out what is optimal for your problem (classification threshold and distance weighting power) file-based cross validation was used to measure predictive performance; simulate the process for having the ground truth from other files and see how well you can predict it; applied the model to all the files (25 million examples); For each example in the file (an example is a 1/32nd second), we classify every example. To classify the time slice: compare it to all known slices and compute the distance between it and all known slices; computing the distance for 256 bands is 256 values (feature space has 256 dimensions in it); when we compute the distance, we are computing the distance between two points in a 256-dimensional space; this means for each dimension, compute the absolute value of the difference between each feature pair and then sum all the differences using the power of 1 (city block, taxi cab distance, hamming distance [straight lines] -- [Euclidean is 2; as the crow flies]) http://taxicabgeometry.net/general/basics.html After computing the distance, convert the distance into a weight; the weight for every example is derived from its distance 1/distance(raised to a power) [which in our case is 15]. [not using other predictions from near windows, previous or subsequent, to determine classes] Use the classes of all the examples and all of their weights to determine a “vote” for the current example; formula for the vote, single class probability: sum up (the actual class [0 or 1] times its weight) then divide by the sum of all weights; This creates a weighted prediction where some examples get more weight than others; [if you set it to 0 every slice would get the same weight]; ground truth is not taken out; [you will see perfect examples in the results;]
- #25 Low voice between 820 and 845
- #26 Blue = sung; green = spoken; red = instrumental