Auditory Interfaces for Small Screen Mobile Devices
•Download as PPT, PDF•
0 likes•62 views
- The document discusses using sound and auditory interfaces to provide information on small screen mobile devices, especially for the visually impaired. It explores how sound can be used to convey information through properties like pitch, timbre, volume, and rhythm. - The presenters discuss projects they worked on like an auditory calendar and organizing tool that aim to understand how the visually impaired access information without visual cues. - Some challenges of auditory interfaces are discussed like condensing timelines, representing icons and symbols, and allowing for navigation. The potential for auditory interfaces to enhance accessibility and experience is explored.
More Related Content
What's hot
What's hot (18)
Similar to Auditory Interfaces for Small Screen Mobile Devices
Similar to Auditory Interfaces for Small Screen Mobile Devices (14)
More from Spencer Kiser
Recently uploaded
Recently uploaded (20)
Auditory Interfaces for Small Screen Mobile Devices
- 1. Auditory Interfaces for Small Screen Mobile Devices Listening to the blind for clues on interaction design.
- 2. Introductions » Michael Jefferson » NYU - ITP » Assistive Technology » HCI beyond the keyboard and mouse
- 3. Introductions » Spencer Kiser » Sound Design » Interactive Audio » Providing information sonically » Enhancing experience with audio
- 4. Discussion Outline » We are all losing our vision? » The un-tapped properties of sound. » Applications present and future » Q & A
- 5. The Fermata Project » Design a better organization tool » Ignore the visual interface » Learn from the visually impaired
- 7. Listening to the Blind » Designing an auditory calendar » Learning how the visually impaired access information » Observing Text-to-speech translation
- 8. Screen Readers » JAWS » Listen once and remember » Slow in translating detail » No such thing as an overview
- 9. The Problems » Condensing Time » Iconic & Symbolic Representation » A form of Short- hand
- 10. Mobility & Innovation » Innovation » Access to Content » Connectivity » Pioneers in mobile computing » Pac Mate » Braille
- 11. Pac Mate Demo
- 12. Digital | Divide » The cost of mobility » Custom hardware » Universal standard?
- 13. Universal Design? » Is there a universal design layer in mobile sofware?
- 14. Earbud Culture » Entertainment and communication merge in the handheld device » Podcasting and RSS
- 15. » As users, we are all facing the same problem. » As designers, we can learn from each other. » How far can we take the auditory interface? Data as Sound
- 17. What is a Soundscape? » An acoustic environment » Natural and man made sounds » Multiple layers
- 18. Information - list » Outside » Windy day » Busy urban daytime » New York (Brooklyn) accent » Standing on a corner » Intersection » Traffic cycles » Bumpy road » Chinese spoken
- 19. How much can we process at one time? » The Magical Number 7 (+/- 2) » Foreground and Background » Selective Listening
- 20. Selective Listening » Tuning in and out » Finding cues in noise » Selecting information when we need it. Examples are from Ben Rubin, Ear Studio www.earstudio.com QuickTime™ and a Sorenson Video 3 decompressor are needed to see this picture. QuickTime™ and a Sorenson Video 3 decompressor are needed to see this picture.
- 21. What are the advantages of using sound? »Pitch »Timbre »Volume »Rhythm »Localized » Multidimensional:
- 22. The Calendar
- 23. Designing for Interaction with Sound » Process » Mapping - Association of sounds to events » Layering - Playing back multiple sounds simultaneously. » Spatialization - Making individual sounds accessible.
- 24. Clues on Mapping » Assign a recognizable sound » Tie in detailed information QuickTime™ and a Sorenson Video 3 decompressor are needed to see this picture.
- 25. Mapping » Iconic Sounds » Applause » Glass clink » Airplane
- 26. Mapping » Symbolic Sounds » Bells = Business » Piano = Social » Vibraphone = Reminder » Woodblock = Separation in time
- 27. Early Response » Difficult to make dynamic » Navigation? QuickTime™ and a Cinepak decompressor are needed to see this picture.
- 28. Surround Solution » Enter event into calendar » Voice recording = detail layer » Association = summary
- 29. Surround Samples » More information requires symbolic sound associations » Specific sound qualities make appointments/events discernable day (iconic) day (symbolic) week
- 32. Feedback » Aesthetically pleasing » May help to remember QuickTime™ and a Sorenson Video 3 decompressor are needed to see this picture.
- 33. Feedback » Need for user control QuickTime™ and a Sorenson Video 3 decompressor are needed to see this picture.
- 34. Navigation
- 35. Conclusions » The answer lies in understanding how we make sense of information without technology. » Universal translation can and will benefit everyone. » Experimenting and sharing information on auditory design will advance UI design.
- 36. Q&A
Editor's Notes
- Capability has outpaced usability in the migration to handheld mobile devices and this is forcing interaction designers to expand their palate or tool set to meet new demands. The demands are not in generating more information but in accessing, filtering and making sense of what we have. We believe it is time to explore and expand the auditory interface and that clues for how to do so can be found in listening to the blind.
- Interaction Designer Currently at NYU attending ITP program Focus is on assistive technology Expanding HCI beyond keyboard and mouse.
- I’m also at ITP. Before starting there, I worked in sound design. At ITP my focus has been on interactive sound applications and devices, more specifically exploring challenges and advantages of providing information sonically. Started working with visually impaired with Michael on Fermata, our audio calendar project. Now using my skills with interactive sound design to work further with visually impaired.
- Today we want to share what we have learned and how it pertains to this “attention economy.” First: how the blind use technology There is an advantage to working together. Next: working with with sound explore the un-tapped properties of sound Finally: share with you examples of applications, including Fermata and some of the lessons we and other developers have learned.
- Nearly a year ago, Spencer, myself and onother partner from ITP, David Yates, set out to design a better system for keeping track of time and events, essentially, build a better organizer. IT was a response to what we felt was an over-taxed visual interface and to find something new, we decided to throuw out visuals all together. Our method or process began with partnering with the visually impaired.
- We know what data looks like in applications, a full calendar, a full inbox, a long document, but if we take away the screen, what would it sound like?
- This is Karen and Lynette, both are legaly blind, both use technology, infact, they train people on the use of technology and we will be hearing from them more later in the presentation. We spoke with them, and many other members of the blind community to learn how they listen to information on their computers, and how they listen to information in the environment.
- On the PC, the translation of data is all done text-to-speech This is what checking your email inbox sounds like to a blind user with the JAWS screenrteader system. We are not playing this to say that screen readers are bad, they are amazing in what they provide acces to. Elements missing Link in a document for example You have to hear it once and remember. Demands attention. Expansion in time.
- From these observations, we had defined our challenge. To build a better system using sound we would have to figure out… How can we condense time. Find a form of representation beyond text and speech? Define a shorthand rule we can provide to all users?
- To some extent, there are some solutions on the market. This is a PDA One in a long history of technical solutions to information access. Braille represents one of the answers in the translation of data, it’s a form of short hand, it let the user get through information quickly.
- (Play clip of Karen, master of the Braille Note.) What works here - Quick access Tactile shorthand What are we lacking - Overview, speed.
- These devices are impressive, the down side is that there is an even greater digital divide among the disabled community. The answer is not in hardware. An accessible PDA can run nearly five grand. While Braille offers an advantage but only to few. Wee need a mian-stream application,
- Can one design or enhanced design layer work for many? And how much would need to be translated? The lever vs knob. Sound can be the lever. Are we ready for an auditory interface?
- We think the answer is yes, if not today then soon. (play clip of commuters) Earbud culture. Mobile devices have gotten smarter and they want us to listen. C of devices in our pockets. Astor Place observation.
- So here we are with rich media and social software literally in the palm of our hand without visuals. We are now just like Karen and Lynette and we want the same thing, efficient, meaningful access. If we listen and share ideas, we can find new ways of making sense of complex, multi-layered information.
- Hand off to spencer
- One term we will be using often in the remainder of the talk is soundscape. A soundscape is the sonic landscape, the acoustic environment that surrounds us at any given moment, an environment from which we can glean data if we pay attention to it. It is made up of multiple layers which include all the natural and manmade sounds in our environment. We will now play a one-minute soundscape that I recorded, and I want you to see what you can find out about the environment based on the auditory clues you hear. After clip - anyone want to take a stab at where that might have been recorded or what kind of things were happening in the environment?
- -outside, kind of windy day -sounds of the city -might not know which city, but if you listen closely you might have heard a Brooklyn accent at the beginning -standing on a corner (sound of plastic scraping on the ground from sharp right to the front, then sharp left) -traffic cycles might have clued you in on the fact that I was standing on a corner - sharp left to in right in front of me, then off at a distance in front of me to sharp right. -also squeaky truck brakes -that might have clued you in to the fact that I was standing at an intersection -then at very end you might have heard a little Chinese being spoken -Indeed I was standing at a corner on Canal & Mulberry about two weeks ago in Chinatown Point of this is that there are tons of streams of audio hitting you at once, all the time. All of them are offering information that you could use if you needed it. Combine with photo.
- But how much can we really take in? In our research for the Fermata Calendar project, we came across the Magical Number Seven, Plus or Minus Two. this was a topic published in Psychology Review back in 1956 by George Miller. His finding was that humans can keep seven different words, sounds, terms, numbers in their head at once if they really put their mind to it. That helped inform us as to how much we can stream to the user before they become bogged down, and unable to process the information. Traditionally, sounds in applications are in the foreground, they serve to grab your attention, alarm and notify you. But as we just experienced in the soundscape, there is room for a whole lot more. Sounds can move in and out of foreground and background, and we can tune in to what we want. When you know what you’re looking for or maybe when you are ready to take in the information, it’s easier to find the what you need. Getting back to the soundscape example - if you put your mind to it, you could tell whether or not it was raining, maybe whether it’s safe to cross the street, or whether you’re in Chinatown or the West Village.
- So now we’ll play you some examples of research done by Ben Rubin, from Ear Studio in New York. He did a study on the aural feedback given to you in the New York subway stations, and made some suggestions on how to improve what is already in place. So when you are in this situation, What do you want to know? Whether your fare card is working, whether a train is coming, and if so, which way that train is going. <Play clip 1> What can we actually hear in this noisy environment? The current system uses a high pitched tone when you swipe the card. It gives you one tone when you are OK to go through, two tones in succession if the card swipe didn’t work, and three tones if the card worked, but you are running low on funds. <play clip 2> Ben’s proposals certainly sound much nicer and more musical than the current scheme. He is also using tones with certain qualities that help a person pick them out over the din of a New York subway station.
- Now I’m going to talk about the different tools to get what you want across to your user when you are working with sound. The great thing about using sound is that it has multiple dimensions: -Pitch (frequency) High and low lend themselves to mapping well, but mapping to discrete notes may make things difficult for non-musicians. This is also where those pleasing melodies you heard in Ben’s example come in. -Timbre is what give sound it’s character. What makes a sound sound like a trumpet or violin. It’s great because it’s helps to differentiate sounds, and it has instant connotations (eg a trumpet sound will might make you perk up and a violin sounds romantic or tragic.) -Loudness can be used to varying degrees of success. It might be hard to hear differences in volume among sounds in a busy soundscape, but changes in volume can be used effectively to indicate things turning on or off. -Timing or Rhythm - Using rhythm is a great way to quickly get across urgency or busy-ness. -Direction/Spatiality - is a tool we use in our calendar to separate the sounds out to let us play many simultaneously. Each of these can be mapped to data to varying degrees of success. Hand off to MJ
- So this is a good time to come back to our application, the mobile calendar for the visually Impaired, Fermata Map Layer Spatial Placement
- This is exactly the process we used in our design. Mapping - Association of sounds to events Layering - Playing back multiple sounds simultaneously. Spatialization - Making individual sounds accessible.
- Lynettes feedback on mapping symbolic or iconic sounds.
- Iconic sounds are sounds that analogous to the info that you are trying to get across. We use them as headlines for the events in the calendar. These are good: Instant identification because of direct relationship to the data you are representing and offer an opportunity for user customization and . Problems with building in libraries of iconic sounds.
- Symbolic sounds Sounds that have abstracted relationship with event Sound qualities: sharp attack, high frequencies are easier to place Bells: business appts. bells have been used to signify time for centuries (because of these qualities) can hear from far away and don’t need to be facing the tower or need to see it. Piano: social appts. Relaxing Vibraphone: all day events. longer, sustaining tone Woodblock we used to separate the time segments within each view.
- Turned to Surround for more control. So let’s make an appointment and map it. Let’s say I call Lynette about going to dinner and a concert on Saturday Lynette would speak that information into her device and the appointment would exist on two levels, a detail and a summary. (play clip of cork pop)
- Rings of sound In or out Left or right
- Now let’s listen to Our day in Surround. Now let’s check the week. Monday is to the left now and Sunday to the far right. Listen for Saturday night. (play)
- What has become clear to us over this past year is that the answer to the auditory interface lies in understanding how we make sense of information en every day life. When we do find answers, the great news is that audio is scalable, unlike the visual interface an auditory interface can move independent of physical form factor. What benefits the blind will benefit everyone. Last, In digging for answers on this topic it is evident that the knowledge is here to make it happen but it exists in different silos, game design, scoring for film, interaction design, assistive technology. In Manhattan, at Tisch, we have the unique opportunity to have all of this under one roof at 721 Broadway but even then, we are on different floors and departments and what we need is a place to pull these ideas together and stack solutions next to each other. This is why we are so honored to have a chance to share what we have found, in a hope that others can pick up on one area or another and help contribute. Spencer and I have created a WIKI page that we will be filling out with content and we encourage you to contribute or distribute this resource to anyone who may be interested. Thank You.