Visual soundscapes from your augmented reality glasses

605 views

Published on

Slides presented at "CONGres 2012: Sounds in Science", invited lecture at the University of Amsterdam, The Netherlands, October 17, 2012

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Visual soundscapes from your augmented reality glasses

  1. 1. CONGres 2012 - Visual soundscapes from your augmented reality glasses visual soundscapes from your augmented reality glasses CONGres 2012 October 17 2012, Amsterdam Sounds in Science Peter Meijer, Metamodal BV www.seeingwithsound.com © 2012 Peter B.L. Meijer 0% 2 camera display you in 2014? © 2012 Peter B.L. Meijer 2%E-mail: feedback@seeingwithsound.com (Peter Meijer) URL: http://www.seeingwithsound.com 1
  2. 2. CONGres 2012 - Visual soundscapes from your augmented reality glasses 3 ? © 2012 Peter B.L. Meijer 4% 4 visual soundscapes & augmented reality Overview Overview •• introduction introduction •• sensory substitution sensory substitution camera glasses •• visual soundscapes visual soundscapes •• brain plasticity brain plasticity •• augmented reality augmented reality •• conclusions conclusions © 2012 Peter B.L. Meijer 6%E-mail: feedback@seeingwithsound.com (Peter Meijer) URL: http://www.seeingwithsound.com 2
  3. 3. CONGres 2012 - Visual soundscapes from your augmented reality glasses 5 visual soundscapes & augmented reality Overview Overview •• introduction introduction •• sensory substitution sensory substitution camera glasses •• visual soundscapes visual soundscapes •• brain plasticity brain plasticity •• augmented reality augmented reality •• conclusions conclusions © 2012 Peter B.L. Meijer 9% 6 bionic eye retinal and cortical implants www.jensnaumann.com “Jerry” with Dobelle brain implant Jens Naumann with Dobelle brain implant Mark Humayun with Connie Schoeman (RP) Retina: Retina Implant AG, Second Sight, Boston U., and many others Cortex: Dobelle Institute (until 2004), CORTIVIS, U. Utah., IIT, U. NSW © 2012 Peter B.L. Meijer 11%E-mail: feedback@seeingwithsound.com (Peter Meijer) URL: http://www.seeingwithsound.com 3
  4. 4. CONGres 2012 - Visual soundscapes from your augmented reality glasses 7 non-invasive approaches computer mobile brain vision computing plasticity ubiquitous augmented cameras reality © 2012 Peter B.L. Meijer 13% 8 Robotic navigation aid for the blind? James Weiland, USC Viterbi School of Engineering URL SLAM (Simultaneous Localization and Mapping), USC on NTDTV July 4 2011 © 2012 Peter B.L. Meijer 16%E-mail: feedback@seeingwithsound.com (Peter Meijer) URL: http://www.seeingwithsound.com 4
  5. 5. CONGres 2012 - Visual soundscapes from your augmented reality glasses 9 computer vision & vision technology for the blind (news headlines in 2011) More examples MS Kinect KinectFusion next? Stereo vision NAVI - Universität Konstanz, Germany EYE21- Universitat Politècnica de València, Spain Kinect for the Blind - Moscow Institute of Physics & Technology, Russia (MS Imagine Cup 2011) - few technologies ever make it out of the lab - cost of commercialization, bulky, unreliable, physical limitations (Kinect fails in sunshine), … © 2012 Peter B.L. Meijer 18% 10 visual soundscapes & augmented reality Overview Overview •• introduction introduction •• sensory substitution sensory substitution camera glasses •• visual soundscapes visual soundscapes •• brain plasticity brain plasticity •• augmented reality augmented reality •• conclusions conclusions © 2012 Peter B.L. Meijer 20%E-mail: feedback@seeingwithsound.com (Peter Meijer) URL: http://www.seeingwithsound.com 5
  6. 6. CONGres 2012 - Visual soundscapes from your augmented reality glasses 11 sensory substitution tactile or auditory display camera ear buds camera tongue display Wicab’s BrainPort® The vOICe © 2012 Peter B.L. Meijer 23% 12 sensory substitution with auditory display Michael Proulx © 2012 Peter B.L. Meijer 25%E-mail: feedback@seeingwithsound.com (Peter Meijer) URL: http://www.seeingwithsound.com 6
  7. 7. CONGres 2012 - Visual soundscapes from your augmented reality glasses 13 setups for The vOICe cost < $1000, or < $500 when using a netbook PC Michael Proulx © 2012 Peter B.L. Meijer 27% 14 visual soundscapes & augmented reality Overview Overview •• introduction introduction •• sensory substitution sensory substitution camera glasses •• visual soundscapes visual soundscapes •• brain plasticity brain plasticity •• augmented reality augmented reality •• conclusions conclusions © 2012 Peter B.L. Meijer 30%E-mail: feedback@seeingwithsound.com (Peter Meijer) URL: http://www.seeingwithsound.com 7
  8. 8. CONGres 2012 - Visual soundscapes from your augmented reality glasses 15 The vOICe image-to-sound mapping Frequency Spectrographic synthesis Time © 2012 Peter B.L. Meijer 32% 16 The vOICe image-to-sound mapping Could this really work? Theory Yes! (at least technically) Check: map sounds back to images © 2012 Peter B.L. Meijer 34%E-mail: feedback@seeingwithsound.com (Peter Meijer) URL: http://www.seeingwithsound.com 8
  9. 9. CONGres 2012 - Visual soundscapes from your augmented reality glasses 17 The vOICe image reconstruction “Soundscape” © 2012 Peter B.L. Meijer 37% 18 perceptual resolution beat 20/200 legal blindness barrier? PLoS ONE 7(3), 2010, ``Visual’ Acuity of the Congenitally Blind Using Visual-to-Auditory Sensory Substitution, by Ella Striem-Amit, Miriam Guendelman and Amir Amedi: : “Group performance differed statistically from chance level at all visual acuities up to 20/320” ” “The visual acuity of the individual participants varied between 20/200 and 20/600” “Interestingly, 5 of the 9 participants (55%) had visual acuity that exceeded the visual acuity threshold for blindness as defined by the WHO” (20/400) © 2012 Peter B.L. Meijer 39%E-mail: feedback@seeingwithsound.com (Peter Meijer) URL: http://www.seeingwithsound.com 9
  10. 10. CONGres 2012 - Visual soundscapes from your augmented reality glasses 19 The vOICe mental imagery? Warning: Representation may dramatically affect recognition! © 2012 Peter B.L. Meijer 41% 20 The vOICe demo sounds (1 of 5) Frequency Time © 2012 Peter B.L. Meijer 44%E-mail: feedback@seeingwithsound.com (Peter Meijer) URL: http://www.seeingwithsound.com 10
  11. 11. CONGres 2012 - Visual soundscapes from your augmented reality glasses 21 The vOICe demo sounds (2 of 5) Frequency Time © 2012 Peter B.L. Meijer 46% 22 The vOICe demo sounds (3 of 5) Frequency Time © 2012 Peter B.L. Meijer 48%E-mail: feedback@seeingwithsound.com (Peter Meijer) URL: http://www.seeingwithsound.com 11
  12. 12. CONGres 2012 - Visual soundscapes from your augmented reality glasses 23 The vOICe demo sounds (4 of 5) Frequency Time © 2012 Peter B.L. Meijer 51% 24 The vOICe demo sounds (5 of 5) Informational Frequency masking? Apply mental saccades Time ``Slow motion’’ © 2012 Peter B.L. Meijer 53%E-mail: feedback@seeingwithsound.com (Peter Meijer) URL: http://www.seeingwithsound.com 12
  13. 13. CONGres 2012 - Visual soundscapes from your augmented reality glasses 25 Grasping exercise (can be mastered in a few hours) © 2012 Peter B.L. Meijer 55% 26 Grasping exercise (video clip from PLoS ONE 2008) Source: Proulx MJ, Stoerig P, Ludowig E, Knoll I, 2008 Seeing ‘Where’ through the Ears: Effects of Learning-by- Doing and Long-Term Sensory Deprivation on Localization Based on Image-to-Sound Substitution. PLoS ONE B.L. Meijer © 2012 Peter 3(3) 58%E-mail: feedback@seeingwithsound.com (Peter Meijer) URL: http://www.seeingwithsound.com 13
  14. 14. CONGres 2012 - Visual soundscapes from your augmented reality glasses 27 [early blind] Metamodal LLC, California: NSF-funded SBIR Phase I project with The vOICe, Pasadena 2010 © 2012 Peter B.L. Meijer 60% http://www.youtube.com/playlist?list=PLDE409461FD43A4D0 28 [late-blind] Metamodal LLC, California: NSF-funded SBIR Phase I project with The vOICe, Pasadena 2010 © 2012 Peter B.L. Meijer http://www.youtube.com/playlist?list=PLDE409461FD43A4D0 62%E-mail: feedback@seeingwithsound.com (Peter Meijer) URL: http://www.seeingwithsound.com 14
  15. 15. CONGres 2012 - Visual soundscapes from your augmented reality glasses 29 The vOICe What do blind users see? > In the bottom left hand corner is 3 short dark columns and above it a V > shaped band. These columns are shorter than the trees. To the right of > the small columns are several short and long horizontal lines on a > bright short band. I dont think they are columns but not sure what > they are. > In the top right hand corner of the picture is something which I > recognise as architecture. It has 3 light coloured vertical columns > below a sloping roof or sloping structure at the front above the > columns. It is surrounded in shadow, especially on the right. > > Given that this is an architectural feature and not something smaller, Photograph: courtesy Michael Larsen > that must mean that the vegetation on the top left of the picture is one > or several trees. © 2012 Peter B.L. Meijer 65% 30 The vOICe Artificial synesthesia? PF, late blind female, US, after hearing about visual cortex implants: Just sound?.... No, It is by far more, it is sight ! There IS true light preception generated by the vOICe. When I am not wearing the voice the light I perceive from a small slit in my left eye is a grey fog. When wearing the vOICe the image is light with all the little greys and blacks. Yet a definite light image. True it is not color but it is definitely like looking at a black and white TV show. The light generated is very white and clear then it erodes down the scale of color to the dark black. I dont really see adiffrence in this light as compaired to the "light phosphenes " they are talking about. See also: Ward, J. and Meijer, P., ``Visual experiences in the blind induced by an auditory sensory substitution device, Consciousness and Cognition, Vol. 19, 2010, pp. 492–500. © 2012 Peter B.L. Meijer 67%E-mail: feedback@seeingwithsound.com (Peter Meijer) URL: http://www.seeingwithsound.com 15
  16. 16. CONGres 2012 - Visual soundscapes from your augmented reality glasses 31 visual soundscapes & augmented reality Overview Overview •• introduction introduction •• sensory substitution sensory substitution •• brain plasticity brain plasticity •• augmented reality augmented reality •• conclusions conclusions © 2012 Peter B.L. Meijer 69% 32 Neuroscience: Sound can activate “visual” cortex of blind Kujala, Teija et al. (1995) “Visual cortex activation in blind humans during sound discrimination,” Neurosci. Lett. 183, 143–146 Early-blind person detecting pitch changes © 2012 Peter B.L. Meijer 72%E-mail: feedback@seeingwithsound.com (Peter Meijer) URL: http://www.seeingwithsound.com 16
  17. 17. CONGres 2012 - Visual soundscapes from your augmented reality glasses 33 The vOICe in Neuroscience 2007: Cross-modal plasticity & visual sounds dx.doi.org/10.1038/nn1912 Amedi et al., Nature Neuroscience, June 2007: Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex LOtv acts as a metamodal operator for shape Percent signal change analysis of LOtv LOtv V1 Late-blind expert user of The vOICe (Talairach normalized inflated cortex reconstruction) © 2012 Peter B.L. Meijer 74% 34 The Nature of Things CBC television 2008 Harvard Medical School using The vOICe, Boston 2007 © 2012 Peter B.L. Meijer 76%E-mail: feedback@seeingwithsound.com (Peter Meijer) URL: http://www.seeingwithsound.com 17
  18. 18. CONGres 2012 - Visual soundscapes from your augmented reality glasses 35 Dobelle brain implant crossmodal perception? Dennis, one of the recipients of a Dobelle brain implaint, complained during test stimulation of his visual cortex that "the beeps were too loud" This only happened when multiple electrodes were simultaneously stimulated. When individually stimulated, there were only visual sensations (phosphenes) for him Dennis was 26 when he received the implant, after being blind for 6 years Source: Jens Naumann, ``Search for paradise: A patients account of the artificial vision experiment, www.jensnaumann.com, and personal communication © 2012 Peter B.L. Meijer 79% 36 visual soundscapes & augmented reality Overview Overview •• introduction introduction •• sensory substitution sensory substitution •• visual soundscapes visual soundscapes •• brain plasticity brain plasticity •• augmented reality augmented reality •• conclusions conclusions © 2012 Peter B.L. Meijer 81%E-mail: feedback@seeingwithsound.com (Peter Meijer) URL: http://www.seeingwithsound.com 18
  19. 19. CONGres 2012 - Visual soundscapes from your augmented reality glasses 37 camera earbuds The vOICe? © 2012 Peter B.L. Meijer 83% 38 augmented reality camera view overlays Layar Wikitude Smartphone-based visual overlays • Wikitude • Layar • … For those without eyesight, a sonic overlay © 2012 Peter B.L. Meijer 86%E-mail: feedback@seeingwithsound.com (Peter Meijer) URL: http://www.seeingwithsound.com 19
  20. 20. CONGres 2012 - Visual soundscapes from your augmented reality glasses 39 The vOICe for Android augmented reality for the blind live soundscapes barcode reader Google Goggles foveal enlargement talking locator (GPS) talking compass color identifier/filters tactile feedback voice commands multilingual (Russian, Spanish, Chinese, …) © 2012 Peter B.L. Meijer 88% 40 © 2012 Peter B.L. Meijer 90%E-mail: feedback@seeingwithsound.com (Peter Meijer) URL: http://www.seeingwithsound.com 20
  21. 21. CONGres 2012 - Visual soundscapes from your augmented reality glasses 41 The vOICe for Android Google Analytics & mobile vision niche, yet in August 2011: 90,000+ total installs 4-week use: 135 countries October 2012: 180,000+ total installs Google Play © 2012 Peter B.L. Meijer 93% http://www.nibib.nih.gov/Research/Highlights/Connectome 42 training & motivation… © 2012 Peter B.L. Meijer 95%E-mail: feedback@seeingwithsound.com (Peter Meijer) URL: http://www.seeingwithsound.com 21
  22. 22. CONGres 2012 - Visual soundscapes from your augmented reality glasses 43 visual soundscapes & augmented reality Overview Overview •• introduction introduction •• sensory substitution sensory substitution •• visual soundscapes visual soundscapes •• brain plasticity brain plasticity •• augmented reality augmented reality •• conclusions conclusions © 2012 Peter B.L. Meijer 97% 44 visual soundscapes & augmented reality conclusions: R&D ongoing affordable non-invasive available worldwide http://www.seeingwithsound.com http://www.seeingwithsound.com with thanks to all blind users and research partners around the world! © 2012 Peter B.L. Meijer 100%E-mail: feedback@seeingwithsound.com (Peter Meijer) URL: http://www.seeingwithsound.com 22

×