This document discusses sensory substitution, which is using input from one sensory modality to acquire information for another sensory modality. It provides examples of auditory-vision substitution using the vOICe system and tactile-vision substitution using tactile arrays. Tactile-vestibular substitution is also discussed as a way to provide balance information to subjects with bilateral vestibular loss. The strengths are that training with these systems can lead to externalization of sensations and accurate identification of percepts. However, the precise neural mechanisms are not fully understood. There is potential for inexpensive and accessible implementations to expand human senses and study brain plasticity.

1. Natasha Gupta

2. 
Neuroplasticity
 Adaptive capacities of
the central nervous
system
 Its ability to modify its
own structural
organization and
functioning

3. 
Sensory Substitution
 Using input from one sensory modality
to acquire information for another
 Eg - Persons who become blind do not
lose the capacity to see. Usually, they
lose the peripheral sensory system (the
retina), but retain central visual
mechanisms
 The input from a sensory substitution
system can reach many brain structures
 Offers an opportunity to restore
function.
 In such restoration of lost senses,
information from an artificial receptor is
coupled to the brain via a human–
machine interface (HMI), replacing
information usually carried to the brain
from an intact sense organ.
 Sensory substitution is thus only possible
because of brain plasticity.

4. 
Common Examples
Braille
Reading
Using a Cane

5. 
Auditory-vision sensory
substitution: seeing via the ears
 Meijer (1992) - The vOICe system
 Webcam, Headphones, Computing
Device
 software subsamples the image into a
an array of pixels
 The image is heard column by
column, panning from left to right
over 1 s
 Bright pixels sounding loud (i.e.
brightness to loudness mapping)
 Pixels high in the view being higher in
pitch
 A white ’/’ would be heard as a single
tone rising in pitch
 A white ‘X’ would be heard as two
simultaneous sound streams, one
ascending in pitch and one
descending

6. 
Tactile-vision substitution:
seeing via skin receptors
 Tactile-Visual sensory substitution
(TVSS) explorations of Bach-y-Rita and
colleagues
 The original device used an array of
tactile pins typically mounted on
participants’ backs
 A later version was based on electrical
stimulation of the tongue
 Participants used a hand-held or head-
mounted camera to explore the scene.
 The visual view of the camera was then
converted into an array of pixels, and
the brightness levels of the different
pixels were mapped to corresponding
levels of tactile stimulation by the
tactors in a tactile array.
 Although the tactile array was normally
placed on the back, within 5–15 h of
training the participants reported an
externalization of their sensations in
front of them, i.e. in front of the camera.

8. 
Tactile-vestibular sensory substitution:
balancing via skin receptors
 Rehabilitation of subjects with bilateral
vestibular loss (BVL) extremely difficult.
 System includes the following:
 A miniature accelerometer was mounted
on a low-mass plastic hard hat.
 Anterior-posterior and medial-lateral
angular displacement data were fed to a
previously developed tongue display
unit (TDU) that generates a patterned
stimulus on a electrotactile array held
against the superior, anterior surface of
the tongue.
 Subjects readily perceived both position
and motion of a small ‘target’ stimulus
on the tongue display, and interpreted
this information to make corrective
postural adjustments, causing the target
stimulus to become centred.

10. 
Evaluation
 Strengths
 Demonstrated the capacity
of the brain to adapt to
information relayed from an
artificial receptor via an
auditory or tactile HMI
 With training and with
motor control of the input
by the subject, percepts are
accurately identified and
spatially located
 Limitation
 Precise neural mechanisms
have not been identified

11. 
Implications
 Inexpensive implementations of
the technology to make it
accessible to a wide range of
patients suffering sensory loss
 It should be possible to use the
same technology to expand
human sensibilities, for example,
enabling the use of night vision
apparatus without interfering
with normal vision
 The technology enables a whole
range of noninvasive low-risk
experiments with human subjects
to gain a deeper understanding of
brain plasticity and cognitive
processes
Paul Bach-y-Rita