This study investigated how audio feedback influences tactile perception using a device that can vary friction between a finger and surface. In experiments, subjects explored the surface and rated perceived roughness with different audio and tactile conditions. Results showed that increasing high frequency audio led to higher roughness ratings both with and without sound from the finger. This suggests audio may bias touch through nonlinear transduction of signals or technical limitations. Further analysis is needed to understand the interaction between senses.

1. 
Abstract— We present a combined audio-tactile surface
haptic device, and utilize thisdevice to probe the effect of audio
bias on touch perception. The device itself utilizes an
electroadhesive effect to vary the amount of friction between a
fingerand the surface while sliding. While little is still known
about the full capabilities of this device, we hypothesized that,
due to its ability to render high frequency audio content, the
device could be used in a similar manner at the well-known
parchment skin illusion. With this in mind, we devised and
carried out an experiment with 6 different subjects to test their
perception of roughness with varying amounts of high
frequency content playing though their finger. We also altered
the amount of tactile range content being played though the
device, and whether or not subjects heard the varying audio
through theirfinger, or a constant audio level through a pair of
headphones. In all normal cases (audio heard though finger),
subjects reported increasing roughness for an increase in high
frequency audio gain. However, in the cases with headphones,
subjects still reported increases in roughness with increases in
high frequency audio gain. Furthertesting is needto determine
what cue subjects are using in the headphone cases.
INTRODUCTION
Illusions can tell us a lot about human perception. In the
case of haptics, one of the more well-known illusions is
called the parchment skin illusion[parchment ref]. The
illusion works by altering the high frequency audio content
that is heard when a subject is rubbing his or her hands
together. As the high frequency is boosted, subjects
spontaneously began reporting changes in their tactile
perception. As there were no actual changes going on with
the tactile inputs the subject was feeling, it is quite amazing
that the perceptual change is so apparent.
Indeed, it is these instances when our perceptual systems
fail, when they are tricked by manipulating different types of
sensory input, that give us the most flexibility in the design
and control of reproducing realistic experiences. In the field
of surface haptics this idea already has one example: the
haptic bump illusion [original nature paper]. In this case, a
subject’s kinesthetic cues are able to be overcome by lateral
force cues, and they are able to perceive a virtual bump or
hole while the surface they are feeling is perfectly flat. This
illusion has since been analyzed and controlled to give users
strong perceptions of shape and curvature on a 2D surface
[steven most recent paper].
It is with this design focus that we turned our attention to
manipulating and controlling sound output from a subject’s
finger as it is moved across the surface. We aim to expand
the current capabilities of surface haptic devices by adding on
a wide bandwidth audio channel that can play sound directly
through the finger. In addition, we aim to show that audio
played through this channel is able to modify tactile
perception, in a similar manner as the parchment skin
illusion.
I. BACKGROUND
A. Audio-Tactile Illusions
Multisensory interactions provide complementary
information of the environment. In active touch, auditory
senses like noise or tones may directly influence the
perception of tactile. For instance, tactile sensation of skin
roughness could be affected by sounds when subjects rub
their palms. If feedback sounds had lower intensity and high
frequency gains, subject reported a rougher surface of the
palm [1]. Based on the results, it is evident that auditory
sensation contributes to illusions in roughness perception.
Later on, a psychometric method was implemented in
experiments. Subjects were asked to discriminate roughness
of pairs of sandpapers with feedback sound manipulation.
Results showed that attenuating high frequencies without
modifying intensity led to a bias toward an increased smooth
perception [2]. However, the direction of the bias is
inconsistent to the results done by Jousmaki and Hari [1]
because composite responses which are rough/wet and
smooth/dry were used in Jousmaki and Hari [1] instead of
only roughness and smoothness so the chosen responses for
subjects had an effect on how human reported the results.
(note : they also did rubbing hand experiment to proof chosen
response did matter.) Instead of using direct feedback sounds
as auditory stimuli, white noise with different intensity was
presented when subjects explored fingers on 14 silicon
carbide abrasive papers with different particle sizes. When
auditory stimuli are white noise, the slope of perceived
roughness versus particle diameter in log scale is different
from the slope which auditory stimuli are 1 kHz beeps. It
indicated that auditory stimuli would affect sensation of
roughness even though the stimuli were irrelevant to tactile
[3]. Later on, to investigate how intensity of sound affected
tactile perception, the same process of psychophysical
experiments was conducted but with different intensity of
auditory stimuli. The weak-white noise resulted in smoother
sensation for both fine and coarse surface; the loud-white
noise increased roughersensation for coarse surface [4]. As a
results, audiotactile crossmodal sensation has been
investigated with different stimuli. In this paper, we attempt
to present audiotactile experiments with an innovative
variable friction surface. In our design, tactile and audio
stimuli are two independent variables. The method details are
provided in Section 3 and Section 4 describes the results of
how subjects rate their response to the sensation of texture. A
more detailed discussion is presented in Section 5.
Sound Biased Touch with a Combined Audio-tactile display
Craig Shultz Janeen Williams Cheng-Hsien Lin

2. B. Sound and Touch Neurological basis
Psychophysical experiments often indicate an exchange
between audio and tactile processing [3]. Results of
interaction between the two senses are also supported by
physiology. A 2005 study by Kayser et al showed that with
high resolution fMRI, the auditory cortex was activated with
both auditory and tactile stimuli [4]. Moreover, activation of
the auditory cortex was greater with coincident tactile and
auditory stimulation than with auditory stimulation alone.
Heightened activation was not seen in the somatosensory
cortex. This could be attributed to the smaller cortical regions
stimulated in this study,the hand and foot. Another potential
reason is to diminish the effect of redundancy on sensory
processing. Beyond the cortex, multisensory processing has
been shown at the level of the neuron. Fu et al showed in
nonhuman primates that single neurons within the auditory
cortex can be identified for activation at both tactile and
auditory stimuli [5]. These neurons produced distinct firing
patterns for auditory and tactile stimuli and was identified
with a receptive field on the back of the hand. In the same
study, the auditory cortex was shown to have biased tactile
representation of the skin surfaces of the hand and neck. A
study by Schurmann et al showed that pulsed-tactile and
vibrotactile stimuli activated different regions of the auditory
cortex in human subjects[6]. These findings in macaque
monkeys and humans are evidence to multimodal regions of
the auditory cortex. In particular, that posteromedial terminus
of the transverse temporal gyrus are a point of auditory and
somatosensory convergence in humans [7].
3 Methods
3.1 Methods
Fig 1 White noise signal with 0-1 volts is generated by
laptop and equalized. Signal is amplified to 0-10 volts by
line-amplifier and sent to High voltage amplifier which
drives the variable friction, electroadhesive surface. Subjects
freely explore with their finger-tips on the variable friction
surface.
An electroadhesive surface and setup similar to the one
used by Shultz et al. was used at the main surface haptic
device for these experiments. This devices works by
electroadhesively modulating the friction force on a human
finger as it is moved across the surface. This movement turns
the change in electroadhesive force into a larger change in
lateral force on the finger, thus producing vibrations sensed
by the subject. The signal used to drive the electroadhesive
surface was first generated as white noise via a laptop
computer. This white noise was then passed through a 9 band
audio equalizer, which was capable of boosting orattenuating
each frequency range by +12dB or -12dB respectively. After
shaping the white noise, the signal was then fed into a line
amplifier in order to calibrate the overall gain of the system.
From this amplifier, the signal was passed through a high
voltage capable, transconductace amplifier, where the input
voltage was converted to an output current, and a DC offset
was applied. The signal was then fed into the anodized
aluminum electroadhesive surface. The subject’s finger was
grounded,and a circuit was completed when they touched the
surface. The maximum current flowing through the finger
was approximately 100uA, roughly one order of magnitude
below the human electro cutaneous threshold. The system
was calibrated so that the highest gain to be applied in the
experiment would not saturate the capabilities of the high
voltage amplifier.
3.2 Rating scale and reasons behind it, adjectives we
chose (Janeen)
A literature review of similar studies showed that
perceptual changes might be identified as an escalating scale
of “roughness” [2, 3 4]. Each subject was presented with a
scale for each trial. They were asked to indicate their
perception on a Likert scale from 1 - 5.
3.3 Procedure - Set of Three separate experiments
(Janeen)
The experimental setup was designed to evaluate constant
tactile feedback with changes in audio feedback. Experiment
1 demonstrated constant, boosted or normal tactile feedback
with changes in audio feedback. Experiment 2 presented
constant, boosted or normal tactile feedback without audio

3. feedback. The tactile range was considered to be frequencies
between 0 and 1k Hz. Normal tactile feedback was delivered
at 0 dB, whereas boosted tactile feedback occurred at 12 dB.
Audio range feedback occurred at high frequencies between 2
kHz and 16 kHz. Each experiment involved 30 presentations
of the stimuli. Subjects were equally divided into boosted or
normal tactile feedback groups. The graph below
demonstrates the experimental setup.
Experimental Setup
Fixed
Normal Tactile
Feedback
Fixed
Boosted Tactile
Feedback
Experiment 1:
Changing Audio
Feedback with sound
- 12 dB
0 dB
+ 12 dB
- 12 dB
0 dB
+ 12 dB
Experiment 2:
Changing Audio
Feedback without
sound
- 12 dB
0 dB
+ 12 dB
- 12 dB
0 dB
+ 12 dB
Procedure followed
The experimental methodology was developed to reduce
the influence of external factors such as experimenter
suggestions, environmental distractions, and subject
investigation. Subjects were asked to face the stimulus in a
seated position. A foam apparatus visually blocked the
subject from the experimenters and experimental
equipment. To reduce subject curiosity and biasing, the
subject was provided a brief description of the experimental
procedure and apparatus. This was followed by a period of
free exploration with the pointer finger of the dominant hand
for 5 second intervals.At the conclusion of this presentation,
the experiments began.
Subjects were randomized to begin with or without
headphones. The subject was prompted to explore a stimuli
for 5 seconds. At the conclusion of Trial 1, the subject was
asked to indicate the perceived tactile roughness of the
stimulus on a Likert scale from 1 to 5. This process was
repeated for all 30 trials of the experiment. The second
experiment followed the same procedure. At the conclusion
of Experiments 1 and 2, the subject provided qualitative
feedback on their experience.
II. RESULTS
A. Quantitative
Results the experiments are shown in Fig 2 and Fig 3. In
Fig 2, we see the data from each subject, averaged across all
of their trials (n=10 for each data point). The error bars
represent +/- SEM. In general, we see that for every normal
case (that is, without the use of headphones and white noise),
subjects reliably reported an increase in perceived roughness
with an increase in high frequency audio gain. Additionally,
and unexpectedly, subjects also reported a similar increase in
tactile roughness with an increase in high frequency audio
gain when the noise coming fromtheir finger was masked out
by white noise in headphones.A few possible reasons forthis
are given later in this paper.
When the data is then averaged across subjects (n=3 each
point), what results is shown in Fig 3. Error bars represent +/-
SEM between subjects. Once averaged, we see some of the
same general trends that were present in the individual data.
For instance, in the tactile boosted case, users seemed to be
more confused by the stimuli during the trials with
headphones on. In particular, subjects 1 and 5 did not record
monotonically increasing responses.Additionally in this case,
the smoothest stimuli appears to feel rougher, and the
roughest stimuli appears to feel smoother, indicating that
subjects may not have been able to distinguish between the
different audio gain levels as well as they could without
headphones on.
Looking at the tactile attenuation cases, we see little to no
differences in the responses of the subjects between the
normal and headphones conditions. There is only a slight
increase in overall roughness reported.Indeed,subjects 3 and
4 performed almost exactly the same in each case.
Brief interviews on subject experience after the
experience provided a large range of results. Some subjects
reported a sense of learning; a perception that their
performance improved in subsequent trials. Other subjects
did not feel they could distinguish between the stimuli,
although the data suggests this is not the case. In particular,
one subject’s account was that the experience was
challenging and choices on roughness were made as guesse.
III. DISCUSSION
Analysis of the quantitative data shows that, even when
subjects wore headphones, which were intened to block all
sound cues coming from the finger, they still were able to
discern between the -12/0/12dB high frequency audio cases.
This fact could have a variety of explanations behind it.
First and foremost, the method of transduction from the
current controlled signal going into the electroadheasive
surface could introduce nonlinearities into the force signal
actually rendered to the finger. These nonlinearities could
rectify some of the high frequency components of the current
signal down into the tactile range. This rectification could
occur electrically or mechanically. If this were the case, then
the assumption of constant tactile stimulation between the -
12/0/12dB high frequency audio gain stimuli would be
invalid. More work is needed to understand this transduction
mechanism more clearly, and more measurements could be

4. made to analyze and quantify the actual mechanical signal
produced at the finger (i.e. the frequency content of the
vibrations of the skin).
Another area that may present error is in the signal
equalization and amplification itself. Careful steps were taken
in order to minimize any distortion that might occur by signal
clipping, but it is still possible that one or more of the
components in Fig 1 were also introducing nonlinearities,
noise, or generally not acting as expected.
For example, the frequency width of the equalization
bands on the equalizer used were not exactly specified, and it
is possible they are wide enough as to boost or attenuate
frequencies in the tactile range when the lowest band (4kHz
band)was adjusted.If this effect was happening, it could first
be measured by a careful frequency analysis of the output
current as high frequency audio bands are boosted or
attenuated. If confirmed, this could be mitigated in the future
by using different equalization equipment. Subject 6,
however, was run with a different equalization setup, and still
showed similar trends as all of the others.
Additionally, the high voltage amplifier used can, in some
cases, (typically for very low control currents) introduced
60Hz noise that is coupled in from mains power. This noise
term could have in turn influenced participants’ rating of
roughness, especially if it is affected at all by additional high
frequency content.
Separate from technical limitations of the setup, subjects
ratings of roughness may have been influenced by the
volume level of the white noise played through their
headphones. As shown by Suzuki et al. [3],[4], white noise
can, by itself, either increase or decrease roughness
depending on the volume of the white noise, and what tactile
stimuli it is paired with. This may explain why, in the
attenuated tactile case, the overall roughness seemed to
slightly increase.
Finally, the most unlikely reason subjects would be able
to still discern roughness with headphones on would be if
they are still somehow sensing the frequency high frequency
content directly (as opposed to some sort of rectification to
low frequency tactile stimulation). This, however, would
require some sort of separate sensing mechanism (as the
mechanoreceptors in the skin are not responsive above
approx 1kHz) or it would require a separate pathway to the
cochlea (perhaps through skin or ligament conduction).
ACKNOWLEDGMENT
The preferred spelling of the word “acknowledgment” in
America is without an “e” after the “g”. Avoid the stilted
expression, “One of us (R. B. G.) thanks . . .” Instead, try
“R. B. G. thanks”. Put sponsor acknowledgments in the
unnumbered footnote on the first page.
REFERENCES
[1] Jousmäki, V. & Hari, R. Parchment-skinillusion: sound-biasedtouch.
Curr. Biol. 8, R190 (1998).
[2] Guest, S., Catmur, C., Lloyd, D. & Spence, C. Audiotactile
interactions in roughness perception. Exp. Brain Res. 146, 161–171
(2002).
[3] Suzuki, Y., Gyoba, J. & Sakamoto, S. Selective effects of auditory
stimuli on tactile roughness perception. Brain Res. 1242, 87–94
(2008).
[4] Suzuki, Y. & Gyoba, J. Effects of sounds on tactileroughness depend
on the congruency between modalities. Proc. - 3rd Jt. EuroHaptics
Conf. Symp. HapticInterfaces Virtual Environ. Teleoperator Syst.
World Haptics 2009 150–153 (2009).
[5] Kayser, C., Petkov, C. I., Augath, M. & Logothetis, N. K. Integration
of touch andsoundin auditory cortex. Neuron 48, 373–384 (2005).
[6] Schürmann,M., Caetano, G., Hlushchuk,Y., Jousmäki, V. & Hari, R.
Touch activates humanauditorycortex.Neuroimage 30, 1325–1331
(2006).
[7] Hackett, T. a. et al. Sources of somatosensoryinput tothe caudal belt
areas of auditory cortex. Perception 36, 1419–1430 (2007).
[8] Fu, K.-M. G. et al. Auditory cortical neurons respond to
somatosensory stimulation. J. Neurosci. 23, 7510–7515 (2003).
[9] Foxe, J. J. Multisensory Integration: Frequency Tuning of Audio-
Tactile Integration. Curr. Biol. 19, R373–R375 (2009).
[10] Shultz C., Peshkin M., & Colgate E. “Surface Haptics via
Electroadhesion: Expanding Electrovibration with Johnsen and
Rahbek,” submitted for publication.
[11] Manual S. et al. “Coincidence avoidance principle in surface haptic
interpretation,” PNAS 2015 112 (8) 2605-2610
[12] Robles-De-La-Torre G. & HaywardV. "Force Can Overcome Object
Geometry In the perception of Shape Through Active
Touch". Nature (2001) 412 (6845): 445–8.
TABLE TYPE STYLES
Table
Head
Table Column Head
Table column subhead Subhead Subhead
copy More table copya
a. Sample of a Table footnote. (Table footnote)