Synaesthsia

Synaesthesia Course: Sensors in Mobile Systems
Edinburgh Napier University Matric: 40099589 Page 1
Student Name : Calum James Beck
Matriculation Number : 40099589
Edinburgh Napier University
Title : Synaesthesia
OVERVIEW/ABSTRACT
Synaesthesia is a system created to facilitate the traversal of small journeys for those of hard of
sight to work in conjunction with a White Cane or Seeing Eye Dog. This system should allow
users to avoid obstacles of waist height and alert them to moving obstacles within a distance of
3m.
The system works by reading values from three ultrasonic sensors connected to an Arduino
board and sends the values to an Android device to interpret the distance and play
corresponding tones of varying speed and volume. This is then sent to a Bluetooth headset to
allow for stereoscopic sound to alert the user as to where the objects are approximately
located.
This project endeavours to deliver a reliable solution to these objectives in a portable
environment. A physical circuit incorporating an embedded system and an Android app are
part of this solution. These are described herein.

TABLE OF CONTENTS
TABLE OF FIGURES.............................................................................................................................................3
INTRODUCTION....................................................................................................................................................4
BACKGROUND/ CONTEXT ..................................................................................................................................4
AIMS & DELIVERABLES ......................................................................................................................................5
CHAPTER 1.........................................................................................................................................................6
DESIGN DELIBERATIONS AND COMPONENT ANALYSIS.......................................................................................6
1 INTRODUCTION.............................................................................................................................................6
1.1 COMPONENT COMPARISONS ...................................................................................................................6
1.1.1 Distance Detection: Infrared versus Ultrasonic..............................................................................6
1.1.2 Object notification: Servo Motors Versus Audio ............................................................................7
1.1.3 Wireless Communication: Bluetooth ..............................................................................................7
CHAPTER 2.........................................................................................................................................................8
DESIGN OF THE SYNAESTHESIA.........................................................................................................................8
2 INTRODUCTION.............................................................................................................................................8
2.1 THE SYSTEM COMPONENTS AND THEIR FUNCTIONS.................................................................................8
CHAPTER 3.........................................................................................................................................................9
IMPLEMENTATION OF THE PROJECT....................................................................................................................9
3 INTRODUCTION.............................................................................................................................................9
3.1 SOFTWARE INTERFACE .........................................................................................................................13
3.1.1 Code Listing..................................................................................................................................15
3.1.1.1 Arduino Code........................................................................................................................... 15
3.1.1.2 Android Code ........................................................................................................................... 15
3.2 INTERFACE TEST PLAN..........................................................................................................................17
RESULTS.......................................................................................................................................................... 25
CONCLUSIONS AND FUTURE WORK................................................................................................................ 25
EVALUATION OF ACHIEVEMENT....................................................................................................................... 26
REFERENCES................................................................................................................................................... 27
APPENDICES.................................................................................................................................................... 28
APPENDIX A- INITIAL PROJECT PROPOSAL.....................................................................................................28
BACKGROUND AND RATIONAL......................................................................................................................... 29
DELIVERABLES................................................................................................................................................. 29

TABLE OF FIGURES
FIGURE 1 – CONCEPTUAL ILLUSTRATION OF IDEA.............................................................................................4
FIGURE 2 - SIMPLIFIED COMPARISON OF EMISSION BETWEEN IR AND ULTRASONIC........................................7
FIGURE 1 - THE HARDWARE INTERFACE BLOCK SCHEMATIC..........................................................................8
FIGURE 2 - INITIAL TESTING ENVIRONMENT........................................................................................................9
FIGURE 5 - WIRING DIAGRAM OF ARDUINO BOARD AND BREADBOARD........................................................ 10
FIGURE 6 - FINAL WIRING IMPLEMENTATION .................................................................................................. 11
FIGURE 7 - SYSTEM IN PROTECTIVE BOX........................................................................................................ 12
FIGURE 8 - FULLY PORTABLE SYSTEM .......................................................................................................... 12
FIGURE 9 - DEBUGGING AND FINAL APPLICATIONS ..................................................................................13-14
FIGURE 10 – HAND GUIDANCE SYSTEM IDEA ................................................................................................ 27

INTRODUCTION
BACKGROUND/ CONTEXT
In everyday commuting, it is easy to take our sight for granted when getting from A to B. With busy town centres in
particular, there are always obstacles we move around without any thought and moving in and out of crowds is reliant
on intuition rather than contemplating each move. For those hard of sight, this is not as simple.
‘Synaesthesia’ aims to help those who find themselves in this situation without being too bulky or intrusive to everyday
travel. It is designed in order to help those in this situation and not to completely combat the problem, hence, it should
be used in conjunction with a White cane but should allow pre-emptive movement by indicating obstacles within a 3m
radius.
Figure 3 - Conceptual illustration of idea.
The system should allow those of hard sight to hear where objects are at waist height whilst still being vigilant of that
on the ground thanks to the White cane. This system is also good for those with such conditions such as macular
degeneration where the sight comes in pin hole views and it is hard for the mind to interpret all objects seen;
‘Synaesthesia’ provides a digital horizontal field of vision to aid those in this circumstance.

AIMS & DELIVERABLES
This project aims to build and document a system which will allow the user to perceive objects via sound in a
stereoscopic environment from multi-directional origins in order to allow the user to prioritise direction and paths based
on sound alerts. The main project will focus on building an environment to cater the conception, implementing a
system to facilitate collecting the necessary data and a practical solution for processing this data.
As such, the final product will feature the appropriate equipment to enable implementation and will allow users to
traverse paths and obstacles without the use of sight. It will feature a system to allow stereo sound to be manipulated
to show proof of concept that an audio based system can allow those hard of sight to commute with a higher degree of
ease, to create a feeling of sight through sound; Synaesthesia.

CHAPTER 1
DESIGN DELIBERATIONS AND
COMPONENT ANALYSIS
1 INTRODUCTION
In elaborating on the main concept behind the project, key areas were targeted and highlighted in order to create an
adequate system fit for purpose. The driving forces behind this were data collection for distance, communication between
devices and physical alerts for the user. All of this had to be facilitated in a portable environment as well so no wired
instruments could be used independently without a method of wireless communication.
Hence, research was needed for a point of reference to compare and contrast the growing idea. On researching the subject
of digital sight aids, other projects similar to the concept of this project. With similar traits, a project revolving around the
white cane was created, allowing those with hard of sight to use their walking stick (white cane) to extend their range. With
sensors fitted to the shaft and motors vibrating in different places, the user could feel the direction that obstacles lay. Dr. G.
Prasanthi (2015) theorised that this would allow the user to interact with their surroundings with the least impact to their daily
lives. From the standpoint of this project however, this approach would make the white cane much more bulky and would
require modifiying users personal belongings [1]. This was felt to be a good idea, but outwith the scope of the project. It is
important at this point to mention that the scope of the this project was to incorporate the use of sensors in mobile systems
and to apply these to an idea rather than take existing objects and modify them..
In search for visual impairment aids there were many other applications that used a camera on a phone to track lines and
help the user navigate a indoors where there is no GPS [2] [3]. This however uses advanced tracking and did not allow for a
hands free experience seeing as one hand would be occupied by a White cane. Another project fitted ultrasonic sensors to a
cap and projected from the head. This was good, but the range would surely be limited pointing at the ground. [4]
At this point, it was concluded that the system should be used around waist height using a proximity sensor with either
vibration of sound as an indication of distance. This could all be inclusive as a standalone system or incorporate a larger
processing power on the premise it would all still be portable.
1.1 COMPONENT COMPARISONS
1.1.1 DISTANCE DETECTION: INFRARED VERSUS ULTRASONIC
Both Infrared (IR) and Ultrasonic sensor are readily available for the build environment being used and are well documented
on how to implement. When choosing to build a system which is designed to help those wishing to traverse obstacles relying
on data procured by the device, then the accuracy of the device is incredibly important. Both Infrared (IR) and Ultrasonic
sensors are readily available for the build environment being used so a comparison was made between both in order to
attain the best option for this project.
Mustpha, Zayegh and Begg (2013) conducted a study into the uses of IR and ultrasound in an project with similar aim to this
where the distance detection tools were interchanged to ascertain the ramifications of using each. Of the main points
brought up, in order to use IR efficiently “knowledge of the surface properties must be known beforehand. In other words,
the nature in which a surface reflects and absorbs infrared energy is needed to interpret the sensor output as distance
measure” . [5] This restricts the ability to use the system in everyday circumstances but it should be mentioned that
Ultrasonic is noted in the same article as being affected by mirrored surfaces as well as weather elements.

Despite Ultrasonic waves being affected by humidity and wind however, IR is severely hampered by harsh sunlight and can
only be used at dusk and night outdoors. However, the preceding factor was the scope of the beams. Infrared only returns
object in a precise direction whilst Ultrasonic spreads as it travels capturing a larger area.
Figure 4 - Simplified comparison of Emission between IR and Ultrasonic
1.1.2 OBJECT NOTIFICATION: SERVO MOTORS VERSUS AUDIO
In order for the user to receive feedback, a method of converting the digital data into physical feedback was needed. Two
choices were narrowed down to a vibration effect against sound. Using servos means only having to build a small
embedded system with no outside help whilst Audio would require a higher processing power. At this point, stereo was the
more favourable choice for the developer purely to further their knowledge and build an android application to handle sound
manipulation and audio streaming.
1.1.3 WIRELESS COMMUNICATION: BLUETOOTH
Using Bluetooth meant an easy connection procedure for the user and a large amount of readily available headsets to take
advantage of. Connecting using a node would be simple and Bluetooth does not use heavy encryption which is ideal for
devices such as Arduino which have low memory capacities and processing power. This keeps processing overheads down
resulting in less power consumption. As well as this, transferring sequential data from Ultrasonic sensors would be idea for
Bluetooth, with the developer not expecting data transfer to exceed 1Mb/s.
From these conclusions, a basic design was created using ultrasonic sensors as the data source, a Bluetooth module as
communication and an Android app to facilitate audio generation via a Bluetooth headset for the user.

CHAPTER 2
DESIGN OF THE SYNAESTHESIA
2 INTRODUCTION
As previously discussed, the components selection was crucial to a working system. The component chosen are
displayed in the block diagram below with the addition of an Arduino Uno board to connect needed sensors.
Figure 5 - The Hardware Interface Block Schematic
2.1 THE SYSTEM COMPONENTS AND THEIR FUNCTIONS
The idea was for the Ultrasonic sensors to collect data and pass it to the Arduino board which would convert the data to a
ASCII value and broadcasts a serialised stream of the data using a Bluetooth (BT) Module. This part of the system would all
be powered using a 12v battery.
The broadcast is then picked up by the Bluetooth receiver in the Android device. This would then in turn, create or play a
sound that would correspond to the values the Ultrasonic sensors are producing (i.e. in relation to the distance). This sound
would then be played through the BT headset and, depending on which sensor sent the data, played through the correct
stereo channel.
With this design, Synaesthesia conforms to the constraints placed by the conceptual design and was very close to the
solution initially outlined in the Initial Project Proposal (documented as appendix C).

CHAPTER 3
IMPLEMENTATION OF THE PROJECT
3 INTRODUCTION
The main components driving this system are the Arduino Uno embedded code and the Android device receiving the data.
This meant a supplementary application was needed to process the data coming in. This started out only using two
Ultrasonic sensors and the Bluetooth module.
Figure 4 - Initial Testing Environment
In the final environment, each ultrasonic sensor is connected to a ‘tiny breadboard’ to access 5v power pin and connect to
the GND pin. They are then directly connected to pin on the Arduino board above the first two pins (pins 0 and 1). The
Bluetooth module sits on the breadboard and is connected to the 3v pin and the other GND. This can be seen in the
diagram below.

Figure 5 - Wiring Diagram of Arduino Board and Breadboard
The way the system works is as so: The data read in from the ultrasonic sensors and the Id of that sensor is added to the
value (i.e. if the sensor has an Id value of 1, then a reading of 1cm is changed to 11). This is then sent over Bluetooth
where the app is looking for a BT signal and establishes a connection with only specific devices corresponding to name of
the BT module.
From here the app attempts to create an input stream and takes the current byte packets by Arduino and converts it into a
string value. An if else statement is then run checking the value of the of the string. It then picks the first digit of the string
and plays the corresponding tone depending on the distance from the Ultrasonic sensor. The initial number read
corresponds to the channel output for the sound (e.g. left = left channel). The sound is then streamed to a Bluetooth headset
to be heard by the user.

Figure 6 - Final Wiring implementation
Once the System was complete, it was compacted into a small box to protect components and then mounted in a portable
bag for the user to carry it in. In order to avoid colliding waves from the sensors, they were mounted at angles of 20 degrees
away from each other, the middle pointing forward.
Figure 7 - System in protective box

Figure 8 - Fully Portable System

3.1 SOFTWARE INTERFACE
The main software composes of an Android App that loads data from the Arduino board. Throughout
development, a development app was used in conjunction with the Logcat console to debug the code as the
project progressed.

Figure 9 - Debugging and Final Applications
The buttons correspond to Opening a connection to Bluetooth, Sending a signal to the Arduino Serial console, closing the
Bluetooth connection and, in the debugger, clearing the white space below. The white space was a Text field which updated
with the serial data being streamed. The ‘Serial Data’ label in the corner also displayed incoming data in both versions.

3.1.1 CODE LISTING
Full Code listings can be obtained on request. Some has been left in order to convey some aspects of how the system
worked overall.
3.1.1.1 ARDUINO CODE
Pin setup example
//Sonar 1
int echoPin1 =11;
int initPin1 =12;
int distance1 =0;
Setup code
void setup()
{
pinMode(initPin1, OUTPUT);
pinMode(echoPin1, INPUT);
/*
*/
//Setup usb serialconnection to computer
Serial.begin(9600);
// initialize all the readings to 0:
for (int thisReading = 0; thisReading < numReadings; thisReading++)
readings[thisReading] = 0;
//Setup Bluetooth serialconnection to android
bluetooth.begin(115200);
bluetooth.print("$$$");
delay(100);
bluetooth.println("U,9600,N");
bluetooth.begin(9600);
}
Get Distance function
int getDistance (int initPin, int echoPin){
digitalWrite(initPin, HIGH);
delayMicroseconds(10);
digitalWrite(initPin, LOW);
unsigned long pulseTime = pulseIn(echoPin, HIGH);
int distance = pulseTime/58;
return distance;
}
3.1.1.2 ANDROID CODE

Setting up SoundPools for each stream
//Set up streams for each feed
sp = new SoundPool(1, AudioManager.STREAM_MUSIC, 0);
sp2 = new SoundPool(1, AudioManager.STREAM_MUSIC, 0);
sp3 = new SoundPool(1, AudioManager.STREAM_MUSIC, 0);
//Add tones to Array forretrieval
spMap = new SparseArray<Integer>();
//left ear/Ultrasonic sensor
spMap.put(closest, sp.load(this, R.raw.closest, 1));
spMap.put(closer, sp.load(this, R.raw.closer, 1));
spMap.put(close, sp.load(this, R.raw.close, 1));
spMap.put(aw ay, sp.load(this,R.raw.away, 1));
spMap.put(far_away, sp.load(this, R.raw.far_away, 1));
//middle ear/Ultrasonic sensor
spMap.put(closest_quiet, sp2.load(this, R.raw.closest_quiet, 1));
Listener for data stream
w orkerThread =new Thread(new Runnable()
{
public void run()
{
w hile(!Thread.currentThread().isInterrupted() && !stopWorker) try {
int bytesAvailable = mmInputStream.available();
if (bytesAvailable >0) {
byte[] packetBytes = new byte[bytesAvailable];
//Read packets
mmInputStream.read(packetBytes);
for (int i = 0; i < bytesAvailable; i++) {
byte b = packetBytes[i];
if (b == delimiter) {
byte[] encodedBytes =new byte[readBufferPosition];
System.arraycopy(readBuffer, 0, encodedBytes, 0, encodedBytes.length);
//Data received saved as a string
finalString data = new String(encodedBytes, "US-ASCII");
//String converted to an integer value
finalint numData = Integer.parseInt(data.replaceAll("[D]",""));
Play sound function
public void playSoundLeft(int playTone) {
//Left Ultra
//Sets playtone so same value is not played tw ice
currToneLeft =playTone;
//Clears soundpool so new tone can play
clear(sp);
/*Gets value on method call as sets as playtone w hich corresponds
to value in spMap, left stream value (1 is full volume, 0 is off),
right stream value, priority value, repeat value, frequency of play
*/
sp.play(spMap.get(playTone), 1, 0, 1, 0, 1f);
}

3.2 INTERFACE TEST PLAN
A test driven development cycle was adopted throughout the whole project as it was classified as experimental to the
developer, not having worked in a multi-platform environment before. Here is a log of the methodical process taken to
develop the system.
Test 1 : Get distance from an Ultrasonic sensor and
print in Serial Monitor in Arduino Console
Result:
(Pass/Fail)
Purpose
To understand the values from the sensors in order to use them in later development and test
wiring was correct
Method
Use supplied example code from Arduino software to ensure the values of the sensors is coming
through.
Steps include setting up Trigger & Echo pins and pinging device
Project Files Used : Test Filename(s) : ultrasonic_sensor.ino
Expected Result
Output in Serial Monitor to reflect proximity to sensor
Actual Result
Expected
Pass
Date of Test:
10/03/2015

Test 2 : Get distance from many Ultrasonic sensors and
print in Serial Monitor in Arduino Console
Result:
(Pass/Fail)
Purpose
Ascertain the possibility of using multiple Ultrasonic sensors in the one system.
Method
Adapt supplied example code from Arduino software to allow more than one input source
Steps include adding another set of Trigger & Echo pins and pinging devices
Project Files Used : Test Filename(s) : mult-ultrasound.ino
Expected Result
Output in Serial Monitor to reflect proximity on multiple sensors
Actual Result
Expected
Pass
Date of Test:
10/03/2015
Test 3 : Create method for feedback from different
Sensors
Result:
(Pass/Fail)
Purpose
Associate sound with proximity to a sensor
Method
Add piezo sound effects to readings less than 10 cm
Steps include adding piezo device to board and pin definitions as well as implementing if statement
Project Files Used : Test Filename(s) : mult-ultrasound.ino
Expected Result
When output in Serial Monitor drops below 10cm, reflect proximity using sound effect
Actual Result
Expected
Pass
Date of Test:
10/03/2015

Test 4.1 : Connect bt module to ‘tiny breadboard’ Result:
(Pass/Fail)
Purpose
Wire BT module to board with Ultrasonic sensors connected
Method
Add BT module to board and show power. Connected to pin 0
Expected Result
BT module is auto configured so should show up on other devices scan
Actual Result
Error: avrdude: stk500_getsync(): not in sync: resp=0x00
Cannot upload code to Arduino
Fail
Date of Test:
12/03/2015
Test 4.2 : Connect bt module to ‘tiny breadboard’ Result:
(Pass/Fail)
Purpose
Wire BT module to board with Ultrasonic sensors connected
Method
Add BT module to board and show power. Changed pin connections
Expected Result
BT module is auto configured so should show up on other devices scan
Actual Result
Expected
Pass
Date of Test:
12/03/2015

Test 5 Use Bluetooth Module to connect to Bluetooth
Headset
Result:
(Pass/Fail)
Purpose
Start a connection Headset to send rudimentary sounds
Method
Pair Bluetooth module with headset and send system beep to headset
Project Files Used : Test Filename(s) : btConnect.ino
Expected Result
Sound played
Actual Result
Could not send sound using serial bluetooth module
Fail
Date of Test:
20/03/2015
Test 6 : Communicate via Bluetooth module to Android
device and vice versa
Result:
(Pass/Fail)
Purpose
Start a connection with Android device to ensure data can be transferred by sending data from
Arduino monitor.
Method
Send a message from Arduino to Android and vice versa
Project Files Used : Test Filename(s) : btConnect.java
BT_and_Ultra.ino
Expected Result
‘Hello’ displayed in text field and in serial monitor
Actual Result
Expected
Pass
Date of Test:
22/03/2015

Test 7 : Stream Serial data from UltraSonic sensors to
Android Device
Result:
(Pass/Fail)
Purpose
Assure that sending multiple values is possible from
Method
Print to Bluetooth stream rather than Serial monitor
BT_and_Ultra.ino
Expected Result
Stream of numbers appearing on Android text field
Actual Result
Expected
Pass
Date of Test:
22/03/2015
Test 8.1 : Trigger Sound on reading change in Android
using AudioManager
Result:
(Pass/Fail)
Purpose
Play sound based on data from Ultrasonic sensors.
Method
Use if statement to play different method depending on data in
BT_and_Ultra.ino
Expected Result
Sound played changes depending on data in from sensors
Actual Result
Sound repeatedly plays closest association sound
Fail
Date of Test:
02/04/2015

Test 8.2 : Re attempt Trigger Sound on reading change
in Android using Soundpool
Result:
(Pass/Fail)
Purpose
Play sound based on data from Ultrasonic sensors.
Method
Same as before but preload sounds into Soundpool instance
BT_and_Ultra.ino
Expected Result
Sound played changes depending on data from sensors
Actual Result
Sound played once for each range identified
Note: Changed array to Sparse array for faster read times
Pass
Date of Test:
02/04/2015
Test 9 : Implement audio structure to read from Array,
loop sounds and manipulate frequency and volume to
create constant alert
Result:
(Pass/Fail)
Purpose
Change sound depending on distance
Method
Used sound pool parameters to gain effect
Project Files Used : Test Filename(s) : ‘Arduino_App’
Main_Activity.java
BT_and_Ultra.ino
Expected Result
Sound played changes depending on data in from sensors and manipulated used Android
processing
Actual Result
Sound clashed and loaded to slowly whilst repeatedly playing over each other
Fail
Date of Test:
05/04/2015

Test 10 : Stop audio stream collisions in looping
sounds by pausing audio stream
Result:
(Pass/Fail)
Purpose
Use AudioManager to control Soundpool and pause when needed
Main_Activity.java
BT_and_Ultra.ino
Expected Result
Sound played changes depending on data in from sensors and manipulated used Android
processing and would pause to allow change
Actual Result
Sound clashed less than before but data was still lost when streaming to Bluetooth headset
Pass
Date of Test:
10/04/2015
Test 11: Increase delay on Arduino board when reading
Ultrasonic sensors to allow Android to keep up with
changing events
Result:
(Pass/Fail)
Purpose
Loops were loading logic to slow to cope with data coming in
Method
Use messages in logcat to ascertain the frequency at which the audio streams were being played
as audiostreams were different from audio id
Expected Result
The value of streams would increment once per loop.
Actual Result
Expected
Pass
Date of Test:
10/04/2015
Test 12 : Manipulate .ogg files and read data incoming
sequentially to reduce collisions and play sounds at
different intervals for easier directional recognition
Result:
(Pass/Fail)
Purpose
To speed up the process, do not use Android function to change the tone, pitch and frequency, but
load in pre-set sounds to speed up overall system
Method
A program called ‘Audacity’ was used to engineer tones of different volume, tone and frequency
Pass
Date of Test:
17/04/2015

Test 13 : Play two sounds at once using Soundpool Result:
(Pass/Fail)
Purpose
Play all different tones through head set to create stereo sound depending on input
Method
Used sound pool parameters, change the channel depending on input and change clip which
played depending on distance
Main_Activity.java
BT_and_Ultra.ino
Expected Result
Sound played changes depending on data in from sensors and is played through correct ear piece
Actual Result
Sounds came out in correct ear and correct files were played but frequency was off and volume
needs calibration.
Pass
Date of Test:
18/04/2015
Iterative Test : User Testing Result:
(Pass/Fail)
Purpose
To ascertain whether the system is fit for purpose
Method
Use system in everyday use
Expected Result
Stereo sound would yield an intuitive solution to object awareness
Actual Result
Sound did help, but was deemed slightly confusing.
Fail
Date of Start:
10/04/2015
Date of
Completion:
24/04/2015

RESULTS
From the results of implementation and iterations through the design, the concept is sound as far as the idea goes. Using
ultrasonic sensors to gauge distance is a viable solution for those making small systems in conjunction with Arduino. The
main implementation of the physical components was also easily understandable with a basic knowledge in electrical
circuitry and pin handling.
Streaming serialised data from digital sources over Bluetooth works well and is efficiently fast, with precautions taken in
order to manage the speed the data is read in. It was found during the late testing stages that (if incorrectly logically driven),
the sound files saved on the android would rapidly try to replay with the data coming in twice as fast at the loop could
handle.
Once understood though, it can be concluded that the audio features built into Android are fit to play multiple tones at once
through different channels but more research is needed on the subject. With a more in depth development of the audio files,
a system could be built providing a more intuitive reaction to obstacles.
CONCLUSIONS AND FUTURE WORK
After many amendments and tests, it was concluded that although the structure technically works and demonstrates the
possibility of a successful system, it was unfortunately not fit for purpose. The ultrasonic sensors worked reliably in short
distances and the sound cues are understandable when the user expects the result produced, but using the system in the
initial practical sense it was designed for is not intuitive enough for the general user.
This stems from the sound cues and the multiple streams being delivered at once. This was the initial idea of the system (a
stereo feed leading to a depth of sound) but in testing, it was discovered that directly using 3 sources of sound was not
sufficient enough. It only creates a pseudo-surround sound and is very much a stereo feed in the most literal sense. This
was the initial goal of the project but humans learn to sense sound in a fully 3-D environment so hearing sounds from only 3
sources is slightly confusing. If more ultrasonic sensors were used (reaching around the back of the user as well as the
front), a better effect could be realised, but significant audio manipulation would have to take place dynamically, fading
sounds into one another to reach the desired effect. In the initial concept of the system (see Appendix C), an additional
Android device was not considered but after implementing the system, it would be favourable for a much longer Android
development window.
As stated, the current system is very good for short distances and on a much smaller scale than that of the outside world. As
such, using an ultrasonic sensor to aid picking up objects or navigating small rooms could prove much more successful.
One idea would be to limit the amount of Ultrasonic sensors to one and use it as a hand navigator and may be good for
those who adjusting to sight loss or in recovery of sight.
Figure 10 - Hand Guidance system idea
As stated, using one sound is much easier to understand as well as implement and debug. This idea could also be applied
to those who use their phones a lot whilst walking.

EVALUATION OF ACHIEVEMENT
As far as this project goes, there are positive aspects and negative; the final product being of the latter. However, in terms of
learning, this project has yielded many good results. I believe that I underestimated the amount of work that would go into
software development leading to a less than perfect outcome in my eyes. Making an Android application to work in
conjunction with the system was never in the original scope but was a great learning experience nonetheless.
Building the systems physical components was simple enough and fairly straight forward. It was an enjoyable and
immensely rewarding experience for myself when components worked and communicated with each other, and certainly
has intrigued me in terms of future development. This was ultimately a bad aspect to the project however as it created a
false sense of security in terms of the difficulty of the implementation of the Android side. In hindsight, the application is not
difficult to develop however, there are certainly components that just need to be known in order to progress (such as
Bluetooth methods and audio streaming) and information found on sound producing methods for Android were scarce and,
at times, conflicting. I have attempted a Bluetooth App in the past and did not succeed to have it working so I was very
happy when saw the fruition of my work over the past couple of years, This all contributed the learning experience and have
been beneficial for future endeavours.
Although it was forewarned when entering this project that time management would be critical, I believe I underestimated
the overall implementation time. I would also change the final product next time into a soldered version and if it is a system I
wish to use for myself and not just for proof of concept, I would invest in a smaller board, Having the wireless breadboard
was great for development but is not as stable as soldered components.
Ultimately, I am very happy with what I learned and the direction the project took towards the end. I would say to get the
product I had envisioned, a lot more time would have to be applied over months to give it a professional feel. I believe I
attempted too much for a first project with a lack of understanding of the work involved. Despite this, I am glad I went ahead
with the project and am happy with the outcome despite it not creating a wave of synesthetes.

REFERENCES
[5] A. Z. R. K. B. Baharuddin Mustapha, First International Conference on Artificial Intelligence, Modelling & Simulation,
2013. [Online]. Available: http://ijssst.info/Vol-15/No-2/data/3251a439.pdf. [Accessed 4 April 2015].
[3] arXiv, “MIT Technology Review,” 8 January 2014. [Online]. Available:
http://www.technologyreview.com/view/523401/app-turns-smartphone-into-virtual-cane-for-the-blind/. [Accessed 25
March 2015].
[1] P. Dr G. Prasanthi, “Sensor assisted stick for the blind people,” Transactions of Engineering and Sciences, 2015.
[4] Reuters - Innovations Video Online, “Wearable sensor 'illuminates' surroundings for the blind,” 10 December 2014.
[Online]. Available: http://www.sciencedaily.com/videos/40068be4cc3ee850f30614d96c7431b8.htm.
[2] S. H. S. M. Lisa Ran, “DrishtiL An integrated indoor/Outdoor Blind Navigation System Service,” Computer & Information
Science & Engineering Department University of Florida, Gainesville, 2005.

{ new page}
APPENDICES
APPENDIX A - INITIAL PROJECT PROPOSAL
An Initial Project Overview of a system which helps
people with sight problemspreemptivelyidentify risks
in their path through the use for Stereo sound

BACKGROUND AND RATIONAL
Whilst working with Ultrasonic sensors, I started to think of the ramifications of using such a technology to help those that
did not have the innate ability to perceive distance. Whilst my grandfather was in his later years, he suffered from Macular
Degeneration leading to a loss of sight and visual awareness. At the time, I had thought to create a device that could help
him commute with more confidence but in a discrete manner.
With this in mind, I aim to create a system of Ultra sonic sensors connected to an Arduino board using a 9v battery to power
the system; to allow for portable usage. The aim will be to connect to a Bluetooth head set to the system and use the
sensors to gauge distance and create stereo sounds to alert those with sight difficulties how far an object lies ahead in their
path.
In researching the project, it would appear that I not the first to attempt such a system. Whilst searching I found a system
with the same idea at heart. Instead of having sensors on the person, this group decided to modify a walking stick to vibrate
and move in the direction needed (Dr G. Prasanthi, 2015). The problem I could see with this is that it was too bulky and
apparent. Other ideas included a belt, a similar system with audible sounds created by a speaker system (Reuters -
Innovations Video Online, 2014) and a system that recorded GPS signals to relay voice commands through a headset (Lisa
Ran, 2005).
The system I intend to create attempts to make the aid discreet whilst creating a fully functional guidance system for
standalone use. This way, those that need help will be able to use a system without feeling like they are some sort of
attraction to others walking around them. By making a stereoscopic sound effect, I intend to allow those with sight difficult ies
a better grasp of their surroundings as a whole, rather than have a system which gives commands as to where permanent
hazards are, but not temporary or moving.
DELIVERABLES
I intend to create a system which helps those with visual problems identify risks in their path or at hand. This is going to be
accomplished using stereo sound via a Bluetooth headset.
The system will first send out ultrasonic waves which are then used to record the distance travelled until returned. The
system will then convert that distance into a level of sound or frequency of a sound and transmit that to a Bluetooth headset.
With multiple sensors pointing in different directions worn at the chest area at the front, a stereo sound should be formed for
the user to allow a pseudo 3-D area in which to understand. This is the main directive of the project.
Another recommendation is to allow for another single senor which will be attached to the hand or wrist (not unlike the
USTRAAP system1). Using a button on the Bluetooth headset, the user will be able to switch to this one sensor system and
survey objects in close proximity (i.e. picking items off a table). This can be a system on its own but would be highly
beneficial to include in this build.
The system will be predominantly built in C++ to cater for the Arduino environment. Testing in the system will include the
range sensors (Ultrasound), latency issues with sound, multiple input on same frequency and user testing to ensure the
system is helpful for those it’s aimed at. The additional hardware requirements are listed at the end of the Method section.
This system must not interfere too much with the sound around the person so must be discreet and unobtrusive. In order to
achieve this, many different sounds, sound levels and frequencies will be applied during the build. The system must be
portable and easy to use, so once the hardware and software have been distinguished and proven to be sound, then
designing the system to be confortable worn will be important.
It is to be noted that this system is intended to be used as a mediator to allow preemptive movement and not to be
completely relied upon for unassisted travel. A cane or walking aid should be used so as not to miss lower lying objects.
1 https://nfb.org/blog/atblog/ustraap-system
http://www.fastcompany.com/company/ustraap

METHOD
In order to complete this project and gauge how much can be achieved within the give timeframe, a time line of
milestones has been created in order to depict what aspects should be completed by what time. This is documented
below;
 Week 1
o Create relational sound to distance using ultrasound and Bluetooth headset
 Create a way to set frequency of sound, or sound level depending on distance input
 Distinguish Bluetooth protocol needed for Bluetooth Audio Module 2
 Create system which works with one Ultrasonic Sensor
 Establish list of additional components not noted in IPD needed for full design
 Week 2
o Create basic build with Arduino for multiple Ultrasound sensors and battery operation
 Review angles for Ultrasound devices so as not to interfere with one another
 Begin programming for stereo sound based on multiple inputs and Bluetooth connectivity
 Week 3
o Update system to use more than one Ultrasound device to gauge three distances at once
o Attempt to mount system on person
 Review wiring and board positioning
 Week 4
o Create algorithm which uses relational sound from three sensors to create a stereo sound or 3d at best, 1
week
 Continuous
o Testing – Iterative
 Extra – Program on and off button and additional sensors for hands
o Use Bluetooth buttons to change between hand and walking systems
Preliminary work can be conducted in the lab such as programming distance input and programming needs. However,
for the final product, a new system will be built comprising of the following
 Arduino , Bluetooth audio module, Bluetooth headset, ultrasound proximity sensors (x4), 9v battery adapter,
soldering iron, breadboard, cables, small case for board
Although many piece of equipment are standard, a picture of the Bluetooth headset is placed below to give a better idea. In
this example, the cord between the headphones works especially well so the user can remove the set but not misplace. The
buttons on the side are hardcoded to give a standard signal to phones and mp3 players to move songs, but for this research
project, can be used to turn the sounds on and off in the system.
2 Research had already started on components at this time - http://www.electrodragon.com/product/pkb-pcba-bluetooth-
stereo-audio-module-for-music-speaker/

Synaesthsia

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (10)

Similar to Synaesthsia

Similar to Synaesthsia (20)

Recently uploaded

Recently uploaded (20)

Synaesthsia