Reality System for Social
TOM GURION1 AND NORI JACOBY1,2
1BAR-ILAN UNIVERSITY, RAMAT-GAN
2HEBREW UNIVERSITY, JERUSALEM
Augmented reality in
the audio domain?
Introduction and framework:
Interactive music systems
ADA: The Intelligent Space (2002)
Introduction and framework:
Interactive music systems
AutoRap by Smule (2012)
RjDj – Sonic Experiences (2008)
“A flash mob is a group of
people who assemble
suddenly in a public place,
perform an unusual and
seemingly pointless act for a
brief time, then quickly
disperse, often for the
purposes of entertainment,
satire, and artistic
expression. Flash mobs are
media, or viral emails.”
“Silent disco has become a
common name for
a disco where people dance
to music listened to on
Rather than using a speaker
system, music is broadcast
via an FM-transmitter with
the signal being picked up
by wireless headphone
receivers worn by the
Propose and implement an audio-only augmented reality system for
Evaluate the system in the context of silent disco party.
More generally, explore the potential of new ways of music consumption
through mobile devices.
The six balloons bundles Bluetooth beacon
App Screenshot The partyApplication page in Google Play
Our positioning requirements
What we need:
Accuracy of about one meter
What we don’t need:
Bluetooth Based Relative Positioning system
Repeatedly search for
Extract received signal
strength indicator (RSSI)
from the Bluetooth device
discovery and use it as an
estimation of the distance
between the user and the
Send the device address
and RSSI value to Pure
Data patch using libpd API
Each one of the six beacons
correspond to pre defined
sound zone player.
The patch receives RSSI
values from the app and
route them to sound zone
player according to
Each sound zone uses the
RSSI value differently to
manipulate the music in
real-time (E.g. volume,
music filtration, etc.)
Evaluation: preliminary results
Participants were randomly
assigned to two groups.
Both groups started the
In the control blocks
participants heard pre
composed music based on
the music of the interactive
They were informed that the
experiment consists of
interactive and control
Group A consists of 8 participants (4 females and 4 males) with mean age of 36.7
(s.d = 12.3); group B consists of 10 participants (3 females and 7 males) with
mean age of 29.6 (s.d = 10.2). Participants had a diverse musical background with
4.7 mean years of musical training (s.d = 5.2).
Each participant filled pre/post party surveys that included questions
regarding their musical background and preferences as well as system
Bluetooth discoveries count
Counting the number of Bluetooth device discoveries made by the
participants’ phones during the interactive and the control blocks. In
order to eliminate edge effects, we analyzed only the two middle blocks
of the experiment.
Video capturing of the party from above the dance floor. Analyzed offline.
We found that using the
participants tends to dance
less with people they knew
significantly higher levels of
movement using the system.
We didn’t find supportive
results to show that
participants dance more with
Dancing with known people: paired t-test, t(14) = -2.5, p_value = 0.01
Changing location in space: paired t-test, t(15) = 3.9, p_value < 0.01
We found higher counts
during the interactive blocks
of the party compared with
the control blocks.
paired t-test, t(16) = 1.7, p_value = 0.06, non-significant
manually in Pure Data to
generate positioning tables.
We didn’t find significant
difference between the
position in the interactive
and control blocks. However
position was indicative to the
social relations between
We asked participants for
their social relationships to
We compared the resulted
map with video correlation
The results shows that the p_value < 0.001
Our system demonstrate a simple way to use Bluetooth technology for
relative indoor positioning.
Our preliminary results demonstrate the potential for audio-only
augmented reality to significantly enrich the experience of music
consumption and social interaction.
We show that the social implications of the system can be validated in a
We show that positioning tracking can be used to obtain social
relationships within large group of people.
We will use the devices accelerometers to asses entrainment.
Larger and more comprehensive experiments with a different design.
“Master student… Thesis”
“We are all familiar with the concept of augmented reality and you are probably know much more about that then me. But note that most of the different project in this area are in the visual domain.”
“I was interested if it possible to extends our perception and interaction with the real world using audio-based experience instead of visual.”
“Lets start with an introduction.Music technology changed music creation and consumption in the last 60 years. I will focus on the last decade.On the left you can see… Switzerland”
“…it is called ‘Scenes’ in the RjDj glossary”
“Now I want to talk about some social phenomena that relate to my project. The first one is the flash mob… Note that this social phenomena is highly dependent on new technology, especially on social media… Commercial use of this idea.”
After 1:“Using the system, participants will be able to interact with one another as well as with system components to affect the music in real time.”
“So how does the system work"
“There are six balloons bundles on the dance floor. Every one of them… Inside every one of the bundles there is… This is a completely standard 10$ beacon but any other Bluetooth enables device will work just fine” NOKIA JOKE “…except for these beacons, all the other equipment required for the system is the mobile devices of the participants.”
“Participants come to the party with their own Android devices and headphones. They install our application from Google Play. After turning Bluetooth on… Each bundle creates a virtual sound zone around it… in your headphones. In addition, participants can move the sound zones… Imagine…”
“Here is a video from a party using the system. You can’t hear anything because… The system is here with me and some of you have already tried it. For now I will show you a short simulation that uses the same music we use in our system. You will see how participants can interact with the sound zones and actually choose which one to listen to. In addition you will hear the influence of different participants on one another when one virtual participant takes different sound zones with him during this simulation.”
“As you saw in the demonstration, a crucial component of the system is the ability to find positioning of participants on the dance floor. When we started to consider different options for indoor positioning we realized that we…”“We don’t need the absolute positioning of the participants in space, just the distances between the participants and the sound zones.”“…That we developed.”
“RSSI is part of the Bluetooth protocol and therefore implemented on every mobile device.”“Pure data is both programming language and a powerful sound engine. We use Pd for the audio part of the application.”
PHASE CHANGE“Most of the research in augmented reality nowadays does not describe formal user evaluation of the system. We wanted to check if we could evaluate the system in a controlled experiment. Again, we want to discover the properties of the social behavior using the system as opposed to standard silent disco party.”
After 1: Describe the experiment in details. About the blocks etc.
What I want to get from the different measurements. Find differences in social behavior. More counts shows that participants interact more with system components.
Regarding the right graph:“In the right graph we see the counts during the control compared with interactive blocks. As we can see, most of the participants have higher counts during interactive blocks and therefore are bellow the main diagonal (marked with red X’s).”
After 3:“We used Monte Carlo reshuffling to check how similar those two maps are and we found…Actually, participant 7 and 8 are a married couple…There is a lot of data in this slide but the bottom line is this:…”
“We will use this measure as an implicit measurements of engagement.”“We hope that with better experiment design we will find much more significant results.”