Motto: Prototyping is the new wireframe
Objectives
Our objective is to create a unconventional human-computer interaction using a physical
computing platform based on Arduino, using different outputs and inputs. Our goal is to build a
hardware platform to express different emotions or moods of the user interacting with it, providing
both a software and hardware input for the system. As a software input we are going o use different
variables that might have an important role in establishing the overall state of the user. Also we are
going to use appropriate hardware part, such as LEDs and/or LCD screen to show as accurate as
possible the state of user. A more precise scenario on the interaction with the system we are going to
build as we start developing the project itself.
Because the hardware part plays an important role in the final outcome of the project we will
explain the way in which Arduino interacts and uses the different outputs and inputs and how we are
going to use them in building our hardware platform. Exploring several hardware prototypes and parts
will helps us in building a more complex and accurate prototype. Also in the our search for the perfect
prototype we are considering keeping it simple due to hardware limitations.
As we previously pointed we are going to explore several prototypes and testing the
communication between the hardware and the software to establish the best solutions for our
objectives.
In the first part we setting the foundation of our project by describing some basic concepts we
are going to use also providing a short introduction on how the Arduino device works and the electric
parts we are using in conjunction with the device.
The second part will consist of an analysis on how the device we had develop interacts and
reacts to the user's emotions.
1

Introduction
In our everyday interactions with other humans, we encounter emotional responses all the time,
whether it be others’ emotions, or our reactions to their activities. These emotions influence all parties’
responses and approaches to interacting with each other; they pervade every aspect of interchanges
between people so deeply that we are often only consciously aware of extreme ones in such exchanges
– of anger, rage, upset, sudden love, and so on. But the influence of emotion is deeper than this – it
affects our attention, our perceptions, our memory and decision-making abilities. It also makes us
somewhat uncomfortable when emotion is absent, when we don't get a feedback on our problems, or
when we can't find someone that can send a feeling back to us.
According to Scott Brave & Clifford Nass emotion is: a reaction to events deemed relevant
to the needs, goals, or concerns of an individual. As we notice emotion is defined as a reaction to a
certain action, which can consist of absolutely anything. Taking into consideration the field of
computer science emotion can be associated to a reaction of the user to a certain software or an error,
and other similar action. We also have to consider the fact that any action performed by a software of
any hardware is a response to a user input. Although certain AI systems can perform on their on, they
still need to interact with the user in order to perform an action.
As we mentioned in the previous paragraphs we are concerned with the fact that the
“computers” can't respond with emotions or even show a some level of affection to the user.
Considering the fact that humans can recognize emotions and moods in almost anything, it should be
easy to display an emotional reaction from the system towards the user. Still existing methods of
interacting with a computer don't provide a useful, portable and flexible environment. Building an
unconventional interaction platform provides the appropriate response to our problems. In fewer word
we are building a physical computing system.
Physical (or embedded) computing, in the broadest sense, means building interactive physical
systems by the use of software and hardware that can sense and respond to the analog world, in our
case the user. Our physical system will consist of an Arduino board and a shield built for it. In the
following pages we are going to give a basic introduction on what and how we are using these
hardware parts.
2

Arduino a Prototyping Platform
We are going to start with an example to explain our choice for Arduino and why physical
systems are an improved version of wireframes. Consider the following: without a computer, you can
connect to a button being pressed to a light turning on. With a computer, you can make the relationship
between the button and the light more complex. For example, you can make the light’s turning on
dependent on the number of times the button was pressed, for how long it was pressed, or whether it
was pressed in conjunction with other buttons in other rooms or on other continents. The possibilities
are infinite, and this is done using just a button, computer or micro controller and an LED or light
bulb.
Arduino is an open-source electronics prototyping platform based on flexible, easy-to-use
hardware and software. It's intended for artists, designers, hobbyists, and anyone interested in creating
interactive objects or environments.
Arduino can sense the environment by receiving input from a variety of sensors and can affect
its surroundings by controlling lights, motors, and other actuators. The microcontroller on the board is
programmed using the Arduino programming language (based on Wiring) and the Arduino
development environment (based on Processing). Arduino projects can be stand-alone or they can
communicate with software on running on a computer.
Arduino is composed of two major parts: the Arduino board, which is the piece of hardware we
work on when you build our objects; and the Arduino IDE, the piece of software we run on our
computer. We use the IDE to create a “sketch” that we upload to the Arduino board. The sketch tells
the board what to do. We can we available Libraries depending what we are working with, LCD, or
other devices that require advance handling on the communication part. The libraries are provided
within the IDE and are uploaded to the board with the “sketch”.
Figure 1. Arduino Prototype board (drawing from www.arduino.cc)
There are many versions of this board; the one we'll use throughout this book is the Arduino
Duemilanove, which is the simplest one to use and the best one for learning on. However, these
3

instructions apply to earlier versions of the board, including the more recent. In figure Figure 1 we
illustrated the Arduino Duemilanove, an older version of the board.
The board consists of:
• Digital I/O pins 0 – 13;
• Pins 3,5,6,9,10,11 can also be used as analogue out pins because they ca use PWM (pulse
with modulation)
• Analogue In pins 0 - 5
• USB communication port – to upload the sketch
• AC adapter port
• other pins like GND (ground), 5v (which gives 5volts), Reset port (which we are to refer to
further in the paper.)
The I/O ports are used in conjunction with other electronic parts such as, LEDs, potentiometer,
resistor, transistors, LCD, motor, Arduino shields, and other advanced electronic parts such as sensors
and touchscreen. It can interact with almost anything.
We previously mentioned Arduino shields, which consist of and PCB (printed circuit board)
with a special function such as Ethernet adapter, wireless adapter, touchscreen and so on. These shield
can be added to our device to extend its capabilities.
Although we will not use the processing language in our project we will mention it due to its
important part in developing a IDE that can be easily used by anyone to build software prototypes.
Processing relates software concepts to principles of visual form, motion, and interaction. It
integrates a programming language, development environment, and teaching methodology into a
unified system. Processing was created to teach fundamentals of computer programming within a
visual context, to serve as a software sketchbook, and to be used as a production tool. Students, artists,
design professionals, and researchers use it for learning, prototyping, and production.
The Processing language is a text programming language specifically designed to generate and
modify images. Processing strives to achieve a balance between clarity and advanced features.
Beginners can write their own programs after only a few minutes of instruction, but more advanced
users can employ and write libraries with additional functions. The system facilitates teaching many
computer graphics and interaction techniques including vector/raster drawing, image processing, color
models, mouse and keyboard events, network communication, and object-oriented programming.
Libraries easily extend Processing’s ability to generate sound, send/receive data in diverse formats, and
to import/export 2D and 3D file formats. The most important part is, from our point of view is the fact
that it can easily communicate with Arduino boards, without any additional software support.
As we mentioned the IDE we are using in developing our sketch is based on the same IDE as
the Processing language, but giving a detailed description of it is not the subject of this paper.
The Arduino IDE comes with a C/C++ library called "Wiring" (from the project of the same
name), which makes many common input/output operations much easier. Arduino programs are
written in C/C++, although users only need to define two functions in order to make a runnable
program:
• setup() – a function run once at the start of a program which can be used to initialize settings,
• loop() – a function called repeatedly until the board is powered off.
Wiring is both a programming environment and an electronics prototyping input/output board
for exploring the electronic arts and tangible media. It illustrates the concept of programming with
electronics and the physical realm of hardware control which are necessary to explore physical
interaction design and tangible media aspects in the design and arts disciplines.
4

Building the Prototype
We are using the Arduino Diecimila version in our project, but is basically has same structure
and ports as the Arduino Duemilanove.
The first part of building any kind of prototype is making sure we have the right tools, in our
case we had to establish the inputs and the outputs of the system. The second part is building enough
proof of concepts to make sure we are on the right path. For now we will exclude the software part and
we will focus on the wiring and positioning of the parts, providing explanation for each element used
in the system.
Before we can start building we would have to select our parts. For a first basic prototype we
selected the following electronic parts and started experimenting with them:
• LEDs – we selected three colors: Green, Red and Blue
• Buttons
• Resistors
The most important part of this basic prototype are the LEDs, because as we previously
mentioned, human can recognize emotions in anything, this is why we had to built a metaphor based
on the color of each LED. Each color represents a certain emotion, or reflects the state of the system.
In a more advance version we will use color combination in order to create a more advance pallet of
feelings .
Stage 1
After a few rapid prototypes we developed the following circuit as shown in Figure 2. As we
can see the circuit is divided into groups for a better understanding of the functions. As we can see we
have :
• Idle LEDs – which are use to describe the idle state of the system, meaning no button
was pushed to trigger an event.
• Wake up group – consists of a LED and a Button. When the button is pressed the LED
turn and the Idle LEDs turn off. In this stage we say that the system is ready for input.
• Control group – consists of three LEDs: yellow, red and green, each with its own
button. These LEDs have a direct influence on the Mood LEDs, depending how much
light the control LEDs give. Setting the intensity of the control LEDs is done by
pressing down the corresponding buttons. The mood LEDs update their state in real
time. These are used to express the emotions of the user thus affecting the mood of the
system in the Mood LEDs.
• Mood LEDs – set the overall mood of the system, reflecting it through the LEDs.
Although the mood is total different concept from emotions we decided to set the name
of the LEDs like this because the color metaphor give more the feeling of a mood than
of emotions.
5

Figure 2. First prototype (the circuit).
Stage 2
Although different scenarios can be built on the circuit presented in Figure 2, we focused on
one that we presented in Figure 3 but we are also going to describe in the following paragraphs.
Before we jump into the scenario we will define the following:
1. Color Code:
• Green – the user has a positive emotions regarding the system: happy, joy,
excitement
• Red – the user has a negative emotions regarding the system: sad, angry,
disappointment
• Yellow – gives a boost to the positive emotions, it could also stand for an outside
influence on the user
• Blue – sets the system status, not used to set the mood
2. Hardware Input:
• Control LEDs – with variable intensity they have a direct action on the Mood
Leds
• Control buttons – setting the intensity for the LEDs
3. Hardware Output:
• Control LEDs – they show the intensity – in our case the emotion of the user
regarding the system
• Mood LEDs – reflect the mood of the system depending on the input the
intensity of the LEDs is given by simple formulas:
▪ Green LED = (Green Control LED – Red Control LED) + Yellow
Control LED
▪ Red LED = Red Control LED – Green Control LED
6

4. Software Output:
• Twitter.com – the system will communicate via USB port with the computer to
send a message to a twitter account when the system in on, the wake up LED in
on, and when wake up LED is off. In the last case we will send the state of the
mood using the following format:
▪ if the green LED is on: System has a var % happy mood (var = value of
the intensity)
▪ if the red LED is on: System has a var % sad mood (var = value of the
intensity)
• Flickr.com – as an additional reaction to the user input we want the system to
constantly improve the state of the user. This is why depending on the outcome
we will search for images to make the user happy or show compassion to the
user state. Searching photo tags and returning the most relevant photos might
have a positive action on the user.
Considering all the outputs and inputs described above we built a scenario explaining the flow
in Figure 3 . The figure illustrates a single flow of the application, but the system is in a continuous
search of improving its state. This is why the system and the user form some kind of bond, this leading
to an improvement in the interaction between a computer an a human being. With a Jef Rasking
quote in mind: “An interface is humane if it is responsive to human needs and considerate of
human frailties.” we can say are building a more humane interface. As we notice the AI of the system
is rather primitive, and its constantly waiting for a user action, basing it's reaction on predefined action.
Now we can only imagine the progress we will have in improving the user emotions when we
add more humane outputs and inputs. With this thought in mind we decided to improve the hardware
and give it a more humane look and different inputs.
Figure 3. First Scenario.
7

Stage 3
In Figure 4 we illustrated the final outcome of the prototype we built. The system is currently
showing a positive mood.
Figure 4. The physical prototype made on a breadboard.
Stage 4
As we said we were searching to constantly improve our outcome and give an accurate
description of the user emotions and also interacting with him in order to improve his state. As we
found out the emotions are more complex and our original color code for showing emotions is not that
appropriate. Also our original input lacks the humane interaction that Jef Rasking was referring to.
This is why we decided to choose a more humane input.
1. Color Code - We initially planned to use RGB LEDs that give a more complex outcome,
but due some hardware issues we resumed with doing color combinations by turning on
groups of LEDs at the same time:
• Green – the user has a positive emotions regarding the system: happy, joy, excitement
• Red – the user has a negative emotions regarding the system: sad, angry, disappointment
• Blue – often used in combination with green and red, it can also express indifference.
2. Hardware Input
• Potentiometer – used to set the intensity of the LEDs in this case it expresses the
user reaction to the mood of the system.
3. Software Input:
• As we can see in Figure 5 we decided to set a different input coming from the user.
Although it's not a perfect representation of how you can retrieve emotions from a
human being, it gives a more accurate method of identifying them. As described in the
8

figure we have as an input the response to three questions :
▪ How do you feel ? - the input would be a slider selecting a percentage
▪ Do you like the weather outside ? - the weather plays sometimes an important
role in establishing the emotions
▪ Are you bored ? - boredom can be considered the source of negative emotions
this is why the user must have an activity to improve it's feelings.
Figure 5. Software Input.
4. Hardware Output:
• LED groups – Red, Green, Blue. The color code we proposed at point 1 is used as an
output of the mood of the system. The mood is set using the software input. The system
will display different combinations of colors depending on the input, giving a more
diverse range of feelings, unlike the previous prototype when we had only two moods.
• LCD – the LCD screen can display different message to the user, thus making the
interaction more complex. The messages displayed on the screen are predefined
depending on the input, but more advance version would display a message based on
the combinations of color from the output.
5. Software Output:
• Twitter.com – the system will communicate via USB port with the computer to send a
message to a twitter account via the software when the user answers the input.
▪ System has a var % happy mood
▪ System has a var % sad mood
• Flickr.com – as an additional reaction to the user input we want the system to
constantly improve the state of the user. This is why depending on the outcome we will
search for images to make the user happy or show compassion to the user state.
Searching photo tags and returning the most relevant photos might have a positive
action on the user.
9

Figure 6. Second prototype (the circuit).
Figure 7. Second Scenario.
10

The circuit from Figure 6 follows the description of the input and output methods used in our
prototype. As we can see it mas a more complex wiring requiring multiple pins from the Arduino
board. This might cause some malfunction in the way the system behaves due to the fact there are too
many electricity consumers tied to the board. In a future version of the project we will consider these
points, so that the system will not have too many consumers.
Regarding the scenario, we tried not to alter the original scenario, but did make some changes,
in order to fit the new hardware profile. As we can see in Figure 7 we modified some actions of the
system, am simplified the interaction with the user, setting a more compact flow of the application, and
requiring less intervention from the user in setting the mood. Thus we a great improvement in the
human-computer interaction.
Figure 8. Working Prototype.
Although the system we just build can be seen as an unconventional interaction, the system
provides a multimodal interaction, because of the different outputs and inputs. Still the fact that the
system we build stands out from the usual interaction with the user we can place it in the category of
unconventional interactions.
Software Communication
Communicating with Arduino is done via USB, more precised a simulated serial interface via
USB. The serial communication is don rather easy, the only problem we faced was the Auto-Reset
function from the device, which we disabled with a 120 Ohm resistor.
As a programming language we chose Python, which a also provides an easy way of interacting
with Twitter API and Flickr API.
We used a web server to generate the input for the device. The application send signals to the
device to set the mood and receives the message which is displayed on the LCD.
11

Figure 9. Software control.
Conclusion Remarks
Making the interaction with a computer more humane, takes a great improvement in the
method we interact and use our computer, even our devices (mobile phone, game console,
touchscreen). But the first step is exploring and developing new ways in which the system can respond
to user's emotions. Even though with technological improvement in the field, such as, facial
recognition, determining emotions in texts, we can't make the computer behave and adapt to every
human. Learning every option and emotion of every user, take a lot of processing and time, that is why
we have to combine old methods of interaction with the new one and create multimodal systems, that
can easily adapt to each user.
Although this paper explores a small part of prototyping these new kind of systems, we first
have to consider that possibilities of creating different interactions are unlimited. That is why we have
to constantly create and explore unconventional interaction in order to create systems that can adapt to
any requirement of the user.
12

Bibliography:
[1] Massimo Banzi, Getting Started with Arduino, O'Reilly 2008
[2] Christian Peter , Russell Beale, Affect and Emotion in Human-Computer Interaction From
Theory to Applications, Springer 2008
[3] Dan O'Sullivan, Tome Igoe, Physical Computing, Thomson Course Technology, 2004
[4] Tom Igoe, Making Things Talk,O'Reilly, 2007
[5] www.arduino.cc
[6] www.wiring.org.co
[7] www.processing.org
[8] www.freeduino.org
Online Resources of the project:
http://www.flickr.com/photos/blankdots/sets/72157617049599313/
http://www.vimeo.com/4563793
http://www.vimeo.com/4930674
http://twitter.com/feeltagged
13