Thesis docuementation2

EMOTIONAL CONNECTION
USING OBJECT MEDIATED COMMUNICATION
BY YANJUN LYU

Emotional Connection
EMOTIONAL CONNECTION
USING OBJECT MEDIATED COMMUNICATION
BY YANJUN LYU

Copyright© 2016 by Yanjun Lyu
Printed in the United States
Designed by Yanjun Lyu
using InDesign CC 2017
Typeface: Minion Pro, Avenir Next.
The uncredited photographs
in this book belong to the author.
Further information about the full series
can be seen on yanjunlyu.com
Edited by Lisa Rosowsky
Printed by ACME BOOKBINDING
Hardcover Trade Book 7x9.5 IN
Dynamic Media Institute
Massachusetts College of Art and Design
To my grandpartents, parents, and all the
people that are separated by distance.
That separation inspired me to explore how
to use technology to enhance emotional
connections through multisensory experi-
ences in remote communication.

I simply believe that the most promising and pleasing
future is one where technology infuses ordinary things with
bit of magic to create a more satisfying interaction and
evoke an emotional response.
- David Rose
Study the past, if you would divine the future.
- Confucius

Thesis Committee
This thesis is submitted in partial fulfillment of the
requirements for the degree of Master of Fine Arts in
Design and approved by the MFA Design Review Board
of Massachusetts College of Art and Design in Boston,
Massachusetts.
December 2016
Jan Kubasiewicz, Project Advisor
Professor of Design
Lisa Rosowsky, Thesis Advisor
Professor of Design
Gunta Kaza
Professor of Design
Joseph Quackenbush
Coordinator of Graduate Program in Design
Professor of Design

Table of Contents
Introduction
Inspiration Stories
Inspiration Technology
An Overview of My Thesis
Abstract
Contextual Background
A. Nonverbal Communication Study Subjects
B. Communication Technology
C. Nonverbal Communication
a. Object-mediated communication with metaphor
b. Haptic communication
119 127
Case Study
Communication Via Physical Plant
Digital Platform
Dialogue Glove
Addendum: HumInterface
131
Conclusion
Learning Outcomes
Findings
Acknowledgments
59
Bibliography
05 19

Recent advances in technology have shifted the way people
communicate with each other from primarily face-to-face
interaction to interactions mediated through screen-based
devices. Technology makes the communication very effi-
cient: it’s quick, immediate, and spans great distances. But
this convenience has trade-offs. The emotional quality of this
technologically-mediated communication is less subtle and
rich. One reason is that there’s little tactile interaction. What
if touch could be distributed over space and time, recorded
and broadcast like streaming data? Digital communication is
also optimized for highly verbal and visual people.
But what about those for whom expressing emotion in words
and pictures is very difficult? How can we infuse warmer,
more tactile, and emotionally- resonant forms of communi-
cation in our electronic exchanges? How can we enhance the
regular way we communicate via computers and comput-
er-driven machines? My thesis seeks to provide a subtle and
sensual shared experience of connected space in daily life for
people who live apart through non-verbal communication.

A. Insipiration Stories
B. Inspiration Technology
C. An Overview of the Book

7
A diary about a plant
growing by hand
drawing. The day that I
left home, Zhangjiakou
at 08/05/2014.
Introduction
The First Story. I was born in China and grew
up in a traditional Chinese family. Due to my
parents’ work I lived with my grandparents until I was
14 years old, so I had a very deep emotional connection
to my grandparents. My grandmother is a traditional
Chinese homemaker; the family is her life. Her days
revolve around taking care of my grandfather and me.
My grandfather is a very strict person and is not good
at expressing his emotions. Whenever I did something
wrong, he would criticize me. He never praised me,
although I got good grades in school. In contrast, my
grandmother always stuck up for me. I was scared to
talk with my grandfather until I grew up and slowly un-
derstood how he expressed to me his hidden emotions.
In the summer of 2014, I moved to Boston to pursue a
master’s degree, and I left my plants in the care of my
grandfather. During the two years that I’ve been away,
my plants have grown to 1.3 meters high and now have
to be moved into bigger pots. My grandmother says he
spends lots of time and effort taking care of my plants
each day, watering, changing containers and turning
the plants to face the sun. Because he is 83 years old,
this has been much work for him on my behalf. He told
my grandmother that when he cares for these plants,
he feels that I am still with him as if I never left, and he
places his hopes and concerns for me in the plants. He
hopes I am becoming stronger and more independent
than before, like those tall plants. The silent love of my
grandfather today rings in my ears with a deep echo.
Sometimes I like to imagine him taking care of my
plants, especially when my grandmother talks about
it. The plants are a metaphor for me, and his care for
them exemplifies his love for me, in a non-verbal way.
I call this kind of emotional expression “metaphorical
expression.” My grandfather’s care of my plants was a
major inspiration for my thesis topic.
The Inspiration Stories
The Second Story. In the winter of 2015-2016, I had to
return to China. My grandfather was in the hospital to
be treated for brain atrophy, and he could hardly speak.
Instead, he held my hand tightly and used his eyes and
hand gestures to “talk” to me. At such moments, I was
able to feel what he was trying to say, and even sense his
emotions and gain information from my connection
with his hand. During the days of his illness I took care
of him, and I was able to understand almost entirely
what he wanted without his using any verbal expres-
sion. I define this type of communication as “haptic
communication” in my thesis documentation. Haptic
communication provides information through texture,
surface and the sense of touch. It is a component of
communication in emotional relationships, and can
be nonverbal as well as non-visual. It may even involve
telepathy instead of direct perception.
6

8 9
Introduction | plant daily

11
10
Introduction
My Plant Diary (above)
My plant grows from a “baby” plant to
“adult” plant in three months.
Taking care of the plant has already
become a necessary part of my life. I treat
the plant as an intimate friend. I expect to
see it grow. It reflects my mood every day
as well. Some days I cannot take care of
my plant because I am anxious due to my
grandfather staying in the hospital. As a
result, my plant starts to whither away by
February.
Both of these experiences inspired me to explore how
subtle emotions are expressed nonverbally, which may
be the main way emotions are expressed by people who
are not good at directly showing emotion, especially to
convey subtle and sensual feelings and thoughts, like
my grandfather. When he lay on the bed, unable to
speak, I was worried about what would happen if he met
with trouble when no-one was around—how he would
indicate an emergency to a doctor or his family without
words. I imagined at that moment, that if he could grab
a small object with his hand he could just use simple
hand gestures (swipe, tap or squeeze) to send a signal
for helping via the object. Similarly, individuals who
live apart and cannot conveniently chat with friends or
family using voice technologies, could benefit from the
ability to communicate with each other non-verbally.
My friend and her boyfriend live far away from each
other, one in Beijing and the other in Italy. She says that
distance and time make their love and emotion feel
further away. “However,” she says, “I just want to ensure
that he is good, that everything is fine, and that he is
safe —even just to remind him to take his medicine
before meals.”
When people get together, even if they are silent, eyes
connecting, a hug, or a kiss, all have message value
which can eliminate misunderstandings. However, dis-
tance and time prevent them from connecting through
touch and perception—they can only connect verbally.
I envision ways to enrich and enhance communication
between remote couples or families.
The Initial but Most Important Ideas
My Grandfather’s home
My grandfather usually likes to spend a
half day to stay in his garden. He really
loves taking care of these plants.
My grandfather always said to my grand-
ma: “taking care of Dan Dan’s (my nick
name) Lily and seeing it grow taller and
taller each day, I feel as if she never left
me. I hope she is doing well in America
and grows stronger and more indepen-
dent everyday.”
Hand drawing by Yanjun Lyu

13
This is my hand drawing
about my grandfather
and me using Wechat to
connect with each other
every two weeks.
Introduction
12 The majority of current interfaces are based on a com-
puter or screen-based machine as a connecting medium
between two individuals. As a study abroad student,
I am separated from my home and family. Because of
the twelve-hour time difference between Beijing and
Boston, we have to find a common time for us to speak
together. The best overlap time for us, when we are all
awake, is less than 2 hours. Even when I Skype with
them, I feel very distant and mechanistic through the
screen. Technology makes communication very effi-
cient: it’s quick, immediate, and spans great distances.
But this convenience has trade-offs. The emotional
quality of this technologically-mediated communication
is unsatisfactory.
Similarly, individuals who live apart and cannot
conveniently chat with friends or family using voice
technologies, could benefit from the ability to commu-
nicate with each other nonverbally. My friend and her
boyfriend live far away from each other, one in Beijing
and the other in Italy. She says that distance and time
make their love and emotion feel further away. “How-
ever,” she says, “I just want to ensure that he is good,
that everything is fine, and that he is safe —even just to
remind him to take his medicine before meals.” When
people get together, even if they are silent, eyes connect-
ing, a hug, or a kiss, all have message value which can
eliminate misunderstandings. However, distance and
time prevent them from connecting through touch and
perception—they can only connect verbally. Although
current communication technologies help people to
Inspiration Technology

14 15
connect over distances, I imagine how we might infuse
warmer, more tactile, and emotionally resonant forms
of communication in our electronic exchanges in order
to enrich and enhance communication between remote
couples or families.
While current communication tools such as Skype and
FaceTime exist, traditional methods of connection are
still used by many. Some write letters to maintain their
long-distance relationships. Compared with current
technology, letters are not a fast or convenient way to
communicate, and yet a recipient can distinguish the
writer’s emotion in the moment through the style of
his or her handwriting, the tactile feel of different types
of paper and the variety of paper shapes that he or she
chose. Some older people still prefer handwritten letters
as interfaces to communicate with their families. Of
course, I do not reject current communication technol-
ogies, but I envision an interactive and emotional in-
terface that would allow people to use touch to directly
sense surface, texture and temperature across a distance.
I was also curious about why communication has
evolved to be limited to the screen of a computer,
and what alternatives may replace the screen-based
medium in new communication media. How can we
combine traditional analog communication technolo-
gy and digital communication?
The two stories about my family; life experiences for me
and my friends; the barriers that our current communi-
cational tools bring; and the knowledge and skills that I
learned in DMI, together crystallized my thesis concept:
In order to understand my thesis work, it is necessary
for the reader to be able to visualize the problems I hope
to solve, and define the users that would be benefit from
my work and the steps I took in my case study. These
three steps will be addressed in contextual research and
through the case study.
Defining Problems and Users. At the beginning of
the contextual research, I did a broader user survey to
collect data and define the user-type. I needed to give
myself a broader understanding of the importance
of emotional connection, not only from my personal
experience, but also how it could benefit others. Each
of the interviews focused on three core questions:
what kinds of nonverbal strategies did the users use to
communicate remotely? In which kinds of scenarios did
they prefer to use nonverbal communication rather than
verbal communication? What were their feelings about
nonverbal communication? I collected these data using
six surveys of people from various cultures and with
different customs.
An Overview of My Thesis
To explore how to enhance the emotional connection
in remote communication through creating mediated
object connections.
Introduction

16 movements such as those we use in caring for our plants
in daily life: watering gestures, hands moving the pot
and pruning leaves. The interactive plant is a connect-
ing medium between two individuals to enable subtle
emotional communication.
The third project focuses on haptic communication
between two people who live at a distance from each
other . In this project, users wear circuit-wired gloves
that help them to sense temperature, pressure, and
surface from another gloved hand in a distant location.
The purpose is not only to explore how one might use
an everyday object to replace screen-based machines,
but also to think more about how to use haptic feedback
to create a more direct immersive experience, one that is
more magical and almost telepathic.
The fourth project was HumInterface (see section Case
Studies. D. Addendum). It was a team project that I
did with another three students when we took the MIT
Tangible Interface class. We investigated how the rep-
resentation of the user’s body parts could be altered to
amplify their capabilities for teleoperation. The project
will be a meaningful influence on my future work. In
conclusion, I have summarized my thoughts about re-
mote communication. This research has made me aware
of the issues that I still have to explore. Knowing what I
need to work on is one of the important outcomes of my
prototyping process and my work over the past two and
a half years. Let’s begin.
Based on my research about the limitations of current
technology and my own experience, I eventually
decided to use two main strategies to solve the problems
that I identified from the user survey. Mediated object
communication and haptic communication are two
types of nonverbal communication, which can enhance
emotional experience in remote communication.
Mediated object focuses on how to make an everyday
object become extraordinary and more memorable or
more “enchanted” in interpersonal communication.
Haptic communication explores how to create a tactile
illusion and synchronized for a remote interpersonal
connection.
What I did. My descriptions of the four projects in the
case study will explain how I enhanced the human emo-
tional connection through two types of communication:
object mediated and haptic. The first two case studies
introduced a physical and screen-based object that was
designed for long-distance communication between
people. The screen-based object explored how object
expression functioned as a kind of visual language to
communicate in virtual space. The object I chose, both
physically and digitally, was a plant that grows and
thrives in direct proportion to how well you care for
it. The plant is a metaphorical connection between my
grandfather and me; his caring for the plant reflects
his care for me. The project draws from our real life
experience and recreates it as a digital communication
experience. The intention is to help users span the gap
between distance and emotion through several hand
19
Introduction

19
C. Nonverbal Communication
Object Mediated Communication
Haptic Communication
A. The Non-verbal Communication
Study Subjects
B. Communication Technology

21
There are people like my grandparents and parents
who are not good at verbal communication.
However, their day-to-day actions, such as knitting
sweaters for someone, cooking their loved one’s favorite
foods, waking up early to cook breakfast for others, all
communicate their love wordlessly. I investigated six
different relationships among my study subjects in order
to explore and collect data:
Two of these relationships were especially interesting
to me. The first subjects are a young couple, who lived
in separate cities, Beijing and Rome. There is a six-hour
time difference. They told me that they connect through
an Online sharing game. They can use their time dif-
ference to continually play the game; the girl is going to
bed when the boy wakes up, so they connect with each
other through sharing a common game. I asked them
how they feel about playing the game together. While
their verbal communication delivers content clearly
and efficiently between them, sharing a virtual game
provides them with a more general sense of the status of
their relationship. Another example is how people can
express their feelings for one another nonverbally across
the longest distance of all: the span between the living
and the dead.
In China, people like to burn cash, letters, or whatever
they want as offerings to the dead because they believe
How do they express their emotions nonverbally?
What do they use to talk to each other and what is their
intention?
The Nonverbal Communication Study Subjects
20
The goals:
How do they express their emotions
nonverbally? What do they use to talk
to each other and the reasons.

Contextural Research

22 23
As with the young couple above, the action of sharing
a game makes them know what one another is doing at
that moment, which is enough to satisfy their require-
ments of emotion.
During interviewing my subjects, I was thinking how
we might infuse warmer, more tactile and emotionally
resonant forms of communication to those users who
rely on remote communication. Who would benefit?
1. Users who are not good at expressing subtle and
sensual emotion directly, like my grandfather.
2. Elderly people, for whom verbal communication is
not an easy way to interact and get help.
3. People who lived far apart, for whom verbal or
screen-based communication only emphasizes their
sense of distance.
that smoke can convey their emotions and words to the
other world. For example, there is an old couple, the
wife alive, the husband dead. The woman continues to
write letters about daily things, then burns them on her
late husband’s birthday each year. The actions of burn-
ing transmit her love and memories to him.
In China, people like to burn cash, letters, or whatever
they want as offerings to the dead because they believe
that smoke can convey their emotions and words to the
other world. For example, there is an old couple, one
alive, the other dead. The women continues to write
letters about daily things, then burns them on her late
husband’s birthday each year. The actions of burning
transmit her love and memories to him.
During the user survey, the outcomes of the six situa-
tions were that what kinds of situations and what types
of users that non-verbal communication is more practi-
cal and than verbal communication. Besides the usual
way they communicate via computer and computer
drive machines, is there another way they connect and
express emotion to their lovers? Compared to verbal
dialogue, the purpose of non-verbal expression is not
so much to transmit accurate information, although
of course it can, but to concentrate more on delivering
signals of emotions such as happiness or anger to others.

25
24
Relationship
Elderly couple
Distance and Time
Life - Death
Strategy
Writing love letters, then burning them.
Contents
The woman uses this to “tell” her hus-
band what happens to her each day.
Reason
Chinese Culture and Customs
Object
Love Letters with hand writting
Relationship
Parents and Son
Distance and Time
Boston - Boston
Strategy
He checks his mom’s refrigerator once a
week, and fixes some broken furniture .
Intention
He needs to know if her food is fresh
or not, and then decide if his mom is
feeling well or not.
“My mom knew I came, when she saw
the fresh food.”
Reason
His mom is already over 89 years old.
Verbal communication is not the best
way to understand her health.
Object
Food, furniture, every day objects.
Relationship
Parents and Son
Distance and Time
London - Japan | 9 hours
Strategy
Sharing Google Calendar or common
space
Intention
She wants to know what her son does
every day in order to know which time is
best to talk with him.
Reason
To determine when they can speak with
each other.
Object
Calendar
Relationship
A young couple
Distance and Time
Beijing - Rome | 6 hours
Strategy
Game sharing, Postcard, Mailing each
other small gifts as a surprise.
Intention
His girlfriend reminds him to take
medicine on time. She calls him to wake
up on time.
Reason
To enhance their relationship
Object
Physical gift; Printed postcard
Relationship
Grandfather and me
Distance and Time
Beijing - Boston | 12 hours
Strategy
Places his love on the plant, taking care
of the plant everyday.
Intention
He wants to know what’s going on with
me. He reminds me to have dinner on
time. My grandfather asks me whether I
have enough money or not.
Reason
He worries if he will brother me when he
wants to give me a call.
Object
Plant
Relationship
Friendship
Distance and Time
Boston - London | 4 hours
Strategy
Mailing postcard.
Intention
She wants to invite me to her graduation
exhibition.
Reason
Sharing her happiness with me.
Object
Post cards
Six Study Subjects

From Prehistory to Current Social Media
important carriers of primitive information. In times
of war people will put important messages into a fish’s
belly, or hide them inside food, in order to ensure the
safety of the message enroute to the right receivers. In
addition to these examples, there are myriad other tech-
niques of ancient communication, such as secret signals,
letters, flags, light, and so forth.
All these communication tools and methods—whether
the belly of a fish, bamboo poles, hand gestures, or
secret signals as communication medium—help people
to connect with each other, to link individuals into a
community, to extend a small area to a big city, and to
draw people closer together, despite physical distance.
Of course, I am not encouraging people to return to tra-
ditional types of communication! But I am suggesting
that we think about the purpose of human interaction
and ways to facilitate it: to invent new ways to commu-
nicate, and to improve the quality of communication.
Current Social Media. In the 18th and 19th centu-
ries, the telegraph (1792), telephone (1890) and radio
(1891) initiated new eras in sending and receiving
messages over a long distance. Our current era is the
age of the Internet, in which social media are based on
web-based and mobile technologies, computers, and
computer-driven machines. Such software engineer-
ing creations as Myspace, FaceTime, and email have
addressed the problem of time, space, and even safety.
People are able to immediately communicate remote-
ly with families without pigeons, letters, or smoke.
Communication has changed rapidly in a very short
time. Now, it often means social media, such as
Face book, Skype, Twitter, and so forth. However, the
history of communication offers diverse insights even
for modern times. Beginning about 40,000 BCE, ancient
people used drawing on cave walls to express their
thoughts. Piled up stones represented directions and
boundaries. People used smoke signals by day and bea-
con fires by night in ancient China, Egypt, and Greece.
Drums also were employed in many parts of the world
to extend the range of the human voice for commu-
nication. Pigeon post was a major method in tradi-
tional China to communicate between distant points;
people used the speed of flight and easily distinguished
direction to deliver a message tied to a pigeon’s foot to
transmit important or urgent messages.
Creative development of communication extended to
ways to deliver fairly specific information. Among some
ethnic minorities of China, bamboo was used to spread
information during a funeral to represent identity and
age of the dead. One sound on bamboo was for an un-
married person; two sounds signified a married person
who had a wife and children; three were used for the
shaman, and so forth. Mimicry and hand language were
popular among primitive hunters; when they found the
mark of an animal, they use hand language to exchange
information silently. Such gestures are often the most
prominent feature of prehistoric humans. Ethnological
research suggests that this kind of mimicry and hand
gestures are ubiquitous in ancient societies, and are
Technology Communication
26
27

28 29
Furthermore, applications such as Skype and Face Time
bridge the communication gap among like-minded
individuals, offering a platform for Online users to
find others who share the same interests and to build
virtual communities based on those common interests.
With the availability of social media technologies and
services, content sharing and user interaction have
become relatively easy and efficient, and span time and
space limitations.
Drawbacks of Social Media
Fig. 1
People in the sub-
way. Bejing, China.
In addition, people today have the opportunity to
"brand" themselves via Facebook status updates and
tweets, so that now, Vaughn says, “ instead of talking
to each other, they're more likely to read about each
other” 1
Today, most people, especially millennials and
teenagers, rely on social media for keeping in contact
with their peers. A report published in 2012 Teens,
Smartphones & Texting by Amanda Lenhart reveals that
“ texting is the favorite form of communication among
young people ages 12 to 17. Some 63 percent use texting
to chat with others every day. About 39 percent call and
receive calls on their cellphones; 29 percent swap mes-
sages on social network sites — such as Facebook and
Twitter — and 22 percent send instant messages”2
Social media have already disrupted our natural forms
of communication. Frequent use of social media has
been shown to have an influence on worker productiv-
ity, and even break down family ties. As a user of social
media, I ask myself whether I like the current technol-
ogies that people use to interact, whether only through
a cloud, a fixed screen, or typing on the keyboard. And
I conclude no, because the current kinds of communi-
cation contribute to a colder, more isolated, less human
world. Perhaps they are more efficient, but I am not hap-
py. I lived in Beijing for many years. A common scene in
the Beijing subway, which is a crowded but quiet place,
is everyone “communicating” on their mobile phones.
People were often surprised to find, via his posting in
social media, that a friend took the same subway just a
few minutes ago.
What is the meaning of Tangible User Interface(TUI)?
This is the question that Hiroshii Ishii asked us when we
took the Tangible Interface class in the Fall semester. He
shared his own experience and understanding in order
to inspire us to explore our own personal understand-
ing. He is originally from Japan and speaking English
as his second language made him think about the many
ways we could make our communication clearer and
more tangible and specific. Communication, whether
it is from human-to-human, or human-to-computer, is
the purpose of TUI. What advantages can design offer
us in enabling human communication? What would be
the best user interface?
1. Cati Vaucelle, Leonardo Bonanni and Hiroshi Ishii. “Design of Haptic Interfaces for Therapy,” Proceedings of the 27th International Conference on Human
Factors in Computing Systems - CHI 09 (2009), 30.
2. Stephen Walsh “The Cultural Impact of Computer Vision.” Journal of Information Technology 1, no. 4 (1986): 30. doi:10.1057/jit.1986.37.

31Over millennia, as humans have worked with textile,
wood, bamboo, and metal to craft clothing, or drums,
or signal lights, we have developed specialized tools for
specific jobs. For example, people have used bamboo as
a tool of sound transmission, as they did in the past for
Chinese funerals. But in today’s world everything flows
through smartphones and screen-based computer. We
have five senses, touch, smell, sight, hearing and taste.
These five senses guide us to interact with the physical
world to enrich and influence our life experience.
However, in our obsession with our screens, we have put
aside some of our senses. Why have we given up using
our multiple, natural human senses? Usually, when
we interact with an object, we are using more than one
sense. For example, before we enjoy food, we first see
and smell it. We taste it with our eyes and noses and
see it if it looks and smells good. Thus, smell, taste and
sight combine together to create the experience; it is not
isolated to one sense.
Similarly, why are we limited only to sight and sound
(and sometimes touch) when we interact with others
through a screen? For example, when I Skype with my
family, sound and visual interaction alone do not satisfy
my perception. We need more emotional communica-
tion, not just interaction for the purpose of transmitting
information. Technology, I believe, should help make
human beings, and the world we live in, more capti-
vating and more enchanting, rather than widening the
emotional gaps between us.
Human’s Five SensesTangible User Interface(TUI)
As Ishii often reminded us in our class: “Design is not
for computers, design is for humans.”
Our current computer-mediated communication,
especially online social media, focuses on the expression
of opinions and filters out the important elements of hu-
man communication, such as haptic, emotion, physical
space and individuality. Online communication today
is disrupting natural communication of face-to-face
connections. It has isolated the emotional connection
with high speed.
What kinds of user interfaces would be able to make hu-
man communication more natural, simple, and allow us
to overcome language barriers? This is why he created
the Tangible Interface Group at MIT. This is also what
I explore in my thesis. I believe Tangible Interface is
the best way to connect with each other. It uses human
senses and peripheral attention to make information
directly manipulatable and intuitively perceived. We all
want our world to become warm, magical and human,
instead of cold, mechanical, screen-based.
30
Fig. 2
Tangible Interfaces.
source: Hiroshi ISHII
media.mit.edu

32 33
GUI & TUI & Racical Atoms
One of the groups at the MIT Media Lab is called the
“Tangible Media Group,” whose projects have had
a profound influence on my thoughts and projects
created. I took the class in the MIT Tangible Inter-
face Group in the fall semester of 2016. They focus on
exploring the different forms of human-to-human and
human-to-computer interaction, and highlight how to
create multiple experiences of interaction in the digital
world.
“ …Tangible User Interface (TUI)… is based on the
physical embodiment of digital information and com-
putation to go beyond the current dominant paradigm
of Graphical User Interface (GUI).”3
The tangible
user interface gives physical form to digital data and
computation, facilitating the direct manipulation of
information. In other words, it allows people to manip-
ulate digital information directly to perceive its physical
embodiment through their different senses. There is a
suitable example that Udayan Umapathi mentioned in
the class, who is one of the teacher assistances,
From GUI to TUI
In graphic interface, we cannot take advantage of our
skills for manipulating diverse physical objects. With
traditional GUI, we are tied to the screen, the windows
and the mouse and keyboards. These separates us from
the interactions that take place in the physical world,
tangible interface creates a natural connection with
physical devices. Graphical user interfaces represent
information(bits) through pixels on bit-mapped dis-
plays. By decoupling representation from control, GUIs
supports the malleability to graphically mediate diverse
digital information and operations. However, TUIs are
inconsistent with our interactions with the physical
world. “Tangible interface takes advantage of our haptic
communication and our peripheral attention to make
information directly manipulatable and intuitively per-
ceived through our foreground and peripheral senses.”4
Touch has a high emotional impact: touching a cat's
fur, or immersing are fingers into a sticky substance
provide intense emotional and perceptual communi-
cation. An example of a haptic communication created
by the Tangible Media Group is a cup, which can be a
medium of physical perception. It directly transmits
a kiss (movement) between lovers. When a woman
drinks water from the cup, her boyfriend can perceive
her motion as a kiss when he drinks from his cup at the
same time. This is an example of how tangible interface
enriches our human interactions. (Fig.4)
Tangible design expands the affordances of physical
objects so they can supports direct engagement with
digital world. The example below that Hiroshi Ishii
made is a good example to explain. Ishii mentioned that
“Imagine an abacus, the simplest form of a dig-
ital computation device. All of its information
is represented by an array of beads, so users can
directly touch, manipulate, and feel its data.”
Fig. 4 New long dis-
tance lover’s cup.
by Jackie Lee and
Hyemin Chung of MIT
Media Lab
Two users are playing
with the Lover’s Cup,
when they both hold
and drink via the cup
to celebrate a remote
kiss.
Fig. 3
RADICAL ATOMS
TUI
Tangible User
Interface
GUI
Graphic User
Interface
MIT Tangible
Interface Group
3. “Tangible Media Group.” Tangible Media Group. Accessed December 07, 2016, http://tangible.media.mit.edu/vision/. 4. Hiroshi Ishii, David Lakatos, Leonardo Bonanni, and Jean-Baptiste Labrune. “Radical Atoms.” Interactions 19, no. 1 (2012): 35. doi:10.1145/2065327.2065337.

34
35
Fig. 5 musicBottles
By Ali Mazalek, Jay
Lee and Hiroshi Ishii,
(2000).
his mother used to collect small and rare bottles. His
habit inspired him to create a tangible interface through
imagining a user scenario which enabled his mother
to check the weather expected for the next day in her
home city of Sapporo, Japan.
The MusicBottle by Hiroshi Ishii contained weather data
inside glass bottles. Physical manipulation of the bottles
by opening and closing them is the primary mode of
interaction with digital content. (Fig. 5) In addition, the
stopper of the bottle is a trigger (affordance) for guiding
users to show the action, opening and closing with
the hand. Every time the cork of a bottle is removed,
a sound of birds (a signal for sunny day) or the sound
of rain forecasts the weather of a current location. The
unexpected outcome of the bottle, and the bird or rain
sounds, give a sense of “surprise” which surpasses the
original understanding for the affordance of the bottle.
The Bottles project explores the transparency of an
interface that weaves itself into the fabric of every day
life. Seamless extension of physical affordances and
metaphors into the digital domain is a key principle of
the objects’ design, which provides knowledge and un-
derstanding to users learning how to manipulate them.
What I learned. The knowledge and technology that I
studied in DMI and the MIT Tangible Interface Group
has taught me how to transfer a concept into useful pro-
totypes, and how to weave digital technology into the
fabric of the physical environment and make computa-
tion invisible. In my thesis, inspired by the story of my
personal experience, I aim to explore how to use current
technology to create multiple experiences using periph-
eral senses and telepathy to foster human-to-human
connections, especially in remote communication.
My thesis promotes two strategies as powerful tools to
enrich human beings’ emotional experiences through
object mediated communication: the metaphoric and
haptic. Both of this start from a broader idea -Nonver-
bal Communication- which supports a broader concept
for includingbody movement as a language of dynamic
media that can manipulate the tangible interface in
interpersonal communication.
Contextual Research

37
The Definition
36
Nonverbal Communication
When we interact with others, we contin-
uously give and receive wordless signals.
All of our nonverbal behaviors—the ges-
tures we make, the way we sit, how fast
or how loud we talk, how close we stand,
how much eye contact we make—send
strong messages. These messages don’t
stop when you stop speaking either. Even
when you’re silent, you’re still communi-
cating nonverbally.5
The man at a crowded lunch counter who looks straight
ahead, or the airplane passenger who sits with his eyes closed,
are both communicating that they do not want to speak to
anybody or be spoken to; their neighbors usually “get the mes-
sage" and leave them alone. Thus, “One cannot not communi-
cate. Activity or inactivity, words or silence all have message
value.” 6
Wordless communication in human beings is defined
nonverbal communication.
There are many different types of nonverbal communication.
Together, the following nonverbal signals and cues communi-
cate your interest and investment in others: Facial expressions,
the human face is extremely expressive and able to express
countless emotions without saying a word; Body movements
and posture, the way you move and carry yourself communi-
cates a wealth of information to the world; Gestures, gestures
are woven into the fabric of our daily lives. We wave, point,
beckon, and use our hands when we’re arguing or speaking
animatedly—expressing ourselves with gestures often without
thinking. In the arts, dancers and conductors use hand
gestures to transmit information and control the musical
performance; Touch, all materials provide tactile and thermal
feedback from smoothness, roughness, softness and warmth.
Jurgen Ruesch, who is an american psychiatrist states “object
language” is another nonverbal forms of codification in her
book, Nonverbal Communication.
“Object language comprises all intentional and non-intention-
al display of materials things, such as implements, machines,
art objects, architectural structure, and human body and
whatever clothes or covers it.” 7
“
” 5. Paul Watzlawick, Janet Beavin Bavelas, and Don D. Jackson. Pragmatics of Human Communication: A Study of Interactional Patterns, Pathologies, and
Paradoxes. (New York: Norton, 1967), 78.
6. Ibid, 90.
7. Jurgen Ruesch and Weldon Kees. Nonverbal Communication: Notes on the Visual Perception of Human Relations. (Berkeley: University of California Press,
1956), 40.
Contextual Research

38
39
Fig. 6-7
SmartStone
Andreas Forsland
To send different
messages by do-
ing corresponded
hand drawing via
the SmartStone.
need you” or “It's an emergency” or “I am doing fine.”8
The SmartStone gave me a clue about how to apply
nonverbal expression as a powerful tool to build remote
connection tools for human communication.
Furthermore, it motivated me to think about what kinds
of situation are good for using non-verbal communica-
tion rather than verbal, and what kinds of hand move-
ments can convey and transmit information to another.
I also imagined that I could substitute other metaphor-
ical objects for the stone, which could convey special
meaning for interpersonal communication. Through
my research and writing, I will explain the reason that I
focused on mediated objects as powerful tools in human
communication, and how to apply it in my case studies.
Physical Telepresence. Physical telepresence was created
by the Tangible Interface Group in 2014 by Daniel
Leithinger, Sean Follmer, Alex Olwal and Hiroshi Ishii.
They introduced the concept of computer-mediation
for physical telepresence that allows remote people and
objects to communicate and collaborate.
“Physical telepresence allows users with shared work-
spaces to capture and physically render the shapes
of people and objects in real time, remotely.”9
A table
consist of an array of 120 individually addressable pins,
whose height can be actuated and read back simulta-
neously, thus allowing the user to utilize them as both
input and output. Users can interact with the table using
Of all these forms of non-verbal communication, I have
chosen to focus on object mediated communication
with a metaphorical meaning and haptic communica-
tion.
Two Examples
SmartStone. I was originally inspired by an example of
nonverbal, hand manipulated technology. SmartStone
is an example of how gesture and body language can be
used as powerful tools of communication using current
technology. SmartStone uses miniaturized technology to
allow users to use simple hand gestures to communicate
or get emergency help by manipulating a stone-shaped
object. It is already sold on Amazon and is fit for remote
communication. It is a simple, smart solution that lets
your family and friends know that you're well.
The durable “stone,” a palm-sized device, is packed with
LED lights, a gesture recognition library, touch inter-
face, gyroscope, speakers and Bluetooth connectivity.
Its interface connects with an app that allows users and
loved ones to program it to respond to specific gestures.
For example, the stone could be programmed to text a
loved one “Thinking of you” when tapped twice. The re-
ceiver of the message could enter a response, which the
SmartStone would translate into biofeedback, blinking
light or vibration. There is a communication language
of 12 basic hand gestures for users to manipulate the
object, allowing them to send such messages as “Hey, I
Fig. 9
Daniel Leithinger
Sean Follmer Alex
Olwal† Hiroshi
Ishii, MIT media
lab.
System diagram of
the tele-manipula-
tion system using
shape capture
and remote shape
display.
Fig. 8
Deneal Levine
playing with it
as user testing.
Source Yanjun lyu.
9. Leithinger, Daniel, Sean Follmer, Alex Olwal, and Hiroshi Ishii. “Physical Telepresence.” Proceedings of the 27th Annual ACM Symposium on User
Interface Software and Technology - UIST ‘14, 2014. doi:10.1145/2642918.2647377.
8. Rose Ellen O’connor. “Palm-Sized Device Lets People Who Can’t Speak Communicate.” Notimpossiblenow.com. Accessed December 09, 2016.
http://www.notimpossiblenow.com/lives/palm-sized-device-lets-people-who-cant-speak-communicate. doi:10.1145/2642918.2647377.

41
40
Fig.10
Physical Telepresence
Mit Media Lab.
“It provides physical em-
bodiment, remote manipu-
lation and new capabilities
through computer-medi-
ated teleoperation. Here,
local and remote users
physically interact with a 3D
car model.”10
Fig. 11
An user manipulating
a sphere through his
remote physical
embodiment.
their gestures or through direct manipulation. There are
different versions. For example, remote participants in a
video conference can smoothyly move a physical object
in a remote location through displaying different hand
movements. Likewise, “Cityscape” connects to a pin
screen that can allow remote users to “walk through” a
3D model of a cityscape. (Fig. 10-11)
Physical Telepresence supports a clue about how to
translate body movement into a paritcualar forms in
remote connection. The meaning of the translation crys-
talized our information with the phyical version, which
not only made the remote communication more effec-
tive and clearly, but also it is broke the limitation that
traditional video chatting made in remote connection,
it is made our information and imagination become a
physical forms. Imagined an remote meeting scenarios,
they are able to use the physical telepresence to display
a dynamic changing of the cityscape, or their ideas
with a clear structure in real time, instead of a blurred
statements or silent photos.
What I learned. SmartStone gave me a clue to think
about a mediated object with a metaphor to build the
connection in the distance. Telepresence gave a direct
stimulation to think about physical embodiment,
remote manipulation and new capability through com-
puter-mediated teleportation. The telepresence also gave
me a teleology idea, which use kinetic to capture the
hand movement and send signal to the object through
the linkages, which gave me a sample to build my forth
prototype in case study.
In addition, whatever the SmartStone or Telepresence,
they both encouraged me to think about how the rep-
resentation of user’s body parts, especially hands can be
altered to amplify the capabilities in remote connection.
Thus, the manipulation of hands for nonverbal commu-
nication became one of the most significant “control-
lers” in my thesis.
10. Daniel Leithinger, Sean Follmer, Alex Olwal and Hiroshi Ishii. “Physical Telepresence.” Proceedings of the 27th Annual ACM Symposium on
User Interface Software and Technology - UIST ‘14, 2014. doi:10.1145/2642918.2647377.

43
Definition
Metaphors can create sharp mental images that can be
used to great effect in everyday communications and
thinking. What kinds of problems lend themselves to
metaphorical expression for solutions, and how are new
ideas produced in metaphorical thinking? Both ques-
tions shape how I am going to explore my understand-
ing of metaphorical communication.
An Example of metaphorical communication. I had a
meaningful talk with my technology supporter at Mass
Art, Fred Wolflink. He shared his experience of meta-
phorical connection between his mother and him. His
mother is 89 years old. Accurate communication and
speech can be difficult for an elderly person, so Fred has
to interpret her statements about daily life and health by
observing the surroundings, including her refrigerator.
Some food has usually already expired when he visits, so
he throws this out and brings fresh food the next time
he comes. Sometimes he also fixes small items, such as a
broken handle on a pot, or a loose wire in the lamp. As
Fred said:
“A metaphor is a direct comparison between
two unrelated or indirectly linked things.”11
“With the amount of time that I work on her food, clean
her fridge or fix some dangerous everyday object, it is
easy to distinguish her health and mood. When she
hasn’t organized and cleaned her fridge for a long time,
then I have to tell her doctor and try to talk with her.
The spoiled food is a clue to my mother’s health.”
11. Mind Tools Editorial Team “Metaphorical Thinking: Using Comparisons to Express Ideas and Solve Problems.” Creativity Techniques from Mind-
Tools.com. Accessed December 07, 2016. https://www.mindtools.com/pages/article/newCT_93.htm.
The fridge is Fred
Wolflink’s mom. He checks
his mom’s fridge once a
week, and usually fixes
some broken furniture .
Photo by Fred wolflink
at 2016.
42
Object-Mediated Communication with Metaphor
Contextual Research

44 45
I believe most people have similar life experiences, in
the sense that it is easy to know people’s characteristics
through their movements or the objects surrounding
them, even if you never talk with them.
I am interested in types of nonverbal communication
and the emotional expression associated with it, and
why people need it. Fred is an individual who is not
good at directly expressing emotion to people. As I have
worked with him in the past two years, we haven’t had
much conversation, but he has almost always prepared
everything to help me before I would start to work with
him. His nonverbal encouragement has given me lots of
help, especially for a beginner. Through his behavior to
his mom, you can understand he is not a talkative per-
son. The emotional expression of this kind of communi-
cation may be silent, but makes people feel warm from
observing what they did. The object, of course, is the
transmitted signal of emotional context in the process of
nonverbal communication. As Wolflink said:
“When my mom returns home and
opens the fridge, she will know I
came.”
I can perceive and imagine this is a warm moment
for his mom when she sees the fresh food. I had the
same feeling when I saw the growing plant that my
grandfather took care of for two years. Both the food
and the plant are the “transmitted signal” of emotional
expression between people. These objects function as
emotional metaphors, as well as the communication
medium between two people when verbal expression
is difficult.
1. First, metaphorical communication can help peo-
ple indirectly express subtle emotions that are hard to
directly speak to one another.
2. Second, metaphorical objects usually encompass
people’s memories and imagination, which is a power-
ful artists’ tool for enhancing emotional connection.
In the case of the Beijing-Rome couple, the Online
game became their metaphorical object. One type of
metaphorical object is the “enchanted” object.
Enchanted Object
From everyday object to “Enchanted object.” I first
encountered the term enchanted object when I read
David Rose’s book, Enchanted Objects: Design, Human
Desire, and the Internet of Things. His book explains
how everyday objects function as a tangible connect-
ed interface to realize and enhance the extraordinary
capability of human senses in communication. Objects
of daily use in our lives help people improve the hu-
man-to-human and human-to-computer connection in
various situations.
His purpose is to achieve telepathy, teleportation, or
a remote transmission through so-called enchanted
objects. His concept is to think about how to make ordi-
nary objects become extraordinary objects that satisfies
Fig. 12
Enchanted Object,
David Rose.

46 our emotional needs and our need for communication.
Everyday objects such as a table, a wallet, a car, a shoe,
a pillow or a cup govern our lives. Everyone needs such
things—the horn that can summon help across vast
distances, shoes that enable people to walk without
hurting their feet, the cup that allows individuals to
drink. Such objects are familiar to us, and their purpose
coincides with our daily goals. But what if the ordinary
thing is then augmented and enhanced through the
use of emerging technologies such as sensors, wireless
connectivity, or computing? Then it becomes “extraordi-
nary.” As Rose states:
“The enchanted object then gains some remarkable
power or ability that makes it more useful, more delight-
ful, more informative, more sensate, more connected,
more engaging, than its ordinary self.” 12
The example of the wallet illustrates what an “enchanted
object” is. As an everyday object it holds cash and credit
cards, but as an enchanted object it has a wireless link
to a user’s Online bank. Users can receive biofeedback
when going over their budget or overusing credit. The
enchanted wallet looks like a regular wallet and works
like a wallet but is much more than just a wallet.
Eample: SuperShoes
Another example which Rose cites, called SuperShoes,
is not only functional and useful, it records a user’s
steps, provides a user with direction using a GPS which
Fig. 13-14 Super
Shoes
by Dhairya Dand.
MIT Media Lab.
Fig. 15 Movie, Her
Spike Jonze
Shortcut for the
scenario:
Installing Samantha
12. David Rose Enchanted Objects: Design, Human Desire, and the Internet of Things. (New York, NY: Scribner, 2014), 40.
”
sends physical vibration signals to feet, and measures
heart rate which is then immediately sent to his or her
doctor. In addition, SuperShoes allow the wearer to put
down their mobile device and “immerse in the beauty of
reality.” 13
(Fig. 13-14)
The enchanted object, which achieves a beautiful fusion
between the digital and analog world, also offers oppor-
tunity for emotional interaction between users.
Biofeedback as Emotional Response
What is physical, emotional feedback and why it is so
important for human communication? The movie Her
(2013) is American drama film written, directed, and
produced by Spike Jonze, which is about a man who
falls in love with a female-voiced intelligent computer.
The computer can understand his hidden thoughts and
communicate with the emotions in the depth of his
heart through accurately programming with code. Then
the computer responds with a soft, smooth, female,
patient voice.
The movie reflects on human emotional needs and
invites us to consider how people use today’s technol-
ogy to communicate with emotion and humanity. The
man who falls in love with the machine is ridiculed, but
the film shows us that people need emotional response
to be psychological satisfied. Enriching users’ subtle
emotional experiences and satisfying their psychological
needs, rather than just simply fulfilling a function, is an
important aspiration for the future of human-computer
interaction.
13. “Dhairya Dand | SuperShoes.” Accessed December 07, 2016. http://dhairyadand.com/works/supershoes.
14. Rose, David Enchanted Objects: Design, Human Desire, and the Internet of Things. (New York, NY: Scribner, 2014), 56.
“Maybe today’s objects aren’t dis-
tracting and don’t attempt to probe
into deep emotion, but they do en-
courage connections and enable the
surfacing of emotional content with
people you care about.14
Rose, David
”
47

48 49
When I read it, my magic glasses pop up in my mind,
which I imagined and drew in Gunta Kaza's Design
Experience class at Massachusetts College of Art and
Design in 2014. The glasses helped me to peer into the
mood of the boy with whom I had secretly been falling
in love with for a year. I wasn’t brave enough to express
my emotion to him; I couldn't even look him in the
eyes. The glasses helped me understand his hidden
emotions and transmitted the signal to me so that I
could better plan when and how I would interact with
him, and even alter my behavior in response. I felt more
ready to offer support and understanding during his low
moments and challenging times.
Although this idea never came to fruition, I was
motivated to think about the importance of emotional
communication between people. What I hope to do
is to have a better understanding of hidden thoughts
and emotions that are relevant to us and would be
substantial for us to know, and to be able to tune into a
mutually beneficial telepathy based on currently tech-
nology. I hope that enchanted objects will play a role in
the process of communication by remapping social data
into objects that are a part of our daily routine and by
enabling the awareness of people we care about through
receiving transmissions of emotional signals.
Eample: Enchanted Jackets
The “enchanted jacket” created by MIT Tangible Media
Group explored how to help people add emotional
experience through biofeedback from everyday objects
based on social networks. “People send and receive
signals with our postings, photos sharing, tweeting and
testing. However, the problem is, although we share too
many thoughts with far too many friends, we don’t care
about them as close friends”15
Indeed, we lived in a blizzard of text and images, but
we still feel we will slowly lose the intimate feeling with
friends who live at a distance. The jacket is trying to
remedy the emotional gap caused by distance. Its wearer
connects with Facebook, and the jacket, with the aid of
electronic sensors, is inflated to give the wearer emo-
tional biofeedback in the form of a hug when a friend
likes their Facebook post. The real hug makes the wearer
directly feel the concerns of others. (Fig.16)
Today’s technology helps people transform virtual emo-
tion into conscious experience. There is a fascinating
imagined scenario in David Rose’s book:
“You had an enchanted wall in your room that could
display your loved ones’ moods or hidden emotion
through lines of colored light, the trends patterns… If
you could understand that your spouse or child had a
regular pattern of emotions and began to see how they
connected to the environment, times of day, events,
even your moods. A more efficient caregiver? Would it
make you a more attentive partner?”16
Fig. 16 Enchanted Jackets
MIT Media Lab
15. Rose, David. Enchanted Objects: Design, Human Desire, and the Internet of Things. (New York, NY: Scribner, 2014), 34
16. Ibid, 38.

50 51
Object Expression
“The principal medium of communi-
cation is the shape of an object, but
the color, texture, material and other
visual properties also play a major
role. The totality of these object prop-
erties makes an impression on us and
exerts emotion.”17

Fig. 18
Expressive KineticOb-
jects. Hiroshi Ishii and
Andrew Dahley (1998).
MIT Media Lab.
What is an object expression? Objects convey meaning
through physical design attributes such as their form,
color, or materials. Artists and designers are sensitive
to these aspects of perception and express the character
of the objects to evoke meaning in their creations. The
form of the object is the first dimension of a design we
perceive and is also one of the first features expressed by
designers. We can readily understand information and
emotive expressions through the form of objects.
Emotive Modeler is a project by a group of experts in
the design language of objects, called the Object Based
Media Group, developed at MIT in 2013. They built on
existing research into emotive shapes through analyzing
the attributes and features of basic physical objects such
as jars and vases. “The Emotive Modeler UI (user-inter-
face) accepts an inputted ,word, analyses the emotional
associations to that word, and combines the relevant
design elements to generate a 3D model.” 18
So one
can input the word “angry,” for example, and emerge
with one or more 3D renderings of an “angry” vase. It
translates words into visual language by manipulating
the forms of the object. (Fig.18)
The Emotive Modeler is an interesting exploration of
explicit visual language. All languages in the world have
logical grammar. To an interactive designer, building the
grammar of the object language means guiding users to
interact with the works. In other words, how do you use
object expression (visual symbolism) to convey and lead
users to interact with it?
What I learned. The Emotive Modeler project gave me
a clue about how to embody an abstract object with
specific meanings. It used form, size, color and texture
as different transmitted signals to communicate. My first
prototype of the virtual object explored the relationship
between emotional hand movements and the various
forms of an abstract object.
17. Schade, Brittany Diane. Emotion and the Designed Object. 2013.
18. Philippa Mothersill and V. Michael Bove. “The EmotiveModeler.” Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in
Computing Systems - CHI EA ‘15, 2015. doi:10.1145/2702613.2725433.
Fig. 17
Emotive form
design taxonomy
for eight primary
emotions. Anal-
ysisa and design
attributes for
multi-emotion
words.
MIT Media Lab

52 53
Fig. 19
Ambient dialogue
Alexander Wang
DMI, Studio, Mass Art.
Ambient Dialogue. lightOn is an interactive lamp for
people in different locations, conceived by Alex Wang.
His work focused on the transmission of emotional
content between individuals in separate locations. His
lightOn project was based on network-connected lamps
communicating users’ messages by turning a pair of
lights on or off. Two lightOn lamps, set up in different
cities, are connected and respond when either lamp in
the pair is turned on and off. As he said in his thesis,
“Light not only illuminates the en-
vironment but also carries the warm
message that someone is waiting for
you”.19
Light is a beautiful metaphor for the quality of inter-
personal connection in his project. He uses light as a
metaphorical emotional connection, which makes users
feel warm and safe. It transmits the signal of “I’m home”
to his wife.
Wang expanded the diversity of lightOn’s emotional
transmissions, adding color and pulses of light to its
repertoire. The flicker frequency of light, or the changed
color of the lamp carried different visual messages. As
he stated in his book: “The changed color and frequency
of light become the symbol of someone’s presence”.20
The light was used as a symbol—in Wang’s case, it
symbolized love between him and his wife. I appreciate
his notion for metaphorical communication, and love
his choice of light to symbolize the relationship between
a couple. Turning on a light is an indirect expression of
their love. His concept pushed me to think about how
to create the metaphorical connection in a certain sce-
nario. In his case, his scenario is when he and his wife
return home, they both first turn on the light.
19. Alexander Wang “Ambient Dialogue.” “Ambient Dialogue.” http://www.dmistudio.org/thesis/2011/Wang_Alex.pdf.
20. Ibib.
Related Works
of Metaphorical Communication

55
Definition
Haptic communication refers to the ways in which
people communicate and interact via the sense of
touch. Touch is imperative for humans and is vital in
conveying physical intimacy. Haptics is a component
of communication in an interpersonal relationship
that is nonverbal and non-visual. In comparison with
metaphorical communication, the use of haptic commu-
nication in my projects harnesses the tactile experience
of hands to express emotion or get information.
Haptic communication, to me, is a type of psychological
connection for users who communicate with others
in a different location through touch. It is more direct,
non-visual and more “magical” than currently limited
technologies in general use.
Touch Telepathy and Communication
The human hand, as well as being a principal vehicle
of motor activity, is the main device of the fifth sense,
touch. Along with the eye, the hand is our primary
source of contact with the external environment using
the our skin. It can examine the environment using
touch, plus it can perceive things around corners and
operate in the dark.
In addition, the hand is part of the human communi-
cation system. It is used to convey not only words but
emotions and ideas. You probably already are familiar
with how a hug from a loved one can lower your blood
pressure and make you feel valued and important, or
21. Malcolm McCullough Abstracting Craft: The Practiced Digital Hand. Cambridge, MA: MIT Press, 1998, 4.
”
Haptic Communication
54
Hands are underrated. Eyes are in
charge, mind gets all the study, and
heads do all the talking. Hands type
letters, push mice around, and grip
steering wheels, so they are not idle, just
underemployed. 21
“
“

56 57
give you comfort, without any words. A firm handshake
with a friend or family member can create a psycholog-
ical connection, transmitting high and positive energy
through hands alone. A friendly hand touch is able to
solve a misunderstanding for close friends; without
words, they can understand each other.
There is a Chinese proverb,十指连心, which expresses
the blood relationship for parents and children. Its
literal translation is your ten fingers are linked with the
heart of your parents. People sincerely believe that every
small part of the body is inseparable from the heart of
parents. When one part of one’s body is hurt, a parent
can perceive that pain just with the heart, without using
language or any other explicit method of communi-
cation. Thus, in Chinese culture, people do believe in
telepathy especially for blood relationships.
But beyond that, another wordless communication with
touch is for physicians and patients. Following tradi-
tional Chinese medical science, a doctor takes the pulse
with his or her hand to assess general body health. Pulse
diagnosis helps physicians determine other information
about the body, from pulse speed, intensity, and rhythm
(even or not). (Fig. 20)
“Modern pulse condition study confirms that the forma-
tion of pulse condition mainly depends on the function
of the heart, blood vessels, and the function, quality and
quantity of the blood.” 22
The heart is the power organ
which forms pulse condition so that the pulse reflects
the pathological changes of the heart in the first place;
then doctors can infer another disease based on that
collection of data through touch. In addition, touch is
also the most important tool for helping people to build
mental and physiological connection.
Touch as a fundamental building block to our emotional
health is the premise for various touch-based treatments
for mood, anxiety and mental disease, which collectively
are called touch therapies. Psychologist Harry Harlow
demonstrated that
“Maternal touch was critical to development of infant
rhesus monkeys. Those who could see but not touch
their mothers developed erratic behaviours.
some artificial materials can be more effective at
simulating maternal touch than others, for example,
the infant monkeys had more affectionate responses to
a surrogate mother made from soft cloth than to one
made from hard steel wire.” 23
(Fig.21)
The article, “Infant’s need for touch” shows that mater-
nal stimuli can improve cognitive function and stress
resilience.
“Because an infant’s brain is still building connections
in these communication zones, large blasts or long-term
amounts of stress can permanently limit full develop-
ment, increasing the risk of anxiety, depression and
dementia later in life.” 24
“Thousands of orphans in communist Romania were
raised in overcrowded facilities where the absence of
Fig. 20 Traditional
Chinese Medical
Science Diagnosis.
Whether your pulse
races or slugs along, a
practitioner can deter-
mine your health.
By Bill schoenbart and
Ellen shefi.
Fig. 21 Wire and
cloth mother surro-
gates Rhesus monkey
from Harry Harlow’s
studies. http://www.
cerebromente.org.br/
n13/experiment/affec-
tive/monkey2a.jpg
22. Oren Traub. “Shock.” http://www.merckmanuals.com/home/heart-and-blood-vessel-disorders/low-blood-pressure-and-shock/shock.
23. Harry Harlow and Margaret Harlow. “The Young Monkeys.” PsycEXTRA Dataset. doi:10.1037/e400042009-004.
24. Ibid.

58 59
comforting touch led to the development of serious
social and emotional problems.” 25
Thus, touch as a form of non-verbal communication has
an irreplaceable function in emotional transmission and
mental therapy.
Related Works of Haptic Communi-
cation
In developing my projects, I focused on how to convey
emotional content in communications between people
in separate locations, trying to create a kind of illusion
of consciousness that could allow people to feel the
warmth of their families, wherever they are. I found
inspiration for various ideas from the work of others.
Eample: InTouch. For example, Scott Brave and Andrew
Dahley at the MIT Media Laboratory created InTouch,
which is a tangible medium for haptic interpersonal
communication. (Fig. 22) They reported that
“Several machine interfaces have been developed to
simulate touch, for example, to use in training surgeons
and pilots, for sculpting digital models and even in
video game controllers.” 26
The idea behind inTouch is to build an illusory human
connection between two individuals, demonstrating
that “a sense of human presence can be established
through haptic communication.” 27
They created two
connected objects, each consisting of three rollers,
each of them corresponding to a roller on another side.
When one of them is rotated, the corresponding roller
on the remote object will rotate in the same way.
“The richness of the interaction then comes not from
the representation of form, but from the representation
of movement as mediated by the coupled objects…
With inTouch, the idea is not to create a device to
represent the physical form of the user at the other end,
but rather to create a physical link for expressing the
movements or gestures of that person.” 28
Eample: TapTap. The majority of haptic interface
technologies merely provide abstract feedback in real
time from one user to another. However, Taptap, from
MIT’s Tangible Media Group, is a haptic interface
which can record and play back the touch of a lover,
a family member, a therapist. The sense of touch is
essential to emotional health and development. “Taptap
is a wearable accessory that can record, distribute and
play back affectionate touch.”29
Designed to look like
a normal scarf, taptap provides extra warmth and
comfort by simulating the touch of a loved one. One
such wearable haptic device has been developed to assist
people with balance problems. It consists of a vibration
or biofeedback to indicate which way is beneficial for
users. (Fig.23-24)
Fig. 22 InTouch
By Scott Brave and
Andrew Dahley
Source Yanjun Lyu
Fig. 23-24 Taptap
Leonardo Bonanni, Jeff
Lieberman, Cati Vau-
celle, Orit Zuckerman
2005. MIT Media Lab.
A scarf Prototype
The flexible i/o haptic
insert tucks into the
felt scarf (left) and con-
nects to central power
through conductive
steel snaps (right).
25. Ibid, 23.
26. Scott Brave and Andrew Dahley. “InTouch.” CHI ‘97 Extended Abstracts on Human Factors in Computing Systems Looking to the Future CHI ‘97,
1997. doi:10.1145/1120212.1120435.
27. Ibid.
28. Ibid.
29. Leonardo Bonanni, Cati Vaucelle, Jeff Lieberman, and Orit Zuckerman. “TapTap.” CHI ‘06 Extended Abstracts on Human Factors in Computing Systems -
CHI EA ‘06, 2006. doi:10.1145/1125451.1125573.
Contextual Research

60 61
1 Physical Plant
2 Digital Platform
• • Abstract Clay
• • Virtual Plant
3 Haptic Glove
4 Appendix: HumInterface

62 63
Overview
Ideveloped three main personal projects, and a experienced
teamwork project as my appendix in the case study. They were all
designed to build on the concepts and experimental technologies of
metaphorical and haptic communication.
The first prototype was a “Physical Plant,” which aimed to explore
how to use tangible interfaces to send and receive signals and create
emotional connection. The second projects aimed at enabling
metaphorical communication were “Virtual Plant,” which consisted
of two prototypes that changed from abstract object into a specific
plant. The third project explored tactile communication within
intimate relationships across distances, which took the form of
haptic-technology gloves that allow users to communicate remotely
using hand gestures within different scenarios. The fourth project as
an appendix helped me to explore kinesthetic communication, and
how to transfer body movement into a physical object in telepres-
ence. Whatever the communication format, in my case studies I
mainly aimed to replicate physical sensations that people experience
when they are close in order to enhance psychological connection
using bio-feedback, temperature and touch.
Case Study

65
Fig. 25
DMI 15th Celebra-
tion Show
Photo by Yiji He
Physical Plant
As a Tangible Mediated Communication Tool
Keywords: Tangible Interface / Nonverbal /
Remote communication / Emotion multi-sensory
communication
Case Study
64

66 67
Overview | Physical Plant
The Communicational Plant project is an interactive
interface for interpersonal, nonverbal, remote commu-
nication. Compared with the digital plant project (dis-
cussed at next chapter), which created a virtual space for
remote communication, the physical plant project not
only emphasized emotional communication through a
tangible interface, but also it explored how to use cur-
rent technology to connect two objects which exchange
physical data and transform that data into multi-sensory
communication, through sight, touch, and the multiple
experience and perception.
Fig. 26
User, Bing Lin is
playing with the
plant.
Case Study

68 69
Fig. 27
Conceptual sketch
by hand drawing
at 2016.7.8.
Mediated object,
Physical Plant as a
“watcher” assists
me in observing
his health.
I had to return to Beijing during the 2015 winter
holiday, when my grandfather was in the hospital for
heart disease and could hardly speak. Privately, I was
angry with my grandfather and my family because they
had hidden the truth of my grandfather’s health when I
video-conferenced with them from Boston. I under-
stood their “white lies,” but I would have liked for them
to be honest. My grandfather’s health situation inspired
me to create a link that would belong to my grandfather
and me, which would enable me to know his status
firsthand, in real time, not secondhand through others.
For an elderly person who is already over 85 years old,
a phone call or video chat may not be the best way
for him to ask for help directly. When I chat with my
grandparents, I mostly have to pay attention first to
their details of life before my grandfather has a chance
to say whether he has been feeling unwell. Usually my
grandparents will report that their health is good when
my parents ask them, even if my grandfather is not well.
Some types of link would come in very handy for my
parents and me to see their situation immediately so we
could help. Such a link would also help my grandparents
to build a connection with me when we are separated
by distance.
The plant, an everyday object, is an useful communica-
tion tool for my grandparents and me. My grandfather
has always been crazy about taking care of different
types of plants, and has always been proud of his flower
garden. He usually spends a long time each day tending
his plants. I left my own plant with him in 2014, when
I left to study in Boston. He told me that he placed his
hope and love for me in this plant by taking care of it
and watching it grow, and it made him feel that I was
always there with him. Knowing that he was watching
over my plant gave me the same feeling of connection to
him. This led me to envision how I could use the plant
as a tool to observe his daily health: if the plant did not
thrive, indicating that he was not tending it, then the
plant would automatically send a message to me and my
parents.
I designed and implemented three scenarios to explore
how the plant might be used in as a tangible interface in
three ways:
1 To enrich the experience of interaction in human
communication.
2 To improve emotional communication at any given
time.
3 To offer a way to use programming language to
analyze physical data and provide visual, nonverbal
feedback.
How This Idea Begin
Case Study

70 71
70
Arduino board
is a microcontrol-
ler-based kit for build-
ing digital devices and
interactive objects that
can sense and control
physical devices.
Processing
is a programming lan-
guage built on top of
the JAVA programming
language.
Fig. 28
My Lily located in the
east-facing window.
The moisture sensor
connected with an
Arduino board for col-
lecting moisture data,
which is displayed on
the blue screen.
I chose a physical plant to serve as a prototype for my
project on remote communication. My grandfather has
a lily that he keeps in the window on the west side of the
house, while I keep my lily in an east-facing window,
thousands of miles away. (Fig. 28) We had to learn to
communicate with each other through our lilies. The
three scenarios below represent the challenges that I
experienced in communicating with my grandfather.
The First Scenario. My grandfather usually likes to
stick his fingers in the soil to see if the plant needs to be
watered or not. My plant had a device, Neopixel, which
consists of eight LED lights. I had a soil moisture sensor
installed in the second plant, which represents my
grandfather’s plant. (He did not actually have “his” plant
in Beijing during this experiment; I had them both, as I
needed to control and learn from the prototype.)
The moisture sensor connected with an Arduino board,
which is a microcontroller-based kit for building digital
devices and interactive objects that can sense and
control physical devices. For programming the micro
controllers, the Arduino project provides an integrated
development environment based on a programming
language named Processing. Processing is flexible
software, a language for learning how to code within the
context of the visual arts the Arduino board translated
and transmitted the moisture data to the Neopixel. I
programmed different LED colors to represent different
levels of soil moisture. For example, dark red meant
too dry, bright red meant slight moisture, green meant
User Scenarios
Fig. 29
Sketches of three
scenarios. Letter A.
B means different
cities.
Fig. 29

72 73
72
optimal moisture, light blue meant too wet, and dark
blue meant the plant had been over-watered. Thus, the
moisture sensor in my grandfather’s plant controlled the
changes in the LED lights in my plant. (Fig. 30)
When I watered my grandfather’s plant and the green
light came on in my plant, which meant the soil had
reached optimal moisture, I stopped watering. In this
scenario, if my grandfather were taking care of his own
plant, he would also be teaching me how to take care of
mine through by way of sensors and motors. That visual
signal would announce his health to me each day. I felt
very comforted.
The Second Scenario. The characteristic of a lily is that
if no one watered it for a week, the leaves’ color would
change from green to brown. So if grandfather did not
water it for a week, the plant’s leaves would change
to brown, and even begin to fall. I installed a camera
on the second prototype (my grandfather’s plant) to
observe leaf color, which showed to me and my family
that, through the healthy state of the plants, my grandfa-
ther was doing well. The idea is that if the leaves turned
brown, his lily would automatically send an emergency
text message to us, which would be “CALL HIM RIGHT
NOW.”
(Ideally, when the camera would capture the brown
color, it would send that data to Processing, which
would connect with my mobile device—however, I did
not actually implement this step. )
Fig. 31
NeoPixel Stick - 8 x
5050 RGB LED
In the meantime, a motor would vibrate my plant as
an emergency signal to remind me to call his doctor
immediately. If I didn’t take care of my plant for a week,
its leaves would turn brown, and I might receive a
different text message from his plant, “WHAT’S GOING
ON?”(Fig. 33)
The Third Scenario. Generally, lilies bloom in April and
May. My lily bloomed for two weeks, starting May 15.
I imagined a scenario in which my grandfather would
see a virtual flower projected onto his plant at the end of
May, which would signal that I would return to Beijing
in a few days because my spring semester classes would
have finished.
To accomplish this, I set up one projector, two cameras,
and two computers (one connected to each of the two
prototype plants). The camera connected to Processing
software, which was installed on both computers. The
camera would capture the color of the pink flowers of
the plant in bloom, converting it to an RGB color value.
The pink color number is between red (red’s number is
greater than 120), and green (green’s number is less than
200). Thus, when the camera sees a color of flowers in
the range of 120 < pink > 200, the Processing program
would receive and send the signal to the other Process-
ing program in the second computer.
I pre-programmed an image of a lily blooming, so that
when the second computer received the signal sent
from the first computer, the image would pop up on its
screen. The second projector, which was connected to
Fig. 32
The leaves are
withered and
color are changing
from healthy
green to brown in
one week.
Fig. 33
The user received
a message from
the sender’s plant.
The message is “
What’s going on?”
Case Study
Fig. 30
The first scenario.

74 75the second computer, would project the flower image
onto the real plant so that people would see the project-
ed image on the plant.
I planted lily bulbs in pots in autumn in anticipation of
spring blooms. For the location of my pot I wanted to
select a site that gets full sun. To ensure that the plant
develops blooms, lilies need six to eight hours of direct
sunlight a day. If it’s too shady, the stems will attempt
to lean towards the sun or get spindly and fall over, and
there will be no blossoms.
Observing the direct sunlight position. To determine
the amount of sunlight in two possible locations, I
prepared a light sensor to monitor the sunlight of sum-
mer, and connected it with an Arduino board, which
sent the sunlight data to a screen. The data indicated
which position would be best for receiving full sunlight
throughout the day, and what time of day the sunlight is
most intense. (Fig. 36-37) I graphed the results:
From 8:00 am to 10:00 am, the west window had the
most sunlight. From 11:00 am to 1:00 pm I put the
pot near a north window, because it had the most sun
during those hours. From 2:00 pm to 3:00 pm, the east
window offered the sunniest position. For four days
(June 15-18, 2016), from 8:00 am to 5:00 pm each day,
I recorded the sunlight, moved the pots, and kept a re-
cord of the lily’s soil moisture. The location was Boston,
at a latitude and longitude of 42.3601* N, 71.0589*W.
Fig. 35
The code is about
sending the pink
follower data from
Processing to another
on another computer
through changing the
local-host IP address.
As long as one of the
computers captures
the pink color, the pink
followers will pop up
on the screen.
ColorSortingOSC
(Open Sound Control)
is a content format for
messaging between
devices and software
over a network con-
nection.
Watering data visualized as lights. While the plant was
actively growing, it needed to be watered frequently.
(In the winter, when it is dormant, I kept the soil lightly
moist.) In June 2016 I inserted the moisture sensor into
the soil and connected the wire to the Arduino hard-
ware. The Arduino board received and sent moisture
data to a screen-based device, assigning a number value
according to how much moisture was detected in the
soil. Those values were assigned as follows: dangerously
dry (out of water):
Over 940; somewhat dry (in the soil): 610 ~ 930; op-
timal level: 540 ~ 600; somewhat wet: 390 ~ 530; and
dangerously wet: 320 ~ 380.
(These numbers seem counterintuitive, with lower
numbers assigned to higher levels of water and higher
numbers to lower levels of water; however, they are
merely numbers assigned by the Processing program.) .
I then used Arduino’s Processing software to assign
those ranges of numbers to one of three colored LED
lights: red, green, or blue. NeoPixel is an RGB LED
that can display 16 million (24bits) colors and can be
programmed by Arduino software. For example, if the
moisture data were in the number range above 940,
which meant the moisture sensor was dry, the dark red
LED light in the other plant would turn on.
If the number was in the range of 610–890, which meant
semi-dry, more water was needed, and the dark green
light would turn on. Data in the range of 540 –600,
Fig. 36
To test the quality
of direct sunlight
on the east side
from 4:00-5:00pm,
2016.
Fig. 37
The data that
collected from
8:00am - 5:00pm.
The lower number
means higher
quality of sunlight.
Fig. 34
Projector reflects
the virtual flower
onto the real plant.
Notes: the red
color did not usu-
ally turn on during
the experiment, as
I never let the soil
get too dry.
Research and Design Process
Case Study

76
77Physical Perception Number Visual Expression
Dangerous dry over 940
Getting too dry 610–890
Optimal level 540– 600
Kinds of wet 390– 530
Dangerous wet 320– 380
which indicated optimal soil moisture, would turn on
the medium green light. If data were between 390–530,
which means semi-wet, the light green will turn on. If
the data were between 320--380, which means danger-
ous wet, the LED light would be darker blue. (Fig. 38)
Camera captures the color changing of leaves. My
grandfather told me that there are two reasons why
leaves’ colors change from healthy green to unhealthy
brown. One is often due to over-watering the plant,
(which can be solved by the way of transforming the
watering data into visualized lights.), the other is neglect
(which causes the plant to die). It would be very unusual
for my grandfather to neglect to take care of a plant for
a week without any reasons. Therefore, if the “neglect”
happened on my grandfather's side, I would suspect that
he was sick or that something had happened.(Scenario
2) So a week without watering would trigger a color
change, and would provide a good clue to my grandfa-
ther's health.
The camera could detect neglect by recording a color
change in a plant’s leaves. Of course, the moisture sensor
is also able to measure the dried-out soil, but it is not ac-
curate detect the reason of “neglect” for my grandfather.
On one hand, There are some non-artificial elements
influence the moisture of soil, such as temperature,
sunlight (in different seasons) both of them easy to
result moisture sensor send wrong single to my LED
lights. To compare with detecting the quality of leaves
itself, the color changing after one week is enough to
make me suspect what happened in another side. While,
the moisture of soil easy to dried-out, as long as people
don’t watering it in a short time. Thus, it is not enough
to prove my grandfather is really neglect or “neglect”
with some reasons.
The camera was connected with Processing, which
converted the images that the camera received into
RGB numbers that showed how much red, green, and
blue were in the leaves’ color. The RGB numbers would
then be sent as input to the Arduino hardware. Every
color has a corresponding RGB number (0-255) in a
programming language. The image of a “healthy plant
green” would correspond to around less 145 R, 0-30B,
over 230G, while if G<185 and R> 145 the leaves would
be turning brown. Users could clearly see the colors
changing in one week if water was withheld from the
plant. In addition to lighting the LED lights, the device
would also be programmed to activate a vibration motor
when the red LED was turned on, as an additional
warning signal.
Fig. 39
Camera is capturing
the color of plants. The
photo on the computer
screen is an abstract
visual color mapping of
the plant. The mapping
consists of Red, Green
and Blue (RGB), which
reflects the health of
the plant.
Fig. 38
Watering data
visualized as
lights.
Case Study

78
Virtual Flowers
As with capturing the color of leaves, the camera would
capture images of the plant’s flowers, then Processing
would assign an RGB value to their color. Fred Wolflink
and I determined the interval for the particular pink
color of my flowers, which is G< 200 and R>120, or in
other words, between 120 and 200. Thus, if the color
that the camera captured was in this range, two virtual
flowers (created in Photoshop) would pop up on the
computer screen. I assumed the two locations would
have two different networks, which would require
entering two IP addresses and ports of two comput-
ers. Of course, the IP addresses and ports have to be
programmed in Processing, which would be running in
two computers in different locations. A projector would
need to be connected to each computer and aimed at the
plants. Then, when one user’s plant bloomed with real
pink flowers, the another user who lived in another city
would see the corresponding virtual flowers projected
onto his/her leaves.
Case Study
Fig. 40
The abstract color photo was
generated automatically through
Processing. The camera connected
with Processing captured the
health color of leaves in one week.
The color is changing from health
green to brown. (From left side to
right side)
79

80
These three scenarios used the health of the plant as a
visualization tool, and in the process a certain amount
of data was sent. Unlike in most data collection
efforts, not all the data were made visible to the users.
Objective, factual information was not the point of the
communication. This project emphasized the perceptive
and metaphorical quality of remote communication,
through the support of electronic technology.
Taking traditional physical measurements of soil
moisture and plant posture are difficult for those who
are inexperienced. Electronic technology helps improve
accuracy in these measurements, and makes it possible
to take that which is beyond verbal description into
visual and dynamic communication. The prototype was
a success in helping two people, separated by a great
distance, to share common emotional but nonverbal
communication. It also shows promise as a way for adult
children to unobtrusively monitor the health of their
aging parents.
Case Study
Conclusion
8180

83
Case Study
Digital Platform
A Sharing Mediated In Remote Communication
Keywords: Nonverbal / Hand manipulation /
Interpersonal communication / Multi-sensory communication /
Sharing space
82

85
Project Overview
As with the physical plant project, my digital plant
project also explores emotional communication, but
through hand manipulation. The digital plant project
is devoted to building a virtual platform: first, for an
abstract plant-like object; next, for a plant; and finally,
for a ideal garden in which people select different types
of plants as their "emotional metaphors" in order to
connect with their families. Since the "digital garden"
grows in virtual space, users are able to see growth
quickly, and can spend as much time as they like taking
care of it.
The first prototype was an abstract object, which I creat-
ed to explore how to use the semiotic visual language of
different forms in dialogue with another person. Due to
the current limitations of technology and user testing,
the visual of the abstract object did not make enough
sense as a communication signal in the emotional
expression of the users.
However, the second prototype—a “magic” plant—of-
fered a more effective metaphorical expression of users’
emotions. The magic plant aimed to make human emo-
tion visible through the interaction of the users’ hands
with the growing virtual plant. The users were asked to
choose one type of plant or flower in a digital garden.
Each user had his or her own plant, which communi-
cated the user’s emotions to be shared with his or her
partner. For my prototype, I chose a lily.
Case Study
84

The description below presents the development of
the two prototypes, a summary of my field research on
emotional hand gestures and how I used it in this proj-
ect, user study results, and a conclusion to the project,
with discussion for future work.
Prototype One: Abstract Object
Have you ever seen clay being shaped on a potter’s
wheel? At first it is a lump, then the potter adds water
to the lump (what a mess) and begins to work with it.
Potters skillfully form the clay into a beautiful vase or
pot. But if the potter makes a mistake and it is spoiled,
she smashes it back into a lump and the beauty is
gone. It is a wet lump of clay again. Like a piece of clay,
our hearts, attitudes and emotions can be shaped by
designers.
My abstract object transfers the physical experience of
playing potter into a digital and interactive experience
using sensors and software in virtual space.
Abstract Object Overview
Users manipulate the abstract object on their screens
and control its form and size with simple hand gestures.
The object is controlled alternately by two individuals.
If one of them allows a long time to elapse without
manipulating the object, the object, like the clay, will
slowly return to its initial lump. In my prototype, the
object will change slowly and return to its initial form
Case Study
Hands enjoying direct
contact with material by
pointing, pushing and
pulling. “ Hands acts as
condurits through which
we extend our will to the
world.”
87
86

88 89
Caring
Overwhelmed ShamedHateful
Trusting
Excited
Blooming Kiss | Love
Love Hopeful
Fig. 41
Hand gestures collecting
and analyzing three
elements: Negative space,
direction and temperature.
Hand Gestures Analysis with Leap Motion
Controller and Max
I started to observe and take photos of the public behav-
iors of people who were unaware that I was recording
them. There are hundreds of gestures or body move-
ments that convey meaning in different parts of the
world, but there are certain basic emotions that can be
universally understood. When we are excited, we wave
our hands. When we are upset with someone, we point
our fingers. When we say goodbye to those we love, we
kiss them with tenderness or passion or give them a
hug. Our emotional expression is transmitted through
Case Study
within a twelve-hour span (the time difference between
China and Boston) if neither individual interacts with
it. In other words, the object’s growth depends on how
much time and effort each of the partners would like
to spend manipulating it. The form's particular shape,
color and size are manipulated by emotional hand ges-
tures. For example, hand movements that express anger,
disappointment or sadness will weaken the growth of
object and the color will change from a light shade to a
dark. In contrast, hand gestures that express happiness
or love will grow the object, and cause it to turn a warm
color. The particular forms of the object itself depend on
the hand gestures that express emotion. I programmed
the emotions associated with specific hand gestures after
studying how people use their hands, as I describe in the
next section.

90 91Leap Motion Con-
troller:
The Leap Motion
Controller lets you use
your computer in a
whole new way. Reach
out and swipe, grab,
pinch, or punch your
way through the digital
world.
The Leap Motion
Controller tracks your
hands at up to 200
frames per second us-
ing infrared cameras.
As the first user, I uploaded some music to the program-
ming interface of the Max software, then played it with
my hand moving; it sounded like a small symphony.
Users look like conductors as they control the rhythm of
the music through hand movements in virtual space.
The digital analysis by the Leap Motion controller was
conducted to prepare for the next step: making a seam-
less match between the abstract digital object and users’
physical hand movements.
Collecting Hand Gestures
Next, I focused on the relationships between emotions
and visualization, hand movements and emotion, and
hand movements and the form of the digital object. In
the beginning, I began to collect hand gestures:
I took some photos of loving gestures in different
situations; I took some pictures on my way home on
the train; and I video recorded and observed strangers
walking along the streets and waving their hands at busy
intersections. I sketched from memories of moments
and events that happened when I was with my family
and friends. For example: I sketched from memory the
hand gestures that my friends used in saying goodbye to
me, such as a telephone calling “keep in touch with me”
or “remember to call me after you arrive home” hand
gesture.
3D Object Making. At first, I started to create analog
shapes with an actual lump of clay, then drew those
shapes with Maya. Then, I used a 3D printer to print a
small model.
Fig. 42:
1 Fred Wolflink
is testing the
two computer
connections
through changing
the Local port via
Max.
2 Max analysis
the data through
receiving the
physical hand
movement with
Leap Motion
controller.
MaxMSP is a visual
programing enviro-
ment built on the met-
aphor of synthesizer
patches.
MaxMSP/jitter is a
visual programming
environment for the
development of
media experience.
Its interface is based
on the metaphor of
synthesizer patches.
body, hand and eye contact to communicate with
people. I am interested in these nonverbal strategies and
their meanings. I tried to encode the functions of non-
verbal expression as powerful tools of communication,
through digital technology used in remote communi-
cation, in order to add more interactive experience to
traditional methods of computer-mediated commu-
nication. Also, I wanted to narrow the emotional gap
between people communicating remotely, by harnessing
their intimate nonverbal expressions.
I recorded a range of hand gestures associated with
emotional expressions and analyzed them using digital
technology. A Leap Motion controller is a device that
tracks users’ hands and fingers and follows their every
movement, so that users can control the virtual envi-
ronment without touching anything. I connected the
controller with Max, a visual programming language for
media, used, for instance, in Online interactive games to
show the movement of the users’ hands on the screen.
Using Max, I analyzed the hand movement data that
I collected from Leap Motion, so I could connect this
hand motion data to the digital object. I was able to pro-
gram such data as hand direction and position (which
was displayed as a coordinate on the screen), and the
number of fingers or palms. I programmed these data
into the Max software. There is a small window on Max’s
interface which displays hand movement, as if in a mir-
ror, as long as users move their hands in the controlling
area of Leap Motion.
Case Study
Maya is an animation
software which offers
a comprehensive cre-
ative feature set for 3D
computer animation,
modeling, simulation
on a highly extensible
production platform.

Thesis docuementation2

Recommended

Recommended

More Related Content

Similar to Thesis docuementation2

Similar to Thesis docuementation2 (11)

Thesis docuementation2