Design beyond the glowing rectangle - EuroIA2010

Design beyond the
glowing rectangle.
What does the internet of things mean for UX people?

Claire Rowland & Chris Browne
September 2010

Tuesday, September 28, 2010
We’ll be covering why this is important for us to start thinking about, what the impact may be, and what some of the key challenges
are.

“Today’s multimedia machine
makes the computer screen into
a demanding focus of attention
rather than allowing it to fade
into the background.”
Mark Weiser

In 1991, Mark Weiser (the ‘father of ubiquitous computing’) said...

“I hope we don’t end
up in a world filled
solely with

slick,
glowing
rectangles”
Timo Arnall

image - The Onion

...19 years later, we’ve not really solved this, have we?

Our day to day lives involve many interactions with objects, but most of our interactions with computing still happen through the
abstracted world of what Timo Arnall calls ‘slick, glowing rectangles’.

UX is
moving
beyond the
screen.

However, we’re starting to see digital dissolve more into our surroundings.

Of course, people like Weiser have been talking about this for a long time now.

But things are now starting to happen in the mainstream, here and now, which pose new challenges for UX and service design.

We think that in the next couple of years, a lot of UX designers are going to have the opportunity to design things that involve not
just screens, but for the world around them.

The Vision?

Slide © Fjord 2010 | Confidential


A lot of the classic ubicomp (ubiquitous computing) research has been focused on trying to understand the ‘bigger picture’ such as
smart environments and homes.

There’s been lots of promise in this but very few good examples have made it out of research and into the mainstream.

Above is a a research project from the University of Washington.
PED (Powerline Event Detection) is a single plug-in sensor that monitors noise on the powerline infrastructure to detect and classify
appliance-level events in real time in the home. this can build up at picture of a user or users activities in the home.

Intel also have a similar system but that is also still in the research phase.

rough slide

Internet of Things

When your shipment is vital,
know it’s vital signs.



The ubicomp approach is that of a top down (understanding the illusive ‘big picture’), there is an emerging bottom up approach
(coined in 1999) as ‘the internet of things’ which talks about everyday objects becoming interconnected and increasingly able to
sense their environment.

Currently 98% of all processors are in embedded devices, not multi-purpose computers.... and they are becoming increasingly
connected.

An example of this is Fedex Senseaware: sensors track the location, temperature of packages and whether they have been opened.
For example, if temperature sensitive medical supplies are in danger of being destroyed en route, the package can be returned and a
new one sent out, saving vital time.

rough slide

Internet of Things



Another example is Glow Cap, which is a smart pill bottle that knows when the user has taken their pills and can share this data with
the user’s doctor or family. It can also remind the user to take them.

However many of these smart objects are closed and propriety systems.

Why should UX designers care?

because lots of other things are coming
online...


Most of the people in this room probably started out designing for PC web/PC apps...


online...


Many of us now design services that are delivered across PC, mobile, interactive TV and perhaps other platforms...


online...


Screen based media aren’t going away. But lots of other types of object are becoming connected and will be forming part of digital
services and the experiences we design.

Some of them will be objects we interact with directly.

They might take lots of forms, and some of them won’t have screens.

Some of these objects will just talk to other objects, but will provide data that’s used by things we do interact with.

Real (and perhaps crazy) examples of these that someone somewhere has made include dog tags that tweet, connected umbrellas,
coffee machines smart home heating and lighting systems.

We’re still experimenting with what can be done here and some of the examples might seem flippant.

The point is that objects that were not connected can now be connected, and even if we’re not designing those objects, they can have
an impact on what we do.

For example, right now, you might design websites that sell washing machines. Sooner or later, you may have sensor data from
those machines that enables much better after sales service.. how does that change the service you provide? Or you may work
for an environmental organisation. What difference would widespread live pollution data make to you?

These are examples of ways in which connected devices enable more complex services, which will make designing those services more
complex, and, we would say, more exciting.

Either way, IA has a lot to offer in terms of top down experience design and bottom up information sensemaking.

Devices are becoming connected
faster than people

New Verizon and AT&T subscriptions
May - August 2010


By way of evidence...

That’s currently things like e readers, vehicle tracking systems and alarm monitoring services

http://www.economist.com/blogs/babbage/2010/08/internet_things

22 billion devices connected
by 2020

IMS Research


... that’s TVs, cars, mobiles and lots and lots of other random things. Actually a conservative estimate - Ericsson predict 50 billion
connected devices.

There are so many things becoming connected that we’re running out of IP addresses... IPV4 only provides for 4 billion
addresses and this will be exhausted by 2012. IPv6, which will provide over 300 sextillion (that’s 300 billion billion
billions) IP addresses, is currently being implemented, which will allow every single object you can think of to have an
internet connection.

http://www.readwriteweb.com/archives/22_billion_phones_tvs_cars_on_internet_of_things.php

What does
this mean for
design?

Over the next couple of years, this stuff is due to hit the mainstream and will affect the work UX designers do on an increasing basis

Here’s what we think this might mean for design...

Give digital service a physical form



We can give digital services a physical embodiment.

With the Oyster card (London's travel card). The whole digital service is perceived to be embodied within the card. Even though the
card on its own is worthless.

Enhance products with digital services



Another way to approach design is enhancing existing products with digital services.

“The Copenhagen Wheel is a new emblem for urban mobility. It transforms ordinary bicycles quickly into hybrid e-bikes that also
function as mobile sensing units. The Copenhagen Wheel maps pollution levels, traffic congestion,
and road conditions in real-time.” and shares this via a web service that can be accessed on multiple devices.”

http://senseable.mit.edu/copenhagenwheel/

Click to edit Master text styles

Live data
enables smarter
services.


We design for and with live data to enrich our services.

“SFpark works by collecting and distributing real-time information about where parking is available so drivers can quickly find open
spaces.
To help achieve the right level of parking availability, SFpark will periodically adjust meter pricing up and down to match demand.
Demand-responsive pricing encourages drivers to park in underused areas and garages, reducing demand in overused areas. With
SFpark, real-time data and demand-responsive pricing work together to readjust parking patterns in the City so that parking is easier
to find.” http://sfpark.org/

Fjord’s work
in this area...


A 3 year EU project across
16 partners & 7 countries

Investigating user-centred
design of interconnected
embedded objects

Fjord’s role covers
developing design,
concepting and research
methods, running trials

We’re currently at the
stage of identifying
challenges and generating
initial concepts


Although we are working with the above partners, this presentation represents Fjord’s view.

We’re at an early stage and this is a massive area of ongoing research, but we are starting to propose some challenges and
suggestions as to how we might overcome them

Key design
challenges.

Here are a few of the key challenges we think UX designers will have to be prepared for, and some suggested ways to do things
differently.

We’re just working this stuff out ourselves... these are some of the issues we hope to be able to research over the next couple of years.

These touch on bigger issues - they’re important for this but each is a huge topic in its own right

1.
Service and
interaction design
needs to scale.

Classic usability tends to focus on one user, one device, one service and one task at a time.

Service designers recognise that services are dynamic but we can often still get away with designing for a limited number of platforms
and quite scripted scenarios.

That’s all right if your interactions are fairly simple, and your service works in isolation.

As embedded components come online, digital services will have to cope with increasing complexity, in 3 ways:

One device, multiple services


Some objects will be touchpoints or data sources for multiple services.

Mobiles will play a huge role in this.

But there will be others which may not have screens to show us what they are up to.

For example, in New Songdo, a Korean ubiquitous city in development, there is a student smartcard that can be used to get into
school, pay for food, as a library card, and as a bus pass (which can be used to notify students’ parents that the student is on the
bus).

Although it’s a simple object, and not even terribly smart, it’s a good example of an object with dynamic functionality, dependent on
context.

Users have to use context of use to infer what it’s doing, and what information it is exchanging, every time it’s swiped.

It also represents different types of meaning: it has monetary value, identity value, knowledge value... and privacy implications (what
if you don’t want your parents to know where you’re going after school)

Multiple devices, one service


Likewise, services need to be usable across more than one device. there are lots of web services that can be used on more than one
device, like Google docs, or YouTube, or Facebook.

But generally it’s not possible for users to start an interaction on one device and continue it on another without losing their place in
the task. Especially not if it is not continuously connected.

Hence getting lots of Facebook notifications on different devices... currently a poor cross platform design experience.

Our Smarcos partners CNR ISTI in Italy are expert researchers in the field of migratory interfaces: applications that can transfer
among different devices while preserving state and giving the sense of a non-interrupted activity. the interface and interaction
capabilities are tailored to the specific capabilities of the device. at the moment the examples we’ve seen are screen based
(mobile to PC) but there is also a need for smooth interactions with services across both screened and non-screened devices.

Example: smart fridge, smart phone, shopping trolley, tagged items in fridge.

Services become interoperable


Finally, services themselves will get more complex to design.

They’ll have to be flexible to accommodate new components being added, such as adding new sensors to an environmental
monitoring system.

They’ll increasingly share data, becoming interoperable. that’s not just technical interoperability, it’s at the design level too: it shapes
the kind of services and experience you can provide.

This is the norm in web apps, and we think it will become the norm in embedded device design too.

For example, a developer called Hans Scharler at iobridge has rigged up his home thermostat to adjust the heating based on location
data from Google Latitude, and weather information from google weather.

http://www.readwriteweb.com/archives/automate_your_thermostat_coffeemaker_as_location_m.php

You might think this is a gimmicky example, but it’s interesting from an openness perspective.

Whilst we may see an Apple style market in which some users are happy to be locked into one provider’s technology, we’ll also see a
whole lot of open services enabling first hobbyists, and perhaps then the rest of us, to mashup and create our own physical
services.

The point is we can’t design services in isolation. When services share data, it will be much harder to define the whole of usage as top
down scripted scenarios.

We’ll have to get used to designing more and more dynamic services across lots of different platforms, and think about how they can
enable other services.

2.
What do we do with
all this data?


Data overload


More and more data is being produced in both the physical and digital space, and can be shared in near real time.

How do we as designers leverage this huge amount of increasing complex data to help enrich the services we design, and aid us in
designing new forms of services?

Aggregation



The traditional web 2.0 way of leveraging this data would be that of aggregation.

Example: Globrix, a property search engine that enriches property info via aggregating data from around the web to give more
information about a property’s area, such as maps, crime stats, nearest school, photos from nearby and housing market trends for the
area.

Thinking about data in new ways


We can move beyond just aggregation. Data can be leveraged in increasingly innovative ways beyond their intended purpose.

An example of using data beyond its intended use: using the number of mobile phones connected to cell towers by main roads in the
US to predict traffic conditions.

Likewise, hedge funds could leverage Foursquare's checkin data for financial forecasting. Foursquare is a great social location based
service, it also shows in near realtime where people from different demographics are spending their time (and possibly money). It is
this data capture that is increasingly valuable to hedge funds or market researchers.

Apps for your house?



When the data your service and others produce can be used in different ways, how do you approach design and the increasing levels
of complexity between datasets?

If you were designing a wireless security system for a home, the same sensors could be used as a health system for the elderly e.g pick
up on if the person has fallen over or has an unusual behaviour/movement pattern (key in diagnosing the early signs of dementia).

What if we start to think of sensors as their own digital platform?

Everything in your home has the potential to be hardware and contain software. We as designers need to be open to this and focus on
how relationships and interactions with and between objects add true value to the service and the user.

Data as a design
material?



As data becomes more open, we as designers need to be more open in the ways in which we think about the data around us and its
potential. Over the coming years it is due to increase at an exponential rate.

Data is an increasingly powerful and abundant raw material for designers to use in new and exciting ways to enrich, extend and
create services.

3.
Understanding data


With some much data being produced we need to start understanding it on a human level.

Ubicomp approach



The ubicomp ‘big picture’ approach is “to understand the user you need to understand their world/environment”

Example: Intel carried out an experiment in which they tagged objects in an elderly person’s home with RFID tags and gave the user
a glove with an RFID reader on. This means that when the user interacted with any object it was digitally recorded. This record was
then cross referenced with a context model that had been created via mining the web to infer an activity, e.g. getting out eggs, a frying
pan and turning on the hob implies frying eggs or making an omelette. This activity is then shared with caregivers and loved ones and
unusual behaviours or potentially dangerous situations can be flagged up to caregivers, allowing the elderly person to remain self
sufficient and in their home for longer.

http://rfid.weblogsinc.com/2004/03/16/intel-showcases-in-home-technology-healthcare-aps/

Applications



Example: Umbrellas are awkward to carry. The best thing an umbrella can do is tell you when you don’t need it - help you not use it.
Violet’s smart umbrella indicates if it is due to rain and you should carry it with you. It does this by grabbing an easily accessible
weather forecast feed... that forecast is in essence the aggregation of data from a huge amount of sensors thats outputs are
understood as weather... but the umbrella doesn’t have to worry about any of these sensors. As their data has been abstracted to a
human level of ‘its likely to rain’ or ‘its not likely to rain’.

Once we have an understanding of the data around us on a human level, designers can start applying it to their work in more
meaningful ways. And this human understanding is a key stepping stone towards moving away from screen.

4.
Ensuring users
retain control of
their data.


“There’s a fine line between
pervasive computing and
invasive computing.”
Victor Rozek


Privacy management is much more than a bunch of tick boxes and security settings. It incorporates less tangible elements such as
'appropriate use' and a user’s ‘comfort level’ etc. These may be determined by the users’ sense of control over their data and its use.
The level of ‘appropriate use’ is again dependent on the user and their culture.

As designers we need to set ‘sensible’ defaults for users and allow them to quickly and easily manage their privacy settings, whilst
clearly communicating the pros and cons of their choices. This is an increasingly difficult challenge as many users can barely
manage Facebook privacy settings.

Please mind
the gap



We can design individual services that use a user’s data ‘appropriately’. However, gaps between individual services can form where
personal data may be leaked.

Example: Please Rob Me - pleaserobme.com leverages the gap between Foursquare and Twitter to say when people are away from
home and where their house is located. This raises a few interesting points such as ‘how aware are we of the data we put out there’
and the fact it would seem ‘appropriate’ for Foursquare to know and share your location, as it is also ‘appropriate’ for Twitter to
share your messages with the world. It is only when the two services are joined that the possibility of inappropriate use appears.

With more objects becoming connected and sharing what’s deemed 'appropriate' data dependent on their service, the potential for
gaps that can leak personal data increases.

Brokers?



Given the complexity of managing privacy across the tools we already have, and the fact that this is likely to get more complicated,
some have proposed the emergence of privacy brokers to help manage our digital presence across services.

Example: Yahoo!’s Fire Eagle - “Fire Eagle allows you to share your locations with other sites and services safely through a secure
server - you are always in control. You can decide to share your location with any site that can use it, and even choose how much
detail to give that application (exact point, neighborhood, city, state, country).” - http://fireeagle.yahoo.net

Fire Eagle can be seen as a very early attempt at the idea of trusting one provider to handle an aspect of your personal data in this
case location. However it hasn’t gained enough traction to preform this task, as users use various other location based services that
aren’t linked to Fire Eagle such as Google Maps, Facebook Places etc.

If brokers do emerge, what kind of organisations would we trust enough to manage this? Some have suggested the mobile
operators... Also, what degree of control would we be happy to hand over to them?

React and respond


How would a privacy broker deal with something like this? (Barbie with a webcam in the chest)

Managing privacy becomes much more complex as both the digital and physical environments collide and interactions and data
production become more passive.

Also how do users manage complex privacy settings for devices that may not have a screen?

We don’t have any solid answers for this yet as it is an area that is constantly going to be testing the boundaries. We can’t rely on law
and the legal system to protect users due the the speed in which issues develop in the digital space. We as service designers we need
to think about the data our service holds as a part of a larger ecosystem, how this data my be misused, and be proactive in the
protection of our users and enable users to make informed decisions in regard to what data they are sharing so they feel in control.

5.
Interactions
become tangible.


Thinking is physical

Cognitive scientists now talk about ‘embodied cognition’*: the idea that the way we think is shaped by, and inseparable from, our
physical experiences of interacting with the world. (Dourish, McCullough)

For example, the idea that up is good and down is bad is rooted in your physical experience of living with gravity. “I’m feeling down
today” is bad. Up (and fast) is good... “I’m feeling upbeat”. We say someone is ‘boiling over with rage‘ or ‘steam is coming out of their
ears’: understanding anger through containment of liquids. These are English language examples, but the principles seem to be
universal.

Cognitive scientists would argue that this perception of up and down is a very fundamental basic level category or building block of
thought used to make sense of other, more abstract things.

Embodied interaction seeks to make physical designs make sense to us through harnessing the way we understand the world through
physical experience. At the moment, much tangible interaction work is happening in R&D labs...

Physical interactions can
convey information


MIT’s information design ecology unit’s Proverbial Wallet aims to ‘unabstract virtual assets’ and provide a tangible link to virtual
money. Connects to phone via Bluetooth, and then to your online banking service.

It has an interface through physical actuators, but no screen

It gets fatter or thinner depending on how much money you have in the bank - direct physical representation of a metaphor (fat
wallet).

The hinge resistance increases when you have less money, making it harder to open. This is a nice example of encouraging positive
behaviour without controlling the user. [metaphor of good angel?]

The wallet buzzes when your bank processes a transaction, creating a conscious connection between handing over your credit card
and the money actually leaving your account. This is a nice example of seamful design... communicating interconnections between
services back to the user.

http://eco.media.mit.edu/proverbialwallets/

Physical interactions can enable
product design

image: www.fabianhemmert.com


This is a prototype phone-shaped box by the tangible design researcher Fabian Hemmert at Deutsche Telekom labs.

It has an actuated back plate (controlled by servo motors) which can be tilted. so it can be thin in your pocket, but tapered
downwards when held in the hand.
this can also be used to convey contextual information: for example to make the phone thick or thin on one side when reading an
ebook to indicate how much has been read and is still to be read. the shape of the phone can also be used to indicate direction when
providing navigation...

http://fabianhemmert.com/projects/shape-changing-mobiles

There’s a lot we can do with tangible design to convey information without screens. As digital designers who’ve only worked with
screens, we should start to think about these new interaction channels and how they can help us reclaim digital interactions into the
real world.

Also, physical things are fun :)

6.
What happens
when it all goes
wrong?


When interconnected services go wrong, they will go very wrong.

...it will go badly wrong


Technology is not reliable... we were trying to share files with a ubicomp researcher over Bluetooth the other day and we couldn’t
even get that working or figure out why it wasn’t working.

Imagine that scaled up to a smarthome consisting of hundreds or thousands of connected objects...

If you connect up lots of bits of hardware, the services are only as good as the combined reliability.

A friend’s Danish friend lives in a smarthome with a system to decides when you could or could not open windows. it broke.

Thankfully, there was an override.

Then the override failed, and the house took over, opening and closing windows on its own. No-one could figure out how to fix it.

So the more interconnected things are, the harder it is to know which bit’s gone wrong. and the harder it is for the system to give
useful information on overcoming errors.

congratulations!


What constitutes acceptable reliability?

“A place where one can solve one’s worries” – a Korean euphemism for toilet

In Korea, there are toilets with sensors for female hormones and the signs of diabetes.

Right now, they don’t seem to include a pregnancy test but it’s not too much of a stretch to imagine.

Home pregnancy tests are 99% accurate.

Should you decide to test yourself once in that year, there’s only a 1 in 100 chance you’ll get a false positive.

If you’re tested 1000 times a year by a smart toilet, your toilet will wrongly congratulate you almost once a month.

You’d stop taking it seriously, wouldn’t you?

How much trust would you place in these things? How dependent will we become along them?

7.
User research and
prototyping
methods.


There are two big challenges here...

Understanding needs for things people are not yet familiar with

Prototyping and testing complex systems

Emergent behaviour and expectations

image: techpin.com

image: mp.natlib.govt.nz


We recently interviewed some designers of smart objects as part of our project, about their methods. Numbers were small, but
tendency was to feel that insight research was difficult to do before prototyping.

We often can’t predict the changing behaviours and attitudes that will emerge from new technology until people actually have it.

Putting washing machines in homes, plus bathrooms, changed ideas of acceptable hygiene - expectations of cleanliness rose.

People didn’t used to get mobiles for social reasons, they claimed it was for emergencies, but once they had them they quickly adapt
to using them for social reasons. (e.g. Edwards and Grinter).

Smart environments, objects and services promise to make many things easier, but will they just move the baseline instead of actually
making life easier?

Cultural values, as we might gain from real ethnography (not ‘ethnography’ in the sense of any old field research), tend to remain
stable over longer periods of time. For example... perceptions of privacy trade offs are different in parts of east Asia from Europe,
and willingness to view technology as a way of mitigating uncertainty.

But observing current tasks may not always help generate detailed requirements for highly novel systems, and you certainly can’t ask
people what they want from something totally new.

But there may be methods we can use...

Participatory design


Participatory drama methods can be used to help shape designs... for example giving users scenarios to imagine novel interactions
with an everyday object, invent interactions with an ambiguous object, or a wand that allows them to imbue other objects with
magical powers. Also bodystorming - acting as if the product existed.

Wizard of Oz prototyping is quick and can help evaluate some key interactions

Prototyping

image: kogoro kotobuki/Flickr


Prototyping is an important part of exploration in tangible design. Need early stage iteration as it’s hard to change complex systems!
but hard to create real enough experiences.

Getting into hardware is a little daunting but there are ways to get feedback before going that far.

You *can* use paper models to mock up and get feedback on quick ideas/key interactions, wizard of oz style. Also useful for
concepting.

You can also use video to mock up things that don’t yet exist to get feedback... this Microsoft Office 2019 one is for marketing but
you can make versions in ways that support concept testing.

Need to work closely with technologists... helps for designers to get some understanding of tech, and for developers to move closer to
understanding user needs.

Testing


Testing methods are geared towards one user, one task, one(ish) devices, that the user is highly focused on. We don’t even have a
good definition of usability for more ambient interactions.

With testing, ecological validity is an issue: you can do very controlled simulations in the lab which are good for key interactions, but
when you’re designing something to work in an environment context of use is really important.

Need to study usage over time as well - services are dynamic, and people may react differently to things once they become normal
than they will to novel technology.

So context, and time are important.

One solution to this is the paratype: a modified diary method (Abowd, 05).

This involves a simulated interaction with a certain technological artifact within a specific setting of real social action, and
documenting the effects of this combination.

- focuses on the social - samples experience using a simulation in real social contexts, e.g. personal audio loop - memory device -
allowed them to study 3rd party reactions to the context.

There’s more work to do to develop methods here, but academic ubicomp literature is trying to tackle this issue.

Perhaps we can think of the old approach as being like monochronic cultures of northern europe - it’s embedded in the culture that
people do one thing at a time without interruptions, and are very focused on efficiency and managing time. we need a shift to
something like the polychronic cultures of the rest of the world: people doing multiple things at once, expecting interruptions.

How can UX
people get
started?

Everything we’ve talked about is happening now, somewhere.

We think this is about to affect the work that many of us do, even if just in small ways.

We’d like to suggest a few ways in which UX designers can start to think about this.

Look for the potential in existing
projects...


Look for potential in other projects by thinking of the ecosystem beyond your service and how this might be leveraged to extend or
enrich your service

When everything around you has the potential to be hardware and run software, the challenge lies in looking for the relationships
between objects and services that provide real value to the user experience.

Emotional/informational objects


A possible starter for creating design briefs...

Use objects as a basis for concepting... if they could tell a story, what would it be?

what personality would they have?
what emotions would they feel?
what information would they know, or share?
how could they have good manners? (i.e. some rules for interaction)

Reduce the need for interactions



Another technique is to look for ways to reduce interactions... (that umbrella again!)

Understanding context will be how we move beyond screens and towards a more intuitive relationship with technology.

Hardware prototyping


Learn some hardware prototyping... we’re doing this at Fjord...


Be open to other skillsets. service design, interaction and product design will have to work together to address new challenges

We’ll be doing a lot more work in this area over the next couple of years and would love to connect with anyone else who’s interested.

Some reading...

Timo Arnall: “Designing for an internet of things”
(http://www.slideshare.net/tmo/designing-for-an-internet-of-things)

ReadWriteWeb: Internet of Things:
http://www.readwriteweb.com/archives/internet-of-things/

Adam Greenfield: “Everyware: the dawning age of ubiquitous computing”

Bruce Sterling: “Shaping Things”

Just out:

Mike Kuniavsky: “Smart Things: Ubiquitous Computing User Experience Design”



Thank you.
chris.browne@fjord.co.uk
claire@fjord.co.uk

Thanks also to Alex von Feldmann, Dom Quigley, Ann Light, Alfred Lui,
Christian Lindholm, Ji-Hye Park, Sam Crosland


Design beyond the glowing rectangle - EuroIA2010

More Related Content

What's hot

Viewers also liked

Similar to Design beyond the glowing rectangle - EuroIA2010

More from Claire Rowland

Recently uploaded

Design beyond the glowing rectangle - EuroIA2010