Fjord @ the Polish IA summit

Design beyond the
glowing rectangle.
What does the internet of things mean for UX designers?

Claire Rowland & Chris Browne
April 2011

Sunday, 10 April 2011

Cześć :)

Claire Rowland Chris Browne
Head of Research Technical Design Lead

Cheshcht :)

Hello, we are ... from Fjord in London. Thanks for having us!

rough slide

A bit about Fjord...
Slide © Fjord 2010 | Confidential

We’re a digital service design agency.

That means we design user experiences for services that are used on more than one platform - often mobile, tablets like the iPad,
digital TV as well as web.

A 3 year EU project across
16 partners & 7 countries

Investigating user-centred
design of interconnected
embedded objects

Fjord’s role covers
developing design,
concepting and research
methods, running trials

We’re currently at the
stage of identifying
challenges and generating
initial concepts


All of our client work to date has been around designing for things with screens. But we’re now working on an EU funded research
project that has given us the opportunity to think beyond the screen.

Although we are working with the above partners, this presentation represents Fjord’s view.

The rest of this talk is a quick tour of some of the issues we’ve been thinking about on this project.

We’re at an early stage and this is a massive area of ongoing research, but we are starting to propose some challenges and
suggestions as to how we might overcome them

“Today’s multimedia machine
makes the computer screen into
a demanding focus of attention
rather than allowing it to fade
into the background.”
Mark Weiser


In 1991, Mark Weiser (the ‘father of ubiquitous computing’) said...

“I hope we don’t end
up in a world filled
solely with

slick,
glowing
rectangles”
Timo Arnall

image - The Onion

...20 years later, we’ve not really solved this, have we?

Our day to day lives involve many interactions with objects, but most of our interactions with computing still happen through the
abstracted world of what Timo Arnall calls ‘slick, glowing rectangles’.

UX is
moving
beyond the
screen.

However, we’re starting to see digital dissolve more into the physical world.

Of course, people like Weiser have been talking about this for a long time now.

But things are now starting to happen in the mainstream, here and now, which pose new challenges for UX.

We think that in the next couple of years, UX designers are going to have the opportunity to design things that involve not just
screens, but services and physical objects for the world around them.

We thought it was going to be like this...



A lot of the classic ubicomp (ubiquitous computing) research has been focused on trying to understand the ‘bigger picture’ such as
smart environments and homes.

There’s been lots of promise in this but very few good examples have made it out of research and into the mainstream.

rough slide

But it’s actually more like this...

When your shipment is vital,
know it’s vital signs.



‘The internet of things’: everyday objects becoming interconnected and increasingly able to sense their environment.

Currently 98% of all processors are in embedded devices, not multi-purpose computers.... and they are becoming increasingly
connected.

An example of this is Fedex Senseaware: sensors track the location, temperature of packages and whether they have been opened.
For example, if temperature sensitive medical supplies are in danger of being destroyed en route, the package can be returned and a
new one sent out, saving vital time.

rough slide

...and this



Everyday objects getting smarter

This is RTMM: real time medication monitoring - made by Evalan, one of our partners on Smarcos...

...a smart pill bottle that knows when the user has taken their pills and can share this data with the user’s doctor or family. It can also
remind the user to take them, via SMS or phone call

However many of these smart objects are still closed and proprietary systems.

Why should UX designers care?

because lots of other things are coming
online...


Who works on PC apps/websites?

It’s not just PCs...

online...


Who works on mobile, interactive TV... tablet?

It’s everything

online...


Screen based media aren’t going away. But lots of other types of object are becoming connected and will be forming part of digital
services and the experiences we design.

Some of them will be objects we interact with directly.

They might take lots of forms, and some of them won’t have screens.

Some of these objects will just talk to other objects, but will provide data that’s used by things we do interact with.

Real (and perhaps crazy) examples of these that someone somewhere has made include dog tags that tweet, connected umbrellas,
coffee machines smart home heating and lighting systems.

We’re still experimenting with what can be done here and some of the examples might seem flippant.

The point is that objects that were not connected can now be connected, and even if we’re not designing those objects, they can have
an impact on what we do.

For example, right now, you might design websites that sell washing machines. Sooner or later, you may have sensor data from
those machines that enables much better after sales service.. how does that change the service you provide? Or you may work
for an environmental organisation. What difference would widespread live pollution data make to you?

These are examples of ways in which connected devices enable more complex services, which will make designing those services more
complex, and, we would say, more exciting.

Either way, IA has a lot to offer in terms of both top down experience design and bottom up information sensemaking.

Devices are becoming connected
faster than people

New Verizon and AT&T subscriptions
May - August 2010


By way of evidence...

As of February 2011, both these networks now have more machine subscribers than human subscribers.

That’s currently things like e readers, vehicle tracking systems and alarm monitoring services.

http://www.economist.com/blogs/babbage/2010/08/internet_things

22 billion devices connected
by 2020?

... or 50 billion?
IMS Research, Ericsson

... that’s TVs, cars, mobiles and lots and lots of other random things.

This is actually a conservative estimate - Ericsson predict 50 billion connected devices.

300 000 000 000 000 000 000 000
IP addresses

"If the earth were made entirely out of 1 cubic millimetre grains of
sand, then you could give a unique [IPv6] address to each grain in 300
million planets the size of the earth"
Wikipedia
image by Flickr user !iLauriin



New standard of IPv6 will provide over 300 sextillion (that’s 300 billion billion billions) IP addresses, is currently being
implemented, which will allow every single object you can think of to have an internet connection.

What does
this mean for
design?

Over the next couple of years, this stuff is due to hit the mainstream and will affect the work UX designers do on an increasing basis

Here’s what we think this might mean for design...

Give a digital service a physical form



We can give digital services a physical embodiment.

A simple example is London’s Oyster card - a passive RFID chip embedded in a plastic card, linked up to a service which allows you
to charge it up with credit.

The card itself is just an identifier, it has no intrinsic value. You wave it over readers and credit is deducted from your account, which
lives remotely in a database.

But in users’ minds, the card *is* the service.

Give a physical product a digital form



Another way to approach design is enhancing existing products with digital services.

“Webkinz are toy stuffed animals that were originally released by the Canadian Ganz company on April 29, 2005. The toys are
similar to many other small plush toys, however, each Webkinz toy has an attached tag with a unique "Secret Code" printed on it that
allows access to the "Webkinz World" website. On Webkinz World, the Secret Code allows the user to own a virtual versionof the pet
for online play for a limited time (one year).” - http://en.wikipedia.org/wiki/Webkinz

Click to edit Master text styles

Live data
enables smarter
services.


We design for and with live data to enrich our services.

“SFpark works by collecting and distributing real-time information about where parking is available so drivers can quickly find open
spaces.
To help achieve the right level of parking availability, SFpark will periodically adjust meter pricing up and down to match demand.
Demand-responsive pricing encourages drivers to park in underused areas and garages, reducing demand in overused areas. With
SFpark, real-time data and demand-responsive pricing work together to readjust parking patterns in the City so that parking is easier
to find.” http://sfpark.org/

Key design
challenges.

Here are a few of the key challenges we think UX designers will have to be prepared for, and some suggested ways to do things
differently.

We’re just working this stuff out ourselves... these are some of the issues we hope to be able to research over the next couple of years.

These touch on bigger issues - they’re important for this but each is a huge topic in its own right

1.
Device - service
relationship gets
more complex.


Classic usability tends to focus on one user using one device and one service to do one task at a time.

That’s increasingly not what’s actually going on. Our relationship with devices and services is getting more complicated.

It’s really important to think not just about device, but service design: how your user experience works across multiple devices. The
device is no longer the unit of experience... the service is.

As embedded components come online, digital services will have to cope with increasing complexity in several ways...

One device, multiple services

image: Vivotech.com


Some objects will be touchpoints or data sources for multiple services.

Mobiles will play a huge role in this.

- Apps are a way to access different services

- NFC, beginning to appear in phones, enables multiple types of payment service. One interaction could take money from your credit
card, bank account, Paypal... how do you make sure it’s the one you intended to use?

How do you know which service
you’re using?


But there will be others which may not have screens to show us what they are up to.

For example, in New Songdo, a Korean ubiquitous city in development, there is a student smartcard that can be used to get into
school, pay for food, as a library card, and as a bus pass (which can be used to notify students’ parents that the student is on the
bus and expected home soon- no more secretly sloping off to hang out with friends after school).

Although it’s a simple object, and not even terribly smart, it’s a good example of an object with dynamic functionality, dependent on
context.

Users have to use context of use to infer what it’s doing, and what information it is exchanging, every time it’s swiped.

Service coherence across multiple devices


Individual services are being delivered across multiple devices

Service coherence is a key part of UX- the service being in sync between multiple devices.

Facebook notifications are a current example of this working badly.

BBC used to talk about PC web as second screen for iPlayer, then mobile as third screen, but is now talking about creating
orchestrated media - a synchronised iPlayer experience across all screens at once.

This would allow users to start an interaction on one device and continue it on another without losing their place in the task, and
deliver different and complementary experiences across different devices.

Multiple devices, one service experience


Non screened device example... Ford Sync: an in car connectivity system that enables the car to be used as a controller for the phone,
so users can control the phone’s music player with the car’s voice recognition system, and more.

http://www.ford.com/technology/sync/about/

2.
Services become
interoperable.



For a while now, we’ve been seeing a trend for services to integrate with other services for the bits those services already do well,
such as 1Password, which integrates with Dropbox for synchronisation across devices.


Services can be used to integrate objects, too. Withings, the WiFi body scale shown in our video, shares data with a variety of
different fitness and weight loss services, such as RunKeeper, Lose It!, Daily Burn.

http://www.withings.com/en/bodyscale/sharing

3.
New platforms for
services.


Data overload


More and more data is being produced in both the physical and digital space, and can be shared in near real time.

How do we as designers leverage this huge amount of increasing complex data to help enrich the services we design, and aid us in
designing new forms of services?

Aggregation



The traditional web 2.0 way of leveraging this data would be that of aggregation.

Example: Flipboard, which aggregate various web feeds such as news sites RSS, Twitter, Facebook etc into a magazine format.

Organic platforms: user driven

Text



This aggregation is also happening on an organic level, as in the example of the pdxboom hashtag.

Just after 8 PM on Sunday, March 28th 2010, a deafening BOOM was heard throughout SE Portland. Quickly the twitter community
had adopted the hash tag #pdxboom. In less than 30mins there was a google map displaying where individuals heard the bang and
how loud it was. This quickly began to show a pattern and allowed the police to establish the source and cause (it was a pipebomb).

Granted it wasn’t the most scientific approach and there are other established platforms such as Ushahidi that would have been
better suited. Instead a new platform was created around this individual event in almost real time.

Organic platforms: corporate data

Text



The previous example shows these platforms being formed by the community/users but they can also be created out of leveraging
existing data beyond its intended purpose.

This was shown recently after the earthquake and tsunami in Japan, Honda opened up their traffic data service and plotted the data
onto a public Google map. This information showed in near real time the conditions of the roads and which roads were passable and
which ones were not.

This is a service that you wouldn’t have expected from Honda as it leveraged a Honda asset beyond its intended purpose.

---
http://www.thetruthaboutcars.com/2011/03/honda-gives-devastated-japan-something-badly-needed-information/

Physical platforms



We can also leverage hardware beyond its intended use, as it becomes more connected and its data more accessible.

If you were designing a wireless security system for a home, the same sensors used for the security system could be used as a health
system for the elderly e.g pick up on if the person has fallen over or has an unusual behaviour/movement pattern (key in diagnosing
the early signs of dementia). do we need to start thinking about designing ‘apps for you house’?

Software changes the purpose of
hardware

http://www.flickr.com/photos/nolifebeforecoffee/



We can agree to hardware for one reason but it can be used in other ways.

“ in Wellington, for example, we were told that the surveillance cameras that voters approved to help manage traffic congestion had
been repurposed for crime prevention, without a corresponding degree of public consultation.
Let the image stream coming off of them be provided with a facial-recognition algorithm, and you’ve got an entirely different kind of
system on your hands, with entirely different potentials and vastly expanded implications. Yet the cameras, domed or otherwise, look
no different from one day to the next.” Adam Greenfield

As both designers and users we need to be aware of how data is and also could be used in the future, and ensure that the right
safeguards are in place and their is transparency in the service of how data is collected and used.

http://speedbird.wordpress.com/2010/04/28/neopanoptical/
http://www.flickr.com/photos/nolifebeforecoffee/

4.
Ensuring users
retain control of
their data.


“There’s a fine line between
pervasive computing and
invasive computing.”
Victor Rozek


Privacy management is much more than a bunch of tick boxes and security settings. It incorporates less tangible elements such as
'appropriate use' and a user’s ‘comfort level’ etc.

Many people already find managing privacy too difficult on Facebook and share things with people they didn’t mean to share them
with. It’s going to get a lot more complicated.

Passive vs active control


Passive vs active - person to service can be active.|| service to service is going to be passive ||

Google Latitude didn’t take off partly because users didn’t always want passively to share their location. On Foursquare, which
requires active check in, you can simply choose not to check in. But active controls won’t scale up to really complex services or
services that talk to each other.

We can design individual services that use a user’s data ‘appropriately’. However, gaps between individual services can form where
personal data may be leaked.

With more objects becoming connected and sharing what’s deemed 'appropriate' data dependent on their service, the potential for
gaps that can leak personal data increases, especially when data we share with one service ends up being used by another service.
How do you know where your data is going to end up? How can service designers design for transparency - so that users can see
when their data is being shared, and where it will end up?

When aggregated, data can get creepier

http://ilektrojohn.github.com/creepy/

“Creepy is an application that allows you to gather geolocation related information about users from social networking platforms and image
hosting services. The information is presented in a map inside the application where all the retrieved data is shown accompanied with relevant
information (i.e. what was posted from that speciﬁc location) to provide context to the presentation.”

Information that seemed innocent when shared can be aggregated to build up a scarily detailed picture of someone’s daily life.

Will we see the emergence of privacy brokers to help us manage our data in the face of all this complexity?

If brokers do emerge, what kind of organisations would we trust enough to manage this? Some have suggested the mobile
operators... Also, what degree of control would we be happy to hand over to them?

The benefits can outweigh the risks


...there’s huge potential in all the data that can be gathered and mined... to deliver new types of service and create better user
experiences.

... but we need to do more work to understand how to help people manage this.

5.
Interactions
become tangible.


Thinking is physical

Cognitive scientists now talk about ‘embodied cognition’*: the idea that the way we think is shaped by, and inseparable from, our
physical experiences of interacting with the world. (Dourish, McCullough)

For example, the idea that up is good and down is bad is rooted in your physical experience of living with gravity. “I’m feeling down
today” is bad. Up (and fast) is good... “I’m feeling upbeat”. We say someone is ‘boiling over with rage‘ or ‘steam is coming out of their
ears’: understanding anger through containment of liquids. These are English language examples, but the principles seem to be
universal.

Cognitive scientists would argue that this perception of up and down is a very fundamental basic level category or building block of
thought used to make sense of other, more abstract things.

Embodied interaction seeks to make physical designs make sense to us through harnessing the way we understand the world through
physical experience. At the moment, much tangible interaction work is happening in R&D labs...

Physical interactions can
convey information


MIT’s information design ecology unit’s Proverbial Wallet aims to ‘unabstract virtual assets’ and provide a tangible link to virtual
money. Connects to phone via Bluetooth, and then to your online banking service.

It has an interface through physical actuators, but no screen.

It gets fatter or thinner depending on how much money you have in the bank - direct physical representation of a metaphor (fat
wallet).

The hinge resistance increases when you have less money, making it harder to open. This is a nice example of encouraging positive
behaviour without controlling the user. [metaphor of good angel?]

The wallet buzzes when your bank processes a transaction, creating a conscious connection between handing over your credit card
and the money actually leaving your account. This is a nice example of seamful design... communicating interconnections between
services back to the user.

http://eco.media.mit.edu/proverbialwallets/

Physical interactions can enable
product design

image: www.fabianhemmert.com


This is a prototype phone-shaped box by the tangible design researcher Fabian Hemmert at Deutsche Telekom labs.

It has an actuated back plate (controlled by servo motors) which can be tilted. so it can be thin in your pocket, but tapered
downwards when held in the hand.
this can also be used to convey contextual information: for example to make the phone thick or thin on one side when reading an
ebook to indicate how much has been read and is still to be read. the shape of the phone can also be used to indicate direction when
providing navigation...

http://fabianhemmert.com/projects/shape-changing-mobiles

There’s a lot we can do with tangible design to convey information without screens. As digital designers who’ve only worked with
screens, we should start to think about these new interaction channels and how they can help us reclaim digital interactions into the
real world.

Also, physical things are fun :)

6.
What happens
when it all goes
wrong?


When interconnected services go wrong, they will go very wrong.

...it will go badly wrong


Technology is not reliable... we were trying to share files with a ubicomp researcher over Bluetooth the other day and we couldn’t
even get that working or figure out why it wasn’t working.

Imagine that scaled up to a smarthome consisting of hundreds or thousands of connected objects...

A friend’s Danish friend lives in a smarthome with a system to decides when you could or could not open windows. it broke.

Thankfully, there was an override.

Then the override failed, and the house took over, opening and closing windows on its own. No-one could figure out how to fix it.

So the more interconnected things are, the harder it is to know which bit’s gone wrong. and the harder it is for the system to give
useful information on overcoming errors.

congratulations!


If technology becomes a part of your everyday life, reliability is critical.

“A place where one can solve one’s worries” – a Korean euphemism for toilet

In Korea, there are toilets with sensors for female hormones and the signs of diabetes.

Right now, they don’t seem to include a pregnancy test but it’s not too much of a stretch to imagine.

Home pregnancy tests are 99% accurate.

Should you decide to test yourself once in that year, there’s only a 1 in 100 chance you’ll get a false positive.

If you’re tested 1000 times a year by a smart toilet, your toilet will wrongly congratulate you almost once a month.

You’d stop taking it seriously, wouldn’t you?

How much trust would you place in these things? How dependent will we become along them?

7.
Digital business
models hit the real
world.


As the boundaries between the physical and digital world blur, we’ll see digital business models starting to appear in the physical
world.

Some of these may be more or less acceptable to users...

Closer relationships with customers



People will pay more for a cheap toy if a digital service is added.

Engagement models last longer - longer engagement with product and customer

Ability to upsell over time, and have a closer direct relationship with the customer

Freemium objects
Fridge will open
after this message
from our sponsors!



Rethink how we own objects

Freemium services - get basic service free with ads, pay for ad free. Will we see this extended to everyday objects? Will Tesco
provide fridges free to anyone willing to share all their food consumption data, buy food from Tesco, and watch adverts on a regular
basis?

Be an individual, not a demographic

image: kihonmedia/ compensationclaims.com


Pay for what you’re using/what you need, e.g. sensors in cars enable drivers to pay car insurance based on how they actually drive,
not on how the average person of their demographic drives.

8.
User research
methods.


There are two big challenges here...

Understanding needs for things people are not yet familiar with

Prototyping and testing complex systems

Emergent behaviour and expectations

image: techpin.com

image: mp.natlib.govt.nz


We recently interviewed some designers of smart objects as part of our project, about their methods. Numbers were small, but
tendency was to feel that insight research was difficult to do before prototyping.

We often can’t predict the changing behaviours and attitudes that will emerge from new technology until people actually have it.

Putting washing machines in homes, plus bathrooms, changed ideas of acceptable hygiene - expectations of cleanliness rose.

People didn’t used to get mobiles for social reasons, they claimed it was for emergencies, but once they had them they quickly adapt
to using them for social reasons. (e.g. Edwards and Grinter).

But observing current tasks can help us undersatnd cultural values, may not always help generate detailed requirements for highly
novel systems, and you certainly can’t ask people what they want from something totally new.

But there may be methods we can use...

Participatory design/enactment


Participatory drama methods can be used to help shape designs... for example giving users scenarios to imagine novel interactions
with an everyday object, invent interactions with an ambiguous object, or a wand that allows them to imbue other objects with
magical powers. Also bodystorming - acting as if the product existed.

Prototyping


Prototyping is an important part of exploration in tangible design. Need early stage iteration as it’s hard to change complex systems!
but hard to create real enough experiences.

Getting into hardware is a little daunting but there are ways to get feedback before going that far.

You *can* use paper models to mock up and get feedback on quick ideas/key interactions, wizard of oz style. Also useful for
concepting.

You can also use video to mock up things that don’t yet exist to get feedback... this Microsoft Office 2019 one is for marketing but
you can make versions in ways that support concept testing.

Need to work closely with technologists... helps for designers to get some understanding of tech, and for developers to move closer to
understanding user needs.

Testing


Testing methods are geared towards one user, one task, one(ish) devices, that the user is highly focused on. We don’t even have a
good definition of usability for more ambient interactions.

With testing, ecological validity is an issue: you can do very controlled simulations in the lab which are good for key interactions, but
when you’re designing something to work in an environment context of use is really important.

Need to study usage over time as well - services are dynamic, and people may react differently to things once they become normal
than they will to novel technology.

So context, and time are important.

One solution to this is the paratype: a modified diary method (Abowd, 05).

This involves a simulated interaction with a certain technological artifact within a specific setting of real social action, and
documenting the effects of this combination.

- focuses on the social - samples experience using a simulation in real social contexts, e.g. personal audio loop - memory device -
allowed them to study 3rd party reactions to the context.

There’s more work to do to develop methods here, but academic ubicomp literature is trying to tackle this issue.

How can UX
people get
started?

Everything we’ve talked about is happening now, somewhere.

We think this is about to affect the work that many of us do, even if just in small ways.

We’d like to suggest a few ways in which UX designers can start to think about this.

Look for potential in existing projects...


Look for potential in other projects by thinking of the ecosystem beyond your service and how this might be leveraged to extend or
enrich your service

When everything around you has the potential to be hardware and run software, the challenge lies in looking for the relationships
between objects and services that provide real value to the user experience.

A simple example is to start with mobile. Mobile phones are essentially sensor platforms - many of them have GPs that tells them
where they are, motion sensors that tell them when they are moving, and some have compasses so they know which way they are
facing. They also know what time it is, and can take pictures with the camera. If you’re delivering something on a mobile, even
if it’s a website, you can usually access this data from the phone. How can you use that to make the service better?

Anthropomorphise objects


A possible starter for creating design briefs...

Use objects as a basis for concepting... if they could tell a story, what would it be?

what personality would they have?
what emotions would they feel?
what information would they know, or share?
how would they talk to us?
how could they have good manners? (i.e. some rules for interaction)

Learn some hardware prototyping


Learn some hardware prototyping...


Be open to other skillsets. service design, interaction and product design will have to work together to address new challenges

We’ll be doing a lot more work in this area over the next couple of years and would love to connect with anyone else who’s interested.

Some reading...

Mike Kuniavsky: “Smart Things: Ubiquitous Computing User Experience Design”

Timo Arnall: “Designing for an internet of things”
(http://www.slideshare.net/tmo/designing-for-an-internet-of-things)

ReadWriteWeb: Internet of Things:
http://www.readwriteweb.com/archives/internet-of-things/

Adam Greenfield: “Everyware: the dawning age of ubiquitous computing”

Bruce Sterling: “Shaping Things”

BERG London: http://berglondon.com/blog/



Thank you.
@fjord
chris.browne@fjord.co.uk
claire.rowland@fjord.co.uk / @clurr
Thanks also to Alex von Feldmann, Dom Quigley, Ann Light, Alfred Lui,
Ji-Hye Park, Sam Crosland, Martin Charlier, Helen Le Voi

PS: we’re hiring in London


Fjord @ the Polish IA summit

More Related Content

What's hot

Viewers also liked

Similar to Fjord @ the Polish IA summit

More from Fjord

Recently uploaded

Fjord @ the Polish IA summit