Designing a new ecology of mixed
digital and physical environments
Nicolas Nova, Media and Design Lab
Torino, July 2007

research about how to design for hybridization led to:
pervasive computing, ubiquitous computing,
everyware, wearable computing, mobile computing,
things that think, calm computing, continuous
computing, ambient intelligence, digital-physical
computing, internet of things, networked objects,
augmented reality, mixed reality, geospatial web,
locative media, physical metaverse...
a big mess.. that we will focus on tonight
all these words are propelled by dierent institutions/people/labs, they all have a dierent spin (for
instance...), they are the stakeholders of the hybridization, fortunately, reality is complex so none of
them is right
the premises, however are the ones of ubiquitous computing, expressed by Mark Weiser in 1991:
computation that fade in the backround/environment

pervasive computing, ubiquitous computing,
everyware, wearable computing, mobile computing,
things that think, calm computing, continuous
computing, ambient intelligence, digital-physical
computing, internet of things, networked objects,
augmented reality, mixed reality, geospatial web,
locative media, physical metaverse...
(that’s the one we’ll keep)

tonight’s menu:
what technologies?
examples (art/startups/labs)
...but...
design challenges
expected or alternative [near] futures

underlying technologies
• identification (RFID, ipv6, visual
tags...)
• sensors
• positioning (GPS, Wifi,
CellID...)
• other information (pressure,
temperature, etc.)
• communication (tcp/ip, BT/
Wifi, zigbee, redTacton...)
• computation

Output
digital physical
digital
desktop computers 3D printing, rapid
game consoles prototyping
Input
physical
digital overlay (augmented
reality, visualization)
physical activities
locative media
architecture as an interface
lifelogging+virtual worlds
a sort of model....input / output, digital/physical
matrix digital/physical I/O
intentional/unintentional output

augmented/mixed reality: combination of real
world and computer-generated data

visualizing the
hybridization
S.O.U.P provides a visual and acoustic representation of the omnipresent wireless communication networks which surrounds us in urban spaces. think of our application as a
soup ladle spooning arbitrarily up the boiling air and transforming the data into sound and video. A computer terminal shows a map of the surroundings plus basic points of
reference of the city or town that hosts physically our installation. the observer now can jump or fly to some places in the city and choose from different datasets recorded at
different daytimes. the software will provide a visual and acoustic representation using surround sound and a beame
Sky Ear is a non-rigid carbon-fibre \"cloud\", embedded with one thousand glowing helium balloons and several dozen mobile phones. The balloons contain miniature sensor
circuits that respond to electromagnetic fields, particularly those of mobile phones. When activated, the sensor circuits co-ordinate to cause ultra-bright coloured LEDs to
illuminate. The 30m cloud glows and flickers brightly as it floats across the sky.

(Picture: Blast Theory)
location-tracking: tracing people in real space

location-based services
Examples of location-based applications... large array of purposes (security, play, serendipity...)

urban tapestries: digital annotation of space
Urban Tapestries (http://research.urbantapestries.net)

urban tapestries
Urban Tapestries (http://research.urbantapestries.net)

mobile gaming...

environment overlap (physical+digital) but immobile
the overlapping of two environments

Mogi: environment overlap (mobile)
Screenshot of Mogi (Newt Games) taken from “The Network is the Game: Social Trends in Mobile
Entertainment” (Talk by Amy Jo Kim)

Mogi: environment overlap (mobile)
Screenshot of Mogi (Newt Games) taken from “The Network is the Game: Social Trends in Mobile
Entertainment” (Talk by Amy Jo Kim)

explicit invisible/implicit phenomena
(Beatriz da Costa, 2006)
How to use technologies to make certain phenomena that are implicit/invisible more explicit.
Example: the blogging pigeon project (UC Irvine) that use GPS/sensors on pigeons to detect
pollutions on the city of San José and give the citizens a representation of the environment.

Premptive media’s Area Immediate Reading
AIR is a public, social experiment in which people are invited to use Preemptive Media's portable air monitoring devices to explore their neighborhoods and
urban environments for pollution and fossil fuel burning hotspots.
Participants or \"carriers\" are able to see pollutant levels in their current locations, as well as simultaneously view measurements from the other AIR devices in
the network. An on-board GPS unit and digital compass, combined with a database of known pollution sources such as power plants and heavy industries, allow
carriers to see their distance from polluters as well. The AIR devices regularly transmit data to a central database allowing for real time data visualization on this
website.

tripwire (Tad Hirsch)
Tripwire (Tad Hirsch) San Jose International Airport and downtown San Jose, CA. Custom-built
sensors hidden inside coconuts are hung from trees at several public locations to monitor noise
produced by overflying aircraft. Detection of excessive aircraft noise triggers automated telephone
calls to the airport's complaint line on behalf of the city's residents and wildlife. Documentation of
noise incidents is archived for later analysis

pedometers beyond lifelogging
Teku Teku angel and v-migo

ubiquitous computing of today

actually, not quite new?
(“piazza-listening device” by Athanasius Kirchner, 1650)
Athanasius Kircher was a 17th century German Jesuit scholar who invented a large amount of
interesting artifacts ranging from speaking tube, perpetual motion machines to cat pianos. Among
the devices this Renaissance man created, there are two that I found amazingly intriguing, in terms
of ubiquitous computing.
a “piazza-listening device”. As described in this paper, “the voices from the piazza are taken by the
horn up through the mouth of the statue in the room on the piano nobile above, allowing both
espionage and the appearance of a miraculous event“.

UT

“Once devices were
\"Location-aware service
location-aware, business
integration into applications
applications were expected
began. A critical mass of
to take advantage of the
network and device support
capabilities in the next two
will occur through 2006.\"
to five years“
Gartner, 2003
Gartner, 2006
A proximal future...
...infinitely postponed?
Consultants/trendwatchers/analyst keep repeating the same predictions for 4-5 years... but they
postpone the proximal vision of location-aware services. Something is wrong here.
See: Bell, G. & Dourish, P. (2006). Yesterday’s tomorrow: notes on ubiquitous computing’s
dominant vision, Personal and Ubiquitous Computing.
The example her is about location-aware services but analysts say the same with the intelligent
fridge, ambient computing, etc.

why?
design challenges

a common
belief:
The broken cloud of connectivity
Transitions operating smoothly, seamlessness, invisibility, calmness are often believed to be true

infrastructures are not seamless = difficult to
provide a continuous experience
More about this: Dourish, P. & Bell, G. (2007). The Infrastructure of Experience and the Experience
of Infrastructure: Meaning and Structure in Everyday Encounters with Space. Environment and
Planning B,

fragmented locus of interactions
computers, not switches!
When the locus of interaction is everywhere, there are lots of point of entry
more dificult to convey a sense... it’s then more complex than the thermostat/ light bulbs
Because the interaction is in the air, people are not prepared, expectations are more complicated
and the purpose of the interaction is less clear (and these points of entry are much more
complicated than switches)

other actors... surprised

digital is not immaterial = it needs big infrastructures
See Graham, S., Strategies for Networked Cities

odd/wrong assumptions about people needs and desire
(people can’t relate to this sort of graph)
Based on fake/wrong/flawed assumptions about people needs/desire (automation)

recurring instrumental and utilitarian purposes
The intelligent fridge = posterchild, well it’s a bad posterchild

normative futures (infinitely postponed)
More on that: Bell, G. & Dourish, P. (2006). Yesterday’s tomorrow: notes on ubiquitous computing’s
dominant vision, Personal and Ubiquitous Computing.

space is not homogenous
Picture by Patrick Jermann
dierent places, dierent infrastructures, rugosity

often showing negative aspects of life
It seems that ubicomp applications have a fascination towards showing negative aspects of life
(pollution, disappearing of animals...)

alternative visions
I don’t have the answer to all those issue but I do think we can use them as challenges to design
internet of things applications

assist, not automate
allow the system to convey
intentionality!
panopticon versus masquerade
First, there is a big dierence between completely autonomous systems and assisting human users.
The idea is to augment awareness not automate it, the final interpretation should be left to the
humans. A good example of such is Jaiku that respect the intentionality of the message that is
conveyed: rich presence rather than location. Then of course, the cost for the user is high, but the
benefit for the others is high too: they know the intention of the sender! Also an obvious but
important aspect is to facilitate the opt-out and to lie.
In the same line, in terms of privacy what people do not like is the feeling to be seen without the
ability to see (Jeremy Bentham’s Panopticon) as Michel Foucault argued. BUT people are OK to
disclose things when they can control what they see and when they can see others.

utilitarian but less instrumental
“isolatr”, a system that aims at helping where other people are not.
It’s a joke but still is relevant as a way to express some concerns

“we don’t want to invent people’s life” rafi haladjian
be humble... rafi haladjian... open source, “we don’t want to invent people’s life”

offline gaming - tracking physical activity
Project currently being done with Julian Bleecker (USC) to explore ofline gaming (physicality+
asynchrony of the results)

blogject (blog-objects)
blog - object: an object that would be able to create and circulate its own content based on its
interactions with the environment and people
history of things, blogjects, trajectories, add values... POSITIVE HISTORIES WITH OBJECTS

turn history of interactions
in a continuous experience,
object learn from their
histories
object have histories
imagine if objects could learn from the interactions they have with their owner, other people or
other objects
and if this history could be used change the capabilities of the object?

new place that take seams into account
(Simon Schleicher, MIT Medialab, 2006)
Also of interest, some project tries to design/build physical structure that would take seams/
problems into account (for example GPS holes/network problems...)

from lifelogging to new interaction partners
(Picture found on the Internet) Julian Bleecker and myself are working on this project: how to
engage new partners such as pets on the social web: we have a dwarf on World of Warcraft that is
played by a dog (sensors track its physical activities).

new interactions partners: twitter cat
This is another project that we work on with Julian.

conclusions

what changes?
• space
• new layer of information
• being here and there
• hybridization between physical and digitalS
• interaction: fragmented, concurrency
• production and use of traces of interactions
• privacy!

a complex and unpredictable design ecology?
• beyond user-centered design: contexts
(overlapping space, people participating or not)
• problems:
• seams, digital is material
• people’s needs and desires are balanced
between utilitarian and playful

other models than throwing technology to people?
Future/reality is in
between these
thank you - nicolas.nova@epfl.ch