Designing a new ecology of mixed digital and physical environments

Slide 1: Designing a new ecology of mixed digital and physical environments Nicolas Nova, Media and Design Lab Torino, July 2007

Slide 2: research about how to design for hybridization led to: pervasive computing, ubiquitous computing, everyware, wearable computing, mobile computing, things that think, calm computing, continuous computing, ambient intelligence, digital-physical computing, internet of things, networked objects, augmented reality, mixed reality, geospatial web, locative media, physical metaverse... a big mess.. that we will focus on tonight all these words are propelled by dierent institutions/people/labs, they all have a dierent spin (for instance...), they are the stakeholders of the hybridization, fortunately, reality is complex so none of them is right the premises, however are the ones of ubiquitous computing, expressed by Mark Weiser in 1991: computation that fade in the backround/environment

Slide 3: pervasive computing, ubiquitous computing, everyware, wearable computing, mobile computing, things that think, calm computing, continuous computing, ambient intelligence, digital-physical computing, internet of things, networked objects, augmented reality, mixed reality, geospatial web, locative media, physical metaverse... (that’s the one we’ll keep)

Slide 4: tonight’s menu: what technologies? examples (art/startups/labs) ...but... design challenges expected or alternative [near] futures

Slide 5: underlying technologies • identification (RFID, ipv6, visual tags...) • sensors • positioning (GPS, Wifi, CellID...) • other information (pressure, temperature, etc.) • communication (tcp/ip, BT/ Wifi, zigbee, redTacton...) • computation

Slide 6: Output digital physical digital desktop computers 3D printing, rapid game consoles prototyping Input physical digital overlay (augmented reality, visualization) physical activities locative media architecture as an interface lifelogging+virtual worlds a sort of model....input / output, digital/physical matrix digital/physical I/O intentional/unintentional output

Slide 7: augmented/mixed reality: combination of real world and computer-generated data

Slide 8: visualizing the hybridization S.O.U.P provides a visual and acoustic representation of the omnipresent wireless communication networks which surrounds us in urban spaces. think of our application as a soup ladle spooning arbitrarily up the boiling air and transforming the data into sound and video. A computer terminal shows a map of the surroundings plus basic points of reference of the city or town that hosts physically our installation. the observer now can jump or fly to some places in the city and choose from different datasets recorded at different daytimes. the software will provide a visual and acoustic representation using surround sound and a beame Sky Ear is a non-rigid carbon-fibre \"cloud\", embedded with one thousand glowing helium balloons and several dozen mobile phones. The balloons contain miniature sensor circuits that respond to electromagnetic fields, particularly those of mobile phones. When activated, the sensor circuits co-ordinate to cause ultra-bright coloured LEDs to illuminate. The 30m cloud glows and flickers brightly as it floats across the sky.

Slide 9: (Picture: Blast Theory) location-tracking: tracing people in real space

Slide 10: location-based services Examples of location-based applications... large array of purposes (security, play, serendipity...)

Slide 11: urban tapestries: digital annotation of space Urban Tapestries (http://research.urbantapestries.net)

Slide 12: urban tapestries Urban Tapestries (http://research.urbantapestries.net)

Slide 13: mobile gaming...

Slide 14: environment overlap (physical+digital) but immobile the overlapping of two environments

Slide 15: Mogi: environment overlap (mobile) Screenshot of Mogi (Newt Games) taken from “The Network is the Game: Social Trends in Mobile Entertainment” (Talk by Amy Jo Kim)

Slide 16: Mogi: environment overlap (mobile) Screenshot of Mogi (Newt Games) taken from “The Network is the Game: Social Trends in Mobile Entertainment” (Talk by Amy Jo Kim)

Slide 17: explicit invisible/implicit phenomena (Beatriz da Costa, 2006) How to use technologies to make certain phenomena that are implicit/invisible more explicit. Example: the blogging pigeon project (UC Irvine) that use GPS/sensors on pigeons to detect pollutions on the city of San José and give the citizens a representation of the environment.

Slide 18: Premptive media’s Area Immediate Reading AIR is a public, social experiment in which people are invited to use Preemptive Media's portable air monitoring devices to explore their neighborhoods and urban environments for pollution and fossil fuel burning hotspots. Participants or \"carriers\" are able to see pollutant levels in their current locations, as well as simultaneously view measurements from the other AIR devices in the network. An on-board GPS unit and digital compass, combined with a database of known pollution sources such as power plants and heavy industries, allow carriers to see their distance from polluters as well. The AIR devices regularly transmit data to a central database allowing for real time data visualization on this website.

Slide 20: tripwire (Tad Hirsch) Tripwire (Tad Hirsch) San Jose International Airport and downtown San Jose, CA. Custom-built sensors hidden inside coconuts are hung from trees at several public locations to monitor noise produced by overflying aircraft. Detection of excessive aircraft noise triggers automated telephone calls to the airport's complaint line on behalf of the city's residents and wildlife. Documentation of noise incidents is archived for later analysis

Slide 21: pedometers beyond lifelogging Teku Teku angel and v-migo

Slide 22: ubiquitous computing of today

Slide 23: actually, not quite new? (“piazza-listening device” by Athanasius Kirchner, 1650) Athanasius Kircher was a 17th century German Jesuit scholar who invented a large amount of interesting artifacts ranging from speaking tube, perpetual motion machines to cat pianos. Among the devices this Renaissance man created, there are two that I found amazingly intriguing, in terms of ubiquitous computing. a “piazza-listening device”. As described in this paper, “the voices from the piazza are taken by the horn up through the mouth of the statue in the room on the piano nobile above, allowing both espionage and the appearance of a miraculous event“.

Slide 24: UT

Slide 25: “Once devices were \"Location-aware service location-aware, business integration into applications applications were expected began. A critical mass of to take advantage of the network and device support capabilities in the next two will occur through 2006.\" to five years“ Gartner, 2003 Gartner, 2006 A proximal future... ...infinitely postponed? Consultants/trendwatchers/analyst keep repeating the same predictions for 4-5 years... but they postpone the proximal vision of location-aware services. Something is wrong here. See: Bell, G. & Dourish, P. (2006). Yesterday’s tomorrow: notes on ubiquitous computing’s dominant vision, Personal and Ubiquitous Computing. The example her is about location-aware services but analysts say the same with the intelligent fridge, ambient computing, etc.

Slide 26: why? design challenges

Slide 27: a common belief: The broken cloud of connectivity Transitions operating smoothly, seamlessness, invisibility, calmness are often believed to be true

Slide 28: infrastructures are not seamless = difficult to provide a continuous experience More about this: Dourish, P. & Bell, G. (2007). The Infrastructure of Experience and the Experience of Infrastructure: Meaning and Structure in Everyday Encounters with Space. Environment and Planning B,

Slide 29: fragmented locus of interactions computers, not switches! When the locus of interaction is everywhere, there are lots of point of entry more dificult to convey a sense... it’s then more complex than the thermostat/ light bulbs Because the interaction is in the air, people are not prepared, expectations are more complicated and the purpose of the interaction is less clear (and these points of entry are much more complicated than switches)

Slide 30: other actors... surprised

Slide 31: digital is not immaterial = it needs big infrastructures See Graham, S., Strategies for Networked Cities

Slide 32: odd/wrong assumptions about people needs and desire (people can’t relate to this sort of graph) Based on fake/wrong/flawed assumptions about people needs/desire (automation)

Slide 33: recurring instrumental and utilitarian purposes The intelligent fridge = posterchild, well it’s a bad posterchild

Slide 34: normative futures (infinitely postponed) More on that: Bell, G. & Dourish, P. (2006). Yesterday’s tomorrow: notes on ubiquitous computing’s dominant vision, Personal and Ubiquitous Computing.

Slide 35: space is not homogenous Picture by Patrick Jermann dierent places, dierent infrastructures, rugosity

Slide 36: often showing negative aspects of life It seems that ubicomp applications have a fascination towards showing negative aspects of life (pollution, disappearing of animals...)

Slide 37: alternative visions I don’t have the answer to all those issue but I do think we can use them as challenges to design internet of things applications

Slide 38: assist, not automate allow the system to convey intentionality! panopticon versus masquerade First, there is a big dierence between completely autonomous systems and assisting human users. The idea is to augment awareness not automate it, the final interpretation should be left to the humans. A good example of such is Jaiku that respect the intentionality of the message that is conveyed: rich presence rather than location. Then of course, the cost for the user is high, but the benefit for the others is high too: they know the intention of the sender! Also an obvious but important aspect is to facilitate the opt-out and to lie. In the same line, in terms of privacy what people do not like is the feeling to be seen without the ability to see (Jeremy Bentham’s Panopticon) as Michel Foucault argued. BUT people are OK to disclose things when they can control what they see and when they can see others.

Slide 39: utilitarian but less instrumental “isolatr”, a system that aims at helping where other people are not. It’s a joke but still is relevant as a way to express some concerns

Slide 40: “we don’t want to invent people’s life” rafi haladjian be humble... rafi haladjian... open source, “we don’t want to invent people’s life”

Slide 41: offline gaming - tracking physical activity Project currently being done with Julian Bleecker (USC) to explore ofline gaming (physicality+ asynchrony of the results)

Slide 42: blogject (blog-objects) blog - object: an object that would be able to create and circulate its own content based on its interactions with the environment and people history of things, blogjects, trajectories, add values... POSITIVE HISTORIES WITH OBJECTS

Slide 43: turn history of interactions in a continuous experience, object learn from their histories object have histories imagine if objects could learn from the interactions they have with their owner, other people or other objects and if this history could be used change the capabilities of the object?

Slide 44: new place that take seams into account (Simon Schleicher, MIT Medialab, 2006) Also of interest, some project tries to design/build physical structure that would take seams/ problems into account (for example GPS holes/network problems...)

Slide 45: from lifelogging to new interaction partners (Picture found on the Internet) Julian Bleecker and myself are working on this project: how to engage new partners such as pets on the social web: we have a dwarf on World of Warcraft that is played by a dog (sensors track its physical activities).

Slide 46: new interactions partners: twitter cat This is another project that we work on with Julian.

Slide 47: conclusions

Slide 48: what changes? • space • new layer of information • being here and there • hybridization between physical and digitalS • interaction: fragmented, concurrency • production and use of traces of interactions • privacy!

Slide 49: a complex and unpredictable design ecology? • beyond user-centered design: contexts (overlapping space, people participating or not) • problems: • seams, digital is material • people’s needs and desires are balanced between utilitarian and playful

Slide 50: other models than throwing technology to people? Future/reality is in between these thank you - nicolas.nova@epfl.ch