This document discusses three paradigms of human-computer interaction (HCI): 1) Interaction as man-machine coupling, focusing on fixing problems in interaction and improving efficiency. 2) Interaction as information communication, focusing on accurate information transfer and addressing mismatches. 3) Interaction as phenomenologically situated, focusing on supporting existing situated activities without constraining interaction, and considering politics and values in design.

1. Sticky Data and Superstitious Patterns:
Visualization beyond Cognitivism
Dietmar Offenhuber, PhD
Northeastern University
@dietoff
offenhuber.net | northeastern.edu/visualization

2. Seeing as Thinking
"My contention is that the cognitive operations called thinking are
not the privilege of mental processes above and beyond perception
but the essential ingredients of perception itself. I am referring to
such operations as active exploration, selection, grasping of
essentials, simplification, abstraction, analysis and synthesis,
completion, correction, comparison, problem solving, as well as
combining, separating, putting in context.
There is no basic difference in this respect between what happens
when a person looks at the world directly and when he sits with his
eyes closed and ‘thinks’."
Rudolf Arnheim, Visual Thinking, 1969

3. .. he describes his new theory of how the neocortex (the thinking part of
the brain) works: as a self-organizing hierarchical system of
pattern recognizers

4. “A decade-old toasted cheese sandwich said
to bear an image of the Virgin Mary has sold
on the eBay auction website for $28,000.”
BBC news, Nov. 23, 2004

5. Gizmodo - iPhoto Discovers Face in Delicious Chocolate Chip Cookie Dough

6. Riegler, Alexander. 2007. “Superstition in the Machine.” In Anticipatory Behavior in Adaptive Learning
Systems, edited by Martin V. Butz, Olivier Sigaud, Giovanni Pezzulo, and Gianluca Baldassarre,
4520:57–72. Berlin, Heidelberg: Springer.
“Under normal conditions, humans construct information
rather than process it. […] When it fails, cognition
switches to pattern processing.
There is a close relationship between pattern discovery
and superstition, since humans and animals alike excel at
finding structures where there are none.”
— Alexander Riegler

8. Gregor Aisch, 2011

9. Accountability Technologies
collecting data
coordinating volunteers
spreading the message
Guttenplag activity, visualization: User8, reproduced in:
Offenhuber, Dietmar, and Katja Schechtner, eds. 2013. Accountability Technologies
- Tools for Asking Hard Questions. Vienna: Ambra V.

10. 1. Sticky vs. non-sticky data
considering the context of data generation
2. Indexicality and evidence
visualization beyond symbolic conventions
3. System legibility
how representations mediate governance of systems

11. Sticky vs. non-sticky data
considering the context of data generation

12. Masaki Fujihata, Impressing Velocity, ICC, Tokyo 1992

13. wegzeit series
(2000-2002)
interactive software
Sonar 2002, Barcelona, ES
ISEA 2002, Nagoya, JP
Transmediale 2003, Berlin, DE
vectors online journal 2005

14. The meaning is
in the context
Drucker, Johanna. 2011. “Humanities Approaches to Graphical Display.” Digital Humanities Quarterly 5 (1).

15. Sorting Out Cities
Dietmar Offenhuber & Ars Electronica Futurelab
Miraikan Museum Tokyo, 2014

16. Video by YouTube users: Shunichi Kitamura, barmaglot0

17. Cities occupy 3% of the earth’s land surface. If
all urban areas would be combined in one large
city, it would be slightly larger than the size of
Europe, or about half the size of Brazil.
Data:
MODIS 500-m map of global urban extent — Center for
Sustainability and the Global Environment (SAGE), University of
Wisconsin-Madison

18. 53% of the global population lives in cities;
more than half of the global population lives in
Southeast Asia.
Data:
Center for International Earth Science Information Network - CIESIN -
Columbia University. 2014. Gridded Population of the World, Version
4 (GPWv4)

19. Grey Water Footprint
www.waterfootprint.org
Population Density
www.fao.org
Wage Density (Sweden).
(Charlotta Mellander, Kevin Stolarick, Zara
Matheson, José Lobo)

20. NASA black marble
dataset
http://earthobservatory.nasa.gov/Features/NightLights/

21. “even after factoring out country and
year fixed effects the simple correlation
between the standard deviation and
mean of lights is 0.88.”
Henderson, J. Vernon, Adam Storeygard, and David N. Weil. 2009. Measuring Economic
Growth from Outer Space. Working Paper 15199. National Bureau of Economic Research.

22. Sticky data: user-generated data resists the
transcendence of context

24. Categorizations and design
Peter Morville (2014)

25. trashsidewalk
street
left
pick
early
picked
pickup
days
bags
Topic 11
illegal
dumping
street
parking
sign
light
sidewalk
parked
faded
side
Topic 6
abandoned
bridge
bike
lane
stickers
remove
sidewalk
street
paint
car
Topic 22
water
sign
street
tire
bentparking
sidewalk
park
fountain
cars
Topic 7
dead
tree
trash
sidewalk
street
trees
gang
park
front
ice
Topic 20
street
parking
car
parked
sidewalk
light
parkbroken
side
blocking
Topic 12
Latent topics within the general “Other” category

26. Opacity
through
transparency

28. Urban Entropy (2015), Dietmar Offenhuber, Ars Electronica Center

29. Indexicality and evidence
visualization beyond the symbolic convention

30. Peircean Semiology revisited
iconic
symbolic
indexical
Peirce, Charles Sanders, Nathan Houser, and Christian J. W. Kloesel. 1998. The Essential
Peirce: Selected Philosophical Writings, 1893-1913. Indiana University Press.

31. iconic
symbolic
indexical
0
50
100
April May June July
0
50
100
April June
?

34. Kamel Makhloufi “function”, 2010
https://www.flickr.com/photos/melkaone/5121285002/

35. Wilks, Kiesl, Moser, 2007, Garden of Eden, Ars Electronica 2007

36. West Elevator
near Dunkin Donuts
descends to commuter rail
North Elevator
near northern entrance
descends to Northeastern entrance
East Elevator
central breezeway
descends to bus station
upperbreezewayinsideelevatorlowerplatform
Sonic VertiscapesSkye Morét
with Sam Auinger, Bruce Odland & Katrin Emler
coding assistance from Armin Akhavan
Sonic Commons Workshop 2015
Northeastern University + Harvard GSD
Constraints for generative
code.The soundscapes
below represent a 15 sec.
recording of each space.
frequency = step size
intensity (dB) = diameter
At Boston’s Ruggles Station I examined the sonic identity of spaces, listening closely to
soundscapes and their patterns and signatures.
While transiting normal pedestrian paths, I identified clear vertical sonic thresholds.
Elevators function as temporary aural escapes—passages from one soundscape to the
next. Below, I examine their sonic identity with generative code, allowing decibel and
frequency levels to determine the visual scape. For us to transition, as a society, toward a
multi-sensual approach to design, we must understand how our actions and use of space
manifest themselves within the sonic commons.
Skye Moret

37. Lorenzo Scarpelli, Xiaxin Chen, Sound Identity 2015

38. Lorenzo Scarpelli, Xiaxin Chen. Sound Identity. 2015

39. Design principles of indexical visualization:
framing, not mapping a phenomenon
Offenhuber, Dietmar, and Orkan Telhan. 2015. “Indexical Visualization – the Data-Less Information Display.” In Ubiquitous
Computing, Complexity and Culture, edited by Ulrik Ekman, Jay David Bolter, and Lily Diaz. New York: Crc Press.

40. infrastructure legibility and governance
how representations mediate governance of systems

41. Folk theories
Kempton, Willett. 1986. “Two Theories of Home Heat Control.” Cognitive Science 10 (1): 75–90.

42. Trash Track - Senseable City Lab 2009. Image: Christophe Chung

43. Trash Track - Senseable City Lab 2009. image: Dietmar Offenhuber

44. Paths of 2 Printer Cartridges

45. Items reported from this facility between Oct. 22 and Nov. 8.
some objects for several days. Most items came through the AW
recycling center.

47. IMPROSTRUCTURE
Dietmar Offenhuber (NU), Katja Schechtner (ADB), Julia Nebrija (Manila)

48. Photos & video: Konstantin Jagsch

51. Black market of electricity - Paco, Manila

52. Ad-hoc repairs, lighting hybrids (K. Jagsch)

56. Infrastructure Semiotics
Bgy. 849 entrance, Paco Manila

57. The representation of infrastructure shapes its
governance.
Offenhuber, Dietmar. (in press). Waste Is Information - Infrastructure, Representation,
and Accountability in the Digital Age. Cambridge, Mass.: MIT Press.

58. Paradigm 1 Paradigm 2 Paradigm 3
Metaphor of
interaction
Interaction as man-
machine coupling
Interaction as information
communication
Interaction as
phenomenologically situated
Central goal for
interaction
Optimizing fit between
man and machine
Optimizing accuracy and
efficiency of information
transfer
Support for situated action in the
world
Typical questions of
interest
How can we fix specific
problems that arise in
interaction?
What mismatches come up in
communication between
computers and people?
What existing situated activities
in the world should we support?
How can we accurately
model what people do?
How do users appropriate
technologies, and how can we
support those appropriations?
How can we improve the
efficiency of computer use?
How can we support interaction
without constraining it too
strongly by what a computer can
do or understand?
What are the politics and values
at the site of interaction, and how
can we support those in design?
Harrison, Steve, Deborah Tatar, and Phoebe Sengers. 2007. “The Three Paradigms of HCI.” In Alt. Chi. Session
at the SIGCHI Conference on Human Factors in Computing Systems San Jose, California, USA, 1–18.
The third paradigm of HCI