THE WORLD IS THE SCREEN Elements of Information Environments IA Summit 2013 | Baltimore, MD | Andrew Hinton | @inkblurtI’m Andrew Hinton, and I’m an information architect with The Understanding Group.Today we’re going to be talking about information environments -- and as a way into thatconversation, I’ve titled this talk “the world is the screen” -- so let’s start by considering whatI mean by that.
Andrew Hinton | @inkblurtWe’re surrounded by screens these days. They’re proliferating to the point where we’reinteracting with them as often as any other objects in our surroundings.So, in a way, we might say that screens are ﬁlling up the world to the point that it’ll feel likethe world is made of them.There’s some truth to that. But it’s just one facet of the issue.kiosk: kodak.comtable/phone: android.comgps: garmin.com
Andrew Hinton | @inkblurtAnother way the world is the screen is because of technology like Google Glass, whichessentially lays a screen over the world around us, mediating between our perception and thestuff we’re perceiving.This is certainly worth considering, but it, too, is only a facet of how the world is the screen.It’s also the case that these things -- all sorts of device screens and augmented displays -are getting integrated into our environmental experience.left image wired.com / others from google
Andrew Hinton | @inkblurtThe way we use these things isn’t conﬁned to the things themselves. They’re part of a largercontext.Here, on an airliner, I surreptitiously snapped a picture while waiting for take-off. I did itbecause it’s a good example of how we don’t just sit in front of screens alone, in a vacuum.We do it as part of our activity in the world around us.
Andrew Hinton | @inkblurtIn this case, this guy was talking to friends about a football game in progress -- a game thathad a mirror-world of itself happening in a little avatar of itself on a smartphone.There’s a relationship between the digitally generated “information” environments we use,and the non-digital environments we live in. We live in both. I wondered, at the core of howwe understand the world, are there differences between them? Do they matter?
INFORMATIONENVIRONMENTINFORMATIONENVIRONMENT 6 Andrew Hinton | @inkblurtFor example. What is the difference, really, between shopping for office supplies in a brickand mortar retail store, and shopping for them through an online retailer like Amazon?If we frame both of these as “information environments” -- does that help us understand whyone might be eating the other’s lunch?If we think of the physical store as an information interface, how much information isconveyed, and of what kind, through one interface versus another?
A CURATED, ENCLOSED, BLENDEDINFORMATION ENVIRONMENT Andrew Hinton | @inkblurtWhat about when information environments use all sorts of methods for communication, allat once?We’ve had blended information environments for a while. Here’s a marvelous exhibit at theAmerican Museum of Natural History in New York. It has a whole taxonomy -- the old-schoolmeaning of taxonomy, meaning an organized hierarchy of creatures. But this one isinstantiated on the wall. It also has it represented in written form, in a printed document, andin digitally presented form, in a kiosk. (These were taken when I took my daughter theresome years back.)This is a curated, complex information environment.Physical objects, digital interfaces, lots of language and labels around. All connected togetherto form a whole experience.It’s is a highly controlled version of the world we now live in -- which is more emergent,messier, but even more pervasively connected & digitally enabled.photos by andrew hinton
AN OPEN-ENDED, CUMULATIVE,BLENDED INFORMATION ENVIRONMENT Andrew Hinton | @inkblurtHere’s an everyday intersection in Dublin.This is an environment that also has many different layers and modalities, but it’s notcontrolled and curated in the same way as the museum. It’s been added to, streets have beenwidened, signage added, infrastructure installed. And on top of that, lots of other informationis pouring through it in the form of newspapers, or advertisements on buses and vans.The digital signs are something relatively new for our environments. It used to be that signssaid one thing, and you learned what they said, and then you could forget about them untilsomebody put up a new one. These days, we can’t depend on surfaces being stable,persistent homes for written information. The stuff is embedded in all sorts of places. Thisstreet intersection in Dublin has digital signage mixed in with everything else.Pervasive computing technology means that the world is only getting weirder and morecomplex. We’re not talking about just consumer devices, but whole infrastructures, urbannetworks, and wired economies.photo by andrew hinton
What do we mean by “Information Environment”?? Andrew Hinton | @inkblurtFor over a decade, we’ve been saying that IA is, in part, the structural design of sharedinformation environments. But what do we mean by that phrase? It sounded right at the time,because even when most of what we were doing were static websites, we knew the scopecould be bigger, and that the world was going to change toward more complexity.So, here we are in that spot we supposed we’d be in -- with all this pervasive informationcomplexity - and it seems high time to nail this thing down better.
Labels Card Sorting Mental Models Navigation Methods, Facets Tools, Controlled Affinity Diagrams Taxonomies Hyperlinks Processes Vocabularies Thesaurus Task Analysis Hierarchies Ontologies Context Models What’s underneath that makes these things work (or not)? Andrew Hinton | @inkblurtI’m actually a bit worried because I’m not sure our current tool sets are really up to the task.We have a lot of methods, but not a lot of understanding about why or how they actuallywork. (Kind of like antidepressants.)We also tend to talk about a lot of things like “understanding” and “information” and whatnot-- but what do we mean by those things? We need more rigor, more science - I don’t meaninformation science but science about humans.I’ve been working on a book about how information creates and shapes context. And in partof that work, I’ve had the realization that we’re often looking at information andenvironments the wrong way around, by starting with the technology ﬁrst.(8 min)
A STRUCTURALLY DESIGNED ENVIRONMENT ^ INFORMATION Andrew Hinton | @inkblurtEspecially since all the technology is becoming more and more pervasively integrated into oursurroundings, now I’m thinking we should start with something more basic -- how do wecomprehend our environment generally? What if we start with pre-digital structurallydesigned environments?>>And even further: is something like this ﬁeld not just an environment, but an informationenvironment?I believe it is. We’ll get to that, but ﬁrst something from ten years ago.http://commons.wikimedia.org/wiki/File:Derbyshire_Landscape.jpg
STEWART BRAND “PACE LAYERS” IA SUMMIT 2003 Andrew Hinton | @inkblurtBack in 2003, at the IA Summit in Portland Oregon, Stewart Brand gave the opening keynote.One of the things he discussed was “pace layers” - the idea that some layers of human lifemove more slowly than others. Nature changes very slowly, and all the stuff we’ve built upfrom that foundation tends to move and change more quickly -- quicker and quicker still ateach concentric layer.photo: Mike Leehttp://www.ﬂickr.com/photos/curiouslee/15238458/sizes/o/in/photostream/
PACE LAYERS OFINFORMATION ENVIRONMENTS Start Here Andrew Hinton | @inkblurtTaking that idea and running with it, I’m working out a sort of pace layer stack aboutinformation environments.At the root is our perception and cognition of environment -- these are things that don’t, atcore, change much at all over millennia.Then there’s spoken language, something we’ve had with us possibly for over a million years-- to the point that it’s probably a shaping factor in our evolution as a species.Writing and graphical symbolic language come later than speech. They’re technologies, in asense, for encoding, recording and sharing spoken language.And only later do we get into information organization and design, or what we call“information technology” -- digital computing, networks, & devices.We tend to start our work through the lens of the upper two layers - but they’re the ones thatchange and ﬂuctuate the fastest.>> I think we should start with perception/cognition as the lens for understanding the rest.
Information in Three Modes Digital 10100010 10100010 Digital systems transmitting 01001000 01001000 to & receiving from other 01110011 01110011 digital systems. Semantic People communicating with people. Animals (including Ecological people) perceiving the environment. Andrew Hinton | @inkblurtThere’s a long history of people trying to deﬁne information. I’m not into deﬁning things somuch these days -- I’m more interested in describing them.And that frees us up to understand a thing in more than one mode or dimension -- to be OKwith grasping something in all its facets.Rather than deﬁning information, I’d like to describe how it operates. I think informationaffects perception and understanding in three major modes. Let me mention them all, thenwe’ll look at each in more detail.>> First is “ecological” information. It’s about how animals perceive their environment.>> The second is “semantic” information: it’s the mode people use to communicate with oneanother.>> Third is digital information: digital information is information used by digital systems totransmit to and receive from other digital systems. It’s what happens between the blackboxes of our digital infrastructure.Like I said, we’ll look at each of these more closely. Let’s start with ecological information.(12 min)
Ecological Information Digital 10100010 10100010 Digital systems transmitting 01001000 01001000 to & receiving from other 01110011 01110011 digital systems. Semantic People communicating with people. Animals (including Ecological people) perceiving the environment. Andrew Hinton | @inkblurtSo, starting with ecological information. The word ecological means having to do with therelationship between an animal and its natural environment. I’m using the term this waybecause many of the ideas I’m using are based on ecological psychology and embodiedcognition, which is different from mainstream cognitive science.
Mainstream Cognitive Science Assumes the brain ... • Works like a computer to “process information.” • Uses symbolic logic, “images” & representational models. • Is primarily (if not exclusively) responsible for cognition. Andrew Hinton | @inkblurtMainstream cognitive science, which still forms the foundation for most HCI theory andpractice, assumes that the brain works like a computer as a sort of information processor.The brain takes sensory inputs from a sort of dumb, robotic body, processes those inputs as“information” -- representational images and symbols of the world, along with images andsymbols stored in memory -- and once it has ﬁgured out what to do, it tells the body how torespond.This is still the predominant way of seeing how the brain works. It’s part of the assumptionsbuilt into many of our methods and training.
Embodied Cognition (not yet mainstream) Andrew Hinton | @inkblurtThen there’s embodied cognition theory. Embodied cognition argues that cognition is notbrain-exclusive, but actually uses the body and even the environment around the body forcognitive activity.There are many ﬂavors and schools of thought even within the embodiment movement; butone in particular is what some call “radical embodied cognition” that says we should not tryto marry embodiment with the traditional cognitive science perspective, but replace itentirely. Full disclosure: the ‘radical’ or ‘replacement’ camp is the one I ﬁnd myself aligningwith.
James J Gibson - Ecological Psychology of Perception Long sidelined, now hailed as pioneer of embodied cognition. Andrew Hinton | @inkblurtThe so-called ‘radical embodiment’ movement has adopted the work of James J Gibson, whowas a scientist of something called “ecological psychology” in the mid 20th century..He started out studying WWII pilots - and found that centuries-old assumptions about howpeople comprehend their environment were simply wrong. His ideas have been acknowledgedand quasi-appropriated here and there, but now many are starting to see his whole corpus ofthought more clearly -- he was really writing about embodied cognition (but calling itecological psychology).
A few key ideas from Gibson’s theories We perceive elements in the environment as invariant (persistent) or variant (in ﬂux). We perceive the environment in human- scale terms, not scientiﬁc abstractions. We perceive environment as “nested,” not in logical hierarchy. Andrew Hinton | @inkblurtThere’s no way to cover all the important stuff from Gibson in this talk, but here are a fewkey ideas.>> We perceive elements in the environment as invariant or variant. Invariant elements arenecessary for orientation of everything else. They include at the widest scale, the earth andthe sky. Or perhaps a mountain range. Or even the occluding edge of one’s nose. Variantelements are things that are in ﬂux that we don’t rely on for their persistent structures.>> We perceive the environment in human-scale terms, not scientiﬁc abstractions.Perception doesn’t grasp the abstraction of space or time. Our bodies don’t perceive a fallentree limb in terms of centimeters, but in terms of whether it will ﬁt in the hand, or if it’s tooheavy to pick up.>> We perceive the environment as nested. A stream is nested between banks, which arenested between hills, which are nested within a range of larger hills, all of which is nestedwithin the canopy of sky. This is importantly different from strict hierarchy, though. Itoverlaps and shifts depending on the activity of the perceiver. A cave might feel like “inside”but then feel like “outside” when rain starts leaking in. A stone may just be clutter to mewhen I walk by it the ﬁrst time, but when I need a stone to pound something, it becomes anobject I can pick up. Then when I pick it up, it becomes an extension of my body.All of these are important ideas for the structures we make for digital and other systems,because our cognition expects the world to accommodate these ways of perceiving.
AFFORDANCE “... the perceived functional properties of objects, places and events in relation to an individual perceiver.” - JJ Gibson Perception exists only insofar as we perceive affordances. Andrew Hinton | @inkblurtJJ Gibson invented the concept of affordance. Others have since popularized it, but gotten itsomewhat wrong -- mainly because they’re coming at it from a traditional cognitive-scienceperspective, not an embodied perspective.For Gibson, affordance isn’t a thing you add to something. Affordance is the organizingprinciple behind *ALL* perception. We don’t perceive anything unless it affords meaningfulaction for a given context.
Information Pickup Theory Perception Ambient, structured Agent energy arrays Action Ecological Information "Pick-up" The perception-action loop. Andrew Hinton | @inkblurtWe perceive affordance through something called “information pick-up.”A perceiver, or agent, takes *action* in an environment in order to discover its affordances.The action part is very important. Gibson rails against traditional cognitive sciencelaboratories that strap people into chairs to keep their heads still -- cognition doesn’tfunction from stationary positions. We evolved as active, moving, interacting creatures thatperceive through action.And when we act in the environment, we perceive, which then affects our action, which thenaffects what we perceive, in a continuous loop of cognitive activity.This is a very different way of thinking about “information” - but it’s valid, and forms thebasis for all the other sorts of information in our lives.
Sigmund Andrew Hinton | @inkblurtI’ve seen this with my dog, Sigmund. When I try taking him for a walk, he’ll stop as if theground has grabbed him. Sometimes I’ll let him explore to see what’s up, and it’s almostalways something that I didn’t perceive the way he did - either because it wasn’t relevant orbecause I physically can’t perceive it. I’ve learned a lot by watching my dog ﬁgure out theworld. It’s not that different from us. He just doesn’t have the rich layer of language drapedacross the world like we do.It’s that layer of language that humans have added to the environment that makes up thenext information mode.(+7 = 19 min)
Semantic Information Digital 10100010 10100010 Digital systems transmitting 01001000 01001000 to & receiving from other 01110011 01110011 digital systems. Semantic People communicating with people. Animals (including Ecological people) perceiving the environment. Andrew Hinton | @inkblurtThe semantic mode, in short, is language. But I mean language in the broad sense of thingswe put into the environment to communicate with people. This can be all sorts of stuff:speech, gestures, text, iconography, even buildings have semantic qualities.
flickr - uicdigital SEMANTIC INFORMATION CHANGES HOW WE EXPERIENCE ENVIRONMENT Andrew Hinton | @inkblurtInformation (in the sense we tend to mean it colloquially) is what creates and changes muchof what we consider to be contextual reality.Look at this photo -- there’s information everywhere in this scene.>>The lines on the road tell us where to drive; the traffic light is a virtual barrier that affectsour behavior; the road signs give us a layer of instruction that adds meaning to the cityaround us. without the information here, it would quite literally be a different place.Really, you could have civilization without cars, lightposts and buildings, but you couldn’thave it without language. Language is our reality in many ways. And a city is as much aconstruct made of language - speech as well as labels, signs, other semantic artifacts - asone made of atoms.http://www.ﬂickr.com/photos/uicdigital/5410417461/
WE LIVE IN LANGUAGE STRUCTURES Andrew Hinton | @inkblurtIn a presentation last year, I heard Peter Merholz talk about how a cube farm in an officebuilding is like the org chart “made manifest.”That’s due to the fact that language structures are an architecture that we live withintogether, whether we know it or not. Whether these structures are deﬁned explicitly like inthis early IBM management diagram, or deﬁned tacitly through the collective assumptionswithin a shared culture, the way we talk and write about our shared environment is also astructural feature of that environment.
LANGUAGE IS ENVIRONMENT Language is “a form of cognitive scaffolding...” - Andy Clark - Supersizing the Mind Andrew Hinton | @inkblurtLanguage is not information. Language is environment.When I am speaking I’m vibrating the air - affecting the environment, putting structures intoit that weren’t there before. The same goes for writing - it’s environmental structure we’readding to the world.We then pick up information about what those environmental features mean; you hear thevibrations and, because you’ve learned what those words mean, they have affordance for you.
Ecological & Semantic Information In Conflict Andrew Hinton | @inkblurtDon Norman famously talks about the affordances of door handles.In this case, a similar affordance situation can help us understand how different modes ofinformation can be in conﬂict.I was walking into a store and did not even notice the sign.The language of “Pull” had an intended affordance -- I’m supposed to read it and allow it tocontrol my action.But the ecological information I picked up from the structure of the handle had a strongereffect on my action. I was talking with someone as I was entering the store, so my languageperception was preoccupied; also I could see through the glass into the store toward thecontext I intended to enter -- essentially seeing right past the sign.
Digital-Ecological & Semantic Information In Conflict Which red x???? Looks like a “confirm” action. Andrew Hinton | @inkblurtGraphical user interfaces are essentially simulated ecological information. Objects withaffordances, simulated on screens.But they’re also performing semantically. It can get very confusing.Logically speaking, the red X’s in the ﬁrst example are all very different -- but ecologically,they require too much thought to disambiguate. In this app I found myself always deletingrather than declining, closing rather than deleting, etc. When I’m in a hurry, I just reach forthe closest red X to do whatever I’m trying to do - close the message, decline an invite, ordelete it entirely. About half the time, I end up clicking the wrong one if I don’t stop andthink about it explicitly.>>In an unsubscribe interface for fab.com, my wife discovered that she was apparently re-subscribing without realizing it, because that big red button -- like a big berry you can’t helpbut pick -- contextually feels like it’s a conﬁrmation, not a cancellation/re-subscriptionaction.
Which of these will accidentally tweet publicly? Very little semantic or ecological information about what context I’m in Ecological Information / Affordance for action. Andrew Hinton | @inkblurtThe infamous Twitter “DM Fail” problem is largely caused by users responding to DMs viaSMS.In this case, it’s hard to tell: which of these is a Twitter app that will safely allow me to DMsomeone, and which is my SMS app that will tweet to everyone who follows me? Thephysicality of the interface can easily override my perception of the semantic information’sdifferentiating cues.
Digital Information Digital 10100010 10100010 Digital systems transmitting 01001000 01001000 to & receiving from other 01110011 01110011 digital systems. Semantic People communicating with people. Animals (including Ecological people) perceiving the environment. Andrew Hinton | @inkblurtSo the examples we just looked at weren’t just any sort of semantic information, they weresemantic information driven by digital technology. And digital technology relies on digitalinformation.Digital information is how the black boxes talk to the other black boxes. It’s the lifeblood ofinformation technology. The whole point of digital information is to strip human meaning outof it to make it efficient for transmitting and storing encoded information.This isn’t stuff we see face to face very much. Mainly we encounter its *effects* in theenvironment.
Digital Information Mode Leaking into Semantic Environment Andrew Hinton | @inkblurtWe see machines around us trying to get us to perceive what they are saying, or what theywant to hear from us. We see them murmuring to each other in weird, noisy machine-onlysemantics that we do not comprehend either ecologically or semantically.•The gas pump above has to have a sticker added to it that explains what “Enter Data”means.>>The Twitter proﬁle with the iPhone coordinates expresses my location not in a semanticway (the name of a city, for instance) but in a Cartesian grid that I have no contextualorientation for, either semantically or ecologically.>>The Delta app has information that I, as a human, can read, but it gives priority to themachines that I encounter in the workﬂow of the airport.
Digital information enables pervasive semantic place-making. Andrew Hinton | @inkblurtI don’t mean to paint digital information as a villain. It isn’t.The ability to transmit, store and retrieve information in this way is a miracle.An platform I like a lot is Avocado - it lets a couple keep in touch and share a place together,pervasively.It has nice touches that key into embodied experience of semantic information, like sending ahug by touching the screen to your heart.Another nice touch: the couple shares the same password - making a word into a very reallink of co-ownership of the place, like having the same keys to your home.This sort of pervasively available place would be impossible without digital information in thebackground. But it also requires a lot of discipline with semantic information structure tomake the place coherent.
INFORMATION MAKES PLACES, KIND OF LIKE THIS PICTURE MAKES A PIPE. IF YOU COULD SMOKE THE PIPE. Andrew Hinton | @inkblurtThis is the famous Magritte painting -- it says “this is not a pipe”The picture deﬁnitely shows a pipe but it’s not a real pipe you can smoke.>>Information is kind of like this in the way it makes places.>>Except for a key difference that, withInformation, you can smoke the pipe.
LABEL LANGUAGE IS LABEL LABEL INFRASTRUCTURE RULES LABEL LABEL Andrew Hinton | @inkblurtAnd Language is Infrastructure.We essentially make things out of labels, connections and rules.Too often, we assume the labels are something to add later - but in reality they’re the thingwe have to ﬁgure out ﬁrst.This is why issues like ontology and taxonomy are so important - they establish the“invariant” features of the environments we make.
ONTOLOGY What am I? What is my world? How do I exist in it? Please describe a formal, explicit speciﬁcation of a shared conceptualization for purposes of structuring semantic data. 00101011100100101110100101 Andrew Hinton | @inkblurtBehind the scenes of all this is Ontology.Ontology can be the philosophical sort -- about the nature of one’s being and therelationship of the self to the environment.Or it can be the information-technology sort - developed for digital information work, todeﬁne the formal speciﬁcation for data purposes.A big part of what IA should be doing is bridging these two planes of existence.
ONTOLOGY The Thing Andrew Hinton | @inkblurtPresently our various ﬁelds are preoccupied with how to have content and functionality makesense in various contexts.Ontology is at the heart of this problem.In many organizations and project teams, there’s an over-obsession with things like layout ineach of the instantiations of a thing, but not enough discussion about how to deﬁne thenature of the thing in abstract. That requires an ontological perspective.And, done properly, it forms the main structures of an information environment - theinvariant pillars, so to speak - that allow language to stitch together coherence acrosschannels.
What is “card” in this environment? Andrew Hinton | @inkblurtHere’s an ontology example.Lowes launched a service called MyLowes -- that requires the registration of a card. But theyalso have a “Lowe’s Card” that’s a consumer credit card.Conversations at checkout can end up like a “who’s on ﬁrst” routine -- “do you have yourLowe’s card?” “My Lowe’s card? That’s what I’m paying with.” “No I mean your ‘my lowe’s’card.” “This IS my lowe’s card!”
Semantic-information “place” signified by “account” Digital architecture determining ecological & semantic context. Andrew Hinton | @inkblurtIf I walked into a bank and asked to access an account, it’d be clear what I meant. But online,it can mean different things (my proﬁle-account represents me in the digital context -- andneeds a label, which happens to also be “account”). The digital systems behind the scenes atKohls require that these two things we call “account” be separate - requiring disambiguation.The ontology of ‘account’ is in question here. It’s one of the many sorts of things we have tosort out with language, when we’re working in an environment that’s made of almost nothing*but* language.
Shopping Simultaneously in a Store & the Cloud Andrew Hinton | @inkblurtNow that retailers are trying to be in the cloud and on the ground at the same time, contextis especially confounding. It requires a great deal of work to situate the user’s perception ofplace.For many retailers, product price and availability are driven by location - yet shoppers onlinetend to come to the experience as if it’s a cloud-based store, not thinking about geographyyet. It puts the user in a strange environmental position of being in a local store and in anamorphous web-shop experience simultaneously. The ontology of place is dissonant.
Subway station + Food store Andrew Hinton | @inkblurtAnd here we have a situation where a subway station is also ﬁlled with pictures of productsthat you can actually buy -- not unlike Magritte pipes that you can smoke.With the QR code sprinkled throughout -- digital information wrapped in massive simulacraof ecological information -- plus the semantic information of labels/brands. This could havejust been a list of words with QR codes next to them, but perhaps wisely, the retailer decidedto create the place in our image, to help bring the “reality” of shopping for groceries intowhat would otherwise override perception as a subway station.
INFORMATION MODES & PACE LAYERS Digital Semantic Ecological Andrew Hinton | @inkblurtThe examples we’ve looked at are going to seem primitive in a matter of just a few years.So we need ways of breaking down whole environments into their essential elements - andthose elements are bound up in human perception & cognition.This has been a very cursory overview of what I hope are a useful beginning for principlesand frameworks for doing this work into the future.
THANKS! Digital Semantic Ecological Andrew Hinton | @inkblurtThe examples we’ve looked at are going to seem primitive in a matter of just a few years.So we need ways of breaking down whole environments into their essential elements - andthose elements are bound up in human perception & cognition.This has been a very cursory overview of what I hope are a useful beginning for principlesand frameworks for doing this work into the future.