Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Meaning Modes in Design - Fluxible 2016

1,431 views

Published on

Meaning Modes in Design presented at Fluxible conference 25 September 2016

Published in: Design

Meaning Modes in Design - Fluxible 2016

  1. 1. MARSHA HAVERTY @mjane_h Meaning Modes in Design A Visual Tour Fluxible 2016
  2. 2. Invisible things about meaning… We’re going to take a visual tour of some invisible things about meaning, some properties of meaning, that matter with the kinds of designs we are bringing into the world these days.
  3. 3. Two JJGs Jessie James Garrett James Jerome Gibson 1950s -70s Image credit: @UXweek Twitter account Image credit: Wikipedia Before we visualize these properties, I must mention two JJGs. One JJG is Jessie James Garrett, our contemporary in experience design. The other is James Jerome Gibson, the founder of the field of ecological psychology, working half a century ago.
  4. 4. Two JJGs Jessie James Garrett James Jerome Gibson 2002 1950s -70s In 2002, Jessie James Garrett gave us an exploded planes view of the elements of user experience, in which the structure, skeleton, and surface planes treat how we use information as raw material in terms of its architecture, visualization, and perception.
  5. 5. Two JJGs Jessie James Garrett J.J. Gibson 2002 1979 J.J. Gibson gave us a naturalized view of how we perceive information, how we pick it up directly as we act in the environment. (For present work using the ecological framework developed by Gibson, see Golanka S., and Wilson A. (2013). Embodied cognition is not what you think it is, Frontiers in Psychology )
  6. 6. Two JJGs Jessie James Garrett J.J. Gibson MEANING And both of them worry about how information influences meaning.
  7. 7. Experience is impossible to model and fully design Rocket contrail by Seay Photography via Creative Commons Emergent Ineffable Intangible Subjective Jessie James Garrett Closing Plenary, IA Summit 2016 In his closing plenary in the 2016 Information Architecture Summit, Jessie James Garrett lamented with us that experience is impossible to model and fully design. It is emergent, subjective, intangible, ineffable.
  8. 8. Shifting patterns of awareness Frames of meaning An unfolding as a human engages a design Jessie James Garrett Closing Plenary, IA Summit 2016 Rocket contrail by Seay Photography via Creative Commons Experience is an unfolding as a human engages a design that creates shifting patterns of awareness, and frames of meaning that a human assigns to how that particular design in that particular moment-to-moment shifts her patterns of awareness.
  9. 9. How far can we go? Jessie James Garrett Closing Plenary, IA Summit 2016 Shifting patterns of awareness Frames of meaning Rocket contrail by Seay Photography via Creative Commons He challenges us as designers. How far can we go in understanding these shifting patterns of awareness and frames of meaning and apply that to our craft?
  10. 10. How we think about meaning affects what we can see… But, how we think about meaning affects what we can see.
  11. 11. Photo credit: Barcroft Media via Daily Mail, 3D image of human brain connections That these beautiful brains we walk around with have somehow become disconnected from the surrounding environment, the very thing that, over hundreds of thousands of years of human evolution, tuned us as tribal hunter-gatherer poets, is…a little odd.
  12. 12. Photo credit: Pasos Adalente That we somehow take stuff from our senses, process that so we can recreate it inside our brains, process it some more so we can manipulate it into calculated decisions which are then output as behaviors, is a very unnatural way to think about meaning.
  13. 13. From The world is the screen: elements of information environments, Andrew Hinton, 2013 IA Summit I’m ever grateful to Andrew Hinton for showing this slide in a talk he gave in 2013 giving us another way to think about meaning, from our midcentury JJ Gibson, that puts us and our brains back on the ground, immersed in our environment. Meaning doesn’t happen as an isolated thing only inside our brains; we’re soaking in it.
  14. 14. Meaning is ecological J.J. Gibson 1979 From our midcentury J.J. Gibson, we see that meaning is ecological because information is ecological.
  15. 15. Informationis ecological J.J. Gibson 1979
  16. 16. Visual Information is ecological
  17. 17. Visual Information is ecological Chair image credit: Yoichi Yamamoto surfaces edges textures We don’t have to have seen a chair from every possible perspective in every possible light to recognize the chair. It’s the relationships among the surfaces, edges, and textures that we recognize, that we pick up as invariant, that let us recognize the chair.
  18. 18. MechanicalInformation is ecological balance sound * touch cold warm pain
  19. 19. All Information is ecological Perceptual information * Language words on a surface words through the air Sabrina Golanka All perceptual information is ecological. And present-day embodied cognitive psychologists, in particular, Sabrina Golanka, are drawing on Gibson’s ecological psychology to frame how language, or conceptual information, though different from perceptual information, is ecological too. See Sabrina Golonka (2015) Laws and Conventions in Language-Related Behaviors, Ecological Psychology, 27:3, 236-250, DOI: 10.1080/10407413.2015.1068654
  20. 20. All Information is ecological Information
  21. 21. Meaning emerges InformationGoal-Directed Actions Meaning Meaning emerges when we act with the information in the world around us.
  22. 22. Meaning is dynamic InformationGoal-Directed Actions But this is not a static thing. Our goals change, our actions change, and the information in the environment changes. These things must adapt together. Meaning is dynamic, meaning is an event that unfolds over time.
  23. 23. Meaning is flow
  24. 24. Flows have properties
  25. 25. Meaningful action feels different depending on the properties of its flow
  26. 26. Meaning Modes As designers, if we make deliberate decisions about these properties, then we can facilitate particular meaning modes to suit our design needs.
  27. 27. Impact of a drop of water on a water surface by Roger McLassus https://commons.wikimedia.org/wiki/File:2006-01-28_Drop- impact.jpg Is information like water? To get at our first property of meaning modes, we ask the question: is information like water?
  28. 28. temperature pressure solid gas liquid The phase-space of water If we run all the permutations on the different relative amounts of temperature and pressure, we end up with what we know as the phase-space of water. The nature of water is drastically different as a solid vs. liquid or gas.
  29. 29. Perceptual information Language The phase-space of information words on a surface words through the air * What if we do the same thing for the two types of information: perceptual and linguistic? What is it like to interact with different combinations of perceptual and linguistic information? Do we find neighborhoods of combinations in which it is drastically different to engage with the meaning of information?
  30. 30. Perception flows easily Tacit Reflexive photo credit: Jason Pratt via Creative Commons Perceptual information flows easily. It is tacit and reflexive. Once we’ve learned to detect it, we don’t have think about it to engage with it. Perception flows easily. It has a low viscosity, like water.
  31. 31. Language is viscous, more work to flow Awareness Associativity Requires attention photo credit: edibleoffice via Creative Commons Language is viscous and takes more work to flow. It is laden with awareness and associativity, and requires active attention for us to engage it.
  32. 32. intense concentration intense coordination conceptual visceral Perceptual information Language Ease of flow of meaning Generally, the area dominated by perceptual information is reflexive. The area dominated by linguistic information is more attentive. But we can get more granular than that. If we have little or no language, all perceptual information, we are engaging with meaning in a very visceral manner. If we look at the area with little perceptual information, we are operating in the world of concepts. We are thinking conceptually to engage meaning. If we have a lot of language, especially if it’s abstract, we need intense concentration in this highest viscosity state. If we have a lot of perceptual information to deal with, we must engage with intense coordination. And when we are faced with a lot of information of either type, we may become overwhelmed.
  33. 33. intense concentration intense coordination conceptual visceral Perceptual information Language Before images After images Remember Twitter before images? This is a waybackmachine snapshot from 2011 [left]. Tweets were made of words and, because we are context jumping from one unrelated tweet to the next, it is a highly viscous way to engage with meaning. When Twitter introduced images to tweets, that disrupted the behaviors we had honed to engage with the meaning of twitter. Suddenly there were these perceptual swaths mixed in with the text. We got used to it, we developed new behavior for engaging Twitter, and we probably don’t think about it at all anymore, but Twitter changed. It phase-shifted. The nature of engaging with meaning on Twitter is not the same.
  34. 34. intense concentration intense coordination conceptual visceral Perceptual information Language Palette Gear The speaker yesterday showed us Palette Gear, physical controls that can be mapped to actions on tools like Light Room. This design phase- shifts what it’s like to engage with meaningful action in Light Room: it adds more perceptual information. It requires more coordination to engage, but lets photographers engage multiple simultaneous actions and do adjust the qualities of photos faster.
  35. 35. intense concentration intense coordination conceptual visceral Perceptual information Language VISCOSITY TOLERANCE PERSISTENCE PRESENCE Behavior precision That’s our first property of meaning modes: viscosity, or the ease of flow of meaning. Next, we’ll look at tolerance, or how precise our behavior must be to engage the information in a design to maintain the flow of meaning.
  36. 36. WIDE Tolerance Forgiving Information NARROW Tolerance Precise Breaks the flow of meaning Information Tolerance is about precision A design has wide tolerance if we can veer our behavior a bit and still maintain our engagement with the information. It’s forgiving. If a design has narrow tolerance, we must be very precise with our behavior and if we veer too much, we break the engagement with the information and break the flow of meaning.
  37. 37. It’s dark outside. You’re asleep.
  38. 38. You are drawn from sleep by a telltale hum…
  39. 39. You know what to do; you reach in the direction of the thing, tapping around on the glassy surface, trying to make the sound stop. The sound is still going, so you have to you to lift your head a little, open your eyes a little to see onto the bedside table and finally manage to hit the spot.
  40. 40. That snooze button on your phone may be plenty big to tap with your finger when you’re awake and have your vision light adjusted and are looking at the face of the phone at a direct angle. But, the intent of this snooze button, the meaningful engagement with this snooze button, is to extend the continuity of your dream state 8 minutes and 59 seconds longer. Struggling to find a button in the dark, lifting your head a bit, opening your eyes some to try to see, breaks the flow of meaning of extending your dream state. Or worse, in your tapping around, you may accidentally hit stop and break capped nature of the flow of meaning of snoozing.
  41. 41. NARROWTolerance Precise Imprecision as a design value WIDE Tolerance Forgiving Well-lit, Wide-awake behavior Night-blind, Dream state behavior The point of snooze is to keep us in our night-blind, dream state. But that requires a design that is forgiving of the kind of behavior we are capable of in that state: imprecise.
  42. 42. car UI concept and image from: http://matthaeuskrenn.com/new-car-ui/ Let’s look at a different example where imprecision can be a design value. The concept for a car dashboard control panel is just a blank screen. The human puts a finger anywhere on the screen and the UI comes to that spot. Depending on the number of fingers on the screen, the human can control different things: temperature, volume, etc. The human moves her finger generally up or down to make adjustments to the ambient comfort of the car.
  43. 43. Imprecision as a design value NARROWTolerance Precise WIDE Tolerance Forgiving This design has a wide tolerance and is very forgiving of the distracted nature of a driver of a car.
  44. 44. Let’s look at another example of a the common button layout design. You may have some perceptual memory that the climate control button is in the lower right…
  45. 45. But you still have to glance in order to see exactly where you need to press to hit the button and get to the climate controls.
  46. 46. You may also have perceptual anticipation that the temp buttons are in the upper left and right…
  47. 47. But you still must glance to bring your finger to the exact button location. And, if you want to adjust quite a bit, you must lift your finger off the button and return it to the exact spot to press several degrees, at the same skewed angle with respect to where you are sitting and facing in the driver’s seat.
  48. 48. Viscosity & Tolerance work together Tolerance Viscosity Tolerance Viscosity The viscosity is quite low for both of these designs, but the difference in tolerance makes or breaks the flow of meaning for driving a car.
  49. 49. Flow of meaning of driving a car Swath of attention Periphery of distraction Wayfinding Path Events Music/News Passengers The flow of meaning of driving a car involves wayfinding, continuously watching our path; watching out for events in our path that we must react to; we may be listening to music or news, and that can waft in and out of our attention; and we may be chatting with out passengers.
  50. 50. Flow of meaning of driving a car Swath of attention Periphery of distraction Wayfinding Path Events Music/News Passengers Comfort The need to glance at the buttons to engage their precision takes our vision off of watching for wayfinding cues and path events and breaks the flow of meaning of driving a car.
  51. 51. Flow of meaning of driving a car Swath of attention Periphery of distraction Wayfinding Path Events Music/News Passengers Comfort Voice commands What about forgetting the screen altogether and using voice commands? That uses different type of information, entirely different phase-space neighborhood (conceptual instead of visceral) and allows us to focus on the visual information we need for wayfinding and reacting to path events. But, it’s a higher viscosity and requires our active attention, higher still because we have to remember the precise words and speech syntax to verbally engage the environment controls of the system. So, we have to break our attention to listening to news/music and chatting with our passengers. We’re still breaking the flow of meaning of driving a car. We’re breaking less critical parts of that flow, but still.
  52. 52. Flow of meaning of driving a car Swath of attention Periphery of distraction Wayfinding Path Events Music/News Passengers Comfort The concept control panel precision and low viscosity allows us to engage it in the periphery of our distraction, and maintain the flow of meaning of driving a car.
  53. 53. Two sides to tolerance HUMAN BEHAVIOR SYSTEM BEHAVIOR WIDE NARROW TEMP VOL FAN But, there are two sides to tolerance. We’ve looked at the precision required for the human’s behavior. The design also has behavior and that has a tolerance too. We see that human engagement has a tolerance and system response has a tolerance. It’s not one or the other alone that accounts for the the tolerance of the meaning mode, but both feeding off of each other. Even though the human can veer a bit in how she engages the car control panel, the system doesn’t make wild changes in temp or volume. Temp still goes up and down by degree, and volume changes by a fine increment, and the fan speeds shift specifically. So we see that, the system has a precise behavior even though the human is allowed to have imprecise behavior to engage with it.
  54. 54. Two sides to tolerance HUMAN BEHAVIOR SYSTEM BEHAVIOR WIDE-NARROW Forgiving-Precise TEMP VOL FAN We’d call the overall human-system behavioral tolerance a forgiving-precise dynamic.
  55. 55. HUMAN BEHAVIOR—SYSTEM BEHAVIOR WIDE-NARROW Forgiving-Precise Adjusting while distracted Multi-tasking NARROW-NARROW NARROW-WIDE WIDE-WIDE Comfort The forgiving-precise dynamic is just one of the combinations.
  56. 56. HUMAN BEHAVIOR—SYSTEM BEHAVIOR Forgiving-Precise Adjusting while distracted Multi-tasking NARROW-NARROW Precise-Precise Fine-tuning Range of complex actions WIDE-NARROW CODING NARROW-WIDE WIDE-WIDE An example of a precise-precise, human-system dynamic is writing computer code. If we think about the flow of meaning of writing code, it’s a very high viscosity activity requiring lots of intense concentration to wield all of the abstractions; encode them in the precise syntax conventions of the particular language
  57. 57. HUMAN BEHAVIOR—DESIGN BEHAVIOR Forgiving-Precise Adjusting while distracted Multi-tasking NARROW-NARROW Precise-Precise Fine-tuning Range of complex actions WIDE-NARROW CODING NARROW-WIDE WIDE-WIDE Forgiving-Fuzzy Long-lead discovery Play/explore multiple possible worlds Code Review image from GitHub It’s mostly made of concepts and very persnickety syntax. There is some perceptual information in the nested commands, and some color coding for code reviews in some cases. The human must be extraordinarily precise with this language; the system behaves exactly as dictated in the code.
  58. 58. HUMAN BEHAVIOR—DESIGN BEHAVIOR Forgiving-Precise Adjusting while distracted Multi-tasking NARROW-NARROW Precise-Precise Fine-tuning Range of complex actions WIDE-NARROW NARROW-WIDE WIDE-WIDE 3D MODELING Mouse image credit: 3D Connecxion Model Image credit: MaTane Tutorials, Autodesk Inventor Mirroring, via YouTube Where I work at Autodesk, we make tools that help engineers and designers create things in the world, products, machinery, architecture, by way of 3D modeling.
  59. 59. HUMAN BEHAVIOR—DESIGN BEHAVIOR Forgiving-Precise Adjusting while distracted Multi-tasking NARROW-NARROW Precise-Precise Fine-tuning Range of complex actions WIDE-NARROW NARROW-WIDE WIDE-WIDE Forgiving-Fuzzy Long-lead discovery Play/explore multiple possible worlds 3D MODELING Mouse image credit: 3D Connecxion Model Image credit: MaTane Tutorials, Autodesk Inventor Mirroring, via YouTube Direct 3D modeling requires high precision behavior to select particular edges and faces and points on the geometry of the thing being made, and then make very precise adjustments to them. The system updates the visualization very precisely with each action. A lot of our customers use an exquisitely precise 3D mouse. This requires a lot of perceptual coordination and is a very visceral way of engaging with information.
  60. 60. HUMAN BEHAVIOR—DESIGN BEHAVIOR Forgiving-Precise Adjusting while distracted Multi-tasking NARROW-NARROW Precise-Precise Fine-tuning Range of complex actions WIDE-NARROW NARROW-WIDE WIDE-WIDE Forgiving-Fuzzy Long-lead discovery Play/explore multiple possible worlds 3D MODELING Mouse image credit: 3D Connecxion Digital Agent What about when we start offloading some of the modeling work to digital agents? The word agent implies agency, or the ability to act with some degree of freedom.
  61. 61. HUMAN BEHAVIOR—DESIGN BEHAVIOR Forgiving-Precise Adjusting while distracted Multi-tasking NARROW-NARROW Precise-Precise Fine-tuning Range of complex actions WIDE-NARROW NARROW-WIDE WIDE-WIDE Forgiving-Fuzzy Long-lead discovery Play/explore multiple possible worlds Digital Agent Constraints Instead of the human doing all the intricate geometry sculpting, she sets some constraints on what we’re making, and then the digital agent runs through all the different possibilities within those constraints and brings them back to the human to refine further.
  62. 62. Generative design image credit: Autodesk project Dreamcatcher The digital agent can then visualize all the possibilities within the human’s set constraints, and the human can make further adjustments on constraints. If the designer wants to explore options for chair design, the constraints may be a particular type of material, amount of material, requirements for weight that should be supported, that it should still have 4 legs. And the the agent comes back with the range of possibilities that the designer can filter further.
  63. 63. Generative design image credit: Autodesk Airbus partition project Or in designing a partition for a jet plane that is much lighter, uses less material, but still meets very important stress criteria.
  64. 64. HUMAN BEHAVIOR—SYSTEM BEHAVIOR NARROW-WIDE Precise-Fuzzy Quick-hit discovery Unknown possibilities GENERATIVE DESIGN Forgiving-Precise Adjusting while distracted Multi-tasking NARROW-NARROW Precise-Precise Fine-tuning Range of complex actions WIDE-NARROW NARROW-WIDE WIDE-WIDE 3D MODELING This type of generative design phase shifts our customers from the highly perceptual engagement with the direct geometry of the design that requires intense coordination in a precise-precise style of engagement between human and design tool, over to a conceptual mode of engaging in which our customers are no longer directly conduct the actual geometry of the design itself. Instead they precisely constrain the conceptual facets of the design and offload the geometry decisions to the design tool as digital agent. The tool becomes an agent that has agency to run through permutations and come up with possibilities for the designer to then refine further. These two different styles of modeling have different meaning modes in terms of viscosity and tolerance.
  65. 65. HUMAN BEHAVIOR—SYSTEM BEHAVIOR NARROW-WIDE Precise-Fuzzy Quick-hit discovery Unknown possibilities GENERATIVE DESIGN Forgiving-Precise Adjusting while distracted Multi-tasking NARROW-NARROW Precise-Precise Fine-tuning Range of complex actions WIDE-NARROW NARROW-WIDE WIDE-WIDE Forgiving-Fuzzy Long-lead discovery Play/explore multiple possible worlds 3D MODELING MACHINE LEARNING If we add some machine learning to the digital agent’s agency, then we open up the wide-wide tolerance, or forgiving-fuzzy dynamic for 3D modeling. The agent can tolerate less precision on our part because it is learning from other things like usage patterns, material properties, all sorts of things that can expand into feedback loops leaving the human to task of making tweaks as desired based on aesthetics or higher-level requirements.
  66. 66. intense concentration intense coordination conceptual visceral Perceptual information Language NARROW-WIDE Precise-Fuzzy Forgiving-Precise NARROW-NARROW Precise-Precise WIDE-NARROW NARROW-WIDE WIDE-WIDE Forgiving-Fuzzy VISCOSITY TOLERANCE PERSISTENCE PRESENCE Coordinating vs. Adjusting Our next property is persistence: are we relying on the information in the design to help coordinate our behavior, or are we using it to make adjustments or decisions? (This property is based on: Sabrina Golonka (2015) Laws and Conventions in Language-Related Behaviors, Ecological Psychology, 27:3, 236-250, DOI: 10.1080/10407413.2015.1068654 )
  67. 67. COORDINATING BEHAVIOR ADJUSTING HUMAN BEHAVIOR—SYSTEM BEHAVIOR Forgiving-Precise Adjusting while distracted Multi-tasking NARROW-NARROW Precise-Precise Fine-tuning Range of complex actions NARROW-WIDE WIDE-NARROW 3D MODELING Precise-Fuzzy Quick-hit discovery Unknown possibilities GENERATIVE DESIGN WIDE-WIDE Forgiving-Fuzzy Long-lead discovery Play/explore multiple possible worlds Notice that in 3D modeling, the human is using the design tool to coordinate behavior over time as she directly sculpts the geometry. In generative design, the human is simply making adjustments, decisions about which of the possibilities best suit the situation and constraints. This difference between relying on information to coordinate our behavior vs. to make adjustments is our next property of meaning modes. (Notice that when precision is about conceptual information (or the language-dominated region of the phase-space), precision is not about usability of the interface itself; the literal act of engaging, but in the conceptual act of engaging. How conceptually precise must the human be to engage the information and make adjustments (in the case of generative design). In the case of controlling the car dashboard panel, the questions of tolerance (precision) involved how precise the human had to be in physical behavior to engage the information. This is an important distinction.)
  68. 68. COORDINATING BEHAVIOR ADJUSTING Three Dancers, pen and ink by Chris Carter When we think about gestural input, which one is that suited to? Which style of persistence?
  69. 69. ADJUSTING Hololens gesture image credit: Microsoft Gesture as CONCEPT Forgiving-Precise When I make gesture with my arm or hand or eye gaze, I’m not able to do it exactly the same angle and speed, etc., every time; there’s too many degrees of freedom. There has to be some invariant structure the system picks up to get the gesture. That makes them a lot like concepts. Directions; adjustments.
  70. 70. COORDINATING BEHAVIOR Sonar haptics image credit: MIT Media Lab Gesture as SCULPTING Precise-Precise MIT and other groups are starting to use focused sound ways to create a mechanical, haptic effect that we can feel. We could actually touch and feel holograms. This is just proof of concept now, but we could imagine that it could then use our behavior, our gestures, to directly change it. We could actually use gestures to coordinate our behavior along with the haptic feedback we get from the focused sound waves changing as our motions change the hologram. That shifts what we can do with gestures from just adjusting by evoking concepts, to relying on the system to coordinate our behavior over time.
  71. 71. intense concentration intense coordination conceptual visceral Perceptual information Language NARROW-WIDE Precise-Fuzzy Forgiving-Precise NARROW-NARROW Precise-Precise WIDE-NARROW NARROW-WIDE WIDE-WIDE Forgiving-Fuzzy COORDINATING BEHAVIOR ADJUSTING VISCOSITY TOLERANCE PERSISTENCE PRESENCE Sense of Place Persistence is about whether we are relying on the information in the system to help coordinate our behavior over time, or to make adjustments. Our next property is Presence or sense of place.
  72. 72. Creative Commons free usage forest panorama image From the ecological point of view, place is a collection of affordances that support a meaningful activity (NOTE: see Heft, The Participatory Character of Landscape https://www.researchgate.net/publication/253950787_The_participatory_character_of_landscape ). This forest can be many different places, depending on the activity. If I’m looking to set up camp, place becomes a particular layout of the forest that offers a clearing, a reasonably unbumpy ground, and perhaps a break in the trees to watch the stars.
  73. 73. Creative Commons free usage forest panorama image When I’m ready to make a camp fire, place becomes an extent of the forest floor within a walkable radius from my camp. The affordances of this place become objects that appear twig-ish and stick-ish, with surfaces that appear dry and burnable. A forest contains nested places made of physical information, depending on my activity. We also have places made of digital information. For more, see Information Architecture for the Web and Beyond, Morville, Rosenfeld, Arango, and Pervasive Information Architecture, Resmini and Rosati, and Understanding Context, Andrew Hinton.
  74. 74. Precise-Fuzzy NARROW-WIDE ADJUSTING VISCOSITY TOLERANCE PERSISTENCE One place made of digital information is Amazon. The Amazon website has a viscosity that is a combination of concepts in terms of the categories of products and their facets, and the perceptual cues of the products themselves displayed as images. If you are browsing boardgames or books, you may lean more on the labels, titles, and recommendation text for the flow of meaning; if you are looking for a new shirt, you may engage more heavily on the perceptual information about the surfaces, edges, textures of the shirts portrayed in the images. Amazon has a precise-fuzzy human- system dynamic in that the human must select categories or enter well-formed search terms, but the system is fuzzy in offering “customers also bought” and customer reviews and rec’s for other things, and search term structuring with each additional character typed. The human relys on the information in Amazon to adjust, and make decisions about things to purchase.
  75. 75. MARKETPLACE If we zoom out, this set of activities we engage in Amazon gives it a sense of place. We recognize Amazon as a marketplace.
  76. 76. DIGITALPHYSICAL MARKETPLACE Emergence Image credit: RD Riccoboni We know there is some emergent nature to engaging a digital place like Amazon in whatever our present physical surroundings may be, and the designers of Amazon have to worry about its meaning shape shifting across contexts of devices, and languages, and accessibility modes, but the physicality of the surrounding environment, the objects and layout of the surrounding physical environment, do not participate directly in the meaning of engaging amazon as marketplace. As far as meaningful engagement with the activities in the marketplace that is amazon, our physical surroundings may as well be a painting.
  77. 77. DIGITALPHYSICAL MARKETPLACE Except for the beige boxes that show up on doorsteps.
  78. 78. DIGITALPHYSICAL There is a designed connection between the physical box that contains the thing you bought, and the digital object that is order tracking of the thing you bought in the marketplace. Tracking your box location is part of the meaningful activities in Amazon as Marketplace, and participates in the place that is amazon.
  79. 79. DIGITALPHYSICAL Correspondence --David Benyon, Blended Spaces PRESENCE These designed connections among physical and digital objects and relationships are what David Benyon, working on Blended Spaces in HCI, calls correspondences. This extends the presence of Amazon marketplace to our physical world. Although the meaningful activities we engage in this marketplace are mostly digital, this connection extends the meaningful activities, our sense of presence, our sense of place of Amazon, into the physical world. These designed connections among physical and digital objects and relationships are what David Benyon calls correspondences. See his research including, http://iwc.oxfordjournals.org/content/24/4/219.short (Also, keep an eye on the work of Andrea Resmini, bringing these concepts to his Pervasive Information Architecture work.)
  80. 80. DIGITALPHYSICAL Room image credit: Autodesk Revit rendering; VR game credit: Sony Project Morpheus VR What about virtual reality?
  81. 81. DIGITALPHYSICAL VR Not only does virtual reality deliberately not offer correspondences to our physical surroundings, but deliberately tries to obliterate our ability to shift our attention from the digital place to our physical surroundings.
  82. 82. DIGITALPHYSICAL Correspondence Wall alert VR But, we like to move around. And because we can’t perceive our physical surroundings, we might run into walls or furniture. So, we have to let a little physicality seep into our “virtual” reality. These “wall alerts” take the form of beeping sounds or maybe a visual cue that appears like an aurora in this other place that we are inhabiting. This wall alert is a designed correspondence too, but rather than participating directly in the flow of meaning in the digital place, it disrupts that presence and for a moment asks our presence to co-exist in two parallel places. It’s a very unnatural kind of presence.
  83. 83. DIGITALPHYSICAL BALANCE LOCOMOTION Missing Correspondences VR
  84. 84. Image credit: Microsoft, Hololens What about holograms?
  85. 85. DIGITALPHYSICAL Holograms BALANCE LOCOMOTION OBJECTS No Correspondences LAYOUT Now we can see through to the physical surroundings. Balance and locomotion, and visual information about the layout of physical surroundings is available, but the layout and objects in the physical surroundings don’t participate in the meaning of engaging with the hologram. There’s no correspondence there. Although we can shift attention between the digital place that is the hologram and its meaningful activities, and my physical surroundings, they are not both participating in the same sense of presence.
  86. 86. Image credit: Microsoft, Hololens game Fragments In the game Fragments for Hololens, the system first 3D scans your entire room. The objects in the room actually participate in the game. A detective game where you find clues. So a hologram character can sit on your actual chair, because the chair is recognized, and has affordances for the holograms. Theoretically, the system could place digital things inside physical drawers and cabinets because they could be recognized in terms of the dispositions of the kinds of activities they support.
  87. 87. DIGITALPHYSICAL 3D Room Scanning Mixed Reality Correspondence OBJECTS LAYOUT PRESENCE spans both In the case of 3D room scanning mixed reality (or whatever fancy name gets assigned to that), the physical objects and layout of the room participate directly in the meaning of the place, and there is a designed correspondence between the physical place and digital place, and our sense of presence spans both.
  88. 88. DIGITALPHYSICAL Degree of Physical-Digital Correspondences PRESENCE Nouns & Verbs Affirmations extending Ecosystem -Alaine Mackenzie -Dan Brown OBJECTS + INTENT So, one important consideration for presence is recognizing the degree to which our physical surroundings participate in the meaningful activities of the designed place. Are our physical surroundings (objects and layout) participating in the place we’re engaging, in the meaningful activities we’re engaging, or are they just our surroundings? Two of the speakers yesterday, Alaine Mackenzie’s “nouns and verbs” and Dan Brown’s affirmations extending the ecosystem of the domain, are great ways to help us define what are our meaningful correspondences among physical objects/layout and digital objects/layout? Or, if they need not be present. Defining the ontology of our design domain is a critical step in understanding the scope of place (and therefore of presence when our humans engage them). (For more on ontology, Dan Klyn on ontology-taxonomy-choreography, and Abby Covert’s book, How to Make Sense of Any Mess.)
  89. 89. “One sees the environment not just with the eyes but with the eyes in the head on the shoulders of a body that gets about” Image credit: Designboom (Alison Brooks Architects ‘The Smile’) J.J. Gibson We humans want to explore! We don’t just look at our surroundings, we explore them to meaningfully engage them. From an ecological point of view, we see the environment by moving through it. J.J. Gibson quote from, An Ecological Approach to Visual Perception, p.222 1979. According to JJ Gibson, our legs (or whatever we use to physically move around) are part of our visual systems. It’s part of how we see. See also Heft (2010) “Affordances and the perception of landscape: an inquiry into environmental perception and aesthetics,” in Innovative Approaches to Researching Landscape and Health. An ecological point of view makes a strong distinction between looking at an image and engaging the environment.
  90. 90. Are we just looking at this playground, or can we explore it?
  91. 91. DIGITALPHYSICAL Degree of Physical-Digital Correspondences PRESENCE Explorability of Objects & Layout ACTING WITH LOOKING AT We can ask in our designs, to what degree are there correspondences between the objects and events in our digital place and physical surroundings? How explorable are the objects in our designs? If your augmented reality design is just an informational overlay on an object in a specific geographic location, that’s just a map; a distributed map, but still just a map; it doesn’t have materiality because it isn’t explorable.
  92. 92. WRITING GROUP MARKETPLACE Explorability is not just for physical, “virtual” places. Around 2006, new kinds of digital places popped up, nested within the digital place that is Amazon the marketplace. Some shoppers found humor in some of the products Amazon offers, like Tuscan whole milk and microwave for one. These shoppers re-purposed the product reviews to write humorous literary works, short stories, poems, about the products. Others would riff on these works and add their own creative writing. Shoppers became writers. The product reviews of these products became writing groups, and nested places with distinct meaningful activities very different from Amazon as marketplace, yet within.
  93. 93. *'t?'t?'t?'t? Paradise Redredged By Dr. D. v. Simmental on August 11, 2006 Timeless works often suffer at the hands of translators. One thinks of the numerous and continuing attempts to render Dante's "Divina Commedia" (another early vernacular Italian masterpiece and contemporary to the justifiably obscure Tuscana Latte series) and the struggle with both terza rima and meaning. No so for Tuscan Whole Milk ("La tutta latte"). Few works are better left translation of the redredged thir librettist for two of the earliest p Tuscan series and "Mechanicae *****Milk for a Better Tomorrow By C.A. Little on August 29, 2006 opening nights sometime in the I feel compelled to relay my story with Tuscan Whole Milk. My husband is a middle manager store. In an attempt to move him up the ranks of paper pushers (pardon the pun), we decide That Stilton translated the Tus and invite some of his bosses. Not being much of a homemaker, I was frantic to come upwi1 Emmenthaler, present at the Gr I discovered the response people seem to have to Tuscan Whole Milk. Reisen", returning to his rooms I the Arno, but had been clogged. I ordered a gallon - 128 fluid ounces of pure dairy goodness. I was quite concerned about he massive s complete work.He and he fle � One Friday, Without the Milk, October 30, 2006 By Amazon Customer This review Is from: Tuscan Dairy Whole Vitamin D Milk, Gallon, 128 oz (Grocery) He always brought home milk on Friday. After a long hard week full of days he would burst through the door, his fatigue hidden behind a smile. There was an icy jug of Tuscan Whole Milk, 1 Gallon, 128 fl oz in his right hand. With his left hand he would grip my waist - I was always cooking dinner - and press the cold frostiness of the jug against my arm as he kissed my cheek. I would jump, mostly to gratify him after a time, and smile lovingly at him. He was a good man, a wonderful husband who always brought the milk on Friday, Tuscan Whole Milk, 1 Gallon, 128 fl oz. ed to se, using Tu greet o the kitct edel Crys Iwrought Milk. lwa, ilk! I COUit sticated an, *'Clf:rtrC:i With apologies to Emily By S. Mayo on February 1, 2011 Because I could not stop for milk, It kindly came to me. I ordered it through Amazon, Whole milk; sweet Tuscany. The driver, though, he knew no haste, No Expediter, he! The wait: Four days, and weekend, too. It seemed eternity. My milk, it passed through NYC Where traffic snarled in rings, Slow past the fields of grazing cows Who nodded knowingly. Its expiration date passed, too; And though the truck was chill, Vile germs digested lactose sweet And had their gruesome fill. Rarely mi read (if ne diatribes infamous Grasso") by some s Dickinson' Proust's " Neal; "El Peck and Then there was that Friday, the terrible Friday that would ruin every Friday for the rest of my life. The It paused before my house at last, door opened, but there was no bouyant greeting - no cold jug against the back of my arm. There was can Whole Milk. 1 had to place my This Stilto the flavor spoke littl is not cert One start • 3comm no TuscanWhole Milk in his right hand, nor his left. There came no kiss. I watched as he sat down in (By USPS Ground) a kitchen chair to remove his shoes. He wore no fatigue, but also no smile. I didn't spe'!fo��e!!:!:�=:::!::s :::o:::f:::th=e=fi=m=e=d =a =i �s =a =v =a =n =ts =a =t =T =u =s =ca =n=::::!:=======�===================i back to the beans I had been stirring. I stirred until most of their little shrivelled skins surface of the cloudy water. Something was wrong, but it was vague wrongness that no ar hard thought could give shape to. • Over dinner that night I casually inserted,"What happened to the milk?" "Oh,"he smiled sheepishly, glancing aside,"! guess I forgot today." That was when I knew. He was tired of this life with me, tired of bringing home the Tus r Milk, 1 Gallon, 128 fl oz. He was probably shoveling funds into a secret bank account, I I 228 votes Question: Answer: If I spill it, can I cry? If you do it is best to cry either next to it or below it. Crying over it is useless. By Marcus M. on February 25, 2013 � See more answers (7) apartments in town, casting furtive glances at cashiers and secretaries and waitresses. l,'h,'-��---=--=��=�-----��------�--------------------' knew it was over. Some time later he moved in with a cashier from the Food Mart down the street. Tuscan Whole Milk, tomorrow a And me? Well, I've gone soy. Help other customers find the most helpful reviews Was this review helpful to you? GisJ� Yes � Report abuse -S&J • 7 comments 602 people found this helpful. J The Tuscan whole milk writing group has its own activities embedded within the product reviews. The act of writing a review, the act of selecting stars, rating a review as ‘helpful’ are all activities with very particular meaning in this nested place. It has its own meaning mode that is very distinct from the meaning mode of Amazon as marketplace.
  94. 94. MARKETPLACE TUSCAN MILK WRITING GROUP ADJUSTING WIDE-WIDE Forgiving-Fuzzy ACTING WITHDIGITALPHYSICAL No Correspondences Explorable The Tuscan milk writing group has its own meaning mode. It has a high viscosity: highly conceptual, made of language, creative writing, playing with literary genres and tropes, and riffing off of the works of others with the dual purpose of literary form and humor. We’d call this a forgiving-fuzzy human-system dynamic because a review writer participating in this writing group can introduce a new trope, a new way of riffing on others that other writers can then riff off of in turn. The forgiving-fuzzy dynamic has vast generative power to associate on the work of others and weave in new literary styles. Ten years later, people are still writing Tuscan milk reviews. Engaging the information in this place involves writers relying on the information to adjust their behavior, responding creatively to what is already there, what just appeared, how other writers comment on their work. There are no designed correspondences between the digital place and physical surroundings: the meaning, the set of activities that is this place all occur in the digital layer. This place is highly explorable: writers in this place are acting with the information they engage.
  95. 95. SELECT STARS COMMENT ON OTHERS MARK REVIEWS ‘HELPFUL’ ANSWER QUESTIONS WRITE REVIEWS AS SHORT STORIES, PARABLES, POEMS Writers engage in a variety of meaningful activities that make this an explorable place.
  96. 96. MARKETPLACE TUSCAN MILK WRITING GROUP ADJUSTING WIDE-WIDE Forgiving-Fuzzy LOOKING ATDIGITALPHYSICAL No Correspondences Photostatic NOT SIGNED IN Notice that the explorability changes if the writer is not signed in. Only signed in writers can comment on the writings of others, mark reviews ’helpful,’ and write their own reviews. When a writer visits this place but is not signed in, the place loses its materiality; the writer can only look at this place, she cannot engage the meaningful activities. It’s photostatic, like a painting. This changes the nature of how a writer finds presence in this place.
  97. 97. intense concentration intense coordination conceptual visceral Perceptual information Language NARROW-WIDE Precise-Fuzzy Forgiving-Precise NARROW-NARROW Precise-Precise WIDE-NARROW NARROW-WIDE WIDE-WIDE Forgiving-Fuzzy COORDINATING BEHAVIOR ADJUSTING Physical -Digital Correspondences ACTING WITH LOOKING AT Explorability of Objects & Layout VISCOSITY TOLERANCE PERSISTENCE PRESENCE DIGITALPHYSICAL These are four properties of the invisible things about meaning we can recognize in our designs. We can use viscosity, or ease of the flow of meaning, like a design dial to speed up an interaction or deliberately slow it down and draw awareness and attention when something important needs it, or something engaging that we don’t want to miss. Tolerance is about considering the precision of the human-system behavior dynamic, and which situations those different dynamics are good for. Persistence is about whether we are relying on the information to help coordinate our behavior as we act over time, or if we are using it to make adjustments or decisions. Presence is about our sense of place, and recognizing when we need designed correspondences among our physical surroundings and our digital places and what objects and layouts are participating in meaningful ways in our place. Presence is also about how explorable the objects and layout of digital and physical surroundings are in allowing us to act with them, or just look at them (photostatic). All of these together facilitate very different meaning modes for the humans that engage them.
  98. 98. Meaning Modes Rocket contrail by Seay Photography via Creative Commons These properties of meaning are really abstract things. Why should designers add this to all the other things we already have to worry about in our designs?
  99. 99. J.J. Gibson’s Observer is not in 1979 any more J.J. Gibson 1979 JJ Gibson’s observer is not in 1979 any more.
  100. 100. J.J. Gibson’s Observer is not in 1979 any more
  101. 101. J.J. Gibson’s Observer is not in 1979 any more
  102. 102. J.J. Gibson’s Observer is not in 1979 any more We’ve opened many more wormholes to information with our designs. [Watch icon via bluetip design; VR headset via Michael Beno; iphone icon via Edward Boatman; shoe icon via romzicon; egg carton via Blaise Sewell; toaster icon via Creative Stall; band icon via Nate Eul]
  103. 103. J.J. Gibson’s Observer is not in 1979 any more
  104. 104. Information wormholes are getting all over us And they’re getting all over us; in our clothes, on our kids, and all over the objects around us.
  105. 105. Robots minding all the wormholes Same degree of attention In the same way We don’t want to have to act like robots minding all the information wormholes with the same degree of attention, in the same way.
  106. 106. We are tribal hunter-gatherer poets And we want to act like it
  107. 107. MARSHA HAVERTY @mjane_h Thank You

×