Sensemaking position paper for chi 2005 workshop


Published on

Position paper for a CHI workshop on sensemaking

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Sensemaking position paper for chi 2005 workshop

  1. 1. Sensemaking & Complexity: Position Paper for CHI 2005 Workshop John C. Thomas T. J. Watson Research CenterBackground: The topic of this workshop provides a fascinating and insightful way ofviewing many of the research topics that I have been involved in over time. Although theterm “sensemaking” has not always been applied to my previous relevant work, I’vebeen interested in the substance of this topic for several decades. My dissertation, “Ananalysis of behavior in the Hobbits-Orcs problem” (Thomas, 1974) offered acomparison between actual behavior and the then popular notion that problem solvingproceeded as a step-by-step process of minimizing the difference between the currentstate and the desired state. I was suspicious of this model, partly because of theobservations of DeGroot (1965) who studied grandmaster chess players and found thatthey examined one branch of play, examined another branch of play, and then re-examined the first branch of play. This behavior cannot be a question of “forgetting”;rather, it seemed obvious to me, that they had discovered something general about thesituation that they were in during their examination of the second branch of play thatcaused them to re-evaluate their thinking with respect to the first branch of play.Similarly, in my dissertation, I found that people were operating at two levelssimultaneously. Yes, they were attempting to solve the problem at hand. In addition,they were attempting to “make sense of” and “understand” the nature of the problemat hand. Converging evidence from several sources including latencies to make a move,error probabilities, verbal protocols, and transfer effects all indicated that within theoverall problem of moving from the current state to the goal state, people had to learn ordiscover three basic properties of the situation that they found themselves in. Thesediscoveries about the nature of the problem space took long and highly variable times.Deciding on what “move to make” was done relatively quickly and with a much smallervariance.In my work managing a project on the “Psychology of Aging”, (Fozard, Thomas &Waugh, 1976; Thomas, Fozard, & Waugh, 1977) there were two findings of particularinterest to this workshop. First, healthy older male veterans, across a wide variety ofcognitive tasks, showed not only increased times to complete tasks, but also, incomparison with younger healthy males, increased variance both within and betweensubjects. Second, with many other variables partialed out, there was a significantnegative correlation between years of formal education and intra-subject variance. Forthese subjects, veterans who were retired or came from a variety of white and blue collarprofessions, the battery of experiments to which they were exposed (e.g., choicereaction time, memory scanning, paired associate learning) represented novel tasks.The subjects were “instructed” how to do the tasks and given a few practice trials.However, it is highly unlikely, a priori, that everyone was immediately able to translatethe declarative knowledge of instruction into an optimally constructed procedure foraccomplishing the various tasks. How much of the observed age effect in variance wasdue to biological factors, as opposed to generational factors (younger subjects, e.g.,probably having more practice with more arbitrary tasks such as school requires andhaving carried these out much more recently) is impossible to tell from those data.Similarly, there are alternative explanations about the possible reasons for a correlationbetween having more formal education and having a smaller variance in performance.
  2. 2. What both of these findings do question, however, is the traditional psychologicalassumption that human performance in experiments is measuring somethingfundamental about the “hardware” of the human system. In reality, subjects inexperiments are active participants who quickly attempt to make sense of the situationand often must construct strategies, attempt to gain feedback about the effectiveness ofthose strategies and then modify their strategies accordingly.Perhaps three personal anecdotes can be useful in illustrating the point that subjects areactively attempting to make sense of the experimental situation and thinking about howto optimize their performance in that situation. The first two concern the author as anexperimental subject and the third as an experimenter. In our undergraduateintroductory psychology class, the instructor illustrated paired associate learning bygiving multiple learning trials of a series of ten paired associate CVC’s. After learningall the associates on the first trial, I was amazed how many trials other students took. Inasking some of them about this later, it became clear that what seemed obvious to me;viz., to make a story out of each paired associate, was not at all an obvious strategy toothers. So, e.g., the first pair was MOF-DAQ and I imagined being offered a strawberrydaiquiri by my friend Bob Hoerner. In graduate school, I was a subject in atachistoscopic experiment in which letters or words were presented followed by a“mask” (a random pattern of dots) which is supposed to “erase” the retinal image.My results were atypical because when the letter was presented, I immediately movedmy eyes away thus leaving the retinal image of the target and the mask in two differentplaces on the retina. As an undergraduate, one of my part time jobs was to teach spacescience to sixth graders and another job involved being a research assistant to abehavioral psychologist. In that latter context, I was running subjects in a large“Skinner Box” in an experiment to determine whether or not applying a verbal label toa discriminative stimulus (in this case, a large red circle) made a difference ingeneralization gradients along various dimensions. Before one particular experimentaltrial, one of the kids was simply waiting their turn in the anteroom and so I decidedinstead of just having him sit there with folded hands, I would teach him about theplanets. After the experiment, when I debriefed this subject, to my complete amazementand shock, he interpreted the experiment to be a test of how well he had learned thematerial I presented on the solar system!! From my perspective, the little mini-lecture onthe solar system was simply a way to pass the time and impart some knowledge andhad nothing whatever to do with the operant conditioning experiment. From theperspective of the subject, however, who came to the “University” to a “laboratory”and then proceeded to see a lot of diagrams with circles and names on a chalkboardfollowed by watching a series of red circles and pulling a lever for nickels followed bybeing presented with other colors and sizes of circles and ellipses, pulling a lever andnot getting nickels, this was one single experience which he attempted to make somecoherent sense out of. He constructed a narrative, if you will, that included all the data.When I joined IBM Research in 1973, my first set of studies involved evaluating an ideafor a new query language called, “Query By Example” in which users wrote queriesdirectly into a visual representation of a relational data base. For a variety of reasons, itturned out that this language was remarkably easy to learn and use, in comparison withother available alternatives, in the sense that experimental subjects did well attranslating English questions into Query By Example. There were some troublingexceptions, however, mainly having to do with the directness of that translation process,on a word by word basis. For example, one column in the sample data base waslabeled “Year of Hire.” If the following English question were presented to users:
  3. 3. “Print a list of all the employees hired after 1970,” the results were quite good.However, if the English question were put in this form, “Print a list of all the employeeswith less than three years experience,” the results were much less encouraging.Similarly, given an English question such as, “Print a list of all the items that are bothlarge and red,” the queries tended to be quite accurate. However, when presented with“Print a list of all the large items and all the red items,” the results were much lessencouraging. (In the latter case, subjects tended to use the AND operator in the querylanguage when an OR operation was called for). In a follow-up experiment, subjects(college students) were not given English queries to “translate” into Query ByExample, but instead, were given a fairly complex data base reflecting various relationsin a typical college. Then, they were given a series of “issues” and asked to write theirown queries whose answers might shed some light on those issues. Then, they were totranslate their own English queries into Query By Example. By and large, students atthat time (@ 1975) were fairly clueless about the types of questions that could and couldnot be reasonably answered by a computer system. For instance, in response to theissue, “Many of the younger faculty feel that they are not paid enough relative to theolder faculty,” many students wrote the English question, “Are the younger facultybeing paid enough?” and then attempted to translate that into Query By Example.Given the much more widespread use computers by students today and of googol inparticular, a similar study might yield quite different results now. However, the generalresult remains. Successful use of a system requires more than simply understandingthe syntax of the system; it requires making sense of the situation and how to respond tothat situation (Thomas & Gould, 1975; Thomas, 1983).During the 1970’s, we conducted a series of experiments on “the psychology ofdesign” (Thomas & Carroll, 1978; Carroll, Thomas & Malhotra,1979; Carroll, Thomas &Malhotra,1980; Malhotra,Thomas, Carroll & Miller, 1980). Here too, it became clearthat “solving” a design problem, while difficult, was often not nearly so crucial asfinding and formulating problems. The best designers were able to formulate a problem,attempt to solve it, and then, in the course of solving it, completely redefine andreformulate the problem itself. In most of our formal education, however, this kind ofbehavior is not only unnecessary; it is actively discouraged or penalized. To addressthis issue in my own teaching during undergraduate statistics classes, I sometimesposed the following quiz question --- which many of the students found inordinatelydifficult --- “If I do three t-tests on Monday and four t-tests on Tuesday, how many t-tests have I done in total?” The overall interpretation of the context of statistics formany students is that it is a difficult, complex, and novel situation. Giving a problem inthat context that is easy, simple, and relies only on already acquired skills requires achange of perspective and set. Reflection on this question lead to a more generaldiscussion on the importance of understanding the “deep” (and not just “superficial”)nature of a situation before applying a particular statistical test.More recent and current work relevant to sense-making .Stories. From approximately 1999-2001, I managed a research project on the businessuses of stories and story-telling. Stories are a quintessential way for people to makesense of complex situations. Stories can prove useful in cultural change, personalchange, sales, knowledge creation and sharing (Thomas, 1999). They can be highlymemorable and motivating. On the downside, once a person accepts as “true” aparticular narrative viewpoint of a complex situation, it can be difficult to persuade themto consider alternative ways to make sense of a complex situation.
  4. 4. Pattern Languages. The term “Pattern Language” was first introduced by Alexander(Alexander, et. als, 1977) in the field of architecture. Since, Pattern Languages havebeen applied to such diverse fields as object-oriented programming, management, andhuman-computer interaction. Patterns are named recurring abstract solutions torecurring problems. A Pattern Language is a lattice of inter-related patterns that attemptto provide coverage for the set of recurring problems in a given field. A person familiarwith the Pattern Language in a given field can use them, not only as a guide to solvingspecific types of problems, but also as a sensitization device for finding and formulatingproblems. In this sense, Pattern Languages provide one conceptual tool for sense-making for use by individuals or communities. For the past several years, we have beeninvolved in working collaboratively to develop a socio-technical pattern language alongwith tools to help construct, organize, find, and use patterns.e-learning. In 2002-2003, I was responsible for the user experience for a “DynamicAssembly of Learning Objects” project (Farrell, Thomas, Rubin, Gordin, Katriel,O’Donnell, Fuller, 2004; Farrell, Thomas, Dooley, Rubin, Levy, O’Donnell, Fuller,2003; Farrell, Dooley, Thomas, Rubin & Levy, 2003). Learning Objects are much likelearning modules and also contain metadata that may specify topic, level of difficulty,prerequisites, intended audience, rhetorical purpose, author, length, reading level and soon. Our system enables users to build a kind of personalized mini-course relevant totheir specific learning goals, background and time constraints. Initial interviews indicateda strong need for this kind of personalized but semi-automatically generated course. Inour case, we use metadata added by Subject Matter Experts in conjunction with apedagogically motivated ontology to help select and organize the material. A series offield studies and an experimental study indicated that the system helped considerably insense-making (Thomas & Farrell, 2004).Business Consulting. Currently, I am working with IBM business consultants to buildtools to help them with what they do which is essentially to help their clients withcollective sense-making. The fastest growing business segment in IBM is services anda large part of that is business consulting services. In many ways, the very existence ofbusiness consultants gives support to the importance of sensemaking as well as itsapparent difficulty. After all, should it not be the case that the executives who arerunning a company should know more about it than anyone else? If it were notcommonly done, we might think it very strange that highly competitive and highly paidexecutives would pay outside consultants to help them “make sense” of their owncompany and how it fits into a larger ecological scheme.Business consultants can prove valuable precisely because they are able to seepatterns and use perspectives that are different from the ones that their clients havegrown accustomed to.Making Sense of SensemakingSolving problems, in particular, well-defined problems, often requires a logical, step-by-step approach. Much of our educational process values, trains, and rewards such aprocess. In real life, such approaches can prove useful in solving engineering problems,writing computer programs, or implementing accounting practices. Such approaches,however, are almost completely useless, in my opinion, for discovering engineeringproblems to be solved, understanding what computer programs are worth writing, or for
  5. 5. deciding whether an accounting practice is ethical. Problem finding and problemformulation are much more akin to what are generally considered perceptual rather thanconceptual processes. Doing a good job in problem finding and formulation requirestaking multiple perspectives, being able to distinguish figure from ground, being able tosee patterns, and being able to relate the present situation to relevant past experiences.I believe that technological aids have primarily, but not exclusively, focused on aidingpeople in solving well-defined problems. However, I believe that technological aids canbe designed to help people with sensemaking. The key approaches here are to allowmultiple and flexible representations of situations, to bring to bear multiple perspectives,and to remind people of potentially relevant experiences.References:Alexander, C. A., Ishikawa, S., Silverstein, M., Jacobson, M. Fiksdahl-King, I., andAngel, S. A Pattern Language. New York: Oxford Press, 1977.Carroll, J., Thomas, J.C. and Malhotra, A. (1980). Presentation and representation indesign problem solving. British Journal of Psychology/,71 (1), pp. 143-155.Carroll, J., Thomas, J.C. and Malhotra, A. (1979). A clinical-experimental analysis ofdesign problem solving. Design Studies, 1 (2), pp. 84-92.DeGroot, A. D. (1965). Thought and choice in chess. The Hague: Mouton.Farrell, R., Thomas, J. Rubin, B., Gordin, D., Katriel, A., O’Donnell, R., Fuller, E., andRolando, S. Personalized just-in-time dynamic assembly of learning objects. E-learning2004. November, 2004.Farrell, R., Thomas, J., Dooley, S., Rubin, W., Levy, S., O’Donnell, R., Fuller, E.Learner-driven assembly of Web-based courseware. E-learn 2003 , Phoenix, Arizona,Nov. 7-11, 2003.Farrell, R., Dooley, S., Thomas, J., Rubin, B. And Levy, S. Implementing and extendingLearning Object Metadata for learning-based assembly of computer-based training.Learning Technology Newsletter, Vol.5, 1, January, 2003, 14-16.Fozard, J. L., Thomas, J. C., and Waugh, N. C. (1976). Effects of age and frequency ofstimulus repetitions on two-choice reaction time. Journal of Gerontology, 31, (5), pp.556-563.Malhotra, A., Thomas, J.C. Carroll, J. M., and Miller, L. A., (1980). Cognitive processesin design. International Journal of Man-Machine Studies, 12, pp. 119-140.Thomas, J.C. (1974). An analysis of behavior in the hobbits-orcs problem. CognitivePsychology 6 , pp. 257-269. Thomas, J.C. & Gould, J.D., (1975), A psychological studyof Query By Example, Proceedings of AFIPS, 1974 National Computer Conference,Arlington, VA: AFIPS Press, 44, 439-445.Thomas, J. C., Fozard, J. L. and Waugh, N. C. (1977). Age-related differences in naminglatency. American Journal of Psychology, 90(30), pp. 499-509.
  6. 6. Thomas, J.C. (1978). A design-interpretation analysis of natural English. InternationalJournal of Man-Machine Studies, 10, pp. 651-668.Thomas, J.C. and Carroll, J. (1978). The psychological study of design. Design Studies,1 (1), pp. 5-11.Thomas, J.C. (1983). Psychological issues in the design of data-base query languages.In M. Sime and M. Fitter (Eds.), Designing for human-computer communication..London: Academic Press.Thomas, J. C. (1999) Narrative technology and the new millennium. KnowledgeManagement Journal, 2(9), 14-17.Thomas, J. & Farrell, R. (2004). An experimental investigation of the effectiveness ofindividualized web-based learning based on the dynamic assembly of learning objects.IBM Research Report, 2004.