The document describes a study that compared the effectiveness of two approaches to online learning - a custom course assembly approach that dynamically assembled learning objects, and a query engine approach. Subjects using the custom course assembly produced more extensive designs for a given task and performed better based on expert evaluations, even after accounting for differences in experience. The custom course approach may have led to better performance by encouraging focus, providing more relevant content, and creating a more organized learning experience.

1. Human-Computer Interaction Research at IBM T. J. Watson Research Center John C. Thomas Lehman College October 20, 2004

2. Outline Whirlwind Tour of the Past Whirlwind Tour of the Present In-depth Look at One Project Challenge: A Science of HCI Potential Approaches: Pattern Languages in HCI Fifteen Properties of Great Design Practical Hacks

3. Whirlwind Tour of the Past Query By Example (1973-1975) Pencil and Paper studies Deeper Dive on Quantifiers Natural Language Processing (1975-1976) “ Wizard of Oz” studies Psychology of Design (1976-1979) Spectrum of Techniques Corporate Headquarters (1980-1982) Speech Synthesis (1983-1985) Motivation (1986) NYNEX AI Lab (1986-1998) Expert Systems, Machine Vision, Speech Recognition, Robotics, HCI, Intelligent Tutoring Systems IBM Research – Stories and Storytelling (1999-2001) Dynamic Assembly of Learning Objects (2001-2003) Tools for Consultants (2004--)

4. Whirlwind Tour of the Present Multi-media Everywhere Display Video Parsing Scene Parsing User Interface Technology Speech Recognition Handwriting Recognition Natural Language Research Human Computer Interaction Social Computing New Learning & Teaching Systems Better Tools for SW development, consultants

5. Learning Objects: Vision and Challenges Learning Materials Modularized with added Metadata “ Best” Materials become available to all Re-purposing and re-use (e.g., Tiger Woods video and physics professor) ----------------------------------------------- Copyright Issues How does metadata get added ? Selection, ordering, coherence issues

6. Context Dynamic Assembly allows on-demand material based on background and needs. System allows a person to use a Query Engine and then select from a list of relevance-ordered results. System uses metadata associated with each selected result to construct a pedagogically coherent course from learning objects. Here, domain is WebSphere and Web Services Materials come from IBM Redbooks. Hypothesis: People using Dynamic Assembly will perform significantly better in an on-demand situation than those using the Query Engine alone. Experiment designed to inform this question and also to provide on-going feedback to the design team.

7. Custom Course Delivery

8. Custom Course Delivery

9. Custom Course Delivery

10. Custom Course Delivery

11. Iterative Development Began with a series of PowerPoint “simulations” After several iterations, programming began on working system Pilot subjects with “real system” Followed by extensive IGS Field Trial Users had very positive reaction to the system based on questionnaire results; HOWEVER, it seemed that most were either: trying it because they were told to, kicking the tires, or seeing what it “would be like.” Only a few (and there were some) seemed to be engaged in serious learning.

12. Experimental Design Rationale Trade-offs between “real people” doing “real work” in their “real setting” at their convenience and control Used IBM volunteers willing to spend two hours from their own office connected via phone, shadowing, and/or sametime Gave scenario to motivate learning and performance Used very difficult task to avoid ceiling effect Used multiple measures of background, process, and product of learning Group I used Dynamic Assembly; Group II only used (the same) Query Engine

13. Procedure Volunteers recruited via e-mail Volunteers asked for two hour time slot; given instructions to print out ahead of time; most stayed on phone and spoke aloud during all tasks; some preferred to work silently or send same-time messages. Not all subjects kept appointments; not all subjects turned in all materials – but most (>90%) did turn in all materials.

14. Tasks Subjects read overview of experiment and were asked if they had any questions Subjects read a scenario to motivate them to design a solution to a customer’s problem Subjects had 55 minutes to learn Subjects had 35 minutes to complete a high level design Subjects were then quizzed on objective knowledge of WebSphere and Web Services, answered subjective questions about their experience with the software and asked about their relevant background knowledge.

15. Scenario IBM has the chance for a moderately large software deal with Genysis although … is a strong competitor. You’ve been called in at the last minute because the person who was supposed to give a client presentation was in a serious automobile accident. The client exec wanted to reschedule …Genysis … needs to make a decision now. You will meet with the client in about an hour …provide preliminary high level “design”…. If you can pull this off, your boss would be extremely impressed. Basically, their situation is as follows: Genysis … an international provider of herbs, vitamins, and related health care products; they have a working client-server application that allows registered pharmacists to order their products on-line. …software called “Doser” allows the pharmacist to enter a body weight and get a recommended dosage for particular drugs. A series of recent events, …have led them to want to publish “Doser” as a Web Service available to the general public. Moreover…they would like to offer interactions in at least six major languages… as well as English. Luckily, although the current application is written for an American English, the Model View Controller Pattern was used and formats, help files and messages …separate from …control flow. Genysis… considering possible use of another Web Service PolyGlot to do translation….

16. Subjects Subjects were all IBMers Subjects varied considerably both in IT experience generally and knowledge of Web Services and WebSphere from “complete novice” to “levels 3 to 5.” Subjects also varied widely in “the number of languages I’ve used to write programs of >1000 lines” from 0 to 7.

17. Evaluating the Designs Design Behavior difficult to quantify; hence, use multiple measures: Quantitative Measures (often correlated with quality) Total Number of Pages in Design/Presentation Number of Words, Boxes, Arrows (not counting what is copied) Number of features present from an expert’s design Qualitative Measures Three experts rating the designs

18. Total Pages of Output Mean for Custom Course Group = 6.5 Mean for Query Only Group = 1.83 t(24)=2.10, p<.05 Also, a significant correlation between self-reported experience and number of pages; r=.46, t(24)=2.535, p<.01 (one-tailed), p<.02 (two-tailed) When experience is taken into account, the effect of group increases: t(24)=2.66, p<.014

19. Registry includes herb, vitamins, etc. WSDL document Existing Java beans Import JAR file into AAT WAR file Web Sphere build Deploy Run Manage Client Service Registry Sample Design from one of the Query Only subjects

20. Architectural concept Genysis Doser WebService enabling Sample Design from Custom Course Assembly Subject (p.1)

21. Existing Architecture Doser Business Logic Genysis Drug Database (DB2) Doser Controller Doser View Doser GUI Doser Java Client Doser Java Server Sample Design from Custom Course Assembly Subject (p.2)

22. Requirements WebService enabling of existing Business Logic Multilanguage (Mandarin, French, German, Italian, Spanish and English) Sample Design from Custom Course Assembly Subject (p.3)

23. Assumptions Business Logic of existing solutions runs on server New solution runs exactly the same Business Logic Content of the database is language independent (product names and numbers) Sample Design from Custom Course Assembly Subject (p.4)

24. New Architecture Doser Business Logic Genysis Drug Database (DB2) Doser Controller Doser View Doser GUI Doser Java Client Doser Server RPC Servlet (WebService) HTML Controller HTML View NLS Properties Browser WebService Client SOAP/HTTP HTTP HTTP UDDI register Sample Design from Custom Course Assembly Subject (p.5)

25. Custom Course Group had more total design features that matched expert design suggestion Custom Course Group Mean = 10.8 Query Only Group Mean = 8.6 However, previous reported experience dwarfed this effect, r=.463, t (22)=2.35, p<.05 When experience is taken into account, the superiority of the Custom Course Group is significant t(22) = 2.35, p<.03

26. Experience also correlated with overall “quantity” of design Correlation of Experience with words+boxes+arrows r=.409, t(22)=2.1, p<.05 Custom Course group had more in their designs; when experience taken into account, Custom Course outperformed the Query Only Group t(22)=2.59, p<.02 Experience also correlated separately with boxes used (r=.418, p<.05), arrows,r=.346, p<.1) and words (r=.367,p<.08) With experience accounted for, Custom Course Group also had significantly more words t(22)=2.46, p<.03

27. Designs were graded qualitatively by three experts Designs were graded on presentation, level of detail, accuracy, completeness, depth of understanding, and overall Graders tended to agree overall: r (1,2) = .728, r (1,3) = .72, r (2,3)= .734, p<.0001 for all Dynamic Assembly Group produced better designs for all three raters, but not significantly so. Adjusted for experience, more “good designs” in Dynamic Assembly

28. Conclusions It appears that the subjects from the Dynamic Assembly Group produced more in terms of design and, with experience accounted for, qualitatively better designs as well Individual differences, both in terms of background knowledge and design quality were very large

29. Why might the Custom Course Group have done better? The extra work involved in making a course may have encouraged them to stay more focused and less scattered. The additional metadata presented in the query results page may have allowed them to go to material better suited to their background and goals. The expectation of building and using a “course” may have induced a more reflective cognitive set. The learning objectives and organization may have served as advanced organizers thus improving learning. The ordering of learning objects within a course may have allowed a more coherent learning experience.

30. Conclusions Behavior during the experiment was qualitatively and quantitatively different for the two groups: The Query Only group spent a short amount of time on a larger number of Learning Objects, focusing on search and navigation The Custom Course Group spent longer amounts of time on fewer (and somewhat different) Learning Objects focusing on reading and understanding Subjects varied considerably in their queries, Learning Object selections and visits, probably due to differences in their background

31. “Good” HCI Depends on… Users (People differ in training, ability, etc. Tasks (Speed/Accuracy, Complexity, etc. Context (Work, Play, School, War, Retail… Technology (Typing, Speech Recognition, Gesture Recognition, Virtual Reality, Affect Recognition, etc.

32. “Good” HCI Depends on… Users Tasks Context Technology BUT…. All of these are changing rapidly ALL the TIME !!

33. How then do we “Cumulate” Knowledge and Develop a True Science ? Scientific Truths are Supposedly independent of time and place One Approach: Pattern Language Second Approach: Fifteen Properties

34. Potential Forms of Knowledge Known, Predictable, Unchanging, Simple Unknown, Unpredictable, Changing, Complex Algorithms, Formulae, Programs, Machines Patterns Guidelines Heuristics, Principles, Properties Case Studies Stories Ethical values and fluid intelligence

35. How can we help the designer DESIGN?

36. The Importance of the Social Robert Putnam: Making Democracy Work (Italy) Bowling Alone (America) Impacts health of individual more than smoking Impacts on whether we have a sustainable approach to the world’s resources Impact on war and other miseries Corporations now supporting collaboration and communities of practice Socially defined intelligence: Evan’s Thesis on figures analogies

37. E.g. Washing Dishes Hand Washing Duo Rhythm required Side by side “confessional” Conversation OK Team accomplishes the work High shared stimulus context Using Dishwasher Rhythm not required Unitary better Conversation ? Team or One prepares machine to accomplish the work Moderate shared stimulus context

38. Fixing Dinner Traditional cooking Negotiation Required High shared stimulus context (same meal) Synchronous activity Conversation likely Microwave No negotiation required (separate meals) Asynchronous activity Conversation less likely (person who is ready first starts some other activity)

39. Traditional Queue Some shared context; however… Perceived as competition for limited resource (tickets may run out) People in front are costing you time Face to Back of Head orientation Asynchronous movement reinforces individual identity (cf. rowing)

40. Vibrating Pager Queue The obviousness of the competition has been greatly reduced No requirement to “face the same direction” Face to face interaction possible Conversation is much more likely

41. Enhanced Telephone Help Desk Queue Many more people need help solving technical problem than servers available People describe problem ASR used to group similar problems People are bridged onto a conference call Synthesis announces to group their areas of overlapping interest Group may be able to solve the individual problems When available, help first gives generic advice

42. A Pattern Language Christopher Alexander Architectural “Patterns” that capture recurring problems and solutions Organized into a “Pattern Language” – a lattice of inter-related Patterns. Examples: Eccentric Town Center encourages commuter traffic to stop at Town Center European Pub Gradient of Privacy in homes: porch, entry, living room, dinning room, kitchen, bedroom

43. Some Socio-Technical Patterns Community of Communities Reality Check Radical Co-location Small Successes Early Who Speaks for Wolf? Support Conversation at Boundaries Social Proxy Context-setting Entry Answer Garden Registered Anonymity Anonymized Stories for Organizational Learning Mentoring Circle Levels of Authority Rites of Passage

44. Reality Check

45. Who Speaks for Wolf? Visual by www.PDIimages.com

46. Small Successes Early

47. Support Conversation at the Borders

48. Christopher Alexander’s Fifteen Properties from The Nature of Order 1. Levels of scale. 2. Strong centers. 3. Boundaries. 4. Alternating repetition. 5. Positive space. 6. Good shape. 7. Local symmetries. 8. Deep interlock and ambiguity. 9. Contrast. 10. Gradients. 11. Roughness. 12. Echoes. 13. The Void. 14. Simplicity and Inner Calm. 15. Not-separateness.

49. Can these be applied to the design of social systems? * Levels of Scale: Organizations, Divisions, Departments, Projects, Teams, Individual. * Positive Space: Opposite of “not my job”; better to have contention than gaps * The Void: Need empty space and empty time; perhaps even roles of peace * Roughness: Problems arise when designs presume that they have covered every case.

50. For more information: www.research.ibm.com/knowsoc/ www.truthtable.com/websitewelcome_page_index.html http://www.cpsr.org/conferences/diac02 http://www.welie.com/patterns/plml/ http://www.pliant.org/personal/Tom_Erickson/InteractionPatterns.html http:// www.hcipatterns.org / http:// www.cpsr.org /program/sphere/patterns/ http:// www.ibm.com/developerWorks/patterns / http://jerry.cs.uiuc.edu/~plop/plop2003/cfp2003.html http:/www.cs.kent.ac.uk/people/staff/saf/patterns/gallery.html http://www.groupware-patterns.org / http://www.lmu.ac.uk/ies/comp/research/isle/janetfinlay/ www.truthtable.com/patterns.html/