Hci techniques from idea to deployment
Upcoming SlideShare
Loading in...5

Hci techniques from idea to deployment



Describes a project for individualized learning. CHI 2006 case sudy.

Describes a project for individualized learning. CHI 2006 case sudy.



Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment
  • Recombinant - relating to or exhibiting [data/genetic] recombination the formation by the processes of crossing-over and independent assortment of new combinations of [learning objects/genes] in [derived works/progeny] that did not occur in the [original works/parents]
  • Spreadsheet (easy) Created from Table of Contents Type – Module, Resource, Include, or Exclude Intended Use – optional Modules or Resources Rest of the metadata – required Modules Tried Default modularization at the section level, but SMEs wanted to control. SMEs can adjust up (to chapter) or down (to subsection or subsubsection)
  • If the course is large, we would like to group things into chapters. And offer prerequites information by intended use (e.g., here are some scenarios…). We plan to use intended use, difficulty, and importance to reorder the material for different types of learners as defined by the audiences.

Hci techniques from idea to deployment Hci techniques from idea to deployment Presentation Transcript

  • HCI Techniques from Idea to Deployment: A Case Study for a Dynamic Learning Environment John C. Thomas & Robert A. Farrell CHI 2006 Montreal, Canada April 27, 2006 Other Development Team Members included: Sam Dooley, Bill Rubin, Stephen Levy, Amy Katriel, Doug Gordin & Danny Oppenheim IGS Learning partners: Ray O’Donnell, Ernie Fuller & Steve Rolando
  • Outline
    • Background & Context
    • Pre-project Interviews
    • Scenario-based Iterations in PowerPoint
    • Heuristic Evaluations, Corporate Guidelines & Orienting Values
    • Iterations on the Interface
    • Field Trial Results
    • Scenario-Based Experimental Investigation
    • Lessons Learned
    • References
    • Q&A
  • General Problem
    • Needed Knowledge Changing at a Rapid Rate in many fields; e.g., IT, HCI, Biol, etc
    • Moreover, Knowledge also Branching
    • Many Learners under time pressure
    • Companies under monetary pressure; less willing to send employees away for a week course; also hard on families
    • Yet, there is a great duplication of effort in producing materials and many of them are on the web
  • Learning Object Framework “ Build technology to enable recombinant learning experiences via the production, dynamic assembly, and adaptation of modular learning objects by publishers, instructors, and learners.”
  • Learning Objects Vision & Issues
    • Use vast sea of materials in new and exciting ways (physics of baseball, golf)
    • Able to use the “best of the best”
    • Copyright
      • Who gets credit/money for collage ?
      • Do original authors even want their material used in (any possible) new context?
    • Metadata are costly to provide
    • How do we select and organize materials from different sources into something pedagogically coherent?
  • Our Approach to Issues
    • Copyright: Avoid this issue by using only materials where IBM owns copyright (Redbooks and presentations)
    • Metadata: Combination of approaches
      • Some metadata present in original
      • Some derived from computational linguistics
      • Provided tool for SME’s to simplify task (e.g., inheritance as default)
      • Allow users to collaboratively add metadata
    • Selection and Coherence
      • System and User work collaboratively (“System proposes but User disposes.”)
      • Eventually added ontology to “fill in” missing connective material
      • Eventually added conceptual maps for more novice users
  • Metadata Spreadsheet
  • Metadata and Its Usage Use Values Element Product, language, or operating system Mapping to books IEEE LOM standard (5pt scale) Prerequiste, co-requisite Architecture, Code Listing, Definition, Scenario Motivation, Concept, Process, Procedure Introduction, Conclusion Original context of this data (section, chapter, book) Sequence in book order, maintains source coherence Context Sequence topics by prerequisite and within topic by cognitive level Topic Select proper resource type for job/task Resource Type Select environment to match task Environment Prefer < difficulty when introducing material Difficulty prerequisites first sequencing, assess prerequisites first Relationships Select proper level for job/task, logically sequence Cognitive Level Advance organizer, reduce dangling references Rhetorical Role
  • Specific Context
    • Users:
      • IBM IT personnel constantly need to learn about new technology
      • Their time is often very limited
      • Adult, motivated learners
      • Enough previous knowledge to understand technical materials
      • Much learning is done as team but teams are often global
      • Many workers are often working remotely
    • IBM Goals:
      • Re-use educational materials in new ways to cut costs
      • Reduce travel costs; use f2f when necessary
      • Research technology of possible use beyond IBM
  • Team Background
    • Team Leader
      • Extremely sympathetic to iterative user-centered design
      • Extensive background in CAI former John Anderson and Roger Shank student + a decade of additional CAI development
    • User Experience Lead:
      • Began tutoring other students at age 12
      • First educational experiment at age 16
      • 30 years experience in HCI
      • Taught to sixth graders, undergrads, grad students, post-grads, etc.
    • Approach includes iterative design, test, and re-design
    • However, experience allows for better initial design and more effective recovery from problems
  • Initial Interviews
    • Before initial design, I interviewed potential stakeholders
    • Potential users
    • Course developers in IBM
    • Curriculum designers in IBM
    • External course developer
    • Broad survey about “best educational experiences”
  • On-demand Custom Courseware Original Sources Learning Objects Customize Capture Assemble Share Describe Cleave Custom Course
  • Design “Values”
    • Heard of as technique for making a story coherent in Story by Robert McKee
    • Also heard as technique used in European car company for design integrity
    • Design Values prominently posted above my ThinkPad and referenced during design decisions, interviews, and looking at test results
  • Design “Values”
    • E-Learning
    • Personalized
    • Modern
    • Useful
    • Fun
    • Flexible
    • Targeted
    • Efficient
    • Correct
    • Authoritative
    • Under my (i.e., the user’s) control
    • Easy to Use
    • Worthwhile
    • Secure
    • Private
    • Forest Glen
    • Waterfall
    • Springtime
    • Sunny
    • Dynamic
    • Safe
    • Active
  • Initial Iterations
    • Made initial design in presentation SW
    • Three iterations, each with a small number of users (3-6)
    • Gave each user a motivating scenario for learning based on dilemma for “Joe”
    • For each screen, asked what they saw, what each button would do, which action they would take, then “reset” to sequence by saying what Joe actually did
    • Asked HCI experts in company for “heuristic evaluation”
  • Intranet Standards
    • Functionality and vocabulary tested and changed with presentation software prototypes
    • Our intended users were all familiar with IBM Intranet so, “being forced” to use the IBM Intranet standards made sense
    • These largely determined the UI “look and feel”
    • First working prototype developed
  • Custom Course Delivery
  • Custom Course Delivery
  • Custom Course Delivery
  • Custom Course Delivery
  • Custom Course Delivery
  • Testing before Field Trial
    • The development team tested prototype
      • Attempting to log in twice crashed the system
      • Changing the URL directly crashed the system
    • These were fixed and six more Research personnel tried the system
    • New requirement to “drag and drop” course modules to re-arrange them was implemented
  • Field Trial One
    • 118 users signed up
    • 75 accessed system
    • 81% satisfaction with system
    • 90% found easy to use
    • &quot;I was impressed how well you put the topics together. With the proviso that students understand they receive a smattering of information, where the entire document would provide depth, this is a marvellous way to bring someone up to speed in an area that they need a modicum of knowledge in a short period of time.”
  • Positive Comments
    • Positive Comments: 14
      • … I was impressed…
      • … you can easily find…
      • … very good, precise, and to the point….
      • Well laid out, very easy to follow and understand
      • It’s a fine process. I’m looking forward to more titles.
      • Impressive….
      • This looks slick guys…good job….
      • I like how you can jump to the source…
      • … I like it very much!
      • Being able to select…for your needs is an asset.
      • I think this concept of course development is great.
      • … is excellent….will be a great time saver and productivity boost.
      • … very easy to use and to navigate…Thanks for a great job.
      • … this concept of course development is great…
  • Negative Comments
    • Negative Comments: 5
      • As a novice…I found it very difficult to use the query method to build my unique course. This method may be good for experts…
      • … .so, I might use it if I need to find out the related concept for a specific topic, then go further from there. Meanwhile, I will continue to use Redbooks.
      • Text type and size hard to read…
      • … (hard) to pick the right search criteria
      • … after suspending course…(they disappeared)
  • Suggestions for Improvements
    • Make difficulty filter actually work – 3
    • When you click on related links, open a new IE window – 3
    • Enable one to narrow the results in various ways – 1
    • Be able to send to another person – 1
    • Provide pop-up window for expanding acronyms - 1
    • Have clock show expired vs. estimated time – 1
    • (Improve) sequencing – 1
    • No “backpage” from dead link forces escape and re-logon – 1
    • Add a way to see date of publication of Redbook -1 (without having to go there)
    • Make type font and size customizable -1
    • Make it possible to update course by removing/adding lessons -1
  • Behavior Patterns
    • “ Testing out the query engine”
      • User tries out slight syntactic variations; perhaps doesn’t even play anything
    • “ Playing saved courses”
      • User plays courses without first issuing a query
    • “ Checking out the scene”
      • User enters query; assembles course; plays course but too quickly to actually learn anything
    • “ Real learning”
      • User enters query; assembles course; spends considerable time (apparently)
  • “Real Learning”
    • Suspected: 69 instances that look like “real learning”
    • Suspected 37 different individuals with at least one instance of “real learning”
    • A subset of these were interviewed in more depth
    • A second field trial yielded similar results
  • Field Trial Limitations
    • People liked the system
    • People found it easy to use
    • However:
      • Our materials intersected with their real world needs only for a small percent of users
      • No evidence that it really helped them learn better
    • Chicken/Egg problem: Users will not use extensively without extensive materials converted to learning objects; Education department unwilling to spend a lot of effort converting extensive materials without proof of value
    • Therefore, to supplement these field results, we designed an experimental study
  • Experimental Design Considerations
    • Volunteer users were all busy IBM professionals
    • Their actual learning needs differed considerably
    • We wanted to put them in a motivational state as similar as possible to the actual intended conditions of use
    • Requiring them to learn exactly the same material would be statistically ideal, but not really exercise the functionality of the “custom course” concept
    • Resultant compromise was to engage subjects through a motivating scenario that put them in a situation similar to real work, focused them on a specific area but allowed for considerable individual variation depending on their own background and how they interpreted the situation
    • Verified with five pilot subjects
  • Experimental Design
    • Two Conditions: Query Only and Custom Course Condition (CCC)
    • All subjects given same background questionnaire, same scenario, same design task, same objective knowledge questions, same subjective questionnaire, and opportunity for comments
    • Query Only subjects only had access to the query engine
    • Custom Course subjects also used the system to assemble and use one or more courses
  • Different Functions
    • Though similar in appearance, two groups different in several important ways
    • In CCC, the learning material was referred to as a “course” rather than “results”
    • CCC subjects were shown additional metadata about each module
    • In CCC, each module had learning objectives
    • In CCC, the material was rearranged according to the metadata
    • In CCC, subjects explicitly selected modules and explicitly pushed a button: Assemble Course
  • Experimental Design
    • Procedure
      • Most subjects were run remotely
      • Substantive questions were deflected
      • Subjects free to use tools in preparing designs but were not to use books or colleagues for answers
      • Clear that results would be anonymized
    • Subjects were IBM volunteers
      • Intended 20/20 but only 26 total were run
      • Mostly IT specialists but varied widely in experience
  • Experimental Design
    • Tasks
      • Background questionnaire
      • Read motivating scenario about client needing high level design proposal for globalized Web Service solution
      • Learn what is needed for design
      • Provide requirements and design solution
      • Answer objective fact questions
      • Answer subjective Likert scale questions
      • Provide free-form comments
  • Design Results
    • CCC produced more design behavior; e.g., 6.5 pages vs. 1.83 (t(24)), p < .05.
    • More experienced subjects produced significantly more design behavior (pages, words, boxes, arrows)
    • CCC designs had more features matching an expert solution (n.s.); correlated with experience r=.463, p<.05; if corrected for experience, CCC t(22)=2.35, p<.03.
  • Qualitative Design Results
    • Three experts rated each design on Presentation, Level of Detail, Accuracy, Completeness and Depth of Understanding
    • Graders agreed, r=.72, p<.0001
    • CCC produced “better” designs in the opinion of each rater, but not significantly so
    • If adjusted for experience, more “good” designs in CCC than Query Only Chi Square(1)=4.43, p<.05.
  • Other Findings
    • Custom Course Diversity
      • 29 LO visited by one subject each
      • No LO visited by every subject
      • Every course different, but every course relevant to the scenario
      • Subjectively, it appeared that choices were related to experience level
    • Behavior during experiment
      • CCC used significantly more words related to cognition
      • Query Only used significantly more words related to navigation
      • CCC used fewer queries (5.0 vs. 12.5, t(20)=2.16, p<.05) based on server log data
    • Query Only made longer free form comments than CCC 60.6 words vs. 28.9 t(21)=2.13, p<.05
    • Groups did not differ in subjective scale; however, subjective satisfaction correlated with experience
      • r = .601, t(21)=3.445, p<.0024 (two-tailed)
  • Possible Explanations
    • Course outline provided advanced organizer
    • Proceeding through LO’s in course may have provided a more logical order
    • Additional metadata (difficulty, description, duration) allowed subjects to find more appropriate learning material
    • Extra work involved in constructing a course may have encouraged more focus
    • Expectation of building a “course” may have induced a more reflective cognitive set
  • Lessons Learned: System
    • Content Matters
    • Search Engines are not e-learning
    • Useful tools can be made within the strictures of corporate guidelines for UI
  • Lessons Learned: Process
    • Stories Work
    • Iterate Often and Early
    • Design is Measurable
    • Screening Experimental Subjects is Critical
    • Remote Observation Works
    • Team Early and Often
  • References
    • [1] Farrell, R., Liburd, S.D. and Thomas, J.C. Dynamic Assembly of Learning Objects. Proceedings of the World-Wide Web Conference , (2004) New York, NY.
    • [2] Thomas, J. C. and Farrell, R. G. An experimental investigation on the effectiveness of individualized web-based learning based on the dynamic assembly of learning objects, IBM Technical Report, (2004)RC 23338.
    • [3] Farrell, R. Dynamic Assembly of Learning Materials in a Corporate Context, Educational Technology , vol 46, no 1. (To appear)
    • [4] http://www.alphaworks.ibm.com/tech/dle
  • Redbooks
    • IBM WebSphere V4.0 Advanced Edition Handbook
    • Migrating WebSphere Applications to z/OS
    • WebSphere J2EE Application Development
    • Connecting to CICS from WS V4.0.1 for z/OS
    • WebSphere Solution Bundles
    • WebSphere Studio App Dev Programming Guide
    • Multilingual e-Business Solution Design Guide
    • Enabling Web Services for the IBM iSeries
    • Version 5 Web Services Handbook
    • A Portal Composite Pattern
  • Implementation
    • Entirely in JAVA
    • Runs on web application servers compatible with servlet 2.0
    • Web pages created using JSP’s
    • Learning Object content in XML transformed to XHTML using the XSLT style sheet processor
    • Uses IBM Haifa Lab search engine, JURU which indexes both object content and metadata
  • Metadata
    • 27 SME’s
    • Books are transformed into DocBook XML
    • Documents then cleaved into chapters, sections, subsections
    • All XML metadata uses IEEE LOM standard
    • We submitted extensions to the Learning Technology Standards Committee
  • Ontology
    • Relations between topic entities are declared using RDF description elements
    • Not strictly OWL, but utilized OWL’s properties about relationships
    • RDF files are parsed into JAVA objects at startup