Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

McSig Chi 2008


Published on

A presentation of the McSig system for allowing a teacher to work with a visually impaired student when learning to write. Presented at CHI 2008.

  • Be the first to comment

  • Be the first to like this

McSig Chi 2008

  1. 1. McSig - A Multimodal Collaborative Handwriting Trainer for Visually-Impaired People Beryl Plimmer, Rachel Blagojevic University of Auckland Andrew Crossan, Stephen Brewster University of Glasgow
  2. 2. Outline <ul><li>Motivation </li></ul><ul><ul><li>Why is writing important for a visually impaired person </li></ul></ul><ul><ul><li>Why is a difficult thing to train </li></ul></ul><ul><li>How can we train users to draw a letter shape </li></ul><ul><ul><li>One solution – haptic trajectory playback </li></ul></ul><ul><ul><li>How good are people at recalling a trajectory they’ve been dragged through </li></ul></ul><ul><li>McSig </li></ul><ul><ul><li>Iterative Design </li></ul></ul><ul><ul><li>Evaluation </li></ul></ul><ul><li>Conclusions </li></ul>
  3. 3. Why do Visually Impaired People Need to Write ? <ul><li>Signature </li></ul><ul><ul><li>Difficult without visual feedback </li></ul></ul><ul><ul><li>Important for Job Applications, Legal documents etc. </li></ul></ul><ul><ul><li>One participant described her signature as ‘resembling the meanderings of an inebriated fly’ </li></ul></ul><ul><li>More general problem with spatial data </li></ul><ul><ul><li>Presenting </li></ul></ul><ul><ul><li>Creating </li></ul></ul>
  4. 4. Accessible Computing <ul><li>Magnifiers </li></ul><ul><ul><li>Need some sight </li></ul></ul><ul><li>Screen Readers </li></ul><ul><ul><li>Synthesized speech reads text from the screen </li></ul></ul><ul><ul><li>Not good for non-text information </li></ul></ul><ul><li>Dynamic Braille </li></ul><ul><li>Non-text or spatial data is very difficult to present non-visually </li></ul><ul><ul><li>Maps, charts, graphs, diagrams, web pages </li></ul></ul>
  5. 5. Force Feedback <ul><li>Haptic – Relating to the sense of touch </li></ul><ul><ul><li>Force feedback </li></ul></ul><ul><ul><li>Tactile feedback </li></ul></ul><ul><li>Force feedback </li></ul><ul><ul><li>Forces can be exerted through motors </li></ul></ul><ul><ul><li>Can be arranged to form 3D objects </li></ul></ul><ul><ul><li>Can be used to directly influence the movements of the user </li></ul></ul><ul><li>We use a PHANTOM OMNI from SensAble </li></ul><ul><ul><li>Provides 6 dof in, 3 dof out </li></ul></ul><ul><ul><li>Produces rich feedback suitable for many general purpose applications </li></ul></ul><ul><ul><li>Max 3 Newton force so the user can overpower the device </li></ul></ul>
  6. 6. Force Feedback as an Accessibility Aid <ul><ul><li>Non-visual communication channel </li></ul></ul><ul><ul><li>Can directly affect the user’s exploration </li></ul></ul><ul><ul><ul><li>Potential for collaborative applications </li></ul></ul></ul><ul><ul><li>Research question </li></ul></ul><ul><ul><ul><li>Can trajectory playback techniques be used to communicate shape and trajectory information to visually impaired people ? </li></ul></ul></ul>
  7. 7. Initial Studies <ul><li>Learning abstract trajectories through haptic guidance </li></ul><ul><ul><li>drag a user through a set of pre-defined trajectories </li></ul></ul><ul><ul><li>Initially significantly harder for visually impaired users & congenitally blind participants found the task harder still </li></ul></ul><ul><ul><li>But combining haptic guidance with a sonification of the trajectory (pitch shift & panning a sinusoidal tone) significantly improved learning </li></ul></ul>Audio Pan Pitch Shift Drawing Area
  8. 8. McSig: Multimodal, collaborative handwriting and signature training tool <ul><li>Teacher and student work synchronously on shared representation sat next to each other </li></ul><ul><ul><li>Haptic guidance and audio feedback </li></ul></ul><ul><ul><li>Teacher guides the student to learn letter shapes using words and actions </li></ul></ul><ul><li>Student holds PHANTOM Omni pen, teacher uses a stylus & touchscreen </li></ul><ul><ul><li>Teacher can move student’s pen around the character shape </li></ul></ul><ul><li>Audio feedback as shape is drawn </li></ul><ul><ul><li>Left/right movements: pan, up/down movements: pitch </li></ul></ul>
  9. 9. McSig – system design <ul><li>Simulate standard school learning scenario </li></ul><ul><li>Teacher can choose between three modes </li></ul><ul><ul><li>Collaborative Playback mode: </li></ul></ul><ul><ul><ul><li>Student holds PHANTOM Omni pen </li></ul></ul></ul><ul><ul><ul><li>student dragged through shape as it is drawn by teacher </li></ul></ul></ul><ul><ul><li>Free Stencil mode: </li></ul></ul><ul><ul><ul><li>Teacher draws letter which is used as a virtual stencil which thestudent can then explore </li></ul></ul></ul><ul><ul><ul><li>Can reduce constraining forces as student gets more experienced </li></ul></ul></ul><ul><ul><li>Free drawing mode </li></ul></ul><ul><ul><ul><li>Student moves Omni pen freely within area. </li></ul></ul></ul><ul><ul><ul><li>Movements echoed on teacher’s screen </li></ul></ul></ul>
  10. 10. First evaluation <ul><li>1 blind adult </li></ul><ul><li>Feedback key to learning </li></ul><ul><ul><li>User tried to feel what they had drawn on the paper </li></ul></ul><ul><ul><li>Added Dutch drawing board – shape is raised on the paper </li></ul></ul><ul><ul><li>Can be felt with other hand </li></ul></ul><ul><li>Stencil didn’t work very well </li></ul>
  11. 11. Usability testing <ul><li>4 visually impaired adults (3 blind, 1 partially sighted) </li></ul><ul><li>Familiarization </li></ul><ul><li>Letter set ‘ o,c,a,d,e ’ </li></ul><ul><li>Started with playback mode and then moved on to stencil </li></ul><ul><li>Finally drew the letter unsupported </li></ul><ul><li>Stencil mode hard to use </li></ul><ul><ul><li>Strengthened forces for shapes to give clearer path </li></ul></ul><ul><ul><li>Still didn’t work very well </li></ul></ul><ul><li>Audio feedback useful to some, teacher descriptions most useful </li></ul><ul><li>Speech feedback error prone and reduced confidence of users </li></ul><ul><li>Omni pen difficult to hold, plus pressing button whilst drawing tricky </li></ul><ul><ul><li>Users not used to holding pens </li></ul></ul><ul><ul><li>Gave some pen training before main study </li></ul></ul>
  12. 12. Evaluation <ul><li>Could McSig improve handwriting performance? </li></ul><ul><li>Task designed with teachers </li></ul><ul><ul><li>Some children almost no handwriting skills, some have good skills </li></ul></ul><ul><ul><li>5 characters chosen after discussion with teachers </li></ul></ul><ul><ul><ul><li>o, c, a, d, e </li></ul></ul></ul><ul><li>Participants </li></ul><ul><ul><li>8 children 11-17 years old, read Braille, no other major disabilities </li></ul></ul><ul><ul><li>3 partially sighted, 5 blind </li></ul></ul><ul><li>4 stage study </li></ul><ul><ul><li>Familiarization with McSig, then for each letter: </li></ul></ul><ul><ul><li>Pre-test </li></ul></ul><ul><ul><li>McSig training </li></ul></ul><ul><ul><li>Post-test </li></ul></ul>
  13. 13. Familiarization <ul><li>Participants could feel setup, PHANTOM, mat, PC </li></ul><ul><li>Spatial orientation </li></ul><ul><li>Drew circle, horizontal and vertical lines </li></ul><ul><li>Practised with the pen </li></ul><ul><ul><li>Visually impaired people don’t commonly use pens </li></ul></ul>
  14. 14.
  15. 15. McSig: Pre-Test Training and Post-Test <ul><li>Pre-test: participants asked to draw each letter as best they could </li></ul><ul><ul><li>Some unable to draw one or more of them </li></ul></ul><ul><li>Training: teacher showed participant how to draw letter in Playback mode </li></ul><ul><ul><li>Experimenter wrote shape on screen, child felt it with PHANTOM and scored line on tactile sheet </li></ul></ul><ul><ul><li>Synchronous audio/haptic/tactile feedback </li></ul></ul><ul><ul><li>Number of repeats based on child’s confidence </li></ul></ul><ul><li>Post-test: participants draw character in free draw mode </li></ul><ul><ul><li>If participant could not draw it we trained and tested again </li></ul></ul><ul><li>Time-out after 20 mins </li></ul><ul><ul><li>Stopped earlier if all letters done </li></ul></ul>
  16. 16. Results – partially-sighted children <ul><li>Participants </li></ul><ul><ul><li>All could read enlarged print </li></ul></ul><ul><ul><li>All had deteriorating sight but had learned to write when sight was better </li></ul></ul><ul><ul><li>Did not write now as sight too bad </li></ul></ul><ul><li>Familiarized very quickly, could all do circle, horizontal and vertical lines, no problem </li></ul><ul><li>One participant did all of our letters in the pre-test </li></ul><ul><li>Both of the others started ‘d’ in wrong place </li></ul>Pre Post
  17. 17. Results – partially-sighted children <ul><li>All letter correct except </li></ul><ul><ul><li>One did a normal ‘e’ in mirror image </li></ul></ul><ul><li>Participants had eyes close to drawing surface but did not feel drawing surface with non-dominant hand </li></ul><ul><ul><li>Wanted to use their sight </li></ul></ul><ul><li>All trained quickly and did all letters correctly in post-test </li></ul><ul><ul><li>Completed within 20 mins </li></ul></ul><ul><li>Politely interested but not captivated </li></ul>
  18. 18. Results – blind children <ul><li>Participants </li></ul><ul><ul><li>5 totally blind </li></ul></ul><ul><ul><li>One lost her sight at 3 years, others blind from birth </li></ul></ul><ul><li>Familiarization took much longer </li></ul><ul><ul><li>Pressure on pen – too much/too little </li></ul></ul><ul><li>Interacted with drawing space very differently </li></ul><ul><ul><li>Non-dominant hand for orientation in space </li></ul></ul><ul><li>All but one could draw circle and lines </li></ul><ul><li>Before and after examples </li></ul>
  19. 19. Before and after
  20. 20.
  21. 21. <ul><li>Participant who had sight until age 3 </li></ul>
  22. 22. Discussion - Technology <ul><li>Training </li></ul><ul><li>Student independent </li></ul><ul><li>Input - teacher </li></ul><ul><ul><li>Stylus pen </li></ul></ul><ul><ul><li>Verbalization </li></ul></ul><ul><li>Output </li></ul><ul><ul><li>Visualization on teacher’s display </li></ul></ul><ul><ul><li>Phantom Force feedback </li></ul></ul><ul><ul><li>Sound pan and pitch </li></ul></ul><ul><ul><li>Tactile trace </li></ul></ul><ul><ul><li>Recognition to voice output </li></ul></ul><ul><li>Input </li></ul><ul><ul><li>Phantom pen position </li></ul></ul><ul><li>Output </li></ul><ul><ul><li>Visualization on teacher’s display </li></ul></ul><ul><ul><li>Sound pan and pitch </li></ul></ul><ul><ul><li>Tactile trace </li></ul></ul><ul><ul><li>Recognition to voice output </li></ul></ul>
  23. 23. Discussion <ul><li>Results suggest that McSig could help children to learn </li></ul><ul><ul><li>Especially blind children </li></ul></ul><ul><li>Why didn’t stencil mode work? </li></ul><ul><ul><li>No tactile representation of the letter for non-dominant hand </li></ul></ul><ul><ul><li>Therefore an inconsistent interface </li></ul></ul>
  24. 24. Future Work <ul><li>Cursive handwriting and signatures </li></ul><ul><ul><li>Support move from single letters to cursive </li></ul></ul><ul><ul><li>A signature can be created and then practised to keep it consistent over time </li></ul></ul><ul><li>A self-teaching tool </li></ul><ul><li>Wider context </li></ul><ul><ul><li>Could be used in any application where the teacher wants to guide student </li></ul></ul><ul><ul><ul><li>Geometry, 2D and 3D shapes </li></ul></ul></ul><ul><ul><ul><li>Charts and graphs </li></ul></ul></ul>
  25. 25. Conclusions <ul><li>Hard for visually-impaired people to learn to handwrite </li></ul><ul><ul><li>Signatures difficult to learn and keep consistent </li></ul></ul><ul><ul><li>Required for important aspects of life </li></ul></ul><ul><li>McSig: a collaborative tool that allows a teacher to guide a student to handwrite letter shapes </li></ul><ul><ul><li>Dynamic haptic and audio feedback </li></ul></ul><ul><li>Can improve handwriting in 20 minute session </li></ul><ul><ul><li>All blind students learned at least 2 new letters </li></ul></ul><ul><ul><li>Enjoyed the experience </li></ul></ul><ul><li>Now working on longer-term study to see how learning develops over time </li></ul>
  26. 26. McSig - A Multimodal Collaborative Handwriting Trainer for Visually-Impaired People Beryl Plimmer, Rachel Blagojevic University of Auckland Andrew Crossan, Stephen Brewster University of Glasgow Thanks to our participants, John Williamson and Malcolm Hall at Glasgow University and our sponsors University of Auckland and the EU FP6 MICOLE Project