McSig - A Multimodal Collaborative Handwriting Trainer for Visually-Impaired People Beryl Plimmer, Rachel Blagojevic Unive...
Outline <ul><li>Motivation </li></ul><ul><ul><li>Why is writing important for a visually impaired person </li></ul></ul><u...
Why do Visually Impaired People Need to Write ? <ul><li>Signature </li></ul><ul><ul><li>Difficult without visual feedback ...
Accessible Computing <ul><li>Magnifiers </li></ul><ul><ul><li>Need some sight </li></ul></ul><ul><li>Screen Readers </li><...
Force Feedback <ul><li>Haptic – Relating to the sense of touch </li></ul><ul><ul><li>Force feedback </li></ul></ul><ul><ul...
Force Feedback as an Accessibility Aid <ul><ul><li>Non-visual communication channel </li></ul></ul><ul><ul><li>Can directl...
Initial Studies <ul><li>Learning abstract trajectories through haptic guidance </li></ul><ul><ul><li>drag a user through a...
McSig: Multimodal, collaborative handwriting and signature training tool <ul><li>Teacher and student work synchronously on...
McSig – system design <ul><li>Simulate standard school learning scenario </li></ul><ul><li>Teacher can choose between thre...
First evaluation <ul><li>1 blind adult </li></ul><ul><li>Feedback key to learning </li></ul><ul><ul><li>User tried to feel...
Usability testing <ul><li>4 visually impaired adults (3 blind, 1 partially sighted) </li></ul><ul><li>Familiarization </li...
Evaluation <ul><li>Could McSig improve handwriting performance? </li></ul><ul><li>Task designed with teachers </li></ul><u...
Familiarization <ul><li>Participants could feel setup, PHANTOM, mat, PC </li></ul><ul><li>Spatial orientation </li></ul><u...
McSig: Pre-Test Training and Post-Test <ul><li>Pre-test:  participants asked to draw each letter as best they could </li><...
Results – partially-sighted children  <ul><li>Participants </li></ul><ul><ul><li>All could read enlarged print </li></ul><...
Results – partially-sighted children <ul><li>All letter correct except </li></ul><ul><ul><li>One did a normal ‘e’ in mirro...
Results – blind children <ul><li>Participants </li></ul><ul><ul><li>5 totally blind </li></ul></ul><ul><ul><li>One lost he...
Before and after
<ul><li>Participant who had sight until age 3 </li></ul>
Discussion - Technology <ul><li>Training </li></ul><ul><li>Student independent </li></ul><ul><li>Input - teacher </li></ul...
Discussion <ul><li>Results suggest that McSig could help children to learn </li></ul><ul><ul><li>Especially blind children...
Future Work <ul><li>Cursive handwriting and signatures </li></ul><ul><ul><li>Support move from single letters to cursive <...
Conclusions <ul><li>Hard for visually-impaired people to learn to handwrite </li></ul><ul><ul><li>Signatures difficult to ...
McSig - A Multimodal Collaborative Handwriting Trainer for Visually-Impaired People Beryl Plimmer, Rachel Blagojevic Unive...
Upcoming SlideShare
Loading in...5
×

McSig Chi 2008

1,094
-1

Published on

A presentation of the McSig system for allowing a teacher to work with a visually impaired student when learning to write. Presented at CHI 2008.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,094
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
15
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • McSig Chi 2008

    1. 1. McSig - A Multimodal Collaborative Handwriting Trainer for Visually-Impaired People Beryl Plimmer, Rachel Blagojevic University of Auckland Andrew Crossan, Stephen Brewster University of Glasgow www.cs.auckland.ac.nz/~beryl www.dcs.gla.ac.uk/~stephen
    2. 2. Outline <ul><li>Motivation </li></ul><ul><ul><li>Why is writing important for a visually impaired person </li></ul></ul><ul><ul><li>Why is a difficult thing to train </li></ul></ul><ul><li>How can we train users to draw a letter shape </li></ul><ul><ul><li>One solution – haptic trajectory playback </li></ul></ul><ul><ul><li>How good are people at recalling a trajectory they’ve been dragged through </li></ul></ul><ul><li>McSig </li></ul><ul><ul><li>Iterative Design </li></ul></ul><ul><ul><li>Evaluation </li></ul></ul><ul><li>Conclusions </li></ul>
    3. 3. Why do Visually Impaired People Need to Write ? <ul><li>Signature </li></ul><ul><ul><li>Difficult without visual feedback </li></ul></ul><ul><ul><li>Important for Job Applications, Legal documents etc. </li></ul></ul><ul><ul><li>One participant described her signature as ‘resembling the meanderings of an inebriated fly’ </li></ul></ul><ul><li>More general problem with spatial data </li></ul><ul><ul><li>Presenting </li></ul></ul><ul><ul><li>Creating </li></ul></ul>
    4. 4. Accessible Computing <ul><li>Magnifiers </li></ul><ul><ul><li>Need some sight </li></ul></ul><ul><li>Screen Readers </li></ul><ul><ul><li>Synthesized speech reads text from the screen </li></ul></ul><ul><ul><li>Not good for non-text information </li></ul></ul><ul><li>Dynamic Braille </li></ul><ul><li>Non-text or spatial data is very difficult to present non-visually </li></ul><ul><ul><li>Maps, charts, graphs, diagrams, web pages </li></ul></ul>
    5. 5. Force Feedback <ul><li>Haptic – Relating to the sense of touch </li></ul><ul><ul><li>Force feedback </li></ul></ul><ul><ul><li>Tactile feedback </li></ul></ul><ul><li>Force feedback </li></ul><ul><ul><li>Forces can be exerted through motors </li></ul></ul><ul><ul><li>Can be arranged to form 3D objects </li></ul></ul><ul><ul><li>Can be used to directly influence the movements of the user </li></ul></ul><ul><li>We use a PHANTOM OMNI from SensAble </li></ul><ul><ul><li>Provides 6 dof in, 3 dof out </li></ul></ul><ul><ul><li>Produces rich feedback suitable for many general purpose applications </li></ul></ul><ul><ul><li>Max 3 Newton force so the user can overpower the device </li></ul></ul>
    6. 6. Force Feedback as an Accessibility Aid <ul><ul><li>Non-visual communication channel </li></ul></ul><ul><ul><li>Can directly affect the user’s exploration </li></ul></ul><ul><ul><ul><li>Potential for collaborative applications </li></ul></ul></ul><ul><ul><li>Research question </li></ul></ul><ul><ul><ul><li>Can trajectory playback techniques be used to communicate shape and trajectory information to visually impaired people ? </li></ul></ul></ul>
    7. 7. Initial Studies <ul><li>Learning abstract trajectories through haptic guidance </li></ul><ul><ul><li>drag a user through a set of pre-defined trajectories </li></ul></ul><ul><ul><li>Initially significantly harder for visually impaired users & congenitally blind participants found the task harder still </li></ul></ul><ul><ul><li>But combining haptic guidance with a sonification of the trajectory (pitch shift & panning a sinusoidal tone) significantly improved learning </li></ul></ul>Audio Pan Pitch Shift Drawing Area
    8. 8. McSig: Multimodal, collaborative handwriting and signature training tool <ul><li>Teacher and student work synchronously on shared representation sat next to each other </li></ul><ul><ul><li>Haptic guidance and audio feedback </li></ul></ul><ul><ul><li>Teacher guides the student to learn letter shapes using words and actions </li></ul></ul><ul><li>Student holds PHANTOM Omni pen, teacher uses a stylus & touchscreen </li></ul><ul><ul><li>Teacher can move student’s pen around the character shape </li></ul></ul><ul><li>Audio feedback as shape is drawn </li></ul><ul><ul><li>Left/right movements: pan, up/down movements: pitch </li></ul></ul>
    9. 9. McSig – system design <ul><li>Simulate standard school learning scenario </li></ul><ul><li>Teacher can choose between three modes </li></ul><ul><ul><li>Collaborative Playback mode: </li></ul></ul><ul><ul><ul><li>Student holds PHANTOM Omni pen </li></ul></ul></ul><ul><ul><ul><li>student dragged through shape as it is drawn by teacher </li></ul></ul></ul><ul><ul><li>Free Stencil mode: </li></ul></ul><ul><ul><ul><li>Teacher draws letter which is used as a virtual stencil which thestudent can then explore </li></ul></ul></ul><ul><ul><ul><li>Can reduce constraining forces as student gets more experienced </li></ul></ul></ul><ul><ul><li>Free drawing mode </li></ul></ul><ul><ul><ul><li>Student moves Omni pen freely within area. </li></ul></ul></ul><ul><ul><ul><li>Movements echoed on teacher’s screen </li></ul></ul></ul>
    10. 10. First evaluation <ul><li>1 blind adult </li></ul><ul><li>Feedback key to learning </li></ul><ul><ul><li>User tried to feel what they had drawn on the paper </li></ul></ul><ul><ul><li>Added Dutch drawing board – shape is raised on the paper </li></ul></ul><ul><ul><li>Can be felt with other hand </li></ul></ul><ul><li>Stencil didn’t work very well </li></ul>
    11. 11. Usability testing <ul><li>4 visually impaired adults (3 blind, 1 partially sighted) </li></ul><ul><li>Familiarization </li></ul><ul><li>Letter set ‘ o,c,a,d,e ’ </li></ul><ul><li>Started with playback mode and then moved on to stencil </li></ul><ul><li>Finally drew the letter unsupported </li></ul><ul><li>Stencil mode hard to use </li></ul><ul><ul><li>Strengthened forces for shapes to give clearer path </li></ul></ul><ul><ul><li>Still didn’t work very well </li></ul></ul><ul><li>Audio feedback useful to some, teacher descriptions most useful </li></ul><ul><li>Speech feedback error prone and reduced confidence of users </li></ul><ul><li>Omni pen difficult to hold, plus pressing button whilst drawing tricky </li></ul><ul><ul><li>Users not used to holding pens </li></ul></ul><ul><ul><li>Gave some pen training before main study </li></ul></ul>
    12. 12. Evaluation <ul><li>Could McSig improve handwriting performance? </li></ul><ul><li>Task designed with teachers </li></ul><ul><ul><li>Some children almost no handwriting skills, some have good skills </li></ul></ul><ul><ul><li>5 characters chosen after discussion with teachers </li></ul></ul><ul><ul><ul><li>o, c, a, d, e </li></ul></ul></ul><ul><li>Participants </li></ul><ul><ul><li>8 children 11-17 years old, read Braille, no other major disabilities </li></ul></ul><ul><ul><li>3 partially sighted, 5 blind </li></ul></ul><ul><li>4 stage study </li></ul><ul><ul><li>Familiarization with McSig, then for each letter: </li></ul></ul><ul><ul><li>Pre-test </li></ul></ul><ul><ul><li>McSig training </li></ul></ul><ul><ul><li>Post-test </li></ul></ul>
    13. 13. Familiarization <ul><li>Participants could feel setup, PHANTOM, mat, PC </li></ul><ul><li>Spatial orientation </li></ul><ul><li>Drew circle, horizontal and vertical lines </li></ul><ul><li>Practised with the pen </li></ul><ul><ul><li>Visually impaired people don’t commonly use pens </li></ul></ul>
    14. 14.
    15. 15. McSig: Pre-Test Training and Post-Test <ul><li>Pre-test: participants asked to draw each letter as best they could </li></ul><ul><ul><li>Some unable to draw one or more of them </li></ul></ul><ul><li>Training: teacher showed participant how to draw letter in Playback mode </li></ul><ul><ul><li>Experimenter wrote shape on screen, child felt it with PHANTOM and scored line on tactile sheet </li></ul></ul><ul><ul><li>Synchronous audio/haptic/tactile feedback </li></ul></ul><ul><ul><li>Number of repeats based on child’s confidence </li></ul></ul><ul><li>Post-test: participants draw character in free draw mode </li></ul><ul><ul><li>If participant could not draw it we trained and tested again </li></ul></ul><ul><li>Time-out after 20 mins </li></ul><ul><ul><li>Stopped earlier if all letters done </li></ul></ul>
    16. 16. Results – partially-sighted children <ul><li>Participants </li></ul><ul><ul><li>All could read enlarged print </li></ul></ul><ul><ul><li>All had deteriorating sight but had learned to write when sight was better </li></ul></ul><ul><ul><li>Did not write now as sight too bad </li></ul></ul><ul><li>Familiarized very quickly, could all do circle, horizontal and vertical lines, no problem </li></ul><ul><li>One participant did all of our letters in the pre-test </li></ul><ul><li>Both of the others started ‘d’ in wrong place </li></ul>Pre Post
    17. 17. Results – partially-sighted children <ul><li>All letter correct except </li></ul><ul><ul><li>One did a normal ‘e’ in mirror image </li></ul></ul><ul><li>Participants had eyes close to drawing surface but did not feel drawing surface with non-dominant hand </li></ul><ul><ul><li>Wanted to use their sight </li></ul></ul><ul><li>All trained quickly and did all letters correctly in post-test </li></ul><ul><ul><li>Completed within 20 mins </li></ul></ul><ul><li>Politely interested but not captivated </li></ul>
    18. 18. Results – blind children <ul><li>Participants </li></ul><ul><ul><li>5 totally blind </li></ul></ul><ul><ul><li>One lost her sight at 3 years, others blind from birth </li></ul></ul><ul><li>Familiarization took much longer </li></ul><ul><ul><li>Pressure on pen – too much/too little </li></ul></ul><ul><li>Interacted with drawing space very differently </li></ul><ul><ul><li>Non-dominant hand for orientation in space </li></ul></ul><ul><li>All but one could draw circle and lines </li></ul><ul><li>Before and after examples </li></ul>
    19. 19. Before and after
    20. 20.
    21. 21. <ul><li>Participant who had sight until age 3 </li></ul>
    22. 22. Discussion - Technology <ul><li>Training </li></ul><ul><li>Student independent </li></ul><ul><li>Input - teacher </li></ul><ul><ul><li>Stylus pen </li></ul></ul><ul><ul><li>Verbalization </li></ul></ul><ul><li>Output </li></ul><ul><ul><li>Visualization on teacher’s display </li></ul></ul><ul><ul><li>Phantom Force feedback </li></ul></ul><ul><ul><li>Sound pan and pitch </li></ul></ul><ul><ul><li>Tactile trace </li></ul></ul><ul><ul><li>Recognition to voice output </li></ul></ul><ul><li>Input </li></ul><ul><ul><li>Phantom pen position </li></ul></ul><ul><li>Output </li></ul><ul><ul><li>Visualization on teacher’s display </li></ul></ul><ul><ul><li>Sound pan and pitch </li></ul></ul><ul><ul><li>Tactile trace </li></ul></ul><ul><ul><li>Recognition to voice output </li></ul></ul>
    23. 23. Discussion <ul><li>Results suggest that McSig could help children to learn </li></ul><ul><ul><li>Especially blind children </li></ul></ul><ul><li>Why didn’t stencil mode work? </li></ul><ul><ul><li>No tactile representation of the letter for non-dominant hand </li></ul></ul><ul><ul><li>Therefore an inconsistent interface </li></ul></ul>
    24. 24. Future Work <ul><li>Cursive handwriting and signatures </li></ul><ul><ul><li>Support move from single letters to cursive </li></ul></ul><ul><ul><li>A signature can be created and then practised to keep it consistent over time </li></ul></ul><ul><li>A self-teaching tool </li></ul><ul><li>Wider context </li></ul><ul><ul><li>Could be used in any application where the teacher wants to guide student </li></ul></ul><ul><ul><ul><li>Geometry, 2D and 3D shapes </li></ul></ul></ul><ul><ul><ul><li>Charts and graphs </li></ul></ul></ul>
    25. 25. Conclusions <ul><li>Hard for visually-impaired people to learn to handwrite </li></ul><ul><ul><li>Signatures difficult to learn and keep consistent </li></ul></ul><ul><ul><li>Required for important aspects of life </li></ul></ul><ul><li>McSig: a collaborative tool that allows a teacher to guide a student to handwrite letter shapes </li></ul><ul><ul><li>Dynamic haptic and audio feedback </li></ul></ul><ul><li>Can improve handwriting in 20 minute session </li></ul><ul><ul><li>All blind students learned at least 2 new letters </li></ul></ul><ul><ul><li>Enjoyed the experience </li></ul></ul><ul><li>Now working on longer-term study to see how learning develops over time </li></ul>
    26. 26. McSig - A Multimodal Collaborative Handwriting Trainer for Visually-Impaired People Beryl Plimmer, Rachel Blagojevic University of Auckland Andrew Crossan, Stephen Brewster University of Glasgow www.cs.auckland.ac.nz/~beryl www.dcs.gla.ac.uk/~stephen Thanks to our participants, John Williamson and Malcolm Hall at Glasgow University and our sponsors University of Auckland and the EU FP6 MICOLE Project
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×