3a8 picture driven computing in assistive

  • 622 views
Uploaded on

 

More in: Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
622
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
8
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. PICTURE-DRIVEN COMPUTING IN ASSISTIVETECHNOLOGY FOR ENHANCED ACCESSIBILITY Silvio Bonfiglio1 - George Kourousias2 1. FIMI, a Barco Company, Saronno (Italy) 2. Synchrotron Radiation Facility, ELETTRA, Trieste (Italy) 1st International AEGIS Conference 7-8 October 2010, University of Seville www.aegis-project.eu
  • 2. Domain: Improved Accessibility in ICTWhy? It may positively affect and improve our society PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY 1st International AEGIS Conference 7-8 October 2010, University of Seville
  • 3. Two Paths to AccessibilityDIRECT ACCESS Universal Design (UD) – Design for All i.e. a product that – without add-ons – can be experienced with similar ease of use by any user independently by the disability.INDIRECT ACCESS Multi-modal, Personalized Accessible Design i.e. a product that is made accessible for a specific category of users by incorporating as an add on a specific AT component or an extra interaction modality. PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY 1st International AEGIS Conference 7-8 October 2010, University of Seville
  • 4. Picture Driven Computing We define as Picture-driven Computing the Computing paradigm that has as core Data Abstraction the Visual Information in a 2- dimensional spatial domain. This Visual Information is what intends to be visible on a graphical Output device.•We aim to introduce it as an Assistive Technology•Adding a Modality to the GUI of the computer•Accessibility through Indirect Access PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY 1st International AEGIS Conference 7-8 October 2010, University of Seville
  • 5. Example: a Picture-Driven Search•Using Picture objects instead of keywords PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY 1st International AEGIS Conference 7-8 October 2010, University of Seville
  • 6. Sikuli, a Picture-Driven language•Developed recently at MIT by T. Yeh, T.H. Chang and R.C.Miller for the search andautomation of GUI elements using screenshots•Based on Sikuli-script , a scripting module that allows programmers to usescreenshots of GUI elements to control them programmatically without any API. •Python•Open Source & Multiplatform•We use it as our main paradigm for Picture-Driven Development of AT PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY 1st International AEGIS Conference 7-8 October 2010, University of Seville
  • 7. Sikuli: How it WorksA. a computer vision libraryB. a mechanism to programmatically “dictate” Input actionsC. a mature underlying language (Python) for traditional programatic control PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY 1st International AEGIS Conference 7-8 October 2010, University of Seville
  • 8. Sikuli: An ExampleGoal: clicks all the “minimize window” buttons on the screenPicture-Object Input: target image of the buttoni) then the systems computer vision (A) will locate the x,y position of that on the screenii) (B) the “mouse control” will “send” the mouse on x,y, andiii) through a for or while loop (C) will click (B) for all the targets that (A) has found. PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY 1st International AEGIS Conference 7-8 October 2010, University of Seville
  • 9. Sikuli as an Accessibility Design enablerWe Identify the following 3 classes: Automation Scripts: where Sikuli script - following a predefined set of mouse and keyboard actions- preforms a task that may be difficult otherwise to the user with a disability. Event Raisers: where the system can keep monitoring the users screen for specific graphical objects and raise an event/alarm and consequently trigger a set of programmatic actions. Bridge Systems: where applications can communicate with other software even if an API is not provided. + Hybrid approaches PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY 1st International AEGIS Conference 7-8 October 2010, University of Seville
  • 10. Future work•Further research on Picture-driven Computing.•Comparison of “indirect access” Accessibility Design versus the“direct” one (Universal Design) for the computer as an AT-aid.•Additional development of Sikuli AT Scripts.•Evaluation of the technical issues and performance of Sikuli. PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY 1st International AEGIS Conference 7-8 October 2010, University of Seville
  • 11. Acknowledgments
  • 12. further info: silvio.bonfiglio@barco.comgeorge.kourousias@elettra.trieste.itPICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY 1st International AEGIS Conference 7-8 October 2010, University of Seville