Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Parallel User Interface Rendering
                 ~
Accessibility for Custom Widgets

      Kris Van Hees & Jan Engelen
 ...
Agenda
●
    Introduction
●
    Parallel User Interface Rendering
●
    Widgets
●
    Cosmetic widgets
●
    Conclusions
●...
Introduction
●
    GUIs in all aspects of life: computers, MP3 players,
    home appliances, mobile phones, ...
●
    Inhe...
Introduction (cont...)
●
    Abstracting the user interface offers advantages:
     –   Unified user interaction semantics...
Parallel User Interface Rendering
●
    User interface defined in abstract form (XML file)
     –   Effectively defines da...
PUIR: The AUI engine
●
    Interprets an abstract UI description (XML file)
●
    Implements user interaction semantics
●
...
PUIR: GUI rendering agent
●
    Java Swing based in-process rendering agent
●
    Complex changes to bypass user interacti...
PUIR: Non-visual rendering agent
●
    Remote rendering agent
●
    Basic speech output presentation as proof-of-concept
●...
PUIR: Input processing
●
    Keyboard:
    –   Textual input
    –   Navigation, text selection, editing, …
    –   Proces...
PUIR: Input processing (cont...)
●
    Braille keyboard:
    –   Coded textual input translated into text
    –   Text pro...
Widgets
●
    Two entities per widget:
     –   Abstract widget:
          ●
              Operates entirely within the AU...
Widgets (cont...)
●
    Design:
    –   Define UI in abstract UI description
    –   Typically by means of design tool
   ...
Widgets (cont...)
●
    Runtime construction:
    –   Instantiation of UI based on AUI description
    –   For every widge...
Widgets (cont...)
●
    Rendering:
    –   One-to-one correspondence between abstract widget
        class and presentatio...
Widgets (cont...)
●
    Rendering (cont...):
    –   Rendering agent knows what widgets are supported
    –   Rendering ag...
Cosmetic widgets
●
    Widgets with a very specific presentation
●
    Typically derivative of an existing core widget
●
 ...
Cosmetic widgets (cont...)


●
    Left-to-right:
     –   Four regular buttons
     –   VU-meter style widget: equivalent...
Conclusions
●
    Parallel User Interface Rendering
     –   Powerful technique for Design-for-All
     –   Coherence betw...
Future...
●
    Completion of proof-of-concept
●
    More testing
●
    Public release

●
    Default presentation of cust...
Kris Van Hees & Jan Engelen
  Katholieke Universiteit Leuven
Department of Electrical Engineering
     ESAT – SCD – DocArc...
Upcoming SlideShare
Loading in …5
×

8

359 views

Published on

  • Be the first to comment

  • Be the first to like this

8

  1. 1. Parallel User Interface Rendering ~ Accessibility for Custom Widgets Kris Van Hees & Jan Engelen Katholieke Universiteit Leuven Department of Electrical Engineering ESAT - SCD - DocArch Kris Van Hees & Jan Engelen 1/16
  2. 2. Agenda ● Introduction ● Parallel User Interface Rendering ● Widgets ● Cosmetic widgets ● Conclusions ● Future... Kris Van Hees & Jan Engelen 2/16
  3. 3. Introduction ● GUIs in all aspects of life: computers, MP3 players, home appliances, mobile phones, ... ● Inherent visual interaction model poses a complication for users with disabilities (especially blindness) ● Mixing graphical toolkits within a single environment complicates matters even more (Linux: Qt, GTK, Athena, …) ● Custom widgets: often not supported in assistive technology without fancy scripting (if available) Kris Van Hees & Jan Engelen 3/16
  4. 4. Introduction (cont...) ● Abstracting the user interface offers advantages: – Unified user interaction semantics across toolkits and modalities – Retaining flexibility in rendering for multiple modalities – Assistive technology as 1st generation rendering ● Parallel User Interface Rendering (PUIR) provides for equivalent concurrent renderings of the UI Kris Van Hees & Jan Engelen 4/16
  5. 5. Parallel User Interface Rendering ● User interface defined in abstract form (XML file) – Effectively defines data and how to present data ● Abstract UI description interpreted by an AUI engine – Handles user interaction semantics ● UI presentation rendered by modality-specific agent(s) ● Rendering agent depends on system specific toolkits ● Rendering agents provide presentations concurrently – Provide both visual and non-visual rendering Kris Van Hees & Jan Engelen 5/16
  6. 6. PUIR: The AUI engine ● Interprets an abstract UI description (XML file) ● Implements user interaction semantics ● Encapsulates data in the UI ● Provides focus management ● Dispatches events to the application and rendering agents ● Can be called from application and rendering agents ● Has no knowledge of implementation details of application and rendering agents Kris Van Hees & Jan Engelen 6/16
  7. 7. PUIR: GUI rendering agent ● Java Swing based in-process rendering agent ● Complex changes to bypass user interaction in widgets ● Reduced to provide merely the presentation of widgets ● Some widgets operate slightly differently ● Pointer device UI interaction interpreted in the rendering agent: – Necessary because position is tied to GUI context – Semantic operations mapped to AUI operations Kris Van Hees & Jan Engelen 7/16
  8. 8. PUIR: Non-visual rendering agent ● Remote rendering agent ● Basic speech output presentation as proof-of-concept ● Proxy objects for transparency and caching ● Equivalent with GUI rendering agent: – Receives the same events – Synchronized with other rendering agents Kris Van Hees & Jan Engelen 8/16
  9. 9. PUIR: Input processing ● Keyboard: – Textual input – Navigation, text selection, editing, … – Processed directly by AUI (for proof-of-concept) ● Mouse: – Processed by rendering agent – Position to be interpreted within graphical context – Semantic user interaction delegated to AUI engine Kris Van Hees & Jan Engelen 9/16
  10. 10. PUIR: Input processing (cont...) ● Braille keyboard: – Coded textual input translated into text – Text processed as if entered on regular keyboard – Limited cursor addressable positional input – Positional input interpreted in context – Semantic user interaction delegated to AUI engine Kris Van Hees & Jan Engelen 10/16
  11. 11. Widgets ● Two entities per widget: – Abstract widget: ● Operates entirely within the AUI engine ● Implements user interaction semantics ● Encapsulates data – Widget presentation: ● Usually one per rendering agent ● Implements rendering in output modality Kris Van Hees & Jan Engelen 11/16
  12. 12. Widgets (cont...) ● Design: – Define UI in abstract UI description – Typically by means of design tool – Rendering specific information can be included Kris Van Hees & Jan Engelen 12/16
  13. 13. Widgets (cont...) ● Runtime construction: – Instantiation of UI based on AUI description – For every widget: ● Find built-in abstract widget class ● If none, search in alternate locations ● If none, UI cannot be instantiated Kris Van Hees & Jan Engelen 13/16
  14. 14. Widgets (cont...) ● Rendering: – One-to-one correspondence between abstract widget class and presentation class – For every widget: ● Find built-in widget presentation class ● If none, search in alternate locations ● If none and cosmetic widget, use core alternate (more on this 2 slides from now...) ● If none, use default place-holder widget Kris Van Hees & Jan Engelen 14/16
  15. 15. Widgets (cont...) ● Rendering (cont...): – Rendering agent knows what widgets are supported – Rendering agent can alert user appropriately – Rendering agent can query information from AUI about the unsupported widget – Accidental user interaction is guarded against Kris Van Hees & Jan Engelen 15/16
  16. 16. Cosmetic widgets ● Widgets with a very specific presentation ● Typically derivative of an existing core widget ● User interaction semantics remain the same ● AUI description specifies equivalence between cosmetic widget and core widget ● Core widget can act as default presentation when no custom presentation class is found Kris Van Hees & Jan Engelen 16/16
  17. 17. Cosmetic widgets (cont...) ● Left-to-right: – Four regular buttons – VU-meter style widget: equivalent with numeric label – Four knobs: equivalent with discrete slider – Four indicator lights: equivalent with label – Four regular buttons Kris Van Hees & Jan Engelen 17/16
  18. 18. Conclusions ● Parallel User Interface Rendering – Powerful technique for Design-for-All – Coherence between concurrent equivalent renderings – Generic: not just for non-visual renderings – Robust handling of custom widgets – Support for custom widgets independent from applications and specific uses – Can be used in automated application testing as well Kris Van Hees & Jan Engelen 18/16
  19. 19. Future... ● Completion of proof-of-concept ● More testing ● Public release ● Default presentation of custom widgets merely based on abstract widget information ● Remote operation of applications At all stages, feedback from real users is be crucial! Kris Van Hees & Jan Engelen 19/16
  20. 20. Kris Van Hees & Jan Engelen Katholieke Universiteit Leuven Department of Electrical Engineering ESAT – SCD – DocArch Kasteelpark Arenberg 10 B-3001 Heverlee Belgium kris@alchar.org, jan@docarch.be Kris Van Hees & Jan Engelen 20/16

×