Ray Wang - Gnome Accessibility And Automation Testing


Published on

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Visual: Common, Age, Large Fonts, High Contrast, Magnification, Screen Reader Motor & Speech: Mild -> Serious, Ignore False Activation, Specialized or Alternative Input Devices Synthetic Speech Hearing: Mild -> Serious, Subtitles Cognitive: Reading disorders, Comprehension & Composition Difficulties Speech Recognition Text Highlighting, Careful UI Design, Symbols & Pictures.
  • CSPI: The library for C bindings Pyatspi:The Python bindings atk-bridge: The bridges, of course, need to understand both sides of what it being bridged; so atk-bridge must link to both ATK and AT-SPI. Registryd: Daemon
  • application layer includes productivity software, desktop tools, window manager, panel, etc; these are the same apps found outside ‘accessibility enabled’ environments. Also included in the ‘application layer’ are the GUI toolkits and the APIs necessary to provide accessibility info to the rest of the subsystem. Those APIs may vary among applications and toolkits (i.e. Java, GTK+, mozilla, OpenOffice) but they give adequate information to allow bridging to the common layers below. infrastructure includes what we sometimes call the “accessibility subsystem” and is what people sometimes just call “AT-SPI”. We usually use the name “AT-SPI” more specifically to refer only to the software interfaces and implementations contained in the “AT-SPI” module; either way, the AT-SPI packages are key parts of the infrastructure layer. The infrastructure layer is what enables assistive technologies to interact consistently with different applications. assistive (or adaptive) technologies are the ‘adapter’ programs that help the user operate desktop and apps. Examples: GOK, Orca, dasher Q: are features like StickyKeys assistive technologies? A: It depends who you ask; for our purposes we will call them accessibility utilities or platform accessibility features, and use “AT” to refer to programs that form a direct interface between the end user and the desktop, such as Screen Readers and alternative input systems.
  • We can divide the infrastructure layer into “bridge code” and AT-SPI; the bridging code connects the applications with the AT-SPI subsystem and therefore can be considered part of the infrastructure. Bridges take accessibility support information (services, events) from within the application and converts it to a common format (in this case, the AT-SPI protocol). Examples: java-access-bridge for GNOME; libatk-bridge. Bridges are important because: Applications on the desktop may implement similar features with a wide variety of code: Applications are written in different languages (C, Java, python, etc.) and may use different b“GUI toolkits” to create their interface components (GTK+, Java/Swing, mozilla-gecko, OpenOffice). Their object models and internal APIs may be very different from one another (Gobject, UNO, mozilla) This information needs to be shared or “exported” in a common ABI and via a common IPC protocol. We refer to the code that converts to and from the common format as “bridging” code.
  • Bridges also listen to application events, for instance AtkObject events which are implemented as Gsignals, and convert them into the appropriate AT-SPI event notifications. Events may give the AT some information about a change which has occurred in an application, but in many cases the AT needs to make API calls in response to an event in order to update the information it presents to the user. For instance, if focus moves to a new widget, speech and/or braille output may needed to inform the user of this change and in order to create a useful presentation of the newly focused object for the user (either as a spoken words or braille dots); alternatively if GOK is being used and a menu is focused or pops up, GOK may wish to traverse this menu in order to present the new choices to the user as a set of selectable “GOK Buttons”.
  • GNOME applications implement an in-process accessibility API called ATK. This is ‘bridged’ to the common AT-SPI layer by the ATK-bridge module. Atk-bridge exports ATK information via AT-SPI – it knows and understands the ATK API, but it doesn't know anything about specific applications or even anything about gtk+.
  • GAIL hooks into ATK and implements ATK interfaces on behalf of GTK+. GAIL knows about ATK and GTK+, but neither ATK nor GTK+ know about GAIL. GAIL is dynamically loaded via GTK_MODULES The ATK implementations for the GTK+ widget set are provided by an extermal library called GAIL (or libgail). It is external for mostly historical reasons, and partly in order to allow alternate implementations to be loaded by specialized applications. In the diagram above, note that GAIL gets information from GTK+ via public GTK+ API, and uses that information to fulfill the ATK interfaces on behalf of those widgets. For instance the atk-bridge calls ATK API, but the ATK call is redirected through code in libgail which in turn uses gtk+ API calls to provide the corresponding information.
  • Swing defines a java-specific accessibility API in javax.accessibility The end result is AT-SPI, just as for GNOME apps (interoperability). OpenOffice.org have an internal UNO Accessibility API. Until recently they bridged internally to javax.accessibility; now they bridge to ATK. Java ATK Wrapper
  • Ray Wang - Gnome Accessibility And Automation Testing

    1. 1. GNOME Accessibility & Automation Testing A brief introduction to use GNOME Accessibility to do automation test Ray Wang Software Engineer Novell / rawang@novell.com
    2. 2. Agenda <ul><li>What is Accessibility? </li><ul><li>Definition
    3. 3. AT-SPI </li></ul><li>How to use GNOME Accessibility to implement Automation Test </li><ul><li>Accerciser
    4. 4. LDTP
    5. 5. Orca
    6. 6. Dogtail
    7. 7. Strongwind </li></ul></ul>
    8. 8. What is Accessibility?
    9. 9. What is Accessibility? Definition <ul><li>Abbreviation: A11y
    10. 10. Ability to access
    11. 11. Being able to use a Computer regardless of disability or severity of impairment
    12. 12. Wikipedia </li><ul><li>http://en.wikipedia.org/wiki/Accessibility
    13. 13. http://en.wikipedia.org/wiki/Computer_accessibility
    14. 14. Other stuff... </li></ul></ul>
    15. 15. What is Accessibility? Definition <ul><li>Disabilities </li><ul><li>Visual
    16. 16. Motor & Speech
    17. 17. Hearing
    18. 18. Cognitive </li></ul></ul>
    19. 19. What is Accessibility? AT-SPI <ul><li>Assistive Technology Service Provider Interface (AT-SPI)
    20. 20. GNOME Accessibility infrastructure
    21. 21. AT-SPI </li><ul><li>cspi
    22. 22. pyatspi
    23. 23. atk-bridge
    24. 24. registryd </li></ul></ul>
    25. 25. What is Accessibility? AT-SPI <ul><li>Turn on your GNOME Accessibility </li></ul>
    26. 26. What is Accessibility? AT-SPI <ul><li>What's happened? </li></ul>
    27. 27. What is Accessibility? AT-SPI <ul><li>Divide the view of AT into 3 large components </li></ul>
    28. 28. What is Accessibility? AT-SPI <ul><li>Applications and ATs communicate via a layered architecture </li></ul>
    29. 29. What is Accessibility? AT-SPI <ul><li>ATs receive events and issue IPC calls </li></ul>API calls EVENTS
    30. 30. What is Accessibility? AT-SPI (GNOME) <ul><li>How GTK+/GNOME Applications connect to AT-SPI </li></ul>
    31. 31. What is Accessibility? AT-SPI (GNOME) AT-SPI GAIL <ul><li>How ATK is implemented for GTK+:
    32. 32. The GAIL module </li></ul>
    33. 33. What is Accessibility? AT-SPI (Java) AT-SPI <ul><li>Java application use a different bridge </li></ul>
    34. 34. What is Accessibility? AT-SPI (Mono) <ul><li>Winforms
    35. 35. Moonlight </li></ul>
    36. 36. What is Accessibility? AT-SPI (The Big Picture)
    37. 37. What is Accessibility? AT-SPI (CORBA -> DBus) <ul><li>New AT-SPI (AT-SPI2) </li><ul><li>at-spi2-core
    38. 38. at-spi2-atk </li></ul><li>Report Bugs </li><ul><li>https://bugs.freedesktop.org/enter_bug.cgi?product=at-spi2 </li></ul><li>http://live.gnome.org/Accessibility/BonoboDeprecation
    39. 39. http://www.linuxfoundation.org/en/Accessibility/ATK/AT-SPI/AT-SPI_on_D-Bus </li></ul>
    40. 40. What is Accessibility? AT-SPI (CORBA -> DBus)
    41. 41. What is Accessibility? AT-SPI (CORBA -> DBus)
    42. 42. How to use GNOME Accessibility to implement Automation Test
    43. 43. Automation Test Accerciser <ul><li>Accerciser is </li><ul><li>An Assistive Technology (AT)
    44. 44. an interactive Python accessibility explorer for the GNOME desktop
    45. 45. uses AT-SPI to inspect and control widgets
    46. 46. allowing you to check if an application is providing correct information to assistive technologies and automated test frameworks.
    47. 47. Accerciser has a simple plugin framework which you can use to create custom views of accessibility information. </li></ul><li>Useful AT tool for testing & debugging
    48. 48. http://live.gnome.org/Accerciser </li></ul>
    49. 49. Automation Test Accerciser
    50. 50. Automation Test LDTP <ul><li>Linux Desktop Testing Project
    51. 51. Written in Python
    52. 52. Use python bindings (pyatspi)
    53. 53. Being used by </li><ul><li>GNOME
    54. 54. Ubuntu
    55. 55. VMware
    56. 56. Palm source </li></ul><li>http://ldtp.freedesktop.org/wiki/ </li></ul>
    57. 57. Automation Test Orca <ul><li>A free, open source scriptable Screen Reader
    58. 58. Written in Python
    59. 59. Use python bindings (pyatspi)
    60. 60. Could be used in automation testing </li><ul><li>Has its own test harness
    61. 61. Test samples in the source code (firefox, openoffice,
    62. 62. gtk-demo, gcaltool etc) </li></ul><li>http://live.gnome.org/Orca </li></ul>
    63. 63. Automation Test Dogtail <ul><li>Open source GUI testing tool and automation framework
    64. 64. Written in Python
    65. 65. Use python bindings (pyatspi)
    66. 66. Being used in some companies
    67. 67. Out of maintain? </li></ul>
    68. 68. Automation Test Strongwind <ul><li>GUI test automation framework inspired by dogtail
    69. 69. Written in Python
    70. 70. Use python bindings (pyatspi)
    71. 71. Human-readable log
    72. 72. http://medsphere.org/community/project/strongwind </li></ul>
    73. 73. Automation Test Strongwind <ul><li>How to write a Automation Test by using Strongwind </li><ul><li>Testable Application
    74. 74. Application Wrapper
    75. 75. Test Script(s) </li></ul><li>http://www.mono-project.com/Accessibility:_Strongwind_Basics </li></ul>
    76. 76. Automation Test Strongwind
    77. 77. Automation Test Strongwind
    78. 78. Participation <ul><li>Mailing Lists </li><ul><li>GNOME Accessibility </li><ul><li>[email_address] </li></ul><li>GNOME Desktop Testing </li><ul><li>[email_address] </li></ul><li>Accessiblity-at-spi </li><ul><li>[email_address] </li></ul><li>Mono Accessibility </li><ul><li>[email_address] </li></ul></ul><li>Irc channel: </li><ul><ul><li>irc.gnome.org
    79. 79. #a11y </li></ul></ul></ul>
    80. 80. Reference <ul><li>Python Power Accessibility </li><ul><li>http://live.gnome.org/Accessibility/PythonPoweredAccessibility </li></ul><li>Mono Accessibility </li><ul><li>http://www.mono-project.com/Accessibility </li></ul><li>GNOME Website </li><ul><li>http://www.gnome.org </li></ul></ul>
    81. 82. Unpublished Work of Novell, Inc. All Rights Reserved. This work is an unpublished work and contains confidential, proprietary, and trade secret information of Novell, Inc. Access to this work is restricted to Novell employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of Novell, Inc. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability. General Disclaimer This document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. Novell, Inc. makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for Novell products remains at the sole discretion of Novell. Further, Novell, Inc. reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.