This presentation describes a mechanism for synthesizing meaningful concise descriptions for exploring virtual worlds using a screenreader.

1. Syntherella: A Feedback Synthesizer forEfficient Exploration of Virtual Worlds using a Screen Reader BugraOktay and EelkeFolmer Player-Game Interaction Lab

2. Outline Intro Problem Syntherella Discussion Future Work

3. Introduction Virtual World Screen Reader

4. Introduction Virtual World TextSL

5. Problem Densely Populated Environments Lots of Information to consume Easy with visual perception; overwhelming when transformed into text overwhelming

6. Problem (cnt’d) Command-line interface “describe” “describe objects” Iterative Queries “describe table” Slow interaction (even when there are very few objects) “zzz… zzz…” Consumable

7. Syntherella There are 5 animals, 4 tools, a house and a bike. There are 2 books, a cat, a dog and a house to your west; a dog, a bike … to your east; ... 2 cats are described as “Angora cat with blue eyes”. One of them is interactive and for sale for $20 ...

8. Syntherella: Scanning Scanning objects inside a virtual circle around the user’s avatar where the radius is defined by the user.

9. Syntherella: Filtering Filtering objects that do not contribute to the textual description of the environment (i.e. objects with non-descriptive or meaningless names).

10. Syntherella: Thresholding Good enough 74 table cat laptop book 30 Size / Distance / Name / Description / Interaction Capabilities undescriptive box too small stone too far cat 16

11. Syntherella: Grouping book book cat book chair cat chair table chair table book chair cat cat chair chair chair cat cat chair cat, book, chair, chair, cat, table, chair, book, chair and cat. 3 cats, 2 books, 4 chairs and table.

12. Syntherella: Aggregation There are object 11 objects. animal vehicle … 2 bikes and 9 animals. mammal reptile ground vehicle 2 bikes, 8 mammals and a snake. … more specific cat dog snake bike 2 bikes, 3 cats, 5 dogs and a snake.

13. Syntherella: Detailing There are 2 cats to your north; 1 cat to your south and 2 chairs to your east. Level 2:3 cats are described as “Angora cat with blue eyes”. 2 chairs are described as “black leather office chair”. Level 3:3 cats are described as “Angora cat with blue eyes”. 2 of them are interactive. 1 of them is not interactive. 2 chairs are described as “black leather office chair”. They are not interactive. Level 4:3 cats are described as “Angora cat with blue eyes”. 1 of them is interactive but not for sale. 1 of them is interactive and for sale for L300. 1 of them is not interactive and not for sale. 2 chairs are described as “black leather office chair”. They are not interactive but for sale for L20.

14. Syntherella: Example “describe objects” There are 936 objects: window, object, prim, box_by_vi$boyz, chair, tree, flowerz, book v2.0, stone, dog, cat, book, window, window, object, object, cat, no_name, obj23, owl, table, window, object, notecard dispenser, SL educational places kiosk, landmark_3.4, buy_me_L34, object, window, table, chair, chair, car, chair, bicycle, rabbit, rose, chair, chalk, object, skidzPrimz, object4, box7, window, door, tableXY, object, object, paper clip v4.0, lcd screen with remote, voice activated robot helper, phone booth, object, skidzPrimz, object4, box7, window, door, tableXY, object, object, paper clip v4.0, lcd screen with remote, voice activated robot helper, phone booth, window, object, prim, box_by_vi$boyz, chair, tree, flowerz, book v2.0, stone, dog, cat, book, window, window, object, object, cat, no_name, obj23, owl, table, window, object, notecard dispenser, SL educational places kiosk, landmark_3.4, buy_me_L34, object, window, table, chair, chair, car, chair, bicycle, rabbit, rose, chair, chalk, object, skidzPrimz, object4, box7, window, door, tableXY, object, object, paper clip v4.0, lcd screen with remote, voice activated robot helper, phone booth, object, skidzPrimz, object4, box7, window, door, tableXY, object, object, paper clip v4.0, lcd screen with remote, Pretty long

15. Syntherella: Example After scanning There are 87 objects: frame, object, dog, step, object, object, object, cat, prim, box_by_vi$boyz, object, stone, tree, rose, chair, object, object, frame, nozd_jkie&&jack, stone, step, dog, object, object, object, object, chair, table, cat, object, prim, prim, tree, prim, box, Julia’s Gull, cat, chair, object, object, object, object, object, prim, prim, object, stone, step, object, chair, table, cat, object, prim, prim, tree, prim, box, cat, chair, object, object, object, object, object, prim, prim, object, stone and step. After filtering There are 26 objects: frame, dog, step, cat, stone, frame, tree, rose, chair, frame, stone, dog, chair, table, cat, cat, chair, stone, step, stone, tree, chair, stone, step, Julia’s Gull and frame.

16. Syntherella: Example After thresholding There are 14 objects: dog, cat, tree, rose, chair, dog, chair, table, cat, cat, chair, tree, chair and Julia’s Gull. After grouping There are 14 objects: 2 dogs, 3 cats, 2 trees, rose, 4 chairs, table and Julia’s Gull. After aggregation (word limit: 14) There are 3 plants, 5 animals, 5 furniture and Julia’s Gull. After aggregation (word limit: 10) There are 13 known and 1 unknown objects.

17. Syntherella: Example After detailing There are 2 dogs, 1 cat, 2 trees to your north; 4 chairs, 1 table, 1 rose to your west and 2 cats, Julia’s Gull to your south. 2 dogs are described as “black and brown Saint Bernard”. 3 cats are described as “Angora cat with blue eyes”. 1 tree is described as “large oak tree”. 1 tree is described as “green and tall palmae tree”. 4 chairs are described as “brown stylish dining chair”. Rose is described as “Red rose”. Table is described as “diner table for 4”. Julia’s Gull is described as “Laughing gull that can fly; designed by Julia Doe”.

19. Perceptional Challenges

20. User Studies

21. Performance Measurements

22. Widening the Scope (augmented reality, navigation, web accessibility etc.)

23. Goal/Intention Oriented Feedback Generation

25. Perceptional Challenges

26. User Studies

27. Performance Measurements

28. Widening the Scope (augmented reality, navigation, web accessibility etc.)

29. Goal/Intention Oriented Feedback Generation

30. Exploiting Command Historybugra@cse.unr.edu