Course project for CSE 218 - UCSD

1. Grad Students: Manindra Moharana, Narendran Thangarajan, Soham Shah
Undergrads: Cary Cheng, Christine He, Jessica Cho, Joann Kim,
Koa Nies, Luke Pickett
On-Screen T9 Keyboard for Bob
CSE 218

2. Locked-in Syndrome

3. Motivation
Trends in Interest on LIS by countryTrends in Interest on LIS by news
headlines
Enable people affected by Locked-In Syndrome (LIS)
to interact with any touch based display.

4. Existing Solutions - Tobii Eye Tracker
Pros
● Navigate the web
● Communicate by words
● Skype, emails, music, etc.
Cons
● Expensive
● Complicated to use
● Bad tech support

5. Existing Solutions - EyeGaze Edge Talker
Pros
● Standard keyboard with horizontal
and vertical eye tracking
● Text to Speech
● Quick access to frequent phrases
Cons
● Difficult to use
● Bob’s use case
Image Courtesy : https://www.youtube.com/watch?v=lY22CZ7XP-4, www.eyegaze.com

6. Vision
1. Easy to use tool for
communication
2. Collect data to improve our
tool and foster research.

7. Outline
● Requirements
● Product Description
● Team
● Software Process
● Architecture
● System design and implementation
● Post Mortem

8. Requirement Methods
● Professor
● Articles on LIS
Two types of LIS
○ Classical
○ Total
We focus on classical LIS patients

9. Product Description
● Vertically T9 on-screen
keyboard
● Activate on fixation
● Blink gestures for
switching context
(keyboard - predictions)
● Wink gesture to send
keystrokes to application

10. Team Grads
Undergrads

11. Weekly team
meetings
Facebook group
for communication
Google Docs for
collaboration
Git(Bitbucket) + Eclipse

12. Software Process
● SCRUM
1. Product Owner
2. Scrum Masters
3. Development Team
● Sprints
1. Prototype (Madness
Demo)
2. UX design, bugfixes,
backlog.

13. Architecture
Hardware
● EyeTribe eye tracker
● Microsoft Surface Pro 3
Software
● Java
● HTML/CSS/JS
● Python/mongoDB
Microsoft Surface Internals
EyeTribe
server
eyeTalk
application
backend
eyeTalk
application
frontend
1
2
3

14. System
Four main components :
1. eyeTalk Backend
2. eyeTalk UI
3. T9 and word prediction
4. eyeTalk analytics

15. 1. eyeTalk backend
eyeTalk Backend
Image Courtesy : Eyetribe Docs

16. 1. eyeTalk backend
● Written in Java
● GazeManager and IGazeListener to
communicate with EyeTribe server.
● Technical Challenge : Handle
saccades using running average
filter
● At any point in time, provides the
current smoothened value of X,Y
eyeTalk
Backend
eyeTalk UI
Get gaze data points in real-time (polling)

17. 2. eyeTalk UI
● Built using Processing graphics library
(Java)
● Design focus/constraint - Use only
vertical eye movement for control
● HCI design principles applied
● Dwell time (1.5 sec) based button
clicks - with progress bar animation
● T9 and Manual input mode supported

18. 2. eyeTalk UI
● Modal screen to select from multiple
word predictions
● Integrated TTS (CMU Sphinx)
● On-screen keyboard, works without
staying in focus
● Long blink to send keystrokes to
foreground application (notepad, email
client, etc.)
● Customisable UI

19. ● T9
● Word Completion
○ Higher weights for more
frequently used words
● Word Prediction
○ Learns commonly used phrases
from corpus
● Preprocessing for quick lookups
3. T9 and word prediction

21. 4. eyeTalk analytics
● Written in Python, JS.
DB : mongoDB
● Technical Challenge : Real-time
analytics using map-reduce to calculate
heatmap on demand.
● Future:
○ Streaming API support.
○ Real-time rendering using
websockets.
eyeTalk
Backend
Push data points
map-reduce queries

22. How they all fit together
eyeTalk
Backend
eyeTalk UI
Get gaze data points in real-time (polling)
Get predictionsT9 and
word prediction
algorithm
eyeTalk
analytics

23. Post-mortem
● Lessons learnt
○ Time management with part-time developers.
○ Precision while assigning tasks.
○ Identifying skillsets and exercising comparative advantage.
● What went right/wrong
○ Team split and task allocation.
○ Productive meetings.
○ Avoiding new tools for project management.
● Sub-teams by expertise vs. interest.
● Evaluation

24. Thanks!