The document discusses Assistive Context-Aware Toolkit (ACAT), which was developed by Intel Labs to help Stephen Hawking communicate more efficiently using his computer. ACAT allows customizable extensions to enable communication through keyboard simulation, word prediction, and speech synthesis. It has an open architecture that allows extensions to add new features. Examples discussed include using Intel RealSense cameras and facial expression tracking, as well as a fake server, to create new input methods for ACAT. The document encourages developing new extensions to help more people with disabilities.

1. Assistive Context-
Aware ToolkitFelipe Pedroso

2. #IntelNoTDC
Agenda
• Stephen Hawking and Intel
• What is ACAT?
• Technical Details
• Architecture
• Creating Extensions
• How to?
• RealSense + ACAT
• SharpSenses + ACAT
• Summary and Call 4 Action

3. #IntelNoTDC
• Professor Hawking first met Gordon
Moore during a conference in 1997,
where he was offered a computer
with an Intel processor
• Intel upgrades his computer every
couple of years and also supports
him all his hardware requirements
• A cross-disciplinary team from Intel
Labs developed ACAT to enable him
to communicate and perform all his
computer tasks more efficiently
Stephen Hawking and Intel
Images Sources:
http://www.astrosurf.com/luxorion/hawking-hommage9.htm
http://www.wired.com/2015/01/intel-gave-stephen-hawking-voice/

4. #IntelNoTDC
• Acronym for “Assistive Context-
Aware Toolkit”
• Enable people with disabilities to
communicate through keyboard
simulation, word prediction and
speech synthesis
• Developed by Intel Labs together
with Professor Hawking
• Customizable and Extensible
What is ACAT?

5. #IntelNoTDC
Open Source
http://01.org/ACAT
https://github.com/01org/acat

6. #IntelNoTDC
Tech Details
• Runs on Windows 7 or higher
• Code entirely written in C# using Visual 2012 (works fine on 2015) and
Microsoft .NET Framework 4.5
• Microsoft Office Interop Assemblies are used to interact with Office apps
• Some tools used to develop the project:
• CodeMaid
• StyleCop
• ReSharper

7. #IntelNoTDC
Architecture
Applications
ACAT Core Library
ACAT Extension Library
Extensions
Extension Extension Extension ...

8. #IntelNoTDC
Applications
ACAT Qwerty ACAT Abc ACAT Tryout
Talk Window

9. #IntelNoTDC
ACAT Core Library - Components
Word
Prediction*
Management
UI Management
AnimationsPanels Widgets Themes
Text-to-Speech
Management
Talk Window
Management
*ACAT is powered by Presage, the intelligent predictive text engine created by Matteo Vescov. (http://presage.sourceforge.net)

10. #IntelNoTDC
ACAT Core Library - Agents

11. #IntelNoTDC
ACAT Core Library - Actuators
Actuator
Management
Image Sources: http://www.psfk.com/2013/02/intel-stephen-hawking.html & https://commons.wikimedia.org/wiki/File:Assistive_device.JPG
Actuators
Proximity Sensor Assistive Switch Camera

12. #IntelNoTDC
ACAT Extensions Library
• Allows developers to extend the functionalities from ACAT
• Dynamically loaded during startup using reflection
• Needs to implement specific interfaces/base classes
Extension Types
UI
Word Predictors
Agents
Actuators
Text-to-Speech Spell Checkers

13. #IntelNoTDC
Creating an Actuator Extension
• Create a new “Class Library” project and add references to “ACATCore.dll”
and “ACATExtension.dll”
• Extend the classes “ActuatorBase” and “ActuatorSwitchBase” to interact with
ACAT
• Implement the abstract methods and the use the callbacks (Init, Resume,
Pause, Dispose) to run the code of your extension (threads, config, etc)
• Call the methods OnSwitchActivated (Hold/Press), OnSwitchDeactivated
(Release) or OnSwitchTriggered(Click) to sinalize the interaction from the
actuator
• Deploy your DLL to the extension folder and update the configuration files

14. #IntelNoTDC
Intel® RealSense™
Intel® RealSense™ SDK
Scene
Understanding
Enhanced
Photography
Augmented
Reality
Person & Face
Tracking
Gesture
Recognition
Background
Removal
Object
Recognition
and Tracking
Speech
Recognition
3D Object
Scanning
Hand & Blob
Tracking
User Facing World Facing

15. #IntelNoTDC
Tracking (Position)
Landmarks (78 landmarks)
Pose / Orientation
Expressions
Recognition
Pulse estimation
Intel® RealSense™ - Face Module

16. #IntelNoTDC
• Objective: create an actuator driven
by facial expressions
• Components:
• RealSenseExpressionTracker:
RealSense SDK implementation to
perceive the expressions and notify the
Actuator using events
• RealSenseActuator: translates the
intensity of the expressions to a button
behavior (pressed and released)
Intel® RealSense™ - ACAT Plugin
RealSense Actuator Extension
ExpressionTracker
Actuator Switch
Actuator
Management

17. #IntelNoTDC
• The Camera actuator extension is a
socket client that keeps looking for
the “ACAT Vision” app, that has a
socket server
• Simple protocol and it’s possible to
customize using config files
• ACAT Vision is not yet open-source,
but its code will be released in
upcoming releases
Another way to connect an actuator
ACAT Vision (Server)
Camera Actuator (Client)
Actuator
Management

18. #IntelNoTDC
• SharpSenses* is an open source C#
library that simplifies the
development of Intel® RealSense™
SDK apps.
• Created and maintained by André
Carlucci (Intel Black Belt & Software
Innovator)
• Available on Github* and NuGet*
SharpSenses*

19. #IntelNoTDC
• The idea is to create a ‘fake’ server,
that replicates the communication
between “ACAT Vision” and ACAT
• To notify that the input was triggered,
just send the string
“gesture=trigger;action=CT;conf=75;t
ime=<TIMESTAMP>”
• ACAT also sends some events back
to the server (HideUI, ShowUI,
ExitApp)
SharpSenses* + ACAT
ACAT Vision
Camera Actuator (Client)
Actuator
Management
‘Fake’ Server
SharpSenses

20. #IntelNoTDC
Summary & Call 4 Action
• ACAT is a really nice option to allow people with disabilities to communicate
with others
• Open Source: You can fork and extend ACAT for your needs!
• Extensions: let’s help other people with new actuators, applications agents and
new features 
• RealSense and SharpSenses can also drive acessibility apps.

21. Q/A

22. #IntelNoTDC
References
• Stephen Hawking and Intel
• http://iq.intel.com/behind-scenes-intel-keeps-stephen-hawking-talking/
• http://www.wired.com/2015/01/intel-gave-stephen-hawking-voice/
• ACAT
• https://github.com/01org/acat
• http://01.org/ACAT
• RealSense: https://software.intel.com/en-us/realsense/home
• SharpSenses: https://github.com/SharpSenses/SharpSenses

23. Thanks!

24. BAckup

25. #IntelNoTDC
ACAT Server
SharpSenses + ACAT
SharpSenses
bool canRun = true;
AcatServer acatServer = new AcatServer();
acatServer.Start();
acatServer.AcatNotificationReceived += (sender, a) => {
Console.WriteLine("Notification received from ACAT: {0}", a.NotificationType);
if (a.NotificationType == AcatNotificationType.ExitApp)
{
canRun = false;
}
};
ICamera camera = Camera.Create(Capability.FaceTracking, Capability.FacialExpressionTracking);
camera.Start();
camera.Face.LeftEye.Blink += (s,a) => {
Console.WriteLine("SharpSenses: detected a blink from left eye.");
acatServer.SendGestureMessage("CT", "trigger");
};
while (canRun) { }
acatServer.Stop();