Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Assistive Context-
Aware ToolkitFelipe Pedroso
#IntelNoTDC
Agenda
• Stephen Hawking and Intel
• What is ACAT?
• Technical Details
• Architecture
• Creating Extensions
• ...
#IntelNoTDC
• Professor Hawking first met Gordon
Moore during a conference in 1997,
where he was offered a computer
with a...
#IntelNoTDC
• Acronym for “Assistive Context-
Aware Toolkit”
• Enable people with disabilities to
communicate through keyb...
#IntelNoTDC
Open Source
http://01.org/ACAT
https://github.com/01org/acat
#IntelNoTDC
Tech Details
• Runs on Windows 7 or higher
• Code entirely written in C# using Visual 2012 (works fine on 2015...
#IntelNoTDC
Architecture
Applications
ACAT Core Library
ACAT Extension Library
Extensions
Extension Extension Extension ...
#IntelNoTDC
Applications
ACAT Qwerty ACAT Abc ACAT Tryout
Talk Window
#IntelNoTDC
ACAT Core Library - Components
Word
Prediction*
Management
UI Management
AnimationsPanels Widgets Themes
Text-...
#IntelNoTDC
ACAT Core Library - Agents
#IntelNoTDC
ACAT Core Library - Actuators
Actuator
Management
Image Sources: http://www.psfk.com/2013/02/intel-stephen-haw...
#IntelNoTDC
ACAT Extensions Library
• Allows developers to extend the functionalities from ACAT
• Dynamically loaded durin...
#IntelNoTDC
Creating an Actuator Extension
• Create a new “Class Library” project and add references to “ACATCore.dll”
and...
#IntelNoTDC
Intel® RealSense™
Intel® RealSense™ SDK
Scene
Understanding
Enhanced
Photography
Augmented
Reality
Person & Fa...
#IntelNoTDC
Tracking (Position)
Landmarks (78 landmarks)
Pose / Orientation
Expressions
Recognition
Pulse estimation
Intel...
#IntelNoTDC
• Objective: create an actuator driven
by facial expressions
• Components:
• RealSenseExpressionTracker:
RealS...
#IntelNoTDC
• The Camera actuator extension is a
socket client that keeps looking for
the “ACAT Vision” app, that has a
so...
#IntelNoTDC
• SharpSenses* is an open source C#
library that simplifies the
development of Intel® RealSense™
SDK apps.
• C...
#IntelNoTDC
• The idea is to create a ‘fake’ server,
that replicates the communication
between “ACAT Vision” and ACAT
• To...
#IntelNoTDC
Summary & Call 4 Action
• ACAT is a really nice option to allow people with disabilities to communicate
with o...
Q/A
#IntelNoTDC
References
• Stephen Hawking and Intel
• http://iq.intel.com/behind-scenes-intel-keeps-stephen-hawking-talking...
Thanks!
BAckup
#IntelNoTDC
ACAT Server
SharpSenses + ACAT
SharpSenses
bool canRun = true;
AcatServer acatServer = new AcatServer();
acatS...
Assistive Context-Aware Toolkit (English)
Upcoming SlideShare
Loading in …5
×

Assistive Context-Aware Toolkit (English)

584 views

Published on

Slides from my presentation during TDC Florianópolis 2016 on the .NET Track.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Assistive Context-Aware Toolkit (English)

  1. 1. Assistive Context- Aware ToolkitFelipe Pedroso
  2. 2. #IntelNoTDC Agenda • Stephen Hawking and Intel • What is ACAT? • Technical Details • Architecture • Creating Extensions • How to? • RealSense + ACAT • SharpSenses + ACAT • Summary and Call 4 Action
  3. 3. #IntelNoTDC • Professor Hawking first met Gordon Moore during a conference in 1997, where he was offered a computer with an Intel processor • Intel upgrades his computer every couple of years and also supports him all his hardware requirements • A cross-disciplinary team from Intel Labs developed ACAT to enable him to communicate and perform all his computer tasks more efficiently Stephen Hawking and Intel Images Sources: http://www.astrosurf.com/luxorion/hawking-hommage9.htm http://www.wired.com/2015/01/intel-gave-stephen-hawking-voice/
  4. 4. #IntelNoTDC • Acronym for “Assistive Context- Aware Toolkit” • Enable people with disabilities to communicate through keyboard simulation, word prediction and speech synthesis • Developed by Intel Labs together with Professor Hawking • Customizable and Extensible What is ACAT?
  5. 5. #IntelNoTDC Open Source http://01.org/ACAT https://github.com/01org/acat
  6. 6. #IntelNoTDC Tech Details • Runs on Windows 7 or higher • Code entirely written in C# using Visual 2012 (works fine on 2015) and Microsoft .NET Framework 4.5 • Microsoft Office Interop Assemblies are used to interact with Office apps • Some tools used to develop the project: • CodeMaid • StyleCop • ReSharper
  7. 7. #IntelNoTDC Architecture Applications ACAT Core Library ACAT Extension Library Extensions Extension Extension Extension ...
  8. 8. #IntelNoTDC Applications ACAT Qwerty ACAT Abc ACAT Tryout Talk Window
  9. 9. #IntelNoTDC ACAT Core Library - Components Word Prediction* Management UI Management AnimationsPanels Widgets Themes Text-to-Speech Management Talk Window Management *ACAT is powered by Presage, the intelligent predictive text engine created by Matteo Vescov. (http://presage.sourceforge.net)
  10. 10. #IntelNoTDC ACAT Core Library - Agents
  11. 11. #IntelNoTDC ACAT Core Library - Actuators Actuator Management Image Sources: http://www.psfk.com/2013/02/intel-stephen-hawking.html & https://commons.wikimedia.org/wiki/File:Assistive_device.JPG Actuators Proximity Sensor Assistive Switch Camera
  12. 12. #IntelNoTDC ACAT Extensions Library • Allows developers to extend the functionalities from ACAT • Dynamically loaded during startup using reflection • Needs to implement specific interfaces/base classes Extension Types UI Word Predictors Agents Actuators Text-to-Speech Spell Checkers
  13. 13. #IntelNoTDC Creating an Actuator Extension • Create a new “Class Library” project and add references to “ACATCore.dll” and “ACATExtension.dll” • Extend the classes “ActuatorBase” and “ActuatorSwitchBase” to interact with ACAT • Implement the abstract methods and the use the callbacks (Init, Resume, Pause, Dispose) to run the code of your extension (threads, config, etc) • Call the methods OnSwitchActivated (Hold/Press), OnSwitchDeactivated (Release) or OnSwitchTriggered(Click) to sinalize the interaction from the actuator • Deploy your DLL to the extension folder and update the configuration files
  14. 14. #IntelNoTDC Intel® RealSense™ Intel® RealSense™ SDK Scene Understanding Enhanced Photography Augmented Reality Person & Face Tracking Gesture Recognition Background Removal Object Recognition and Tracking Speech Recognition 3D Object Scanning Hand & Blob Tracking User Facing World Facing
  15. 15. #IntelNoTDC Tracking (Position) Landmarks (78 landmarks) Pose / Orientation Expressions Recognition Pulse estimation Intel® RealSense™ - Face Module
  16. 16. #IntelNoTDC • Objective: create an actuator driven by facial expressions • Components: • RealSenseExpressionTracker: RealSense SDK implementation to perceive the expressions and notify the Actuator using events • RealSenseActuator: translates the intensity of the expressions to a button behavior (pressed and released) Intel® RealSense™ - ACAT Plugin RealSense Actuator Extension ExpressionTracker Actuator Switch Actuator Management
  17. 17. #IntelNoTDC • The Camera actuator extension is a socket client that keeps looking for the “ACAT Vision” app, that has a socket server • Simple protocol and it’s possible to customize using config files • ACAT Vision is not yet open-source, but its code will be released in upcoming releases Another way to connect an actuator ACAT Vision (Server) Camera Actuator (Client) Actuator Management
  18. 18. #IntelNoTDC • SharpSenses* is an open source C# library that simplifies the development of Intel® RealSense™ SDK apps. • Created and maintained by André Carlucci (Intel Black Belt & Software Innovator) • Available on Github* and NuGet* SharpSenses*
  19. 19. #IntelNoTDC • The idea is to create a ‘fake’ server, that replicates the communication between “ACAT Vision” and ACAT • To notify that the input was triggered, just send the string “gesture=trigger;action=CT;conf=75;t ime=<TIMESTAMP>” • ACAT also sends some events back to the server (HideUI, ShowUI, ExitApp) SharpSenses* + ACAT ACAT Vision Camera Actuator (Client) Actuator Management ‘Fake’ Server SharpSenses
  20. 20. #IntelNoTDC Summary & Call 4 Action • ACAT is a really nice option to allow people with disabilities to communicate with others • Open Source: You can fork and extend ACAT for your needs! • Extensions: let’s help other people with new actuators, applications agents and new features  • RealSense and SharpSenses can also drive acessibility apps.
  21. 21. Q/A
  22. 22. #IntelNoTDC References • Stephen Hawking and Intel • http://iq.intel.com/behind-scenes-intel-keeps-stephen-hawking-talking/ • http://www.wired.com/2015/01/intel-gave-stephen-hawking-voice/ • ACAT • https://github.com/01org/acat • http://01.org/ACAT • RealSense: https://software.intel.com/en-us/realsense/home • SharpSenses: https://github.com/SharpSenses/SharpSenses
  23. 23. Thanks!
  24. 24. BAckup
  25. 25. #IntelNoTDC ACAT Server SharpSenses + ACAT SharpSenses bool canRun = true; AcatServer acatServer = new AcatServer(); acatServer.Start(); acatServer.AcatNotificationReceived += (sender, a) => { Console.WriteLine("Notification received from ACAT: {0}", a.NotificationType); if (a.NotificationType == AcatNotificationType.ExitApp) { canRun = false; } }; ICamera camera = Camera.Create(Capability.FaceTracking, Capability.FacialExpressionTracking); camera.Start(); camera.Face.LeftEye.Blink += (s,a) => { Console.WriteLine("SharpSenses: detected a blink from left eye."); acatServer.SendGestureMessage("CT", "trigger"); }; while (canRun) { } acatServer.Stop();

×