Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Charlie Gerard- Mind Control in Javascript (Evolution)

77 views

Published on

A typical interaction with a device or interface involves touching it. Either you're pressing buttons on a controller, swiping on a touchscreen or clicking on your laptop's trackpad. But what if you could control things without the use of your hands? What if you could use your thoughts instead?

Charlie has evolved this interaction using a brain sensor and developed an open source JavaScript framework to allow users to control interfaces or devices using facial expressions and mental commands.

Published in: Technology
  • Be the first to comment

Charlie Gerard- Mind Control in Javascript (Evolution)

  1. 1. CHARLIE GERARD Software Developer @devdevcharlie #EvolutionTW #ThoughtWorks
  2. 2. Charlie Gerard / @devdevcharlie Tuesday 5th June INTERACTIVE TECHNOLOGIES | EPOC.JS | DEMOS | TECH STACK | SAMPLES | LIMITS | POSSIBILITIES | NEXT | RESOURCES MIND CONTROL IN JAVASCRIPT
  3. 3. @devdevcharlieCharlie Gerard
  4. 4. @devdevcharlieCharlie Gerard Explore alternative interactions
  5. 5. INTERACTIVE TECHNOLOGIES1
  6. 6. @devdevcharlie INTERACTIVE TECHNOLOGIES Charlie Gerard Wearables EVOLVING INTERACTIONS
  7. 7. @devdevcharlie INTERACTIVE TECHNOLOGIES Charlie Gerard Wearables EVOLVING INTERACTIONS Voice
  8. 8. @devdevcharlie INTERACTIVE TECHNOLOGIES Charlie Gerard Wearables EVOLVING INTERACTIONS Voice Gesture
  9. 9. @devdevcharlie INTERACTIVE TECHNOLOGIES Charlie Gerard Wearables EVOLVING INTERACTIONS Voice Gesture Computer Vision
  10. 10. @devdevcharlie INTERACTIVE TECHNOLOGIES Charlie Gerard Wearables EVOLVING INTERACTIONS Voice Gesture Computer Vision Biofeedback
  11. 11. @devdevcharlie INTERACTIVE TECHNOLOGIES Charlie Gerard BIOFEEDBACK Gaining greater awareness of many physiological functions primarily using instruments that provide information on the activity of those same systems.
  12. 12. @devdevcharlie INTERACTIVE TECHNOLOGIES Charlie Gerard BIOFEEDBACK Emotiv Epoc
  13. 13. @devdevcharlie INTERACTIVE TECHNOLOGIES Charlie Gerard Intent of subject HOW DOES THE BRAIN WORK? trigger Body part Action Signal
  14. 14. @devdevcharlie INTERACTIVE TECHNOLOGIES Charlie Gerard HOW DOESABRAIN SENSOR WORK?
  15. 15. @devdevcharlie INTERACTIVE TECHNOLOGIES Charlie Gerard EMOTIV EPOC • SDK only available in C++ and Java
  16. 16. @devdevcharlie INTERACTIVE TECHNOLOGIES Charlie Gerard EMOTIV EPOC • SDK only available in C++ and Java • Very small community
  17. 17. @devdevcharlie INTERACTIVE TECHNOLOGIES Charlie Gerard EMOTIV EPOC • SDK only available in C++ and Java • Very small community • No resources
  18. 18. EPOC.JS JavaScript framework to interact with the Emotiv Epoc 2
  19. 19. @devdevcharlie EPOC.JS Charlie Gerard FEATURES Interaction with the EmoComposer emulator.
  20. 20. @devdevcharlie EPOC.JS Charlie Gerard FEATURES Interaction with the EmoComposer emulator. Access to live data: • Hardware (gyroscope, accelerometer)
  21. 21. @devdevcharlie EPOC.JS Charlie Gerard FEATURES Interaction with the EmoComposer emulator. Access to live data: • Hardware (gyroscope, accelerometer) • Performance metrics (focus, excitement, etc…)
  22. 22. @devdevcharlie EPOC.JS Charlie Gerard FEATURES Interaction with the EmoComposer emulator. Access to live data: • Hardware (gyroscope, accelerometer) • Performance metrics (focus, excitement, etc…) • Facial expressions (smile, blink, wink, etc…)
  23. 23. @devdevcharlie EPOC.JS Charlie Gerard FEATURES Interaction with the EmoComposer emulator. Access to live data: • Hardware (gyroscope, accelerometer) • Performance metrics (focus, excitement, etc…) • Facial expressions (smile, blink, wink, etc…) • Mental commands (push, drop, lift, left, right, etc..)
  24. 24. @devdevcharlie EPOC.JS Charlie Gerard TECH STACK • C++ SDK
 • Node.js
 • Nan - Native abstractions for Node.js
 • Node-gyp - Build tool for Node.js native addons
 • Bindings - Helper module for loading native modules’ .node file

  25. 25. @devdevcharlie EPOC.JS Charlie Gerard TECH STACK N-API
  26. 26. DEMOS3
  27. 27. @devdevcharlie DEMOS Charlie Gerard BRAIN KEYBOARD
  28. 28. @devdevcharlie DEMOS Charlie Gerard WEBVR
  29. 29. CODE SAMPLES4
  30. 30. @devdevcharlie CODE SAMPLES Charlie Gerard EPOC.JS # functions.cc
 # ------------
 void Connect(const v8::FunctionCallbackInfo<Value>& args) {
 EmoStateHandle eState = IEE_EmoStateCreate();
 
 int isBlinking = Is_FacialExpressionIsBlink(eState);
 
 v8::Local<v8::Object> event = Nan::New<v8::Object>();
 
 Nan::Set(event, Nan::New(“blink”).ToLocalChecked(), Nan::New(isBlinking));
 }
 
 void Init(v8::Local<v8::Object> exports) {
 exports->Set(Nan::New(“connectToEmoComposer").ToLocalChecked(),
 Nan::New<v8::FunctionTemplate>(Connect)->GetFunction());
 }
 
 NODE_MODULE(epoc, Init);
  31. 31. @devdevcharlie CODE SAMPLES Charlie Gerard EPOC.JS # functions.cc
 # ------------
 void Connect(const v8::FunctionCallbackInfo<Value>& args) {
 EmoStateHandle eState = IEE_EmoStateCreate();
 
 int isBlinking = Is_FacialExpressionIsBlink(eState);
 
 v8::Local<v8::Object> event = Nan::New<v8::Object>();
 
 Nan::Set(event, Nan::New(“blink”).ToLocalChecked(), Nan::New(isBlinking));
 }
 
 void Init(v8::Local<v8::Object> exports) {
 exports->Set(Nan::New("connect").ToLocalChecked(),
 Nan::New<v8::FunctionTemplate>(Connect)->GetFunction());
 }
 
 NODE_MODULE(epoc, Init);
  32. 32. @devdevcharlie CODE SAMPLES Charlie Gerard EPOC.JS # functions.cc
 # ------------
 void Connect(const v8::FunctionCallbackInfo<Value>& args) {
 EmoStateHandle eState = IEE_EmoStateCreate();
 
 int isBlinking = Is_FacialExpressionIsBlink(eState);
 
 v8::Local<v8::Object> event = Nan::New<v8::Object>();
 
 Nan::Set(event, Nan::New(“blink”).ToLocalChecked(), Nan::New(isBlinking));
 }
 
 void Init(v8::Local<v8::Object> exports) {
 exports->Set(Nan::New("connect").ToLocalChecked(),
 Nan::New<v8::FunctionTemplate>(Connect)->GetFunction());
 }
 
 NODE_MODULE(epoc, Init);
  33. 33. @devdevcharlie CODE SAMPLES Charlie Gerard EPOC.JS # functions.cc
 # ------------
 void Connect(const v8::FunctionCallbackInfo<Value>& args) {
 EmoStateHandle eState = IEE_EmoStateCreate();
 
 int isBlinking = Is_FacialExpressionIsBlink(eState);
 
 v8::Local<v8::Object> event = Nan::New<v8::Object>();
 
 Nan::Set(event, Nan::New(“blink”).ToLocalChecked(), Nan::New(isBlinking));
 }
 
 void Init(v8::Local<v8::Object> exports) {
 exports->Set(Nan::New("connect").ToLocalChecked(),
 Nan::New<v8::FunctionTemplate>(Connect)->GetFunction());
 }
 
 NODE_MODULE(epoc, Init);
  34. 34. @devdevcharlie CODE SAMPLES Charlie Gerard EPOC.JS # functions.cc
 # ------------
 void Connect(const v8::FunctionCallbackInfo<Value>& args) {
 EmoStateHandle eState = IEE_EmoStateCreate();
 
 int isBlinking = Is_FacialExpressionIsBlink(eState);
 
 v8::Local<v8::Object> event = Nan::New<v8::Object>();
 
 Nan::Set(event, Nan::New(“blink”).ToLocalChecked(), Nan::New(isBlinking));
 }
 
 void Init(v8::Local<v8::Object> exports) {
 exports->Set(Nan::New("connect").ToLocalChecked(),
 Nan::New<v8::FunctionTemplate>(Connect)->GetFunction());
 }
 
 NODE_MODULE(epoc, Init);
  35. 35. @devdevcharlie CODE SAMPLES Charlie Gerard EPOC.JS # functions.cc
 # ------------
 void Connect(const v8::FunctionCallbackInfo<Value>& args) {
 EmoStateHandle eState = IEE_EmoStateCreate();
 
 int isBlinking = Is_FacialExpressionIsBlink(eState);
 
 v8::Local<v8::Object> event = Nan::New<v8::Object>();
 
 Nan::Set(event, Nan::New(“blink”).ToLocalChecked(), Nan::New(isBlinking));
 }
 
 void Init(v8::Local<v8::Object> exports) {
 exports->Set(Nan::New("connect").ToLocalChecked(),
 Nan::New<v8::FunctionTemplate>(Connect)->GetFunction());
 }
 
 NODE_MODULE(epoc, Init);
  36. 36. @devdevcharlie CODE SAMPLES Charlie Gerard EPOC.JS # functions.cc
 # ------------
 void Connect(const v8::FunctionCallbackInfo<Value>& args) {
 EmoStateHandle eState = IEE_EmoStateCreate();
 
 int isBlinking = Is_FacialExpressionIsBlink(eState);
 
 v8::Local<v8::Object> event = Nan::New<v8::Object>();
 
 Nan::Set(event, Nan::New(“blink”).ToLocalChecked(), Nan::New(isBlinking));
 }
 
 void Init(v8::Local<v8::Object> exports) {
 exports->Set(Nan::New("connect").ToLocalChecked(),
 Nan::New<v8::FunctionTemplate>(Connect)->GetFunction());
 }
 
 NODE_MODULE(epoc, Init);
  37. 37. @devdevcharlie CODE SAMPLES Charlie Gerard EPOC.JS # binding.gyp
 # ------------ 
 {
 "targets": [
 {
 "target_name": “epoc",
 "sources": [ "functions.cc"]
 }
 ]
 } node-gyp build
  38. 38. @devdevcharlie CODE SAMPLES Charlie Gerard EPOC.JS # main.js
 # ------------ 
 const epoc = require('bindings')('epoc');
 
 epoc.connectToEmoComposer(function(event){
 console.log(event);
 }); epoc.connectToLiveData(“path/to/profile/file”, function(event){
 console.log(event);
 }); node main.js // event: {blink: 0}
  39. 39. @devdevcharlie CODE SAMPLES Charlie Gerard EPOC.JS # main.js
 # ------------ 
 const epoc = require('bindings')('epoc');
 
 epoc.connectToEmoComposer(function(event){
 console.log(event);
 }); epoc.connectToLiveData(“path/to/profile/file”, function(event){
 console.log(event);
 }); node main.js // event: {blink: 0}
  40. 40. @devdevcharlie CODE SAMPLES Charlie Gerard EPOC.JS # main.js
 # ------------ 
 const epoc = require('bindings')('epoc');
 
 epoc.connectToEmoComposer(function(event){
 console.log(event);
 }); epoc.connectToLiveData(“path/to/profile/file”, function(event){
 console.log(event);
 }); node main.js // event: {blink: 0}
  41. 41. LIMITS5
  42. 42. @devdevcharlie LIMITS Charlie Gerard TECHNICAL • Needs training for each user.
 

  43. 43. @devdevcharlie LIMITS Charlie Gerard TECHNICAL • Needs training for each user.
 • Delay.

  44. 44. @devdevcharlie LIMITS Charlie Gerard TECHNICAL • Needs training for each user.
 • Delay.
 • Few mental commands available.
  45. 45. @devdevcharlie LIMITS Charlie Gerard USER EXPERIENCE • We want seamless interactions.
  46. 46. @devdevcharlie LIMITS Charlie Gerard USER EXPERIENCE • We want seamless interactions. • No trust in technology as soon as it fails us.
  47. 47. @devdevcharlie LIMITS Charlie Gerard USER EXPERIENCE • We want seamless interactions. • No trust in technology as soon as it fails us. • Real value.
  48. 48. @devdevcharlieCharlie Gerard We haven’t figured out people.
  49. 49. @devdevcharlie EXAMPLE Charlie Gerard MIT PROJECT TED talk - Meet the SixthSense interaction
  50. 50. POSSIBILITIES6
  51. 51. @devdevcharlieCharlie Gerard Augmentation, not replacement
  52. 52. @devdevcharlie POSSIBILITIES Charlie Gerard AUGMENTATION • Accessibility 
 

  53. 53. @devdevcharlie POSSIBILITIES Charlie Gerard AUGMENTATION • Accessibility 
 • Mental health 

  54. 54. @devdevcharlie POSSIBILITIES Charlie Gerard AUGMENTATION • Accessibility 
 • Mental health 
 • Art
  55. 55. @devdevcharlie POSSIBILITIES Charlie Gerard ART
  56. 56. @devdevcharlieCharlie Gerard Useless is not worthless
  57. 57. NEXT STEPS7
  58. 58. @devdevcharlieCharlie Gerard NEXT STEPS • Web bluetooth or Web USB.
 

  59. 59. @devdevcharlieCharlie Gerard NEXT STEPS • Web bluetooth or Web USB.
 • Access to raw EEG data.

  60. 60. @devdevcharlieCharlie Gerard NEXT STEPS • Web bluetooth or Web USB.
 • Access to raw EEG data.
 • Try other device: • OpenBCI • Enobio
  61. 61. @devdevcharlieCharlie Gerard OTHER DEVICES AlterEgo
 

  62. 62. YOU?8
  63. 63. @devdevcharlieCharlie Gerard WHAT CAN YOU DO? • Ask yourself “what if”?
  64. 64. @devdevcharlieCharlie Gerard WHAT CAN YOU DO? • Ask yourself “what if”? • Forget about value
  65. 65. @devdevcharlieCharlie Gerard WHAT CAN YOU DO? • Ask yourself “what if”? • Forget about value • Experiment
  66. 66. @devdevcharlieCharlie Gerard WHAT CAN YOU DO? • Ask yourself “what if”? • Forget about value • Experiment • Have fun
  67. 67. RESOURCES9
  68. 68. @devdevcharlieCharlie Gerard RESOURCES Node-gyp Nan Bindings Node.js addons Emotiv epoc Github repo epoc.js NeuroTechX - Slack channel OpenBCI Enobio TED Talk - Meet the SixthSense interaction MIT Project AlterEgo
  69. 69. RECAP10
  70. 70. @devdevcharlieCharlie Gerard RECAP • Tech is already there
  71. 71. @devdevcharlieCharlie Gerard RECAP • Tech is already there • Useless is not worthless
  72. 72. @devdevcharlieCharlie Gerard RECAP • Tech is already there • Useless is not worthless • Don’t be afraid to experiment - you can do this!
  73. 73. THANK YOU! Charlie Gerard @devdevcharlie

×