A typical interaction with a device or interface involves touching it. Either you're pressing buttons on a controller, swiping on a touchscreen or clicking on your laptop's trackpad. But what if you could control things without the use of your hands? What if you could use your thoughts instead? Charlie has evolved this interaction using a brain sensor and developed an open source JavaScript framework to allow users to control interfaces or devices using facial expressions and mental commands.

1. CHARLIE GERARD
Software Developer
@devdevcharlie #EvolutionTW #ThoughtWorks

2. Charlie Gerard / @devdevcharlie
Tuesday 5th June
INTERACTIVE TECHNOLOGIES | EPOC.JS | DEMOS | TECH STACK | SAMPLES | LIMITS | POSSIBILITIES | NEXT | RESOURCES
MIND CONTROL IN JAVASCRIPT

3. @devdevcharlieCharlie Gerard

4. @devdevcharlieCharlie Gerard
Explore alternative interactions

5. INTERACTIVE TECHNOLOGIES1

6. @devdevcharlie
INTERACTIVE TECHNOLOGIES
Charlie Gerard
Wearables
EVOLVING INTERACTIONS

7. @devdevcharlie
INTERACTIVE TECHNOLOGIES
Charlie Gerard
Wearables
EVOLVING INTERACTIONS
Voice

8. @devdevcharlie
INTERACTIVE TECHNOLOGIES
Charlie Gerard
Wearables
EVOLVING INTERACTIONS
Voice Gesture

9. @devdevcharlie
INTERACTIVE TECHNOLOGIES
Charlie Gerard
Wearables
EVOLVING INTERACTIONS
Voice Gesture
Computer
Vision

10. @devdevcharlie
INTERACTIVE TECHNOLOGIES
Charlie Gerard
Wearables
EVOLVING INTERACTIONS
Voice Gesture
Computer
Vision
Biofeedback

11. @devdevcharlie
INTERACTIVE TECHNOLOGIES
Charlie Gerard
BIOFEEDBACK
Gaining greater awareness of many physiological
functions primarily using instruments that provide
information on the activity of those same systems.

12. @devdevcharlie
INTERACTIVE TECHNOLOGIES
Charlie Gerard
BIOFEEDBACK
Emotiv Epoc

13. @devdevcharlie
INTERACTIVE TECHNOLOGIES
Charlie Gerard
Intent of subject
HOW DOES THE BRAIN WORK?
trigger
Body part Action
Signal

14. @devdevcharlie
INTERACTIVE TECHNOLOGIES
Charlie Gerard
HOW DOESABRAIN SENSOR WORK?

15. @devdevcharlie
INTERACTIVE TECHNOLOGIES
Charlie Gerard
EMOTIV EPOC
• SDK only available in C++ and Java

16. @devdevcharlie
INTERACTIVE TECHNOLOGIES
Charlie Gerard
EMOTIV EPOC
• SDK only available in C++ and Java
• Very small community

17. @devdevcharlie
INTERACTIVE TECHNOLOGIES
Charlie Gerard
EMOTIV EPOC
• SDK only available in C++ and Java
• Very small community
• No resources

18. EPOC.JS
JavaScript framework to interact with the Emotiv Epoc
2

19. @devdevcharlie
EPOC.JS
Charlie Gerard
FEATURES
Interaction with the EmoComposer emulator.

20. @devdevcharlie
EPOC.JS
Charlie Gerard
FEATURES
Interaction with the EmoComposer emulator.
Access to live data:
• Hardware (gyroscope, accelerometer)

21. @devdevcharlie
EPOC.JS
Charlie Gerard
FEATURES
Interaction with the EmoComposer emulator.
Access to live data:
• Hardware (gyroscope, accelerometer)
• Performance metrics (focus, excitement, etc…)

22. @devdevcharlie
EPOC.JS
Charlie Gerard
FEATURES
Interaction with the EmoComposer emulator.
Access to live data:
• Hardware (gyroscope, accelerometer)
• Performance metrics (focus, excitement, etc…)
• Facial expressions (smile, blink, wink, etc…)

23. @devdevcharlie
EPOC.JS
Charlie Gerard
FEATURES
Interaction with the EmoComposer emulator.
Access to live data:
• Hardware (gyroscope, accelerometer)
• Performance metrics (focus, excitement, etc…)
• Facial expressions (smile, blink, wink, etc…)
• Mental commands (push, drop, lift, left, right, etc..)

24. @devdevcharlie
EPOC.JS
Charlie Gerard
TECH STACK
• C++ SDK
• Node.js
• Nan - Native abstractions for Node.js
• Node-gyp - Build tool for Node.js native addons
• Bindings - Helper module for loading native modules’ .node file

25. @devdevcharlie
EPOC.JS
Charlie Gerard
TECH STACK
N-API

26. DEMOS3

27. @devdevcharlie
DEMOS
Charlie Gerard
BRAIN KEYBOARD

28. @devdevcharlie
DEMOS
Charlie Gerard
WEBVR

29. CODE SAMPLES4

30. @devdevcharlie
CODE SAMPLES
Charlie Gerard
EPOC.JS
# functions.cc
# ------------
void Connect(const v8::FunctionCallbackInfo<Value>& args) {
EmoStateHandle eState = IEE_EmoStateCreate();
int isBlinking = Is_FacialExpressionIsBlink(eState);
v8::Local<v8::Object> event = Nan::New<v8::Object>();
Nan::Set(event, Nan::New(“blink”).ToLocalChecked(), Nan::New(isBlinking));
}
void Init(v8::Local<v8::Object> exports) {
exports->Set(Nan::New(“connectToEmoComposer").ToLocalChecked(),
Nan::New<v8::FunctionTemplate>(Connect)->GetFunction());
}
NODE_MODULE(epoc, Init);

31. @devdevcharlie
CODE SAMPLES
Charlie Gerard
EPOC.JS
# functions.cc
# ------------
void Connect(const v8::FunctionCallbackInfo<Value>& args) {
EmoStateHandle eState = IEE_EmoStateCreate();
int isBlinking = Is_FacialExpressionIsBlink(eState);
v8::Local<v8::Object> event = Nan::New<v8::Object>();
Nan::Set(event, Nan::New(“blink”).ToLocalChecked(), Nan::New(isBlinking));
}
void Init(v8::Local<v8::Object> exports) {
exports->Set(Nan::New("connect").ToLocalChecked(),
Nan::New<v8::FunctionTemplate>(Connect)->GetFunction());
}
NODE_MODULE(epoc, Init);

32. @devdevcharlie
CODE SAMPLES
Charlie Gerard
EPOC.JS
# functions.cc
# ------------
void Connect(const v8::FunctionCallbackInfo<Value>& args) {
EmoStateHandle eState = IEE_EmoStateCreate();
int isBlinking = Is_FacialExpressionIsBlink(eState);
v8::Local<v8::Object> event = Nan::New<v8::Object>();
Nan::Set(event, Nan::New(“blink”).ToLocalChecked(), Nan::New(isBlinking));
}
void Init(v8::Local<v8::Object> exports) {
exports->Set(Nan::New("connect").ToLocalChecked(),
Nan::New<v8::FunctionTemplate>(Connect)->GetFunction());
}
NODE_MODULE(epoc, Init);

33. @devdevcharlie
CODE SAMPLES
Charlie Gerard
EPOC.JS
# functions.cc
# ------------
void Connect(const v8::FunctionCallbackInfo<Value>& args) {
EmoStateHandle eState = IEE_EmoStateCreate();
int isBlinking = Is_FacialExpressionIsBlink(eState);
v8::Local<v8::Object> event = Nan::New<v8::Object>();
Nan::Set(event, Nan::New(“blink”).ToLocalChecked(), Nan::New(isBlinking));
}
void Init(v8::Local<v8::Object> exports) {
exports->Set(Nan::New("connect").ToLocalChecked(),
Nan::New<v8::FunctionTemplate>(Connect)->GetFunction());
}
NODE_MODULE(epoc, Init);

34. @devdevcharlie
CODE SAMPLES
Charlie Gerard
EPOC.JS
# functions.cc
# ------------
void Connect(const v8::FunctionCallbackInfo<Value>& args) {
EmoStateHandle eState = IEE_EmoStateCreate();
int isBlinking = Is_FacialExpressionIsBlink(eState);
v8::Local<v8::Object> event = Nan::New<v8::Object>();
Nan::Set(event, Nan::New(“blink”).ToLocalChecked(), Nan::New(isBlinking));
}
void Init(v8::Local<v8::Object> exports) {
exports->Set(Nan::New("connect").ToLocalChecked(),
Nan::New<v8::FunctionTemplate>(Connect)->GetFunction());
}
NODE_MODULE(epoc, Init);

35. @devdevcharlie
CODE SAMPLES
Charlie Gerard
EPOC.JS
# functions.cc
# ------------
void Connect(const v8::FunctionCallbackInfo<Value>& args) {
EmoStateHandle eState = IEE_EmoStateCreate();
int isBlinking = Is_FacialExpressionIsBlink(eState);
v8::Local<v8::Object> event = Nan::New<v8::Object>();
Nan::Set(event, Nan::New(“blink”).ToLocalChecked(), Nan::New(isBlinking));
}
void Init(v8::Local<v8::Object> exports) {
exports->Set(Nan::New("connect").ToLocalChecked(),
Nan::New<v8::FunctionTemplate>(Connect)->GetFunction());
}
NODE_MODULE(epoc, Init);

36. @devdevcharlie
CODE SAMPLES
Charlie Gerard
EPOC.JS
# functions.cc
# ------------
void Connect(const v8::FunctionCallbackInfo<Value>& args) {
EmoStateHandle eState = IEE_EmoStateCreate();
int isBlinking = Is_FacialExpressionIsBlink(eState);
v8::Local<v8::Object> event = Nan::New<v8::Object>();
Nan::Set(event, Nan::New(“blink”).ToLocalChecked(), Nan::New(isBlinking));
}
void Init(v8::Local<v8::Object> exports) {
exports->Set(Nan::New("connect").ToLocalChecked(),
Nan::New<v8::FunctionTemplate>(Connect)->GetFunction());
}
NODE_MODULE(epoc, Init);

37. @devdevcharlie
CODE SAMPLES
Charlie Gerard
EPOC.JS
# binding.gyp
# ------------
{
"targets": [
{
"target_name": “epoc",
"sources": [ "functions.cc"]
}
]
}
node-gyp build

38. @devdevcharlie
CODE SAMPLES
Charlie Gerard
EPOC.JS
# main.js
# ------------
const epoc = require('bindings')('epoc');
epoc.connectToEmoComposer(function(event){
console.log(event);
});
epoc.connectToLiveData(“path/to/profile/file”, function(event){
console.log(event);
});
node main.js // event: {blink: 0}

39. @devdevcharlie
CODE SAMPLES
Charlie Gerard
EPOC.JS
# main.js
# ------------
const epoc = require('bindings')('epoc');
epoc.connectToEmoComposer(function(event){
console.log(event);
});
epoc.connectToLiveData(“path/to/profile/file”, function(event){
console.log(event);
});
node main.js // event: {blink: 0}

40. @devdevcharlie
CODE SAMPLES
Charlie Gerard
EPOC.JS
# main.js
# ------------
const epoc = require('bindings')('epoc');
epoc.connectToEmoComposer(function(event){
console.log(event);
});
epoc.connectToLiveData(“path/to/profile/file”, function(event){
console.log(event);
});
node main.js // event: {blink: 0}

41. LIMITS5

42. @devdevcharlie
LIMITS
Charlie Gerard
TECHNICAL
• Needs training for each user.

43. @devdevcharlie
LIMITS
Charlie Gerard
TECHNICAL
• Needs training for each user.
• Delay.

44. @devdevcharlie
LIMITS
Charlie Gerard
TECHNICAL
• Needs training for each user.
• Delay.
• Few mental commands available.

45. @devdevcharlie
LIMITS
Charlie Gerard
USER EXPERIENCE
• We want seamless interactions.

46. @devdevcharlie
LIMITS
Charlie Gerard
USER EXPERIENCE
• We want seamless interactions.
• No trust in technology as soon as it fails us.

47. @devdevcharlie
LIMITS
Charlie Gerard
USER EXPERIENCE
• We want seamless interactions.
• No trust in technology as soon as it fails us.
• Real value.

48. @devdevcharlieCharlie Gerard
We haven’t figured out
people.

49. @devdevcharlie
EXAMPLE
Charlie Gerard
MIT PROJECT
TED talk - Meet the SixthSense interaction

50. POSSIBILITIES6

51. @devdevcharlieCharlie Gerard
Augmentation, not replacement

52. @devdevcharlie
POSSIBILITIES
Charlie Gerard
AUGMENTATION
• Accessibility

53. @devdevcharlie
POSSIBILITIES
Charlie Gerard
AUGMENTATION
• Accessibility
• Mental health

54. @devdevcharlie
POSSIBILITIES
Charlie Gerard
AUGMENTATION
• Accessibility
• Mental health
• Art

55. @devdevcharlie
POSSIBILITIES
Charlie Gerard
ART

56. @devdevcharlieCharlie Gerard
Useless is not worthless

57. NEXT STEPS7

58. @devdevcharlieCharlie Gerard
NEXT STEPS
• Web bluetooth or Web USB.

59. @devdevcharlieCharlie Gerard
NEXT STEPS
• Web bluetooth or Web USB.
• Access to raw EEG data.

60. @devdevcharlieCharlie Gerard
NEXT STEPS
• Web bluetooth or Web USB.
• Access to raw EEG data.
• Try other device:
• OpenBCI
• Enobio

61. @devdevcharlieCharlie Gerard
OTHER DEVICES
AlterEgo

62. YOU?8

63. @devdevcharlieCharlie Gerard
WHAT CAN YOU DO?
• Ask yourself “what if”?

64. @devdevcharlieCharlie Gerard
WHAT CAN YOU DO?
• Ask yourself “what if”?
• Forget about value

65. @devdevcharlieCharlie Gerard
WHAT CAN YOU DO?
• Ask yourself “what if”?
• Forget about value
• Experiment

66. @devdevcharlieCharlie Gerard
WHAT CAN YOU DO?
• Ask yourself “what if”?
• Forget about value
• Experiment
• Have fun

67. RESOURCES9

68. @devdevcharlieCharlie Gerard
RESOURCES
Node-gyp
Nan
Bindings
Node.js addons
Emotiv epoc
Github repo epoc.js
NeuroTechX - Slack channel
OpenBCI
Enobio
TED Talk - Meet the SixthSense interaction
MIT Project AlterEgo

69. RECAP10

70. @devdevcharlieCharlie Gerard
RECAP
• Tech is already there

71. @devdevcharlieCharlie Gerard
RECAP
• Tech is already there
• Useless is not worthless

72. @devdevcharlieCharlie Gerard
RECAP
• Tech is already there
• Useless is not worthless
• Don’t be afraid to experiment - you can do this!

73. THANK YOU!
Charlie Gerard
@devdevcharlie