Mind Control to Major Tom: Is It Time to Put Your EEG Headset On?

@spoole167 @Luc_At_IBM
Mind Control to Major Tom:
Is It Time to Put Your EEG Headset On?
CON2338

Who we are
Steve Poole aka Obiwan
Java guru, Developer extraordinaire
Delivery lead
Luc Desrosiers aka Lobot (Think Star Wars Cloud city)
Canadian expat living in the UK
Gadget & IoT enthousiast… and Cloud architect

@spoole167 @Luc_At_IBMhttps://www.flickr.com/photos/stawarz/

Who we really are
Terminators from the future…

Here to promote our book series
“Skynet for Dummies”
Today we’ll be talking about the
basics for creating your own
terminator

Skynet will be built with the greatest and best tools
We picked Java because, even in your day it’s on
many, many devices
We will be using Java 8 - because even in the
future Java 9 hasn’t shipped yet

Outline
Ways to interact with your human
understanding your human
controlling your human

https://www.flickr.com/photos/gyduxa/
Ways to interact
with your human

Vision
AuditoryTouch
GustatoryOlfactory
Proprioception
Key input/output components of a
human being

Human being interaction map
Virtual Smell
Electronic noses
Wearables
Haptic sensors
Motion sensors
Molecular
analysis
Virtual Reality
Computer Vision
Text-to-Speech
Speech-to-Text
Touch and
proprioception
Gustatory Olfactory
Auditory Vision
Author: Allan-Hermann Pool

Convincing your human its in a new
world
Teaching your terminator
to understand the real world
Virtual Reality
Augmented Reality
World Domination

Goal is full immersive
sensory replacement.
Environment Awareness
Object and
Facial detection
IBM Watson
Visual Recognition

Virtual Reality, or the many challenges of fooling the
human beings
Images by Pdenbrook - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=31920588
From user interface
You want to reduce motion sickness?
Add a nose!
Pointman

Wait, where did my hands go?
frame.hands().forEach(hand->{
Arm arm = hand.arm();
if (arm.isValid()) {
renderArm(arm);
}
hand.fingers().forEach(finger->{
if (finger.isValid()) {
for(int i = 3; i >= 0; i--) {
Bone bone = finger.bone(Type.swigToEnum(i));
renderCylinder(bone.width(), bone.length(),
bone.prevJoint(), bone.direction());
}
}
});
});

Making the humans feel the virtual world
Sense of touch has been attempted using:
• Vibration
• Small pockets of air balloons
• Electrical impulse
• Inertial sensors
Many startup and currently many failures…

Recognizing objects in a scene
Completely possible… However it requires a lot of
training!
Complexity does not lie in the coding but in getting a
good set of positive and negative images:
- Positive images have the object you want to identify
- Negative images do not
Group positive images together and we call them class.
Group related class together and we call them classifier.

Demo: Visual Recognition with Watson
VisualRecognition service = new
VisualRecognition(VisualRecognition.VERSION_DATE_2016_05_19);
service.setApiKey("{api-key}");
System.out.println("Classify an image");
ClassifyImagesOptions options = new ClassifyImagesOptions.Builder().images(
new File(
"src/test/resources/visual_recognition/car.png")).build();
VisualClassification result = service.classify(options).execute();
System.out.println(result);

Hasta la vista Baby:
it’s not just about vision

Understanding what your human is
saying
Getting your human to
understand who’s the boss
Speech-to-Text Text-to-Speech(aka)
World Domination

Demo: Text to Speech Furbynator speaks
TextToSpeech service = new TextToSpeech();
String text = "Hello world.";
InputStream stream =
service.synthesize (text, Voice.EN_ALLISON, "audio/wav");
OutputStream out = new FileOutputStream("hello_world.wav");

Demo: Speech to Text
SpeechToText service = new SpeechToText();
RecognizeOptions options = new RecognizeOptions().contentType("audio/flac”)
.timestamps(true)
.wordAlternativesThreshold(0.9)
.continuous(true);
File file = new File("audio-file1.flac”);
SpeechResults results = service.recognize(file, options);
System.out.println(results);

Today Virtual
Reality
Computer
Vision
Hand
Tracking
Speech-
to- Text
Text-to-
Speech
Current Applicability 6/10 7/10 8/10 7/10 8/10
Java Coverage 4/10 8/10 9/10 9/10 9/10
Ease of use 4/10 6/10 9/10 8/10 8/10
Reliability 6/10 7/10 8/10 8/10 9/10
Future
Potential for improvement High High Medium Medium Medium
Scorecard

Understanding the
emotional state of
your Human
https://www.flickr.com/photos/pixietart/

Understanding emotion requires combining detection and analysis of
facial features, prosody and lexical content in speech
This is hard!

Recognizing facial features
Two approaches:
Take many pictures of humans in different moods and teach a neural net to do pattern
matching
Detect interesting face points (nose tips, mouth corners, eyes etc and determine
relationship between them.

OpenCV has good Face Detection
VideoCapture camera = new VideoCapture();
CascadeClassifier cc = new CascadeClassifier(
"haarcascade_frontalface_alt.xml");
MatOfRect faces = new MatOfRect();
Mat frame = new Mat();
camera.read(frame);
cc.detectMultiScale(frame, faces);
Rect[] hits=faces.toArray();
for(int i=0;i<hits.length;i++) {
Mat face = new Mat(frame,hits[i]);
Imgcodecs.imwrite("face"+i+".jpg", face);
}

Simple Face Recognition / Emotion detection is much harder
OpenCV has image training capability but it’s not available as a Java API
It’s computationally very expensive
You need a database of images that contain the items you want to detect
You need a database of images that do not contain the items you want to detect
OpenCV training requires creating images from the second set with items from the first set
added in at various scales and rotational angles
http://coding-robin.de/2013/07/22/train-your-own-opencv-haar-classifier.html

Detection alternatives:
Get humans to score a series of pictures of other humans suffering emotions
http://www.ipsp.ucl.ac.be/recherche/projets/FaceTales/en/Home.htm
Apply various point analysis techniques to extract key parts of a face
http://www.codeproject.com/Articles/110805/Human-Emotion-Detection-from-Image
https://github.com/mpillar/java-emotion-recognizer

Creating test images of the various emotions
Challenge:
By Crosa (Flickr: Scream) [CC BY 2.0 (http://creativecommons.org/licenses/by/2.0)], via Wikimedia Commons
https://www.flickr.com/photos/huphtur/
https://www.flickr.com/photos/auroredelsoir/
For some reason humans only ever scream?
https://www.flickr.com/photos/tenaciousme/
https://www.flickr.com/photos/morton/

Analysing emotion from text and speech
This is still hard. Terminators are not good with speech
I’ll be back
asta la vista....baby
Come with me if you want to liveYou’ve been terminated
"Get out."
I need your clothes, your boots and your motorcycle.
Stay here. I'll be back

ToneAnalyzer service = new ToneAnalyzer(
ToneAnalyzer.VERSION_DATE_2016_05_19);
ToneAnalysis tone = service.getTone(text, null).execute();
System.out.println(tone);
Watson Tone Analysis sample

Analysing tone with IBM Watson:
I need your clothes, your boots and your motorcycle.
I’ll be back.
Get out.
asta la vista....baby.
Stay here.
I'll be back.
You’ve been terminated.
Come with me if you want to live
Conscientiousness
Extraversion
Analytical

Analysing tone: the more words the better:
Using your mind to interact with computers is a long-standing desire. Advances in
technology have made it more practical, but is it ready for prime time? This
session presents practical examples and a walkthough of how to build a Java-
based end-to-end system to drive a remote-controlled droid with nothing but the
power of thought. Combining off-the-shelf EEG headsets with cloud technology
and IoT, the presenters showcase what capabilities exist today. Beyond mind
control (if there is such a concept), the session shows other ways to
communicate with your computer besides the keyboard. It will help you
understand the art of the possible and decide if it's time to leave the capsule to
communicate with your computer.

Analysing tone: the more words the better:

But, but, Terminators
can have a personality too!

Maybe, but Personality of Terminators
tends to be too predictable… Exterminate??
So lets look at the personality
of the father of Robotics
https://en.wikipedia.org/wiki/I,_Robot_(film)
Isaac Asimov

This analysis was based from an
extract of the text of iRobot
We could not contact Isaac
next of kin to validate Watson’s
findings…
but Wikipedia seems to agree:
https://en.wikipedia.org/wiki/Isaac_Asimov

Watson Personality Insight sample
PersonalityInsights service = new PersonalityInsights();
service.setUsernameAndPassword("{username}", "{password}");
try {
JsonReader jReader = new JsonReader(new FileReader("profile.json"));
Content content = GsonSingleton.getGson().fromJson(jReader, Content.class);
ProfileOptions options = new ProfileOptions()
.contentItems(content.getContentItems());
Profile profile = service.getProfile(options);
System.out.println(profile);
} catch (FileNotFoundException e) {
e.printStackTrace();
}

Today Tone
Analysis
Personality
Analysis
Current Applicability 8/10 8/10
Java Coverage 9/10 9/10
Ease of use 8/10 7/10
Reliability 8/10 7/10
Future
Potential Improvement Medium Medium
Scorecard

Ways to control with your human
https://www.flickr.com/photos/adforce1/

We tried inserting machines into humans.
Too messy and somewhat obvious.
https://www.flickr.com/photos/auspices/

We tried adding machines
outside humans.
Mixed success
ByCcmsharma2-Ownwork,CCBY-SA4.0,https://commons.wikimedia.org/w/index.php?curid=44920707

43IBM
_
In fact it is much simpler!!! Just hack
the brain!

Capturing brain activity (when applicable) of a
human
IOT Command
• Push
• Pull
• Stop
Bluetooth
BB8.roll
IOT Event
•Push
•Pull
•Neutral

Today Mind
Control
Current Applicability 3/10
Java Coverage 7/10
Ease of use 4/10
Reliability 2/10
Future
Potential Improvement High
Scorecard

So its clear that we’re not quite ready to take
over the world…

“we’ll be back”

Mind Control to Major Tom: Is It Time to Put Your EEG Headset On?

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (7)

Similar to Mind Control to Major Tom: Is It Time to Put Your EEG Headset On?

Similar to Mind Control to Major Tom: Is It Time to Put Your EEG Headset On? (20)

More from Steve Poole

More from Steve Poole (20)

Recently uploaded

Recently uploaded (20)

Mind Control to Major Tom: Is It Time to Put Your EEG Headset On?