Smart Homes Made
Smarter With JS
Joel Lord
Guidebook app: NEPHP2016

www.spiria.com
Robocalypse: Controlling
Nodebots with a Kinect
PresentedBy
JOEL LORD
North EastPHP 2016,Charlottetown,PE
August 5th,2016

@joel__lord
#NEPHP
JOEL LORD
About me,eh?
• Javascript Junkie
• Technology enthusiast
• Tinkerer

@joel__lord
#NEPHP
AGENDA Why
The Goals
What Exists
What I wanted
Gestures 101
Demo

@joel__lord
#NEPHP
MY NODEBOT
Say hello to my little friend

Why?
CONTROLLING A ROBOT WITH A KINECT

@joel__lord
#NEPHP
Why
• Worked with Nodebots for a while now
@joel__lord

@joel__lord
#NEPHP
Why
• Worked with Nodebots for a while now
• Wanted to explore other possibilities of NodeJs
@joel__lord

@joel__lord
#NEPHP
Why
• Worked with Nodebots for a while now
• Wanted to explore other possibilities of NodeJs
• Because controlling a robot with a Kinect is pretty cool
@joel__lord

@joel__lord
#NEPHP
Why
• Worked with Nodebots for a while now
• Wanted to explore other possibilities of NodeJs
• Because controlling a robot with a Kinect is pretty cool
• But mostly… for fame !
@joel__lord

Goals
CONTROLLING A ROBOT WITH A KINECT

@joel__lord
#NEPHP
Goals
• Needed a way to use events to tell the robot to perform an action
@joel__lord

@joel__lord
#NEPHP
Goals
• Needed a way to use events to tell the robot to perform an action
• Gestures seemed the best way to achieve this
@joel__lord

What already exists
CONTROLLING A ROBOT WITH A KINECT

@joel__lord
#NEPHP
TheNodebot Stack
• NodeJs (https://nodejs.org/)
@joel__lord

@joel__lord
#NEPHP
TheNodebot Stack
• NodeJs (https://nodejs.org/)
• Socket.io (http://socket.io/)
@joel__lord

@joel__lord
#NEPHP
TheNodebot Stack
• NodeJs (https://nodejs.org/)
• Socket.io (http://socket.io/)
• Johnny-Five (http://johnny-five.io/)
@joel__lord

A little parenthesis
JOHNNY-FIVE?

@joel__lord
#NEPHP
Johnny-Five
• Uses Javascript to control an Arduino (or other boards) via Standard Firmata
• Also works with a bunch of platforms like Tessel, Raspberry Pi, Particle
• Uses a standard API for all the platforms
@joel__lord

@joel__lord
#NEPHP
Johnny-Five’s Hello World
var five = require("johnny-five");
var board = new five.Board();
board.on("ready", function() {
// Create a standard `led`
// component instance
var led = new five.Led(13);
// "blink" the led in 500ms
// on-off phase periods
led.blink(500);
});
@joel__lord

Back to our regular programming

@joel__lord
#NEPHP
OpenNI and node-openni
• Connects the Kinect to NodeJs using OpenNI
@joel__lord

@joel__lord
#NEPHP
OpenNI and node-openni
• Connects the Kinect to NodeJs using OpenNI
• Very basic event for almost everything
@joel__lord

@joel__lord
#NEPHP
OpenNI and node-openni
• Connects the Kinect to NodeJs using OpenNI
• Very basic event for almost everything
• No gestures !
@joel__lord

What I needed
CONTROLLING A ROBOT WITH A KINECT

@joel__lord
#NEPHP
What Ineeded
• Support for gestures
@joel__lord

@joel__lord
#NEPHP
What Ineeded
• Support for gestures
• Support for a full « skeleton »
@joel__lord

Gestures 101
CONTROLLING A ROBOT WITH A KINECT

@joel__lord
#NEPHP
Gestures 101
• Is the initial condition met?
@joel__lord

@joel__lord
#NEPHP
Gestures 101
• Is the initial condition met?
• Is the condition still met?
@joel__lord

@joel__lord
#NEPHP
Gestures 101
• Is the initial condition met?
• Is the condition still met?
• Is the final condition met?
@joel__lord

@joel__lord
#NEPHP
Gestures 101 - Needs
• A skeleton with an event when it changes
@joel__lord

@joel__lord
#NEPHP
Gestures 101 - Needs
• A skeleton with an event when it changes
• Know the position of the « COM »
@joel__lord

@joel__lord
#NEPHP
Gestures 101 - Needs
• A skeleton with an event when it changes
• Know the position of the « COM »
• Have a « base unit »
@joel__lord

@joel__lord
#NEPHP
Gestures 101 –Introducing kinect-gestures
• Gesture detection using a base class
@joel__lord

@joel__lord
#NEPHP
Gestures 101 –Introducing kinect-gestures
• Gesture detection using a base class
• Tracks the skeleton for an initial condition
@joel__lord

@joel__lord
#NEPHP
Gestures 101 –Introducing kinect-gestures
• Gesture detection using a base class
• Tracks the skeleton for an initial condition
• Checks every 100ms to see if the condition is still met
@joel__lord

@joel__lord
#NEPHP
Gestures 101 –Introducing kinect-gestures
• Gesture detection using a base class
• Tracks the skeleton for an initial condition
• Checks every 100ms to see if the condition is still met
• If the final condition is met, trigger an event
@joel__lord

@joel__lord
#NEPHP
Gestures 101 –Introducing kinect-gestures
• Gesture detection using a base class
• Tracks the skeleton for an initial condition
• Checks every 100ms to see if the condition is still met
• If the final condition is met, trigger an event
• Other classes are used to define the actual gestures
@joel__lord

IT’S CODING TIME!
LET’S GET SERIOUS

IT’S CODING TIME!
LET’S GET SERIOUS

@joel__lord
#NEPHP
Next few steps
8/5/2016
@joel__lord
42
• Return a more friendly coordinate system for skeleton
• Add more information to the returned event (velocity?)
• Add more gestures (jump, wave, kick)

DOCUMENT CONFIDENTIEL, TOUT DROIT RÉSERVÉ
PRESENTED BY
The End !
Questions ?
JOEL LORD
August 5th 2016
TWITTER: @JOEL__LORD
GITHUB: HTTP://GITHUB.COM/JOELLORD