Building Arduino robots is pretty cool, but being able to program a robot in Javascript is even better. However, what is even more awesome is the ability to control a robot with Kinect gestures. This presentation provides attendees with a brief introduction to the awesome world of NodeJs, Johnny-Five and Socket.io. The session will deliver an overview of a Kinect gesture detection library and how to connect it to a node-bot.

1. Smart Homes Made
Smarter With JS
Joel Lord
Guidebook app: NEPHP2016

2. www.spiria.com
Robocalypse: Controlling
Nodebots with a Kinect
PresentedBy
JOEL LORD
North EastPHP 2016,Charlottetown,PE
August 5th,2016

3. @joel__lord
#NEPHP
JOEL LORD
About me,eh?
• Javascript Junkie
• Technology enthusiast
• Tinkerer

4. @joel__lord
#NEPHP
AGENDA Why
The Goals
What Exists
What I wanted
Gestures 101
Demo

5. @joel__lord
#NEPHP
MY NODEBOT
Say hello to my little friend

6. Why?
CONTROLLING A ROBOT WITH A KINECT

7. @joel__lord
#NEPHP
Why
• Worked with Nodebots for a while now
@joel__lord

8. @joel__lord
#NEPHP
Why
• Worked with Nodebots for a while now
• Wanted to explore other possibilities of NodeJs
@joel__lord

9. @joel__lord
#NEPHP
Why
• Worked with Nodebots for a while now
• Wanted to explore other possibilities of NodeJs
• Because controlling a robot with a Kinect is pretty cool
@joel__lord

10. @joel__lord
#NEPHP
Why
• Worked with Nodebots for a while now
• Wanted to explore other possibilities of NodeJs
• Because controlling a robot with a Kinect is pretty cool
• But mostly… for fame !
@joel__lord

11. Goals
CONTROLLING A ROBOT WITH A KINECT

12. @joel__lord
#NEPHP
Goals
• Needed a way to use events to tell the robot to perform an action
@joel__lord

13. @joel__lord
#NEPHP
Goals
• Needed a way to use events to tell the robot to perform an action
• Gestures seemed the best way to achieve this
@joel__lord

14. What already exists
CONTROLLING A ROBOT WITH A KINECT

15. @joel__lord
#NEPHP
TheNodebot Stack
• NodeJs (https://nodejs.org/)
@joel__lord

16. @joel__lord
#NEPHP
TheNodebot Stack
• NodeJs (https://nodejs.org/)
• Socket.io (http://socket.io/)
@joel__lord

17. @joel__lord
#NEPHP
TheNodebot Stack
• NodeJs (https://nodejs.org/)
• Socket.io (http://socket.io/)
• Johnny-Five (http://johnny-five.io/)
@joel__lord

18. A little parenthesis
JOHNNY-FIVE?

19. @joel__lord
#NEPHP
Johnny-Five
• Uses Javascript to control an Arduino (or other boards) via Standard Firmata
• Also works with a bunch of platforms like Tessel, Raspberry Pi, Particle
• Uses a standard API for all the platforms
@joel__lord

20. @joel__lord
#NEPHP
Johnny-Five’s Hello World
var five = require("johnny-five");
var board = new five.Board();
board.on("ready", function() {
// Create a standard `led`
// component instance
var led = new five.Led(13);
// "blink" the led in 500ms
// on-off phase periods
led.blink(500);
});
@joel__lord

21. Back to our regular programming

22. @joel__lord
#NEPHP
OpenNI and node-openni
• Connects the Kinect to NodeJs using OpenNI
@joel__lord

23. @joel__lord
#NEPHP
OpenNI and node-openni
• Connects the Kinect to NodeJs using OpenNI
• Very basic event for almost everything
@joel__lord

24. @joel__lord
#NEPHP
OpenNI and node-openni
• Connects the Kinect to NodeJs using OpenNI
• Very basic event for almost everything
• No gestures !
@joel__lord

25. What I needed
CONTROLLING A ROBOT WITH A KINECT

26. @joel__lord
#NEPHP
What Ineeded
• Support for gestures
@joel__lord

27. @joel__lord
#NEPHP
What Ineeded
• Support for gestures
• Support for a full « skeleton »
@joel__lord

28. Gestures 101
CONTROLLING A ROBOT WITH A KINECT

29. @joel__lord
#NEPHP
Gestures 101
• Is the initial condition met?
@joel__lord

30. @joel__lord
#NEPHP
Gestures 101
• Is the initial condition met?
• Is the condition still met?
@joel__lord

31. @joel__lord
#NEPHP
Gestures 101
• Is the initial condition met?
• Is the condition still met?
• Is the final condition met?
@joel__lord

32. @joel__lord
#NEPHP
Gestures 101 - Needs
• A skeleton with an event when it changes
@joel__lord

33. @joel__lord
#NEPHP
Gestures 101 - Needs
• A skeleton with an event when it changes
• Know the position of the « COM »
@joel__lord

34. @joel__lord
#NEPHP
Gestures 101 - Needs
• A skeleton with an event when it changes
• Know the position of the « COM »
• Have a « base unit »
@joel__lord

35. @joel__lord
#NEPHP
Gestures 101 –Introducing kinect-gestures
• Gesture detection using a base class
@joel__lord

36. @joel__lord
#NEPHP
Gestures 101 –Introducing kinect-gestures
• Gesture detection using a base class
• Tracks the skeleton for an initial condition
@joel__lord

37. @joel__lord
#NEPHP
Gestures 101 –Introducing kinect-gestures
• Gesture detection using a base class
• Tracks the skeleton for an initial condition
• Checks every 100ms to see if the condition is still met
@joel__lord

38. @joel__lord
#NEPHP
Gestures 101 –Introducing kinect-gestures
• Gesture detection using a base class
• Tracks the skeleton for an initial condition
• Checks every 100ms to see if the condition is still met
• If the final condition is met, trigger an event
@joel__lord

39. @joel__lord
#NEPHP
Gestures 101 –Introducing kinect-gestures
• Gesture detection using a base class
• Tracks the skeleton for an initial condition
• Checks every 100ms to see if the condition is still met
• If the final condition is met, trigger an event
• Other classes are used to define the actual gestures
@joel__lord

40. IT’S CODING TIME!
LET’S GET SERIOUS

41. IT’S CODING TIME!
LET’S GET SERIOUS

42. @joel__lord
#NEPHP
Next few steps
8/5/2016
@joel__lord
42
• Return a more friendly coordinate system for skeleton
• Add more information to the returned event (velocity?)
• Add more gestures (jump, wave, kick)

43. DOCUMENT CONFIDENTIEL, TOUT DROIT RÉSERVÉ
PRESENTED BY
The End !
Questions ?
JOEL LORD
August 5th 2016
TWITTER: @JOEL__LORD
GITHUB: HTTP://GITHUB.COM/JOELLORD