Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

[JS Kongress 2016] KittyCam.js - Raspberry Pi Camera w/ Cat Facial Detection

732 views

Published on

About my Raspberry Pi + Node.js project called KittyCam. Source code is available on GitHub at: https://github.com/girliemac/RPi-KittyCam

This talk was given at JS Kongress in Munich on Nov 28, 2016.

Published in: Devices & Hardware
  • Be the first to comment

  • Be the first to like this

[JS Kongress 2016] KittyCam.js - Raspberry Pi Camera w/ Cat Facial Detection

  1. 1. girlie_mac@ KittyCam.js Smile, you’re on camera!Smile, you’re on camera! Tomomi Imura
  2. 2. girlie_mac@ Hello, I’m Tomomi & I’m a: ● San Francisco Dweller ● Front-End Engineer ● Open Web + Tech Advocate ● N00b Hardware Hacker ● Sr. Developer Advocate at Nexmo ● Cat lady of InterWeb
  3. 3. girlie_mac@
  4. 4. girlie_mac@ CC-BY-SA 3.0 https://commons.wikimedia.org/wiki/File:Basic_robot.jpg
  5. 5. girlie_mac@ Arduino ● MCU-based kit ● Open-Source hardware & software (IDE) ● The first developer-friendly boards
  6. 6. girlie_mac@ Sketch ● Language for Arduino ● Loosely based on C ● Wut, C? ● HALP!!1!!! #include <Servo.h> #include <Wire.h> #include <Firmata.h> #define I2C_WRITE B00000000 #define I2C_READ B00001000 #define I2C_READ_CONTINUOUSLY B00010000 #define I2C_STOP_READING B00011000 #define I2C_READ_WRITE_MODE_MASK B00011000 #define I2C_10BIT_ADDRESS_MODE_MASK B00100000 #define I2C_MAX_QUERIES 8 #define I2C_REGISTER_NOT_SPECIFIED -1 #define MINIMUM_SAMPLING_INTERVAL 1 int analogInputsToReport = 0; byte previousPINs[TOTAL_PORTS]; byte pinConfig[TOTAL_PINS]; byte portConfigInputs[TOTAL_PORTS]; int pinState[TOTAL_PINS]; unsigned long currentMillis;
  7. 7. girlie_mac@ Raspberry Pi ● $35 single-board computer ● Broadcom chip with ARM-compatible CPU & GPU ● Runs Linux ● More language choices: C, C++, Python...
  8. 8. girlie_mac@ But… I want to code in
  9. 9. girlie_mac@ Johnny-Five ● JavaScript robotics framework ● Works with Arduino-compatible Boards ● IO plugins for more platform supports ● http://johnny-five.io/
  10. 10. girlie_mac@ Awww, JavaScript all the way!
  11. 11. girlie_mac@ Hello world!
  12. 12. girlie_mac@ KittyCam Raspberry Pi camera with cat facial detection! ● Hardware: Raspberry Pi, camera, and PIR sensor ● Software: Node.js + J5 + More open-source goodies
  13. 13. girlie_mac@ Motion detected! snap! Mmm… donut
  14. 14. girlie_mac@ kittyCam.js 1. Detect motion 2. Take a photo 3. Cat facial detection 4. Store the photo in cloud 5. Real-time view on web 6. SMS the photo link HELL, YEAH!
  15. 15. girlie_mac@
  16. 16. girlie_mac@ Raspberry Pi 3 PIR Motion Sensor Camera Module F/F Jumper Wires (x3)
  17. 17. girlie_mac@
  18. 18. girlie_mac@
  19. 19. girlie_mac@ 1. Raspberry Pi 3 2. 5MP Camera Board Module 3. Pyroelectric Infrared (PIR) motion sensor 4. Female/Female wires
  20. 20. girlie_mac@ Programming Raspberry Pi Pre-installed on Raspbian OS: C / C++
  21. 21. girlie_mac@ Programming RPi with Node.js
  22. 22. girlie_mac@
  23. 23. girlie_mac@ Installing Node ARM $ wget https://nodejs.org/dist/v4.4.5/node-v4.4.5-l inux-armv7l.tar.xz $ tar -xvf node-v4.4.5-linux-armv7l.tar.xz $ cd node-v4.4.5-linux-armv7l $ sudo cp -R * /usr/local/
  24. 24. girlie_mac@ kittyCam.js 1. Detect motion 2. Take a photo 3. Cat facial detection 4. Store the photo in cloud 5. Real-time view on web 6. SMS the photo link HELL, YEAH!
  25. 25. girlie_mac@ kittyCam.js Tech Stack 1. Detect motion w/ Johnny-Five IR.Motion obj 2. Take a photo w/ Raspistill, command line tool 3. Cat facial detection w/ KittyDar 4. Store the photo in Cloudinary 5. Publish & subscribe the url to display on web via PubNub 6. Send a text message via Nexmo
  26. 26. girlie_mac@ $ Raspistill canvas catDetect.js app.js kittyDar Motion detected take a photo Store the photo if cats are detected display photos on web browsers real-time anywhere Johnny-Five w/ raspi-io child process Returns url Notify with SMS via
  27. 27. girlie_mac@ Johnny-Five w/ Raspi-io const five = require('johnny-five'); const raspi = require('raspi-io'); let board = new five.Board({io: new raspi()}); board.on('ready', () => { console.log('board is ready'); ... });
  28. 28. girlie_mac@ Motion const five = require('johnny-five'); const raspi = require('raspi-io'); const board = new five.Board({io: new raspi()}); board.on('ready', function() { // Create a new `motion` hardware instance const motion = new five.Motion('P1-7'); ... }); a PIR is wired on pin 7 (GPIO 4) VCC Ground Data
  29. 29. girlie_mac@ Raspistill Command Line Tool $ raspistill -o myPhoto.jpg
  30. 30. girlie_mac@ PIR Sensor > Run Camera const child_process = require('child_process'); board.on('ready', () => { const motion = new five.Motion('P1-7'); motion.on('motionstart', () => { // Motion detected let filename = 'photo/image_'+i+'.jpg'; let args = ['-w', '320', '-h', '240', '-o', filename, '-t', '1']; let spawn = child_process.spawn('raspistill', args); spawn.on('exit', function() { console.log('A photo is saved as '+filename); ... motion detected! Take a photo! Spawns a new process w/ a given shell command
  31. 31. girlie_mac@ Processing Image spawn.on('exit', () => { let imgPath = __dirname + '/' + filename; // Child process: read the file and detect cats with KittyDar let args = [imgPath]; let fork = child_process.fork(__dirname+'/detectCatsFromPhoto.js'); fork.send(args); // the child process is completed fork.on('message', (base64) => { if(base64) { uploadToCloud(base64); // Send to cloud storage } }); ... Create another worker by running a new instance of V8 engine.
  32. 32. girlie_mac@ detectCatsFromPhoto.js const kittydar = require('kittydar'); const Canvas = require('canvas'); process.on('message', (m) => { fs.readFile(m[0], (err, data) => { ... let canvas = new Canvas(w, h); let ctx = canvas.getContext('2d'); ctx.drawImage(img, 0, 0, w, h, 0, 0, w, h); let cats = kittydar.detectCats(canvas); console.log('There are', cats.length, 'cats in this photo'); if(cats.length > 0) {base64Img = canvas.toDataURL();} process.send(base64Img); process.exit(0); }); Running in an another process a cat detected! KittyDar
  33. 33. girlie_mac@ KittyDar ● Open-source JavaScript cat facial detection written by Heather Arthur ● Takes a canvas obj & calculate the locations of cats in the image let cats = kittydar.detectCats(canvas);
  34. 34. girlie_mac@ KittyDar: Behind the Scene 1. Chops up the image up into many “windows” 2. Extracts data by measuring a set of gradients, from light & dark in order to find edges 3. Compares the direction of these edges to the edges found in known cat images Neural network (JSON w/ vector data) is pre-trained w/ thousands pics of cats & non-cats
  35. 35. girlie_mac@ Cat Facial Detection http://research.microsoft.com/pubs/80582/ECCV_CAT_PROC.pdf
  36. 36. girlie_mac@ Cat Facial Detection http://research.microsoft.com/pubs/80582/ECCV_CAT_PROC.pdf
  37. 37. girlie_mac@ Cat Facial Detection The annotation data sequence: (Number of points, default is 9), (Left Eye), (Right Eye), (Mouth), (Left Ear-1), (Left Ear-2), (Left Ear-3), (Right Ear-1), (Right Ear-2), and (Right Ear-3)
  38. 38. girlie_mac@ Cat Facial Data 9 247 94 294 92 273 127 213 81 207 29 247 53 286 50 320 20 322 74 00000561_012.jpg.cat00000561_012.jpg
  39. 39. girlie_mac@ Cat Facial Data Collection
  40. 40. girlie_mac@ Neural Network Positive Negative JSON data hog-descriptor kittydar.jsCompare data
  41. 41. girlie_mac@ Send the Pic w/ Kitty to Cloud const cloudinary = require('cloudinary'); // the child process is completed fork.on('message', (base64) => { if(base64) { cloudinary.uploader.upload(base64, (result) => { // Done! - Get the URL and do more stuff }); } else deletePhoto(imgPath); });
  42. 42. girlie_mac@ View Photos Real-time via Socket const pubnub = require(pubnub); Publish Subscribe
  43. 43. girlie_mac@ http://www.girliemac.com/RPi-KittyCam/
  44. 44. girlie_mac@ Send SMS via Nexmo const Nexmo = require(nexmo); let nexmo = new Nexmo({//config with API keys}); nexmo.message.sendSms( FROM_NUMBER, TO_NUMBER, ' '+ url, options, (err, responseData) => { if (err) console.log(err); else console.dir(responseData); });
  45. 45. girlie_mac@
  46. 46. girlie_mac@ QA Team Lead QA: Jamie Ginger Basil Alice Yugi Venom @kittenVenom Ozzy
  47. 47. girlie_mac@ github.com/girliemac/RPi-KittyCam
  48. 48. girlie_mac@ Next Steps ● Upgrade Hardware ○ Raspberry Pi 3 ○ NoIR Night Vision Camera ● Upgrade Node (was 0.12) & all dependencies ● ES5 to ES6 ● More features (Maybe) ○ Cat Identification w/ RFID ○ Photo Booth w/ Filter effects & props
  49. 49. girlie_mac@ More Cat Projects w/ Node.js
  50. 50. girlie_mac@ https://http.cat Thanks Rogério Vicente for the .cat domain & the API! Previously: HTTP Status Cats
  51. 51. girlie_mac@ HTTP Status Cats Slack Bot Install it at: http://girliemac.com/ slack-httpstatuscats/ Tutorial on Medium.com: https://goo.gl/saOUl0
  52. 52. @girlie_mac Next Project? ● Selfie bot (à la Mannie the Selfie Cat) Mannie the Selfie Cat by @yoremahm on Instagram https://www.instagram.com/yoremahm/
  53. 53. girlie_mac@ Danke schön! @girlie_mac github.com/girliemac
  54. 54. girlie_mac@ Attribution: Emoji by Emoji-One (CC-BY 4.0)

×