Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

[Longer ver] From Software to Hardware: How Do I Track My Cat with JavaScript

1,382 views

Published on

About my Raspberry Pi + Node.js project called KittyCam
Presented at:
- NodePDX in Portland, OR
- Forward 5 in San Francisco, CA
- Nordic.js in Stockholm, Sweden

Published in: Technology
  • Be the first to comment

[Longer ver] From Software to Hardware: How Do I Track My Cat with JavaScript

  1. 1. @girlie_mac KittyCam.js Smile, you’re on camera!Smile, you’re on camera! Tomomi Imura
  2. 2. @girlie_mac
  3. 3. @girlie_mac Tomomi (@girlie_mac) is a(n): ● San Francisco Dweller ● Front-End Engineer ● Open Web + Tech Advocate ● N00b Hardware Hacker ● Sr. Developer Advocate at Nexmo ● Cat lady of InterWeb
  4. 4. @girlie_mac HTTP Status Cats at https://http.cat Thanks Rogério Vicente for the .cat domain & the API!
  5. 5. @girlie_mac
  6. 6. @girlie_mac CC-BY-SA 3.0 https://commons.wikimedia.org/wiki/File:Basic_robot.jpg
  7. 7. @girlie_mac Arduino ● MCU-based kit ● Open-Source hardware & software (IDE) ● The first developer-friendly boards
  8. 8. @girlie_mac Sketch ● Language for Arduino ● Loosely based on C ● Wut, C? ● HALP!!1!!! #include <Servo.h> #include <Wire.h> #include <Firmata.h> #define I2C_WRITE B00000000 #define I2C_READ B00001000 #define I2C_READ_CONTINUOUSLY B00010000 #define I2C_STOP_READING B00011000 #define I2C_READ_WRITE_MODE_MASK B00011000 #define I2C_10BIT_ADDRESS_MODE_MASK B00100000 #define I2C_MAX_QUERIES 8 #define I2C_REGISTER_NOT_SPECIFIED -1 #define MINIMUM_SAMPLING_INTERVAL 1 int analogInputsToReport = 0; byte previousPINs[TOTAL_PORTS]; byte pinConfig[TOTAL_PINS]; byte portConfigInputs[TOTAL_PORTS]; int pinState[TOTAL_PINS]; unsigned long currentMillis;
  9. 9. @girlie_mac Raspberry Pi ● $35 single-board computer ● Broadcom chip with ARM-compatible CPU & GPU ● Runs Linux ● More language choices: C, C++, Python...
  10. 10. @girlie_mac But… I want to code in
  11. 11. @girlie_mac Johnny-Five ● JavaScript robotics framework ● Works with Arduino-compatible Boards ● IO plugins for more platform supports ● http://johnny-five.io/
  12. 12. @girlie_mac Awww, JavaScript all the way!
  13. 13. @girlie_mac Hello world!
  14. 14. @girlie_mac KittyCam Raspberry Pi camera with cat facial detection! ● Hardware: Raspberry Pi, camera, and PIR sensor ● Software: Node.js + J5 + More open-source goodies
  15. 15. @girlie_mac Motion detected! snap! Mmm… donut
  16. 16. @girlie_mac
  17. 17. @girlie_mac
  18. 18. @girlie_mac
  19. 19. @girlie_mac 1. Raspberry Pi 3 2. 5MP Camera Board Module 3. Pyroelectric Infrared (PIR) motion sensor 4. Female/Female wires
  20. 20. @girlie_mac Programming Raspberry Pi Pre-installed on Raspbian OS: C / C++
  21. 21. @girlie_mac Programming RPi with Node.js
  22. 22. @girlie_mac
  23. 23. @girlie_mac Installing Node ARM $ wget https://nodejs.org/dist/v4.4.5/node-v4.4.5-l inux-armv7l.tar.xz $ tar -xvf node-v4.4.5-linux-armv7l.tar.xz $ cd node-v4.4.5-linux-armv7l $ sudo cp -R * /usr/local/
  24. 24. @girlie_mac kittyCam.js 1. Detect motion 2. Take a photo 3. Cat facial detection 4. Store the photo in cloud 5. Real-time view on web 6. SMS the photo link HELL, YEAH!
  25. 25. @girlie_mac kittyCam.js 1. Detect motion w/ Johnny-Five IR.Motion obj 2. Take a photo w/ Raspistill, command line tool 3. Cat facial detection w/ KittyDar 4. Store the photo in Cloudinary 5. Publish & subscribe the url to display on web via PubNub 6. Send a text message via Nexmo
  26. 26. @girlie_mac $ Raspistill canvas catDetect.js app.js kittyDar Motion detected take a photo Store the photo if cats are detected display photos on web browsers real-time anywhere Johnny-Five w/ raspi-io child process Returns url Notify with SMS via
  27. 27. @girlie_mac Johnny-Five w/ Raspi-io const five = require('johnny-five'); const raspi = require('raspi-io'); let board = new five.Board({io: new raspi()}); board.on('ready', () => { console.log('board is ready'); ... });
  28. 28. @girlie_mac Motion const five = require('johnny-five'); const raspi = require('raspi-io'); const board = new five.Board({io: new raspi()}); board.on('ready', function() { // Create a new `motion` hardware instance const motion = new five.Motion('P1-7'); ... }); a PIR is wired on pin 7 (GPIO 4) VCC Ground Data
  29. 29. @girlie_mac Raspistill Command Line Tool $ raspistill -o myPhoto.jpg
  30. 30. @girlie_mac PIR Sensor > Run Camera const child_process = require('child_process'); board.on('ready', () => { const motion = new five.Motion('P1-7'); motion.on('motionstart', () => { // Motion detected let filename = 'photo/image_'+i+'.jpg'; let args = ['-w', '320', '-h', '240', '-o', filename, '-t', '1']; let spawn = child_process.spawn('raspistill', args); spawn.on('exit', function() { console.log('A photo is saved as '+filename); ... motion detected! Take a photo! Spawns a new process w/ a given shell command
  31. 31. @girlie_mac Processing Photo spawn.on('exit', () => { let imgPath = __dirname + '/' + filename; // Child process: read the file and detect cats with KittyDar let args = [imgPath]; let fork = child_process.fork(__dirname+'/detectCatsFromPhoto.js'); fork.send(args); // the child process is completed fork.on('message', (base64) => { if(base64) { uploadToCloud(base64); // Send to cloud storage } }); ... Create another worker by running a new instance of V8 engine.
  32. 32. @girlie_mac detectCatsFromPhoto.js const kittydar = require('kittydar'); const Canvas = require('canvas'); process.on('message', (m) => { fs.readFile(m[0], (err, data) => { ... let canvas = new Canvas(w, h); let ctx = canvas.getContext('2d'); ctx.drawImage(img, 0, 0, w, h, 0, 0, w, h); let cats = kittydar.detectCats(canvas); console.log('There are', cats.length, 'cats in this photo'); if(cats.length > 0) {base64Img = canvas.toDataURL();} process.send(base64Img); process.exit(0); }); Running in an another process a cat detected! KittyDar
  33. 33. @girlie_mac KittyDar ● Open-source JavaScript cat facial detection written by Heather Arthur ● Takes a canvas obj & calculate the locations of cats in the image let cats = kittydar.detectCats(canvas);
  34. 34. @girlie_mac KittyDar: Behind the Scene 1. Chops up the image up into many “windows” 2. Extracts data by measuring a set of gradients, from light & dark in order to find edges 3. Compares the direction of these edges to the edges found in known cat images Neural network (JSON w/ vector data) is pre-trained w/ thousands pics of cats & non-cats
  35. 35. @girlie_mac Cat Facial Detection http://research.microsoft.com/pubs/80582/ECCV_CAT_PROC.pdf
  36. 36. @girlie_mac Cat Facial Detection http://research.microsoft.com/pubs/80582/ECCV_CAT_PROC.pdf
  37. 37. @girlie_mac Cat Facial Detection The annotation data sequence: (Number of points, default is 9), (Left Eye), (Right Eye), (Mouth), (Left Ear-1), (Left Ear-2), (Left Ear-3), (Right Ear-1), (Right Ear-2), and (Right Ear-3)
  38. 38. @girlie_mac Cat Facial Data 9 247 94 294 92 273 127 213 81 207 29 247 53 286 50 320 20 322 74 00000561_012.jpg.cat00000561_012.jpg
  39. 39. @girlie_mac Cat Facial Data Collection
  40. 40. @girlie_mac Neural Network Positive Negative JSON data hog-descriptor kittydar.jsCompare data
  41. 41. @girlie_mac Send the Pic w/ Kitty to Cloud const cloudinary = require('cloudinary'); // the child process is completed fork.on('message', (base64) => { if(base64) { cloudinary.uploader.upload(base64, (result) => { // Done! - Get the URL and do more stuff }); } else deletePhoto(imgPath); });
  42. 42. @girlie_mac View Photos Real-time via Socket const pubnub = require(pubnub); Publish Subscribe
  43. 43. @girlie_mac http://www.girliemac.com/RPi-KittyCam/
  44. 44. @girlie_mac Send SMS via Nexmo const Nexmo = require(nexmo); let nexmo = new Nexmo({//config with API keys}); nexmo.message.sendSms( FROM_NUMBER, TO_NUMBER, ' '+ url, options, (err, responseData) => { if (err) console.log(err); else console.dir(responseData); });
  45. 45. @girlie_mac
  46. 46. @girlie_mac QA Team Lead QA: Jamie Ginger Basil Alice YugiVenom @kittenVenom
  47. 47. @girlie_mac github.com/girliemac/RPi-KittyCam
  48. 48. @girlie_mac Next Steps ● Upgrade Hardware ○ Raspberry Pi 3 ○ NoIR Night Vision Camera ● Upgrade Node (was 0.12) & all dependencies ● More features (Maybe) ○ Cat Identification w/ RFID ○ Photo Booth w/ Filter effects & props
  49. 49. @girlie_mac Next Project? ● Selfie bot (à la Mannie the Selfie Cat) Mannie the Selfie Cat by @yoremahm on Instagram https://www.instagram.com/yoremahm/
  50. 50. @girlie_mac Tack så mycket! @girlie_mac github.com/girliemac
  51. 51. @girlie_mac Attribution: Emoji by Emoji-One (CC-BY 4.0)

×