Hacking Robots for Fun and Profit

                               presented by Chad Udell
                            at F...
Hacking Robots for Fun and Profit

 • Flash is eminently hackable and ideal for building UIs
    for experiences that inte...
A History of Hacking Flash with Hardware
About the Adler Planetarium Exhibit Project

 • Planet Explorers: 13 Experiences

 • X-Rovers are one of 7 interactive pie...
Planet Explorers: X-Rovers

 • What is it?

 • How does it work?

 • What is the experience?

 • Demos

 • Code Samples
What is it?

 • Emulating a Mars Rover
    like experience
 • Has to have a “mission”

 • Has to be easy to use
Definition of the Experience

 • Origins (KUD)

   ‣ Remote Exploration is
      the Concept
 • Key Points

   ‣ No Crying...
First Thoughts

 • Custom R/C

 • T1

 • Imperial Droid
Basic Limitations

 • Skillset - We aren’t robotics engineers or electricians

 • Budget - Every client is conscious of th...
Choosing a Robot

 • Essentially we need a connected webcam

 • Needs to be cheap

 • Needs to be hackable




           ...
A Diagram of the system
2 Stations, each containing:


                               !"#$%&'()*+,-./+*0)




     UPS (AP...
Rovio to the Rescue

 • Skillset - Essentially like building a mashup
    (documented API, built in webserver)
 • Budget -...
Flash to the Rescue

 • Skillset - We like it.

 • Budget - No extra expensive hardware to be
    designed, no expensive p...
Controlling the Rovio

 • Rovio responds via HTTP (CGI script)

 • Use of key listeners to trigger API calls.
    /rev.cgi...
Controlling the Rovio




          Demo 1: Sending
         Commands to Rovio
Getting the Video

 • Rovio sends via MJPEG via HTTP (CGI script)

 • Set a loader’s source to the URL

    Camera Control...
Getting the Video




      Demo 2: MJPeg stream
                in a Loader
Deeper Research on Video/Image Processing
   Simple color detection -

  •     http://www.interactionfigure.nl/2009/colour...
Technology Evaluations

 • Shopping = Fun




 • Reading Electrical Engineer Documents = Not so Fun

 • Wiring Boards = Fu...
Prototype

 • Software Proof of Concept

 • Hardware Hacking and Integration
Happ Config 1
Happ Config 2
Prototype




     Demo 3: Happ Controls
Alpha

 • Video was ok, but needed some juice.

 • Computer vision was deemed better than AR or
   object detection
 • Pla...
Play Testing Video
Enter Red 5/Xuggle

 • Transcode the Video to H264       (SteamStream)

 • Replace the MJPEG loader with a Netstream objec...
Enter Red 5/Xuggle




      Demo 4: Red5/Xuggle
Processing the video

 • Considered AR

   • Too Inflexible - Markers weren’t attractive

 • Considered Object Detection (...
Terminator Vision
Finding the Targets




    Demo 5: Computer Vision
What does that get you? (The Good)
What does that get you? (The Bad)
How about Creating a Unique Target?

• Eliminate False Positives (was better, but not perfect)

• Ensure Speedy Detection ...
Processing the video

 • Settled on Color Detection via Thresholding

 • Tomek’s Blob Detection

   • http://play.blog2t.n...
Fixing the Video




            Demo 5: Revised
            Computer Vision
In Action
In Action
In Action
In Action
In Action
In Action
Buttoning it up

 • Charging handoff

 • Locking the kiosks down

 • C++ Socket Server for the Lights

 • Startup, shutdow...
Lessons Learned

 • SWF to EXE is not needed anymore. AIR 2!

 • Test early! Test Often! Test Under lots of Conditions!

 ...
Other stuff to hack!

 • Security/WiFi Cameras - http://www.axis.com

 • Tons of X10 Stuff - http://www.smarthome.com

 • ...
More info, code samples, etc.

 • http://electricpineapple.net/ - Erik Peterson’s blog

 • http://ionagroup.com/labs - Ion...
Thank you
• Find Us on:




                                               THE IONA GROUP
                    620 West Jac...
Upcoming SlideShare
Loading in …5
×

Hacking Robots for Fun and Profit

1,268 views

Published on

This is a fun one! Learn how to hack up robots you can buy at a local toy store. You’ll see the methods used to take the video stream out of the robot and turn it into a format Flash likes. You’ll get the lowdown on how to send API commands to control the bot. We’ll show you how to connect it to alternative controllers and use ActionScript for some simple color detection on the video stream.

Published in: Self Improvement, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,268
On SlideShare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Hacking Robots for Fun and Profit

  1. 1. Hacking Robots for Fun and Profit presented by Chad Udell at Flash and the City, 2010
  2. 2. Hacking Robots for Fun and Profit • Flash is eminently hackable and ideal for building UIs for experiences that integrate with hardware • Tons of devices are coming with web connections • Hardware is getting cheaper and easier to get • Hobbyist techniques can be adapted for real world
  3. 3. A History of Hacking Flash with Hardware
  4. 4. About the Adler Planetarium Exhibit Project • Planet Explorers: 13 Experiences • X-Rovers are one of 7 interactive pieces
  5. 5. Planet Explorers: X-Rovers • What is it? • How does it work? • What is the experience? • Demos • Code Samples
  6. 6. What is it? • Emulating a Mars Rover like experience • Has to have a “mission” • Has to be easy to use
  7. 7. Definition of the Experience • Origins (KUD) ‣ Remote Exploration is the Concept • Key Points ‣ No Crying Children ‣ Some Learning ‣ Ages 3-8
  8. 8. First Thoughts • Custom R/C • T1 • Imperial Droid
  9. 9. Basic Limitations • Skillset - We aren’t robotics engineers or electricians • Budget - Every client is conscious of this - duh • Scope - Need to integrate a timer, controls, basic computer vision, long term maintenance solution • Quality - Uptime is crucial. “Out of Order” is not an option
  10. 10. Choosing a Robot • Essentially we need a connected webcam • Needs to be cheap • Needs to be hackable Enter, Rovio.
  11. 11. A Diagram of the system 2 Stations, each containing: !"#$%&'()*+,-./+*0) UPS (APC BR1200) APC AV C2 Happ Kiosk Amp and Speakers 1"234&+56(!11(.7 Rovio Charger Happ Controls Server (Red 5) Netgear Wireless Access Point WG-102 Switch Rovio
  12. 12. Rovio to the Rescue • Skillset - Essentially like building a mashup (documented API, built in webserver) • Budget - They are inexpensive and readily available (important for uptime) • Scope - many of the requirements are handled in some built in fashion already (wayfinding, charging). • Quality - We have a company backing it
  13. 13. Flash to the Rescue • Skillset - We like it. • Budget - No extra expensive hardware to be designed, no expensive proprietary SDKs. • Scope - Still some questions here. Going to have to manage the client’s expectations. • Quality - Debugging and remote config is easy
  14. 14. Controlling the Rovio • Rovio responds via HTTP (CGI script) • Use of key listeners to trigger API calls. /rev.cgi?Cmd=nav&action=value&drive=d_value&speed=s_value Input Parameter value = 18 d_value = 0 (Stop) 1 (Forward) 2 (Backward) 3 (Straight left) 4 (Straight right) 5 (Rotate left by speed) 6 (Rotate right by speed) 7 (Diagonal forward left) 8 (Diagonal forward right) 9 (Diagonal backward left) 10 (Diagonal backward right) 11 (Head up) 12 (Head down) 13 (Head middle) 14 (Reserved) 15 (Reserved) 16 (Reserved) 17 (Rotate left by 20 degree angle increments) 18 (Rotate right by 20 degree angle increments) s_value = 1 (fastest) – 10 (slowest)
  15. 15. Controlling the Rovio Demo 1: Sending Commands to Rovio
  16. 16. Getting the Video • Rovio sends via MJPEG via HTTP (CGI script) • Set a loader’s source to the URL Camera Control – GetData.cgi Description The basic command for acquiring MJPEGGrammar /ChangeResolution.cgi?ResType=value[&RedirectUrl=sUrl] Input Parameter None Privilege None Return Value An instance captured motion image. • Results were marginal (12-15 fps)
  17. 17. Getting the Video Demo 2: MJPeg stream in a Loader
  18. 18. Deeper Research on Video/Image Processing Simple color detection - • http://www.interactionfigure.nl/2009/colour-tracking-with-the-webcam-in-flash/ Color detection in Processing • http://jamesalliban.wordpress.com/2008/11/16/colour-detection-in-processing/ Haar training for object detection • http://note.sonots.com/SciSoftware/haartraining.html Modified facial detection • http://www.quasimondo.com/archives/000687.php
  19. 19. Technology Evaluations • Shopping = Fun • Reading Electrical Engineer Documents = Not so Fun • Wiring Boards = Fun for some • Building PoCs = Super fun!
  20. 20. Prototype • Software Proof of Concept • Hardware Hacking and Integration
  21. 21. Happ Config 1
  22. 22. Happ Config 2
  23. 23. Prototype Demo 3: Happ Controls
  24. 24. Alpha • Video was ok, but needed some juice. • Computer vision was deemed better than AR or object detection • Play control overall was very pleasing • Testing, Testing, Testing
  25. 25. Play Testing Video
  26. 26. Enter Red 5/Xuggle • Transcode the Video to H264 (SteamStream) • Replace the MJPEG loader with a Netstream object • Better results. Lower latency. • Approaching 24-30fps • There is a bit of a hiccup with the initialization of the stream.
  27. 27. Enter Red 5/Xuggle Demo 4: Red5/Xuggle
  28. 28. Processing the video • Considered AR • Too Inflexible - Markers weren’t attractive • Considered Object Detection (Haar Cascades/ OpenCV) • A tad slow-ish • Attempted using simple ColorBoundsRect
  29. 29. Terminator Vision
  30. 30. Finding the Targets Demo 5: Computer Vision
  31. 31. What does that get you? (The Good)
  32. 32. What does that get you? (The Bad)
  33. 33. How about Creating a Unique Target? • Eliminate False Positives (was better, but not perfect) • Ensure Speedy Detection (was a little slower) Not Unique A Lot More Unique
  34. 34. Processing the video • Settled on Color Detection via Thresholding • Tomek’s Blob Detection • http://play.blog2t.net/fast-blob-detection • Soulwire’s Color Averaging • http://blog.soulwire.co.uk/code/actionscript-3/ extract-average-colours-from-bitmapdata
  35. 35. Fixing the Video Demo 5: Revised Computer Vision
  36. 36. In Action
  37. 37. In Action
  38. 38. In Action
  39. 39. In Action
  40. 40. In Action
  41. 41. In Action
  42. 42. Buttoning it up • Charging handoff • Locking the kiosks down • C++ Socket Server for the Lights • Startup, shutdown • Training for the switchover
  43. 43. Lessons Learned • SWF to EXE is not needed anymore. AIR 2! • Test early! Test Often! Test Under lots of Conditions! • Static Electricity Sucks (installation is crazy) • Your Kiosk Can Never Be Too Stable (Bullet Proof Your Kiosks) • Logging, Logging, Logging • Buy Lots of Extras
  44. 44. Other stuff to hack! • Security/WiFi Cameras - http://www.axis.com • Tons of X10 Stuff - http://www.smarthome.com • Other Robots - http://www.spykeeworld.com/ • Rocket Launcher - http://www.thinkgeek.com/ geektoys/warfare/8bc4/?cpg=cj • Ultimarc Controls - http://www.ultimarc.com/
  45. 45. More info, code samples, etc. • http://electricpineapple.net/ - Erik Peterson’s blog • http://ionagroup.com/labs - Iona’s Blog • http://visualrinse.com - My Blog Questions?
  46. 46. Thank you • Find Us on: THE IONA GROUP 620 West Jackson Street • Morton, Illinois 61550 P: 309.263.4662 • F: 309.263.8262 • 888.644.IONA www.ionagroup.com

×