This is a fun one! Learn how to hack up robots you can buy at a local toy store. You’ll see the methods used to take the video stream out of the robot and turn it into a format Flash likes. You’ll get the lowdown on how to send API commands to control the bot. We’ll show you how to connect it to alternative controllers and use ActionScript for some simple color detection on the video stream.
Lilac Illustrated Social Psychology Presentation.pptx
Hacking Robots for Fun and Profit
1. Hacking Robots for Fun and Profit
presented by Chad Udell
at Flash and the City, 2010
2. Hacking Robots for Fun and Profit
• Flash is eminently hackable and ideal for building UIs
for experiences that integrate with hardware
• Tons of devices are coming with web connections
• Hardware is getting cheaper and easier to get
• Hobbyist techniques can be adapted for real world
9. Basic Limitations
• Skillset - We aren’t robotics engineers or electricians
• Budget - Every client is conscious of this - duh
• Scope - Need to integrate a timer, controls, basic
computer vision, long term maintenance solution
• Quality - Uptime is crucial. “Out of Order” is not an
option
10. Choosing a Robot
• Essentially we need a connected webcam
• Needs to be cheap
• Needs to be hackable
Enter, Rovio.
11. A Diagram of the system
2 Stations, each containing:
!"#$%&'()*+,-./+*0)
UPS (APC BR1200)
APC AV C2
Happ Kiosk Amp
and Speakers
1"234&+56(!11(.7
Rovio Charger
Happ
Controls
Server (Red 5)
Netgear Wireless Access Point
WG-102
Switch
Rovio
12. Rovio to the Rescue
• Skillset - Essentially like building a mashup
(documented API, built in webserver)
• Budget - They are inexpensive and readily available
(important for uptime)
• Scope - many of the requirements are handled in
some built in fashion already (wayfinding, charging).
• Quality - We have a company backing it
13. Flash to the Rescue
• Skillset - We like it.
• Budget - No extra expensive hardware to be
designed, no expensive proprietary SDKs.
• Scope - Still some questions here. Going to have to
manage the client’s expectations.
• Quality - Debugging and remote config is easy
14. Controlling the Rovio
• Rovio responds via HTTP (CGI script)
• Use of key listeners to trigger API calls.
/rev.cgi?Cmd=nav&action=value&drive=d_value&speed=s_value Input
Parameter
value = 18 d_value = 0 (Stop)
1 (Forward)
2 (Backward)
3 (Straight left)
4 (Straight right)
5 (Rotate left by speed)
6 (Rotate right by speed)
7 (Diagonal forward left)
8 (Diagonal forward right)
9 (Diagonal backward left)
10 (Diagonal backward right)
11 (Head up)
12 (Head down)
13 (Head middle)
14 (Reserved)
15 (Reserved)
16 (Reserved)
17 (Rotate left by 20 degree angle increments)
18 (Rotate right by 20 degree angle increments)
s_value = 1 (fastest) – 10 (slowest)
16. Getting the Video
• Rovio sends via MJPEG via HTTP (CGI script)
• Set a loader’s source to the URL
Camera Control – GetData.cgi
Description
The basic command for acquiring MJPEGGrammar
/ChangeResolution.cgi?ResType=value[&RedirectUrl=sUrl] Input Parameter
None
Privilege
None
Return Value
An instance captured motion image.
• Results were marginal (12-15 fps)
18. Deeper Research on Video/Image Processing
Simple color detection -
• http://www.interactionfigure.nl/2009/colour-tracking-with-the-webcam-in-flash/
Color detection in Processing
• http://jamesalliban.wordpress.com/2008/11/16/colour-detection-in-processing/
Haar training for object detection
• http://note.sonots.com/SciSoftware/haartraining.html
Modified facial detection
• http://www.quasimondo.com/archives/000687.php
19. Technology Evaluations
• Shopping = Fun
• Reading Electrical Engineer Documents = Not so Fun
• Wiring Boards = Fun for some
• Building PoCs = Super fun!
24. Alpha
• Video was ok, but needed some juice.
• Computer vision was deemed better than AR or
object detection
• Play control overall was very pleasing
• Testing, Testing, Testing
26. Enter Red 5/Xuggle
• Transcode the Video to H264 (SteamStream)
• Replace the MJPEG loader with a Netstream object
• Better results. Lower latency.
• Approaching 24-30fps
• There is a bit of a hiccup with the initialization of the
stream.
28. Processing the video
• Considered AR
• Too Inflexible - Markers weren’t attractive
• Considered Object Detection (Haar Cascades/
OpenCV)
• A tad slow-ish
• Attempted using simple ColorBoundsRect
33. How about Creating a Unique Target?
• Eliminate False Positives (was better, but not perfect)
• Ensure Speedy Detection (was a little slower)
Not Unique A Lot More Unique
34. Processing the video
• Settled on Color Detection via Thresholding
• Tomek’s Blob Detection
• http://play.blog2t.net/fast-blob-detection
• Soulwire’s Color Averaging
• http://blog.soulwire.co.uk/code/actionscript-3/
extract-average-colours-from-bitmapdata
42. Buttoning it up
• Charging handoff
• Locking the kiosks down
• C++ Socket Server for the Lights
• Startup, shutdown
• Training for the switchover
43. Lessons Learned
• SWF to EXE is not needed anymore. AIR 2!
• Test early! Test Often! Test Under lots of Conditions!
• Static Electricity Sucks (installation is crazy)
• Your Kiosk Can Never Be Too Stable (Bullet Proof
Your Kiosks)
• Logging, Logging, Logging
• Buy Lots of Extras
44. Other stuff to hack!
• Security/WiFi Cameras - http://www.axis.com
• Tons of X10 Stuff - http://www.smarthome.com
• Other Robots - http://www.spykeeworld.com/
• Rocket Launcher - http://www.thinkgeek.com/
geektoys/warfare/8bc4/?cpg=cj
• Ultimarc Controls - http://www.ultimarc.com/
45. More info, code samples, etc.
• http://electricpineapple.net/ - Erik Peterson’s blog
• http://ionagroup.com/labs - Iona’s Blog
• http://visualrinse.com - My Blog
Questions?
46. Thank you
• Find Us on:
THE IONA GROUP
620 West Jackson Street • Morton, Illinois 61550
P: 309.263.4662 • F: 309.263.8262 • 888.644.IONA
www.ionagroup.com