Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016

480 views

Published on

This year for the Salone del Mobile at frog, we came up with a funky experiment, based on Drones, Android, and VR. In this talk, your hosts will walk you through our Drone Race experiment, touching topics like real-time computer vision, reactive programming for mobile, indoor positioning and (wheeled) Drones hacking. The variety and complexity of these topics is equal to its coolness though, so you may be puzzled asking yourself: "Where do I start?" We will share experiences and lots of code, so that you can start right away.

Published in: Technology
  • Be the first to comment

Hacking for Salone: Drone Races - Di Saverio; Lippolis - Codemotion Milan 2016

  1. 1. HACKING FOR SALONE: DRONE RACES EMANUELE / SIMONE
  2. 2. EMANUELE DI SAVERIO PRINCIPAL DESIGN TECHNOLOGIST Emanuele started writing software for mobile in the pre-iPhone era. Didn’t stop since then. WHO’S ON THE STAGE SIMONE LIPPOLIS SENIOR DESIGN TECHNOLOGIST Simone started writing software when COBOL was a hype word. Now his interests span from web-bots to data-visualization to embedded applications and IOT.
  3. 3. FROG IS A GLOBAL DESIGN AND STRATEGY FIRM www.frogdesign.com
  4. 4. EVERYTHING STARTED ON A BLACK FRIDAY
  5. 5. WE NEEDED MORE ENTERTAINING TO MAKE IT
  6. 6. A CLASSIC GAME CAME IN OUR HELP 7
  7. 7. TODO LIST TALK TO EACH OTHER SENSE EACH OTHER HARDER, BETTER, FASTER, STRONGER
  8. 8. TALK TO EACH OTHER
  9. 9. COMPONENTS ARCHITECTURE • Drones on wheel (JumpingSumo by Parrot) • An open-security 5 Ghz Wi-Fi AP • Android Smartphones • A WebSocket Server Controller Mobile App JumpingSumo linux+busybox Game Server NodeJS Controller Mobile App JumpingSumo linux+busybox Controller Mobile App JumpingSumo linux+busybox WebSocket . .
  10. 10. The Parrot robots work by creating an 802.11ac themselves - smartphones can connect to that and control though applications built with the Parrot SDK. This disables internet access for the smartphones - we needed to “hack” it. HACKING CONNECTION $ telnet 192.168.1.1 accesses the BusyBox environment to edit config files /etc/init.d/init_manager and dragon.conf Drones will connect to your open WiFi at reboot, where they will find Android phones.
  11. 11. The server doesn’t need to provide much functionality, but rather route events in a soft- realtime fashion to the players, with litte computation. • WebSockets help avoid TCP RTT • nodeJS because it can scale to gazillions of events We have 4 players at at time :) A BROKER SERVER
  12. 12. SENSE EACH OTHER
  13. 13. WHERE IS MY DRONE? BLE BEACON/FINGERPRINTING IR MARKER TANGO
  14. 14. LET DRONES “SEE” The hardware platform features a front camera for still photos and streaming. Specs not great - we’ll need to design around. Camera Spec 640x480 15 FPS Fixed focus Motion JPEG
  15. 15. COLOR BLOBS Using OpenCV as resource for highly-optimized algorithm library, we tried to detect color blobs by • thresholding • finding contours • centroids and applying some heuristics
  16. 16. FIDUCIAL MARKERS We also res B&W Square markers are much more robust and can apply adequate Hamming-distance encodings. As an added benefit, they are more robust with respect to luminosity.
  17. 17. IMAGE PROCESSING PIPELINE PARROT SDK IMAGE
 RE-ENCODE ARTOOLKIT MARKER
 COORDINATES Each of the incoming frames from the drones has to go through heavy processing to detect a marker. 66 ms 3D MODEL ROTOTRASLATE
 COORDINATES
  18. 18. HARDER, BETTER, FASTER, STRONGER
  19. 19. OUT OF THE JAVALAND First problem relates to the sheer computation power needed to conver binary image formats - Java implementation is just too slow. We had to port algorithm implementation to RenderScript to have a fully native implementation that performs acceptably. JVM RS RS_MULTI AVG 283,60 1,73 3,24 STDDEV 38,17 2,49 1,52 MEDIAN 262,11 0,86 2,78 MAX 447,57 10,04 15,18
  20. 20. OUT OF THE GC LAND The main source of delays in real-time applications like games is the Garbage Collector - we’re talking half second freeze on recent hardware. This means we cannot allocate memory in the game loop - Android memory profiler tools will help you discover and eliminate: • Bitmap.create() • new Array[] • Object creations / Autoboxing
  21. 21. FULL CONTROL FLOW PARROT SDK
 (CAMERA) IMAGE
 RE-ENCODE ARTOOLKIT MARKER
 COORDINATES 3D MODEL ROTOTRASLATE
 COORDINATES GAME STATE NETWORK EVENTS GAS PEDAL USER INTERFACE STEERING
 (GYRO) BROKER
 SERVER OTHER PLAYERS ACTIONS
  22. 22. FULL CONTROL FLOW PARROT SDK IMAGE
 RE-ENCODE ARTOOLKIT MARKER
 COORDINATES 3D MODEL ROTOTRASLATE
 COORDINATES GAME STATE NETWORK EVENTS GAS PEDAL USER INTERFACE STEERING BROKER
 SERVER 66 ms 33 ms ~ 15 s 16 ms ~ 50 ms STEERING
 (GYRO) OTHER PLAYERS ACTIONS
  23. 23. Handling all the different events coming in at different rates would a very complex task to handle in a fully procedural way in a standard game loop: REACTIVE STREAMS FTW while (true) { acquire_image(); process_markers(); get_steer_input(); get_other_players_input(); apply_cmds_to_drone(); apply_game_events(); update_ui(); } . . . .
  24. 24. REACTIVE STREAMS FTW Rx provides common semanthics to express and manipulate a collection of asynchronous events
  25. 25. RXJAVA http://www.slideshare.net/hazam/programming-sideways-asynchronous-techniques-for-android (2014) Plenty of good introductions to RxJava in the InterWebs Exercises https://github.com/jhusain/learnrxjava
  26. 26. KOTLIN MAY BE THE ONE We may have found a winner. Kotlin is a deep language with many modern features, including Functional approaches. The three best features for Kotlin on Android: 1. Overhead is at compile time, Runtime Overhead is minimal 2. Closures and SAM 3. Java Interoperability is a breeze One things I’m still undecided if is good or bad: Nullability FUNCTIONAL REACTIVE PROGRAMMING WITH KOTLIN ON ANDROID - BY GIORGIO NATILI
  27. 27. REFERENCES https://github.com/frog/a-kart http://designmind.frogdesign.com/2016/07/ hacking-a-multi-game-user-drone-race/ SIMONE LIPPOLIS EMANUELE DI SAVERIO @simonelippolis @hazam GAME & UI DESIGN, SERVER DEVELOPMENT MOBILE DEVELOPMENT, COMPUTER VISION FEDERICO BRESSAN ELENA MARENGONI RACETRACK, DRONE SKIN DESIGN RACETRACK, DRONE SKIN DESIGN
  28. 28. THANKS!
  29. 29. THANKS! www.frogdesign.com

×