This year for the Salone del Mobile at frog, we came up with a funky experiment, based on Drones, Android, and VR. In this talk, your hosts will walk you through our Drone Race experiment, touching topics like real-time computer vision, reactive programming for mobile, indoor positioning and (wheeled) Drones hacking. The variety and complexity of these topics is equal to its coolness though, so you may be puzzled asking yourself: "Where do I start?" We will share experiences and lots of code, so that you can start right away.

1. HACKING FOR SALONE:
DRONE RACES
EMANUELE / SIMONE

2. EMANUELE DI SAVERIO
PRINCIPAL DESIGN TECHNOLOGIST
Emanuele started writing software for mobile in the pre-iPhone
era. Didn’t stop since then.
WHO’S ON THE STAGE
SIMONE LIPPOLIS
SENIOR DESIGN TECHNOLOGIST
Simone started writing software when COBOL was a hype word.
Now his interests span from web-bots to data-visualization to
embedded applications and IOT.

3. FROG IS A GLOBAL
DESIGN AND
STRATEGY FIRM
www.frogdesign.com

5. EVERYTHING STARTED
ON A BLACK FRIDAY

6. WE NEEDED
MORE
ENTERTAINING
TO MAKE IT

7. A CLASSIC GAME
CAME IN OUR HELP
7

11. TODO LIST
TALK TO EACH OTHER
SENSE EACH OTHER
HARDER, BETTER,
FASTER, STRONGER

12. TALK TO
EACH OTHER

13. COMPONENTS
ARCHITECTURE
• Drones on wheel (JumpingSumo by Parrot)
• An open-security 5 Ghz Wi-Fi AP
• Android Smartphones
• A WebSocket Server
Controller
Mobile App
JumpingSumo
linux+busybox
Game Server
NodeJS
Controller
Mobile App
JumpingSumo
linux+busybox
Controller
Mobile App
JumpingSumo
linux+busybox
WebSocket
. .

14. The Parrot robots work by creating an 802.11ac
themselves - smartphones can connect to that and
control though applications built with the Parrot SDK.
This disables internet access for the smartphones - we
needed to “hack” it.
HACKING CONNECTION
$ telnet 192.168.1.1
accesses the BusyBox environment to edit config
files /etc/init.d/init_manager and
dragon.conf
Drones will connect to your open WiFi at reboot, where
they will find Android phones.

15. The server doesn’t need to provide much
functionality, but rather route events in a soft-
realtime fashion to the players, with litte
computation.
• WebSockets help avoid TCP RTT
• nodeJS because it can scale to gazillions of
events
We have 4 players at at time :)
A BROKER
SERVER

16. SENSE
EACH OTHER

17. WHERE IS MY DRONE?
BLE BEACON/FINGERPRINTING IR MARKER
TANGO

18. LET DRONES “SEE”
The hardware platform features a front camera for
still photos and streaming.
Specs not great - we’ll need to design around.
Camera Spec
640x480
15 FPS
Fixed focus
Motion JPEG

19. COLOR BLOBS
Using OpenCV as resource for highly-optimized
algorithm library, we tried to detect color blobs by
• thresholding
• finding contours
• centroids
and applying some heuristics

20. FIDUCIAL MARKERS
We also res
B&W Square markers are much more
robust and can apply adequate
Hamming-distance encodings.
As an added benefit, they are more
robust with respect to luminosity.

22. IMAGE PROCESSING PIPELINE
PARROT SDK
IMAGE
RE-ENCODE
ARTOOLKIT
MARKER
COORDINATES
Each of the incoming frames from the drones has
to go through heavy processing to detect a marker.
66 ms
3D MODEL
ROTOTRASLATE
COORDINATES

23. HARDER,
BETTER,
FASTER,
STRONGER

24. OUT OF THE JAVALAND
First problem relates to the sheer computation power
needed to conver binary image formats - Java
implementation is just too slow.
We had to port algorithm implementation to
RenderScript to have a fully native implementation
that performs acceptably.
JVM RS RS_MULTI
AVG 283,60 1,73 3,24
STDDEV 38,17 2,49 1,52
MEDIAN 262,11 0,86 2,78
MAX 447,57 10,04 15,18

25. OUT OF THE GC LAND
The main source of delays in real-time applications
like games is the Garbage Collector - we’re talking
half second freeze on recent hardware.
This means we cannot allocate memory in the game
loop - Android memory profiler tools will help you
discover and eliminate:
• Bitmap.create()
• new Array[]
• Object creations / Autoboxing

26. FULL CONTROL FLOW
PARROT SDK
(CAMERA)
IMAGE
RE-ENCODE
ARTOOLKIT
MARKER
COORDINATES
3D MODEL
ROTOTRASLATE
COORDINATES
GAME STATE
NETWORK EVENTS
GAS PEDAL
USER INTERFACE
STEERING
(GYRO)
BROKER
SERVER
OTHER
PLAYERS
ACTIONS

27. FULL CONTROL FLOW
PARROT SDK
IMAGE
RE-ENCODE
ARTOOLKIT
MARKER
COORDINATES
3D MODEL
ROTOTRASLATE
COORDINATES
GAME STATE
NETWORK EVENTS
GAS PEDAL
USER INTERFACE
STEERING
BROKER
SERVER
66 ms
33 ms
~ 15 s
16 ms
~ 50 ms
STEERING
(GYRO)
OTHER
PLAYERS
ACTIONS

28. Handling all the different events coming in at different
rates would a very complex task to handle in a fully
procedural way in a standard game loop:
REACTIVE STREAMS FTW
while (true) {
acquire_image();
process_markers();
get_steer_input();
get_other_players_input();
apply_cmds_to_drone();
apply_game_events();
update_ui();
}
.
.
.
.

29. REACTIVE STREAMS FTW
Rx provides common semanthics to express and
manipulate a collection of asynchronous events

30. RXJAVA
http://www.slideshare.net/hazam/programming-sideways-asynchronous-techniques-for-android
(2014)
Plenty of good introductions to RxJava in the InterWebs
Exercises
https://github.com/jhusain/learnrxjava

31. KOTLIN MAY BE THE ONE
We may have found a winner.
Kotlin is a deep language with many modern features, including
Functional approaches.
The three best features for Kotlin on Android:
1. Overhead is at compile time, Runtime Overhead is minimal
2. Closures and SAM
3. Java Interoperability is a breeze
One things I’m still undecided if is good or bad: Nullability
FUNCTIONAL REACTIVE PROGRAMMING WITH KOTLIN ON ANDROID - BY GIORGIO NATILI

33. REFERENCES
https://github.com/frog/a-kart
http://designmind.frogdesign.com/2016/07/
hacking-a-multi-game-user-drone-race/
SIMONE LIPPOLIS
EMANUELE DI SAVERIO
@simonelippolis
@hazam
GAME & UI DESIGN, SERVER DEVELOPMENT
MOBILE DEVELOPMENT, COMPUTER VISION
FEDERICO BRESSAN
ELENA MARENGONI
RACETRACK, DRONE SKIN
DESIGN
RACETRACK, DRONE SKIN
DESIGN

34. THANKS!

35. THANKS!
www.frogdesign.com