Almost anything that used to run on desktop, now runs in the browser and as of Atwood's law: anything that could be written in JavaScript, will eventually be written in JavaScript. If you have dared imagining to control your toys with code, communicate with the cloud and use advanced computer intelligence, your dreams have now become close at hand. This session is to challenge your fantasy and make you think what you could do with JavaScript. This session is about programming drones with JavaScript and AI capabilities.

1. Nov 23, 2019Sofia
var title = “Flying a Drone with JavaScript
and Computer Vision”;
var info = {
name: “Ivelin Andreev”,
otherOptional: “DJI Tello, Node-Red,
Cognitive Services, CustomVision.ai,
Scratch and DroneBlocks”};

2. Nov 23, 2019
• Software Architect @
o 17+ years professional experience
• Microsoft Azure MVP
• External Expert Horizon 2020, Eurostars-Eureka
• External Expert InnoFund Denmark, RIF Cyprus
• Business Interests
o Web Development, SOA, Integration
o IoT, Machine Learning, Computer Intelligence
o Security & Performance Optimization
• Contact
ivelin.andreev@icb.bg
www.linkedin.com/in/ivelin
www.slideshare.net/ivoandreev
Who is in front of you?

3. Nov 23, 2019
Thanks to our Sponsors:
General Sponsor:
Trusted Sponsor:
Technological Partners:
Hosting Partner:
Automotive Partner:
Innovation Sponsor:
Bronze Sponsors:Silver Sponsors:
Gold Sponsors:
In-Kind:
Strategic R&D Sponsor:

4. Nov 23, 2019
agenda();
• The Drone
• Node-Red
o Flows
o Dashboard
• MIT Scratch & DroneBlocks
• Computer Vision
• Custom Vision
• Demo

5. Nov 23, 2019
Let me Introduce you to DJI Tello
Ryze Robotics
• Tech startup established 2017 (Shenzhen, China)
• First product: DJI Tello, Tello Iron Man, Tello EDU
• Mobile app has 30’000 downloads/month
Reviews
• TechRadar: The smartest toy drone you’ll ever fly
• Digital Trends: Winner: Best drones under $100
Stand-out
• TelloPilots Community (9150+ members, 3500+ threads)
• SDK (UDP Raw, Go, C#, Swift, Python)
• Low-Level Protocol - community decoded
https://tellopilots.com/wiki/protocol/

6. Nov 23, 2019
Auto
Takeoff/Land
Autoland on
lost connection
Vision Positioning
System
5MP Photoes
(2592x1936)
Image
Stabilization
14-core Intel CPU
DJI Technology
VR headset
compatible
Ext. controller
compatible
13Min
Flight Time
100M
Flight Distance
8M/S
Max Speed
720P
1280p @30FPS
2Antennas
Stable Video
300M
With WiFi Booster
Tello Features

7. Nov 23, 2019
3+ Tello Apps
(Free, Paid & Edu)
Videos in 360,
Circle, Up & Away
Throw & Go
8D FlipsBounce Mode
Land on hand
DJI Tello Flight

8. Nov 23, 2019
Highly Customizable

9. Nov 23, 2019
Communication
• Commands: WiFi UDP (outgoing) 192.168.10.1:8889
• State: 192.168.10.1 -> 0.0.0.0:8890
• Image/Video: 192.168.10.1 -> 0.0.0.0:11111
Commands
• Safety: If no command for 15 sec, Tello lands automatically
• State: acc., bat., baro, time, h, temp, tof (time of flight), vgx/y/z (velocity)
• Control: forward 100; back 20; left 50; up 200; cw 360; ccw 180; flip l; emergency
• Set: speed 100, wifi ssid pass (EDU only: mon, moff)
• Read: speed?, battery? time? wifi?, sdk?, sn?
Tello SDK

10. Nov 23, 2019
Program Tello with DroneBlocks
What it Does
• Drag and drop coding on top of DJI SDK
• Ease of Use (Very High), Flexibility (Low)
• For iOS, Android and Chrome extension
Features
• Supports multiple drones (Tello, Mavic Air, Mavic Pro, Spark, Phantom 3/4)
• Create own custom flight logic and algorithms.
• Limitless autonomous mission possibilities
• Save and share missions
Setup
1. Download from https://www.droneblocks.io/app
2. Start from chrome://apps
3. Connect to Tello and design flow

11. Nov 23, 2019
What it Does
• Program interactive games and animations
• Designed by MIT for children 8-16, ease of Use (Med)
• For Windows and macOS
Features
• Create own custom flight logic and algorithms.
Setup
1. https://dl-cdn.ryzerobotics.com/downloads/tello/0222/Tello+Scratch+Readme.pdf
OR https://www.halfchrome.com/program-the-ryze-tello-with-scratch/
2. Requires Node.js server
3. Download Scratch v.2.0: http://llk.github.io/scratch-msi/
4. Tello files: https://dl-cdn.ryzerobotics.com/downloads/tello/20180222/Scratch.zip
5. Install Tello Nodes (SHIFT+File menu)
6. Run PS C:> node [path]Tello.js
MIT Scratch

12. Nov 23, 2019
What it Does
• Flow-based development tool (IBM, open source)
• Intended for wiring HW devices and APIs for IoT
• Ease of use (Med), Flexibility (High)
Features
• Visual and web-based programming tool
• Built on Node.js with JSON format flows
• Built-in nodes hide complexity
• Event-based (with JSON messages)
• Node-red dashboard for live data dashboards
Setup npm install node-red
Node-Red

13. Nov 23, 2019
• Four main sections
Node-Red Editor
1. Header – deploy button,
main menu, user menu
➀
➁ ➂ ➃
2. Palette – available nodes
3. Workspace – flow design area
4. Sidebar – Information,
debug, configuration, context
data

14. Nov 23, 2019
• Generate text from message with Mustache
• Evaluate rule and route message to branch
• Modify message properties with multiple operations:
Set, Change, Move, Delete
• JavaScript code to run on message and execute custom JS
• Print out debug messages in the debug sidebar
• JSON messages usually have payload property
Built-in Nodes
• Manual or automatic flow trigger on interval

15. Nov 23, 2019
• Modules for live data dashboards
• Elements are split into vertical groups
• Widget has the width of group by default
• Themes with minimal customizations
• Multiple individual users NOT supported.
• Widgets:
Node-Red Dashboard

16. Nov 23, 2019
• Noder-Red home (Windows): C:Users[UserName].node-red
• Flows stored in flows_[machinename].json
• Run:
• Node-red editor available at: http://localhost:1880
• Flows could be imported/exported from JSON, enabled/disabled
• Supports user/password authentication and OAuth/OpenID providers
• Subflows allow multiple nodes to be collapsed in a single workspace node
• Extensible
Menu > Manage palette
1. ~2300 available modules
2. Search module
3. Install module
PS C:> node-red
➀➁
➂
Node-Red Tips

17. Nov 23, 2019
NIST Face Recognition Vendor Test
(FRVT) 2019.09.11
https://www.nist.gov/programs-projects/face-
recognition-vendor-test-frvt-ongoing
Microsoft “recognition_02”
won 2019 1:N NIST FRVT
from 203 tested
algorithms, 41 vendors

18. Nov 23, 2019
Computer Vision
What it does
• Advanced algorithms for processing images for information
Use cases
• Detection of object and bounding box
• Description in natural language (try it out with NLP)
• Tag visual features (2000+ objects and actions)
• Categorization on 86 hierarchical categories (parent-child)
• Face detection with tagging of age, gender and bounding rectangle
• Domain – recognize domain members (celebrity, family, landmark)
• Read API in 22 languages, Handwriting - image, TIFF, PDF

19. Nov 23, 2019
Sample: Computer Vision for Object Detection
Cloud Vision
(IBM Watson) Computer Vision
(Microsoft)

20. Nov 23, 2019
Sample Face API Response

21. Nov 23, 2019
Computer Vision is an easy to
use pretrained image classifier.
Custom Vision is end-end AI
service for computer vision in
custom scenarios

22. Nov 23, 2019
customvision.ai – Why?
What it does
• End-end AI service for computer vision in custom scenarios
• Was preview in 2018; was 2x cheaper & less powerful; requires Azure resource
Use cases
• Image classification – assign tags to each image
• Object detection – return the coordinates of object in image
• Compact models execute on mobile device (ONNX, CoreML, Tensorflow)
Pricing
• Free (2 TPS, 2 projects, 5’000 images/project)
• Standard (10 TPS; 100 projects; 99.9% SLA, $2/1K req., $20/compute hour, storage $0.70/1K images)
API- and SDK-based (train, predict, test)

23. Nov 23, 2019
Domains (General, Food, Landmarks, Retail, Compact)
Create project
Upload and tag
• It is easier to upload by the tags (50+ images/tag)
• Use different angles, background, size and balance
Train the classifier
• As easy as
• Set training iteration as default
Performance
• Precision – TP / (TP + FP)
• Recall – TP / (TP + FN)
customvision.ai – What?

24. Nov 23, 2019
Prediction API Request
customvision.ai – How?
Negative image handling
• Resemblance: Automatic negative image handling (0% score expected)
• No resemblance: Build class “Other”
Prediction API Response

25. Nov 23, 2019
Hints
• Make low-level connect communication BEFORE init of SDK mode
• Low level and SDK commands can be mixed together - only if the first
command after power up is the low level init
Prerequisites
• Install Node.js (latest)
• Install node-red
• Start node-red (i.e. from PowerShell)
• Plugins: node-red-dashboard, node-red-node-base64, node-red-contrib-cognitive-services
• Open firewall UDP ports 52955, 11111 and 8890
DEMO
Npm install node-red

26. Nov 23, 2019
Takeaways
• TelloPilots DJI Tello Drone Forum (3500+ threads)
https://tellopilots.com/wiki/development/
• Sample Projects
https://github.com/johnwalicki/Node-RED-DroneViewer/tree/master/telloview
https://dev.to/poojamakes/detecting-objects-with-a-tello-drone-38np
https://developer.ibm.com/tutorials/program-ryze-dji-tello-drone-using-sdks-and-node-red/
https://wesbos.com/drone-javascript/
https://github.com/SovGVD/nodetello/
https://github.com/geaxgx/tello-openpose (Python, OpenCV)
• SDKs
https://tellopilots.com/wiki/development
• Microsoft Cognitive Services Online Demos
https://azure.microsoft.com/en-us/services/cognitive-services/directory/vision/
• Microsoft Computer Vision and Custom Vision - Sample Code
https://github.com/Azure-Samples/cognitive-services-javascript-computer-vision-tutorial

27. Nov 23, 2019
Thanks to our Sponsors:
General Sponsor:
Trusted Sponsor:
Technological Partners:
Hosting Partner:
Automotive Partner:
Innovation Sponsor:
Bronze Sponsors:Silver Sponsors:
Gold Sponsors:
In-Kind:
Strategic R&D Sponsor: