Rapidly Testing ML-Driven Drone Applications - The FlockAI Framework

2/23/23 1
Demetris Trihinas
trihinas.d@unic.ac.cy
1
FlockAI: Rapidly Testing ML-Driven Drone Applications
Department of
Computer Science
Rapidly Testing ML-Driven Drone
Applications
The Framework
Dr. Demetris Trihinas
Department of Computer Science
ailab @ University of Nicosia

2/23/23 2
Demetris Trihinas
2
Department of
Computer Science
“Designing and developing scalable and self-adaptive tools for data analysis by
exploring the intersection between data management, distributed systems and
machine learning”
trihinas.d@unic.ac.cy http:///dtrihinas.info dtrihinas
Dr. Demetris Trihinas
Lecturer at University of Nicosia
Artificial Intelligence Laboratory (AILab)
Open and trusted fog computing
platform that facilitates the
deployment of scalable and
heterogeneous IoT services
Enabling power-efficient Machine
Learning and its applications to
drone technology for handling
time-critical missions
Bridging the early diagnosis and
treatment gap of brain diseases
via smart, connected, proactive
and evidence-based technology
https://ailab.unic.ac.cy/

2/23/23 3
Demetris Trihinas
3
Department of
Computer Science
Drones - Unmanned Aerial Vehicles (UAVs)
• Aircraft without a human pilot.
• Unmanned Aerial System
• Drones, ground control, communication
protocols and optionally satellites
• Degree of autonomy
• Remote-controlled
• Autonomous via on-board computer

2/23/23 4
Demetris Trihinas
4
Department of
Computer Science
The Drone Era…

2/23/23 5
Demetris Trihinas
5
Department of
Computer Science
…largest stake of
funding for
commercial drone tech
(US) is for on-board
data mgmt and
analysis empowered
by AI/ML…

2/23/23 6
Demetris Trihinas
6
Department of
Computer Science
https://www.economist.com/britain/2022/07/21/a-superhighway-for-drones-to-fly-between-british-cities

2/23/23 7
Demetris Trihinas
7
Department of
Computer Science
Intelligence in the Sky
New and emerging applications

2/23/23 8
Demetris Trihinas
8
Department of
Computer Science
Contributing Factors
Drone Hardware
sensors, cameras,
processors, batteries,
communication modules,
…
Artificial
Intelligence
vision, mapping,
navigation, object
detection, tracking,
…
Edge
Computing
Low-power
network protocols,
in-place data
processing, task
and data
offloading,
…
SkyAI
The “Edge”
physical and/or network distance
Less load on
centralized services
LAN/WAN
(one hop away)
Cloudlet
MEC
μ-datacenter
Internet

2/23/23 9
Demetris Trihinas
9
Department of
Computer Science
Precision Agriculture
Weeds
Pests and Weeds
Pests and Weeds
Data collection
plant height, foliage,
canopy size, NDVI
Field mapping
Weeds and pest
detection
Drone Swarm
decision-makers and
sprayers

2/23/23 10
Demetris Trihinas
10
Department of
Computer Science
Crowd Detection
VisDrone 2020 dataset
CNN pre-trained model

2/23/23 11
Demetris Trihinas
11
Department of
Computer Science
Face Detection and Recognition
Image from DroneFaces dataset
Python face_recognition library

2/23/23 12
Demetris Trihinas
12
Department of
Computer Science
Object Tracking and Change Detection

2/23/23 13
Demetris Trihinas
13
Department of
Computer Science
Testing ML-Driven Drone
Applications

2/23/23 14
Demetris Trihinas
14
Department of
Computer Science
Challenges
• Despite great potential of AI/ML algorithms, these are not optimal for dealing
with drones out-of-the-box.
• Why?
• AI/ML are compute-hungry.
• Drones have modest compute
power and are constrained
by their battery.
Log-scale
The
hardware
acceleration
era begins…

2/23/23 15
Demetris Trihinas
15
Department of
Computer Science
Challenges
• Quick fix solution… add a more powerful processor?
• Stronger processors require more power… compute still ends up impacting flight time
• No on-board processing is needed… just stream the data to the cloud.
• Communication is also a significant energy drain
• Also, waiting for a remote answer can lead to catastrophic results for the task
assigned to the drone (e.g., collision).

2/23/23 16
Demetris Trihinas
16
Department of
Computer Science
Power Consumption
• Data processing and dissemination key energy drains in embedded and
mobile devices.
• IoT devices are usually battery-powered which means intense processing
leads to less battery-life…
Low-Cost Adaptive Monitoring Techniques for the Internet of Things. D. Trihinas, G. Pallis and M. D. Dikaiakos, IEEE Transactions on Services Computing, 2018.

2/23/23 17
Demetris Trihinas
17
Department of
Computer Science
ML Comes in Many Different Forms…
More than one model architecture can be used for a task, but the impact on
resource utilization and energy footprint can differ significantly.
TABLE I: Performance comparison of various ML model architectures during inference
Model
Params
(Million)
Size
(MB)
Accuracy
(%)
Mean Inference Time
(ms)
Energy Consumption
(Wh per 5K images)
ResNet50 25.6 99 73 1116 373
SqueezeNet 1.25 5 52 212 51
MobileNetV2 3.4 14 70 143 39
TABLE II: Compute availability impact on ML inference
Device Cores
Mean Inference Time
(ms)
Mean Power Drawn
(W)
Server 3 432 87
6 210 91
9 152 139
12 143 197
RPi 1 693 4.2
2 331 5.1
3 273 5.2
4 269 5.9
oT testing tools. Configuring appropriate power models can
quickly become a larger configuration effort for users of IoT
positions of energy mixes, or, can be a forecast, when testing
for instance scenarios based on the availability of renewable
energy within the next day. Similar to power models, this wil
need to be configured with testing results dependent on these
configurations. Moreover, infrastructures might be supplied by
multiple power sources. Computing infrastructure could, fo
example, be powered by on-site renewable energy sources such
as wind or solar, while also being connected to a public energy
grid. This would make it even harder to configure where
the energy for computational and communication resources is
coming from at any given time and, thus, how much emissions
are associated with any energy consumption.
Towards Energy and Carbon Footprint Testing for AI-driven IoT Services. D. Trihinas, L. Thamsen, J. Beilharz, M. Symeonides, IEEE IC2E 2022.

2/23/23 18
Demetris Trihinas
18
Department of
Computer Science
ML on Drones
• Will it run on my resource-constraint
drone?
• How will it affect energy consumption
and battery autonomy?
• Trade-off between running on-device vs remote inference?
• Could a different (less-intensive) algorithm work by trading
accuracy for flight time?
I have no
idea…
From VisDrone 2020 dataset

2/23/23 19
Demetris Trihinas
19
Department of
Computer Science
The FlockAI Testing Suite

2/23/23 20
Demetris Trihinas
20
Department of
Computer Science
Requirements for a Testing Suite
• Drones come in different flavors
• Resource configurations (i.e., compute, memory, disk)
• Sensing modules (i.e., camera, gps)
• Drones penetrated many application domains
• Large learning curve inhibits adoption
• Drone KPIs entail system, energy and ML metrics
• Repeatable measurements under many cases
Modular Drone SDK
“Ready-to-Go” Templates
ML-Driven Drone Apps
Build on top of Webots
Compute, Memory,
Network, Flight, Energy
and ML Probes
”What-If” Scenario Builder

2/23/23 21
Demetris Trihinas
21
Department of
Computer Science
FlockAI High-Level Architecture

2/23/23 22
Demetris Trihinas
22
Department of
Computer Science
Webots Robotics Simulator
• Open-source 3D robotics simulator.
• Design graphic simulation environments,
denoted as worlds.
• Development toolkit for C++, Java and
Python to program robot control,
denoted as controllers.
• Controller interface can be extended.
https://cyberbotics.com/

2/23/23 23
Demetris Trihinas
23
Department of
Computer Science
Webots Robotics Simulator
• The FlockAIController
hides the low-level interaction of
the simulated drone with the
underlying physics engine.
• Abstractions for drone takeoff,
hovering, landing.

2/23/23 24
Demetris Trihinas
24
Department of
Computer Science

2/23/23 25
Demetris Trihinas
25
Department of
Computer Science

2/23/23 27
Demetris Trihinas
27
Department of
Computer Science
Drone Configurations
• Configure simulated robot so that it resembles actual drone.
• Weight, motor efficiency
• Resource configurations
• Compute, memory, flash storage, battery, wifi, Bluetooth
• Communication
• Wifi, Bluetooth
• Drone-to-Base station, drone broadcasting, drone-to-drone

2/23/23 28
Demetris Trihinas
28
Department of
Computer Science
Drone Configurations
• Sensors and motors
• Propellers, Camera, GPS, gyroscope
• Drone configuration from scratch is tedious, time-consuming and should be
avoided unless you are a drone manufacturer.
• Ready-to-use drone templates!
• DJI Mavic 2, DJI Tello, …

2/23/23 29
Demetris Trihinas
29
Department of
Computer Science
Drone
Configurations

2/23/23 30
Demetris Trihinas
30
Department of
Computer Science
Drone Piloting
• Keyboard-Based
• Autopilot-Based
• Free or within bounded box
• Different navigation (i.e., straight line, zig-zag, random, …)
• [new!] Swarming (leader election)

2/23/23 31
Demetris Trihinas
31
Department of
Computer Science

2/23/23 32
Demetris Trihinas
32
Department of
Computer Science
Experiment Scenario
Builder
• Drone
• Monitoring Probes
• ML
• Workload
• Actions
• Data collection and exporting

2/23/23 33
Demetris Trihinas
33
Department of
Computer Science
ML Algorithm Deployment
• Requirements:
• Training model must be on-boarded or made
remotely accessible
• Runtime monitoring must be accessible
• Inference (periodically or on-demand)
embracing the updated model parameterization.
Furthermore, FlockAI is designed to support the on-boar
ML algorithms, while inference needs not be applied in the sa
note that both the input and the response of the inference task
formats including raw form, csv, or to accommodate the most
in Python, as numpy’s ndarrays and/or pandas’ dataframe
Listing 3: Custom Classifier from FlockAI
1 from flockai.drones.ml import FlockAIClassifier
2 ...
3
4 class CrowdDetectionClassifier ( FlockAIClassifier )
5
6 def__init__ (self , inf_period =10):
7 super ().__init__ ()
8 self. inf_period = inf_period
9 ...
10
11 def load_model (self , model_path , params ,...)
12 ...
13
14 def inference(self , testdata)
15 ...
16 return prediction
17
18 ...
4.4. Energy Profiling
The following provides a comprehensive overview of
parameters one must consider when attempting to obtain en
Through this brief overview of the state-of-the-art in drone
show that there are a plethora of modeling approaches and
must be understood and obtained, while most models will n
• ML SDK flockai.drones.ml
• Base classes for classification, clustering, regression.
• Stay tuned for evolutionary algorithms (i.e., PCO)

2/23/23 34
Demetris Trihinas
34
Department of
Computer Science
Energy Modeling and Profiling (the HornEt toolkit)
• Composable modeling for multi-grain
energy reporting.
• Reusable template repository for
specific drones.
Fig. 1: High-Level and Abstract Overview of HornEt
I/O (e.g., writing to flash drive), while Pss denotes the power
driving peripheral sensing devices with essentially only the
image/video unit consuming measurable energy.
Finally, when the drone simulation starts, the energy model
can be continuously updated at runtime by “pushing” to
HornEt monitoring data. To achieve this, currently, HornEt
accepts data through the monitoring hooks (denoted as probes)
of FlockAI, albeit, if the integration endpoint is implemented,
any other monitoring solution can be used. In regards to
exporting energy measurements, HornEt can print data on the
console of the Webots (stdout/stderr) during runtime and/or
format and export the entire experiment log in plain text or
formatted with both json and pandas dataframes supported.
B. Implementation
Section II gives a brief, but elaborative, view on the
complexity of energy modeling for drones. As shown, imple-
menting such a model requires the definition of a plethora of
parameters and the measurement of the utilization of several
resources. In order to provide a simple “jump-start” energy
Composable Energy Modeling for ML-Driven Drone Applications. D. Trihinas, M. Agathocleous, K. Avogian, IEEE IC2E 2021.

2/23/23 35
Demetris Trihinas
35
Department of
Computer Science
Drone Energy Modeling
consumption for drone technology. A more general model for
a multi-copter drone can be defined as follows. Specifically,
the total energy consumption of a multi-copter drone at the
tth
time interval is denoted as Et and can be modeled by the
following equation:
Et = Emotors + Ecomm + Eproc (2)
In this equation, Emotors denotes the energy consumption
attributed to the mechanical parts tasked with “flying” the
drone, including the electrical motors and rotating propellers.
Naturally, this is the prominent energy draining factor, with
the literature referencing that Emotors may even attribute
to 95% of Et for a drone entrusted with a simple flying
task, and therefore, some consider that Et ' Emotors [9].
However, this does not suffice when a drone is entrusted with
an AI-driven task and an open link is established for real-
time data dissemination. In this case, both Eproc, denoting the
energy consumed for compute tasks (including fligh control
and image/video capture), and Ecomm, the energy consumed
for data exchange (including control messages), cannot be
considered as negligible [10].
Unweaving Equation 2, the energy consumed by the elec-
trical motors, Emotors, to fly the drone can be modeled as:
From
of a d
In t
transm
Simila
a rece
a tran
idle w
signifi
tidle,
be ex
mobile
the ne
the en
In a
drone’
Ep
In th
import DJIMavic2
t
mport EnergyProbe
osPilot(...)
pilot, probes=mon)
lation
Energy").values()
)
s")
ing")
cation")
, eComm)
Fig. 2: Energy Model Dentrogram for Listing 2
In order to avoid having to manually configure multiple
power and/or voltage/current levels just for a ML practitioner
to be able to “see” results on his/her emulator, HornEt comes
Cant ignore with ML
import DJIMavic2
t
mport EnergyProbe
osPilot(...)
pilot, probes=mon)
lation
Energy").values()
)
s")
ing")
cation")
, eComm)
Emotors = Etakeoff + Ehover + Emove + Elanding (3)
In this equation, Etakeoff is the energy attributed to the
drone lifted in the air till it enters a hovering state, and
similar Elanding denotes the energy consumed for the drone
to be safely grounded. In turn, Ehover is the energy attributed
to keeping the drone in the air and is dependent to both
drone and air characteristics, such as the drone mass and
air density. What is more, Emove is the (kinetic) energy
required to set a drone in motion from a hovering state.
1 https://cyberbotics.com/
mobile embedded systems such as the energy consumed by
the network unit if it can be placed in sleep mode along with
the energy consumed for state transitioning.
In a similar fashion, the compute overhead, Eproc, of a
drone’s digital computing systems can be modeled as:
Eproc = Pfc · tfc + Psoc · tsoc + Pio · tio + Pss · tss (7)
In this equation, Pfc denotes the power level of the utilized
flight controller (e.g., 8W for DJI A3), while Psoc denotes the
power utilized by system-on-chip units entrusted with domain-
specific tasks such as the ML-driven algorithms (e.g., object
detection) deployed by users. The Psoc component can be
further decomposed into the power level for when the compute
unit is in active state and the minimal power consumed
when idle. We particularly note that SoC power levels can
drastically differ, where a Raspberry Pi microcontroller will
feature in active state a power level of approximately 8W,
while a Nvidia Jetson Nano with an on-board gpu unit will
consume 57W, respectively. Finally, Pio denotes the power for
landing occur only once during the mission, then the
ng simplification can be considered:
Emotors ' Ehover (4)
to calculate the energy consumed for a drone to hover,
ote that energy consumption is equal to the power to
multiplied by the time in the hovering state Et = P · t.
lowing depicts the power to keep a drone in a hovering
hich is dependent to the mass (m) of the drone, area of
peller (A), air density (⇢), gravity (g) and the drones’
fficiency (⌘).
Phover =
( 2
⇢·A )1/2
· (m · g)3/2
⌘
(5)
Equation 2, the maximum communication overhead
one can be model as:
Ecomm = Px · tx + Pr · tr + Pidle · tidle (6)
is equation, Px is the max power attributed to data
ssion and tx is the time in the transmission state.
ly, Pr and tr are the max power and time consumed in
ving state. When the communication unit is neither in
mitting or receiving state, but still powered on (e.g.,
iting for messages) then it consumes power, albeit
antly less, and is modeled with the values Pidle and
espectively. In more complex settings, Ecomm can
nded to also include components found in general
embedded systems such as the energy consumed by
work unit if it can be placed in sleep mode along with
rgy consumed for state transitioning.
…
A lot of params to
configure…
import DJIMavic2
t
mport EnergyProbe
osPilot(...)
pilot, probes=mon)
lation
Energy").values()
)
s")
ing")
cation")
, eComm)
factor between [0, inf).
Et = tq ⇤ C (1)
Despite it’s “simplicity”, one can easily argue that such a
model ignores significant aspects of what beholds as energy
consumption for drone technology. A more general model for
a multi-copter drone can be defined as follows. Specifically,
the total energy consumption of a multi-copter drone at the
tth
time interval is denoted as Et and can be modeled by the
following equation:
hover multiplied by the time in the hovering state Et
The following depicts the power to keep a drone in a h
state, which is dependent to the mass (m) of the drone,
the propeller (A), air density (⇢), gravity (g) and the
motor efficiency (⌘).
Phover =
( 2
⇢·A )1/2
· (m · g)3/2
⌘
From Equation 2, the maximum communication ov
of a drone can be model as:
Ecomm = Px · tx + Pr · tr + Pidle · tidle
In this equation, Px is the max power attributed
transmission and tx is the time in the transmission
Similarly, Pr and tr are the max power and time consu
a receiving state. When the communication unit is ne
a transmitting or receiving state, but still powered o
idle waiting for messages) then it consumes power
significantly less, and is modeled with the values Pi
tidle, respectively. In more complex settings, Ecom
be extended to also include components found in
mobile embedded systems such as the energy consum
the network unit if it can be placed in sleep mode alo
In a similar fashion, the compute overhead, Epro
following equation:
Emotors = Etakeoff + Ehover + Emove + Elanding (3)
In this equation, Etakeoff is the energy attributed to the
drone lifted in the air till it enters a hovering state, and
similar Elanding denotes the energy consumed for the drone
to be safely grounded. In turn, Ehover is the energy attributed
of a drone can be model as:
Ecomm = Px · tx + Pr · tr + Pidle · tidle (6)
In this equation, Px is the max power attributed to data
transmission and tx is the time in the transmission state.
Similarly, Pr and tr are the max power and time consumed in
a receiving state. When the communication unit is neither in
a transmitting or receiving state, but still powered on (e.g.,
idle waiting for messages) then it consumes power, albeit
significantly less, and is modeled with the values Pidle and
tidle, respectively. In more complex settings, Ecomm can
be extended to also include components found in general
mobile embedded systems such as the energy consumed by
the network unit if it can be placed in sleep mode along with
In a similar fashion, the compute overhead, Eproc, of a
Eproc = Pfc · tfc + Psoc · tsoc + Pio · tio + Pss · tss (7)
In this equation, Pfc denotes the power level of the utilized
flight controller (e.g., 8W for DJI A3), while Psoc denotes the
power utilized by system-on-chip units entrusted with domain-
specific tasks such as the ML-driven algorithms (e.g., object
detection) deployed by users. The Psoc component can be
further decomposed into the power level for when the compute
unit is in active state and the minimal power consumed
What if I want to ignore params of decompose
into deeper levels?
air characteristics, such as the drone mass and air density.
c) energy required to set a drone in motion from a hovering
by Marins et al. [27], Emove can simply be represented as
tly—affecting Ehover when acceleration is imposed to the
weather conditions). So, assuming that movement occurs
g that the energy consumed for movement is only a fraction
ng for stationary hovering, and that the takeoff/landing
ion, then the following simplification can be considered:
Emotors ' Ehover (4)
gy consumed for a drone to hover, we denote that energy
wer to hover multiplied by the time in the hovering state
s the power to keep a drone in a hovering state, which is
he drone, area of the propeller (A), air density (r), gravity
ency (h).
hover =
( 2
r·A )1/2 · (m · g)3/2
h
(5)
mmunication overhead of a drone can be model as:
m = Px · tx + Pr · tr + Pidle · tidle (6)
ower attributed to data transmission and tx is the time in
Pr and tr are the power and time consumed in a receiving
unit is neither in a transmitting or receiving state, but still
or messages) then it consumes power, albeit significantly

2/23/23 36
Demetris Trihinas
36
Department of
Computer Science
Composable Energy Modeling
15 eMotors = metrics.get("motors")
16 eProc = metrics.get("processing")
17 eComm = metrics.get("communication")
18 print(eTotal, eMotors, eProc, eComm)
19 ...
20 mydrone.term()
Listing 1: Exemplary FlockAI Blueprint in Python
1 from hornet.energy import EnergyModel, Etotal
2 from hornet.energy.processing import Eproc
3 ...
4
5 def initEnergyModel():
6 ...
7 model = EnergyModel()
8 model.addComponents([EMotors(),EProc(),EComm()])
9
10 custProc = EProc(name="custom-eproc")
11 custProc.removeComponents(["sensors","io"])
12 efc = custProc.getComponent("fc")
13 efc.setdesc("DJI A3 flight controller")
14 efc.setpower(8.3)
15 esoc = custProc.getComponent("soc")
16 esoc.getComponent("cpu-active").setpower(8)
17 esoc.getComponent("cpu-idle").setpower(4)
18
19 model.updateComponent("proc", custProc)
20 ...
21 return model
22 ...
23 mymodel = initEnergyModel()
24 mydrone.attachEnergyModel(mymodel)
Listing 2: Configuring and Attaching HornEt Model to Drone
with a number of pre-filled energy profiles for a number of
popular drones. In the current release these include drones
from the DJI drone manufacturer. Hence, if a user opts to
use a readily available energy profile then the configuration
presented in Listing 2 can be skipped or reduced just to
a couple of lines targeting any specific component the user
wishes to amend.
IV. EVALUATION
This section provides a comprehensive study demonstrating
the main contributions of our work by introducing a realistic
use-case under-going two different deployments so that a ML
practitioner can truly assess the energy consumption KPIs of
his/her ML-driven drone application are met before introduced
into production.
A. Use-Case Scenario
Suppose a ML practitioner must test the efficiency of his/her
developed ML solution for a hypothetical -but very realistic-
use-case that employs face recognition for tracking human
targets’ from the air (i.e., social distancing). Towards this, the
drone is given a flight plan including a (bounded) area of
interest and a starting location. The drone will then fly above
Python SDK
Create energy model and
add/remove components
Customize components
(naming, add/remove sub-
components, set values, …)
Reusable templates for DJI
Drones

2/23/23 37
Demetris Trihinas
37
Department of
Computer Science
Energy Modeling Accuracy
• Power dominated by motorized components, compute and communication
• Tests show that FlockAI features a 5-9% error.
• Error contributing factors:
• Wind speed!
• Different power models required for take-off
and landing.
• Power for peripherals (especially camera modules
using accelerators).

2/23/23 38
Demetris Trihinas
38
Department of
Computer Science
Experimentation
FlockAI efficacy

2/23/23 39
Demetris Trihinas
39
Department of
Computer Science
Use-Case: Face Detection and Human Counting
• Detect humans during drone flight.
• FlockAI blueprint for the DJI Mavic
2 Pro drone.
• Examine various ML algorithms
(CNN-based, FIG-based).
• DroneFaces dataset – 20 camera
feeds with 30 images each.
DroneFace: An Open Dataset for Drone Research. Hwai-Jung Hsu et al.,
ACM MMSys 2017.

2/23/23 40
Demetris Trihinas
40
Department of
Computer Science
Scenario 0: Drone Flight - No ML (Baseline)
• Emotors dominating component.
• Eproc for flight controller and Ecomm
control msgs and health checks.
• Flight time till landing due to battery
exhaustion 30min 6s

2/23/23 41
Demetris Trihinas
41
Department of
Computer Science
Scenario 1: No ML vs On-Board Inference
• With on-board ML, Eproc “jumps” to 21%.
• This is far from negligible….

2/23/23 42
Demetris Trihinas
42
Department of
Computer Science
Scenario 1: Eproc and Flight Time
• Eactive (79%) denotes camera feed scanning + ML inference.
• Efc denotes energy for flight controller.
Mode Flight Time
No ML 30min 6s
On-Board ML 24min 48s

2/23/23 43
Demetris Trihinas
43
Department of
Computer Science
Scenario 2: On-Board vs Remote Inference
• Ecomm increases by 16% compared to No ML and 12% on-board inference.
• Eproc doesn’t drop to 0… compute dedicated to data preparation and “firing” transmission.
• Flight time (due to energy savings) increases but only by 1min.

2/23/23 44
Demetris Trihinas
44
Department of
Computer Science
Scenario 2: On-Board vs Remote Inference
• The “cost” of remote inference.
• High latency (median 3.15 s) compared to on-
board inference (median 1.36 s).
• Can be disastrous depending on the use-case.

2/23/23 45
Demetris Trihinas
45
Department of
Computer Science
Scenario 3: Testing Different ML Algorithms
• More than one way to achieve a goal (i.e., face detection + human counting).
• Examine performance and accuracy trade-offs between algorithms.
• CNN-based and FIGs-based (from TensorFlow lib) algorithms.
rnet 2021, 13, 317 19 of 24
Table 3. ML algorithm accuracy vs. performance for on-board inference.
Algorithm Accuracy (%) Median Inference Time (ms) Flight Time (s)
CNN 95 1361 1488
FIGs 60 274 1642
Table 4. ML algorithm inference delay (in images of camera feed).
CNN FIGs
19 out of 20
sequences
12 out of 20
sequences
CNN accuracy comes at a 10x ovehead
with more compute-hungry model
FIGs less
intensive
model saves
2min 34s in
flight time

2/23/23 46
Demetris Trihinas
46
Department of
Computer Science
Scenario 3: Testing Different ML Algorithms
• Inference delay (custom metric) how
fast is the crowd detected in the image
sequence (image count)
Future Internet 2021, 13, 317 19 of 24
Table 3. ML algorithm accuracy vs. performance for on-board inference.
Algorithm Accuracy (%) Median Inference Time (ms) Flight Time (s)
CNN 95 1361 1488
FIGs 60 274 1642
Table 4. ML algorithm inference delay (in images of camera feed).
CNN FIGs
Sequence Inf. Delay (img count) Inf. Delay (img count)
seq1 23 -
seq2 26 29
seq3 29 29
seq4 29 29
seq5 23 28
seq6 24 23
seq7 26 -
seq8 27 -
seq9 26 -
seq10 26 29
seq11 29 29
seq12 28 29
seq13 25 -
seq14 26 28
seq15 28 29
seq16 28 -
seq17 26 29
seq18 27 29
seq19 28 -
seq20 - -
Clearly, the CNN algorithm is more accurate but this accuracy comes with a penalty in
the speed, and compute resources consumed, at which the ML task can output a decision
for the captured images. To examine the impact of the additional processing overhead
imposed by the CNN when compared to FIGs, we examine the total flight time for the
drone. Hence, Table 2, depicts the time the drone is in the air when employing either of the
two algorithms. From this, we immediately observe that the FIGs algorithm enables the
drone to stay in the air for an additional 2 min and 34 s. Based on these results, and for

2/23/23 47
Demetris Trihinas
47
Department of
Computer Science
Related Work and SOTA

2/23/23 48
Demetris Trihinas
48
Department of
Computer Science
Table 5. Qualitative comparison of the SOTA in drone testing (M, P, and C in energy profiling denote the energy consumption
attributed to the motorized, processing, and communication components, respectively).
Framework SDK
Drone
Configuration
Algorithm
Testing
Algorithm
Execution
Monitoring
Energy
Profiling
M | P | C
Last
Updat
Drones
Bench
- real drone - -
motorized
components
X | - | - NA
jMavSim java
quad-copter
and hexa-copter
imitations
localization and
camera sensors
custom flight
control with
PX4 autopilot
modules
remote
flight
control
X | - | - 2021
RotorS C++
quad-copter
imitation
localization and
camera sensors
interface for
custom flight
control and
state estimation
on drone
flight
control
- | - | - 2020
MRS C++
drone templates
(dji, tarot)
interface for
custom flight
control and
state estimation
on drone
flight
control
- | - | - 2021
UTSim C#
quad-copter
imitation
interface for
path planning
remote
flight control
comm overhead
- | - | - NA
AirSim
C++
python
java
quad-copter
imitation
localization and
camera sensors
environmental
conditions
interface for
computer vison
remote
flight control
comm overhead
X | - | - 2021
FANETSim java
quad-copter
imitation
comm protocol
- -
flight control
comm overhead
X | X | - NA
Deepbots python
quad-copter
imitation
reinforcement
learning
on drone
inference
accuracy
- | - | - 2020
FlockAI python
customizable
drone templates
lovalization and
camera sensors
autopilot interface
for custom flight
control and state
estimation, regression,
clustering and
classification incl. face
and object detection
on drone
and
remote
flight control
processing
overhead,
comm overhead
inference accuracy
and delay
X | X | X 2021
8. Conclusions and Future Work
In this article, we have introduced FlockAI: an open and extensible framework fa
Drone testing tools
• Plethora of tools.
• Graphical environments, templates
and autopilot testing.
• Gap in ML deployment and in-place
execution.
• Gap in energy profiling for processing
and communication axes.

2/23/23 49
Demetris Trihinas
49
Department of
Computer Science
Conclusions
• Rise of ML-driven drone applications.
• Difficult to create testing environments for drone applications.
• FlockAI embeds a powerful modeling framework for designing experiment
benchmarks so that ML practitioners can configure drone resource capabilities and
deploy their ML algorithm(s).
• Through runtime monitoring, ML practitioners are able to evaluate KPIs including
flight performance, resource utilization, communication overhead, and energy
consumption.

2/23/23 50
Demetris Trihinas
50
Department of
Computer Science
Articles
FlockAI: A Testing Suite for ML-Driven Drone Applications. D. Trihinas, M. Agathocleous, K.
Avogian, I. Katakis, Future Internet, 2021.
Composable Energy Modeling for ML-Driven Drone Applications. D. Trihinas, M.
Agathocleous, K. Avogian, IEEE IC2E 2021.
Demo: FlockAI - A Framework for Rapidly Testing ML-Driven Drone Applications. D.
Trihinas, M. Agathocleous, K. Avogian, IEEE ICDCS 2022.
Towards Energy Consumption and Carbon Footprint Testing for AI-driven IoT Services. D.
Trihinas, L. Thamsen, J. Beilharz, M. Symeonides, IEEE IC2E 2022.

2/23/23 51
Demetris Trihinas
51
Department of
Computer Science
Rapidly Testing ML-Driven Drone
Applications
The Framework
Thank You!
https://unic-ailab.github.io/flockai/

Rapidly Testing ML-Driven Drone Applications - The FlockAI Framework

Recommended

Recommended

More Related Content

Similar to Rapidly Testing ML-Driven Drone Applications - The FlockAI Framework

Similar to Rapidly Testing ML-Driven Drone Applications - The FlockAI Framework (20)

More from Demetris Trihinas

More from Demetris Trihinas (16)

Recently uploaded

Recently uploaded (20)

Rapidly Testing ML-Driven Drone Applications - The FlockAI Framework