The document discusses the design and development experiences related to monitoring AI inference capabilities, emphasizing the importance of trustworthy AI systems. It highlights the necessity for continuous monitoring, human oversight, and compliance with ethical standards, while also addressing challenges such as complexity and resource intensity in implementation. The research is part of a EU-funded project aimed at improving AI effectiveness and safety across various applications, including medical emergency detection and network classification.

1. Design and Development Experiences from
Gauging and Monitoring the AI Inference
Capabilities of Modern Applications.
The SPATIAL Architecture:
A. Ottun and the EU SPATIAL consortium
This research is part of SPATIAL project that has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement No.101021808.

2. Our Team
Collaborators
Security and Privacy Accountable Technology Innovations,
Algorithms, and Machine Learning

3. Background
[Source]https://dribbble.com/shots1 [Source] https://static.wixstatic.com
[Source] https://www.blog.google
[Source ]hhttp://www.kiwibot.com
[Source] waymo.com
AI is fast becoming ubiquitous in our society.
Product recommendation Home automation Autonomous Robot Drone Delivery
https://cdn.rarejob.com
[Source]
Driverless Taxi Chat Agent Health monitor Personal Assistance Social networking
Some day-to-day interaction with AI

4. AI failures: From safety to bias concerns
[Source] https://gbagenlaw.com/drone-related-injuries-are-
becoming-more-common/
[Source] https://www.farrin.com/blog
[Source] Yahoo news
[Source] https://medium.com/@aliborji
An Open Letter - Future of Life Institute
[Source]

5. AI Trustworthy Requirements
Global AI Governance and Framework
are designed to ensure the:
[Source] https://legalnodes.com
Protection of values and rights
Societal safety
AI trustworthiness
Trustworthiness Requirements/ Properties
Human oversight
Robustness
Privacy
Fairness and non-bias
Accuracy
Explainability
Resilience

6. Software Architecture
Classical Architecture
+ Federated Learning
[source] Muccini, H., & Vaidhyanathan, K. (2021, May). Software architecture for ML-based systems: what exists and what lies
ahead. In Proceedings of IEEE/ACM WAIN@ICSE 2021 (pp. 121-128). IEEE.
Classical Architecture
Classical Architecture
+ Machine Learning

7. Software Architecture
Classical architecture
+ Federated learning
[source] Ottun, Abdul-Rasheed., &.. Flores, Huber.
(2023). Detection mechanisms to identify data biases
and exploratory studies about different data quality
trade-offs. Deliverable 3.1, H2020 EU SPATIAL, 2023.

8. SPATIAL Augmentation

9. SPATIAL Deployment
Trustworthy Services

10. Need for continuous monitoring with SPATIAL
Performance
Monitoring
Track performance to
maintain high accuracy
and reliability
Continuous
Improvement
Gain insight into the
application for improvement
Error detection
Enable early detection of
errors for correction
Change
Detection
Detection of changes in data
distribution and operating
environment of the model
Robustness and
security
Detect and defend
application against attacks
Trustworthy and
Compliance
Ensure application is
developed ethically and
compliant to regulations

11. SPATIAL Evaluation
Industrial Use Cases
1
Analysis of emergency fall
detection models used in
a medical e-calling
application
Diagnose models
classifying activities
of internet users
2

12. Medical e-calling app
Autonomous Detection of Medical
Emergency from Falling in the Elderly.
Random Label
Flipping attack
Systematic progression
from 1% - 50% Level
• Random Forest (RF)
• Decision Tree (DT)
• Multilayer Perceptron (MLP)
• Logistic Regression (LR)
• DNN
Trained Classifiers
Dataset
4192
Falling Class
(FALL)
7579
Activities of Daily
Living Class (ADL)
Model Manipulation

13. Network Activity Classification
Diagnosing models used for identifying user internet activities
to assess their behavior when examined on crafted samples.
Neural Network
LightGBM
XGBoost
Dataset
103
Adversarial
samples
382
Web browsing
Interactivity
Video streaming
Models

14. Analysis from SPATIAL- Metric & XAI Services
NN Post-Attack
NN Before Attack
Use case 2: Network Classification Models
Use case 1: Fall Detection Models
0
50
100
LightGBM XGBoost NN
Accuracy
Model Performance
Baseline Attacked

15. Implication of Continuous Monitoring
Explainability Service Resilience Service System Performance
Continuous request for the services impacts system performance in different ways. Some
services tend to be resource-intensive with high response time while others are less intensive
with stable response time.

16. ● Instrumenting AI-based applications
○ Instrumenting AI systems and applications with components like dedicated sensors and metrics for
continuous monitoring of the AI inference process can aptly examine models to help minimize potential
risks and make AI more trustworthy.
● Human oversight:
○ Adopting adaptive and interactive interfaces to tune AI models continuously facilitates complying with
human oversight requirements for ensuring trustworthy AI.
● Implementation complexities:
○ Continuous monitoring of inference can increase the complexities and cost of developing and maintaining
applications due to the need for additional components and infrastructures.
● Resource intensive:
○ Continuously monitoring and analyzing the requirements is resource intensive, depending on the scale of
the development and volume of the input data.
Insights and Lessons Learnt

17. Thank you
MORE INFO
rasheed.ottun@ut.ee
http://spatial-h2020.eu
This project has received funding from the European Union’s Horizon 2020 research and innovation
programme under grant agreement N° 101021808.
Scan QR to read the paper