webinar presentation July 20

1. Horizon 2020
European Union Funding
for Research & Innovation
The COALA Cognitive Advisor
Dr. Evangelos Niforatos
Assistant Professor | AI-Powered Human Augmentation
Faculty of Industrial Design Engineering
Delft University of Technology
Delft, the Netherlands
ForeSee Cluster: Standardization Workshop
20.07.2022 1

2. Digital Intelligent Assistants
2
… support or take over time-consuming,
stressful, and other unwanted activities.
… interact via natural language. … “know” and do things that
humans associate with intelligence.
… are amorphous compound technology.
Basic structure of a conversational agent The amorphous “backend”
20.07.2022 e.niforatos@tudelft.nl

3. The Factory of Today
3
Source: FRATELLI PIACENZA S.P.A. Source: FRATELLI PIACENZA S.P.A.
Source: Whirlpool Corporation Source: MiR / Whirlpool Corporation
• Long distances
• Narrow space
• Noisy
• Manual labor
• Declining workforce
• Automated tasks
• “Smart” machines
• Sensors, IoT & APIs
Source: Diversey Netherlands Production Bv
20.07.2022 e.niforatos@tudelft.nl

4. The COALA Cognitive Advisor (CA)
• (1) Augmented Analytics
• The Cognitive Advisor (CA) collects and analyzes data
from the shop floor
• The users monitor analytics through dialectic interactions
• The CA explains recommendations to improve
trustworthiness and achieve transparency
• Context-aware features streamline interactions
• (2) On-the-job training
• The operators share knowledge through dialectic
interactions with the the CA
• The operators receive advice based on their expertise
• The CA advising behavior adapts to learning progress
• Ultimately, the CA identifies and instills best practices
4
Assistant Benefits
Central access
Customizable
Delegatory
Eyes-free
Instructive & Collaborative
Hands-free
Ubiquitous
Multimodal (text & speech)
Always on
Responsive
Context-aware
Intuitive (no learning curve)
Visit this link to find the related article:
https://ai4manufacturing.com/anatomy-of-a-digital-assistant/
20.07.2022 e.niforatos@tudelft.nl

5. Use Case #1: Augmented Analytics
5
Predictions for a machine’s fluid consumption
Dialog
management
GraphQL
Prediction
module
Intent & Entity
extraction
Intent: “forecast”
Entity values: “hydraulic fluid”, “next 15 days”
GraphQL query with entities + context as
parameters
API call
Context
“machine_id = AF34689543KL”
Result
Shop floor data
• Machine vision
• Machine APIs
• Historical data
• …
20.07.2022 e.niforatos@tudelft.nl

6. Use Case #2: On-the-job Training
6
Novice asking for instructions
Breakdown of task into small steps
Sharing knowledge with the assistant Provide learning content (“nuggets”)
20.07.2022 e.niforatos@tudelft.nl

7. Designing the COALA Cognitive Advisor
Design Aspects
7
Settings Ethics Privacy
Usability is a key concern too but not focused here
20.07.2022 e.niforatos@tudelft.nl

8. Design Aspect 1: Manufacturing Settings
8
COALA prototype testing in BIBA’s
lab shop floor
N
I N
I
Noise management
Voice integration and mobility
20.07.2022 e.niforatos@tudelft.nl

9. Design Aspect 2: Applied AI Ethics
9
• What if the operators uncritically follow the recommendations of the CA?
• What if the CA suggests a task that could harm the user?
• What if the CA recommends a procedure that could damage the equipment?
• Who owns the knowledge generated/discovered with the Cognitive Advisor?
• …
Practice “Ethics-by-Design”
o … but delegation trades autonomy for free time
o … but the workforce in factories contains bias (e.g., males > females)
Bias in training data
Balance of power (Employer <> Employee)
Loss of autonomy
Influencing factors
20.07.2022 e.niforatos@tudelft.nl

10. Design Aspect 3: Operator’s Privacy
10
• What if users name co-workers in a conversation with the assistant?
• What if employer(s) use the CA to track operators on the shop floor?
…
Practice Privacy-by-Design (preferably no personal data)
o … but personalized functions requires knowing personal data
o … but customized assistance requires knowing user pseudonyms
o … but restricting access to an assistant likely requires operators
to register with a system (receive a pseudonym)
Worker council and worker union
General Data Protection Regulation (GDPR)
Labor laws
Influencing factors
20.07.2022 e.niforatos@tudelft.nl

11. Testing the Cognitive Advisor
• Test areas
• User interface and real-world interactions
• Layer one CAI and skills
• Layer two CAI (if any)
• Custom modules (e.g., product quality prediction)
• Test focus
• System usability (e.g., Voice Usability Scale)
• Cognitive workload (e.g., NASA Task Load Index)
• AI trustworthiness (e.g., Assessment List for Trustworthy AI)
• Test users
• Access to testers requires workers’ council permission
• No bias intended but often inherent in workforce
11
Android App
Mycroft
Rasa
Custom module
Layer one
Layer two
20.07.2022 e.niforatos@tudelft.nl

12. Outlook
• Testing in the lab is necessary before
deploying the Cognitive Advisor in factory
settings
12
• Lab testing involves:
• Real tasks
• Simulated machine
interfaces
• Simulated machine APIs
• Simulated floor plans
• Maintenance Experts,
Technicians and Lab
Assistants as operators
20.07.2022 e.niforatos@tudelft.nl

13. Contact
Website: www.coala-h2020.eu
LinkedIn: www.linkedin.com/in/coala-your-factory-assistant
Twitter: twitter.com/coala4factory
Contact: info@coala-h2020.eu
Many more videos about the COALA assistant
www.coala-h2020.eu/index.php/videos
13
e.niforatos@tudelft.nl
20.07.2022
Dr. Evangelos Niforatos
Assistant Professor
AI-Powered Human Augmentation
Faculty of Industrial Design Engineering
Delft University of Technology
Delft, the Netherlands

14. q Objective: Artificial intelligence for manufacturing
q Topic: ICT-38-2020
q Call: H2020-ICT-2018-20
q Lead: BIBA – Bremer Institut für Produktion und Logistik GmbH
q Duration: 36 Months
q Start: 2020/10
The information in this document is provided "as is", and no guarantee or warranty is given that the
information is fit for any particular purpose. The consortium members shall have no liability for damages
of any kind including without limitation direct, special, indirect, or consequential damages that may
result from the use of these materials subject to any liability which is mandatory due to applicable law.
Copyright 2020 – 2023 by the COALA Consortium
Copyright
14
e.niforatos@tudelft.nl
20.07.2022