The interaction between human beings and robotic agents, and the interest towards such topics, have been exponentially growing in the recent years. The purpose of this thesis project is to identify a relation between the behaviours of a humanoid robot placed in a social context, and the emotional responses of a subject interacting with it. In particular, through the use of Brain-Computer Interface (BCI) and gaze tracking technologies, it has been investigated on the relation between the trust towards a robotic agent and the effects it has on the brain signals. In order to evaluate this relation, the framework makes use of the acquired brain signals to extract biometric features, such as attention, stress, and mental workload, along with the visual focus. In order to investigate towards this direction, an interactive game session has been set up for the human-robot interaction. In particular, an instance of the well-known Rock-Paper-Scissors game has been used. The experimental results have been shown a correlation between the behaviours of a robotic agent and the effect of trust on the brain signals of the human user. In particular, the emotional response varies depending the type of behaviours expressed by the robotic agent.
(INDIRA) Call Girl Aurangabad Call Now 8617697112 Aurangabad Escorts 24x7
Design and Implementation of Modules for Biometric Extraction in BCI
1. UNIVERSITY OF PALERMO
POLYTECHNIC SCHOOL
Departmentof Industrial and DigitalInnovation (DIID)
Computer ScienceEngineeringfor Intelligent Systems
Design and Implementation of Modules
for the Extraction of Biometric Parameters
in an Augmented BCI Framework
Master Degree Thesis of:
Salvatore La Bua
WWW.SLBLABS.COMMarch, 2017
2. Introduction
▪What
◦ Investigate the effects of the interaction with a robotic agent
on the mental status of the human player
through brain signal analysis
◦ Acceptance of a robotic agent by the user
◦ Performance improvements over a classical BCI system
▪How
◦ Rock-Paper-Scissors game integration
◦ UniPA BCI Framework based on the P300 paradigm
◦ Augmented by
◦ Eye gaze coordinate acquisition
◦ Biometric feature extraction
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
S. La Bua 2
3. Introduction
Human-Robot Interaction (HRI)
▪HRI as a multidisciplinary research topic
◦ Artificial Intelligence
◦ Human-Computer Interaction
◦ Natural Language Processing
◦ Social Sciences
◦ Design
▪Model of the user’s expectation towards a robotic agent
in a human-robot interaction
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
S. La Bua 3
4. Introduction
Brain-Computer Interfaces (BCI)
▪Direct communication between
brain and external devices
◦ Non-Invasive
◦ Partially-Invasive
◦ Invasive
▪Brain Lobes
◦ Frontal: emotions, social behaviour
◦ Temporal: speech, hearing recognition
◦ Parietal: sensory recognition
◦ Occipital: visual processing
▪Extraction of biometric features from brain signals
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
S. La Bua 4
5. Introduction
Visual Focus
▪Importance of eye gaze for direct interaction in a social
environment
▪Interfaces dedicated to people affected by degenerative
pathologies
▪Entertainment applications, such as games
▪Better advertisement placement
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
S. La Bua 5
6. Methodology
Background Information
▪Problem
◦ Effects of the behaviour of a robotic agent on the brain signals
◦ Trust context in Human-Robot Interaction
▪Feature Extraction
◦ Entropy: as a stress indicator
◦ Energy: as a concentration indicator
◦ Mental Workload: as an index of engagement in the task
▪Brain waves types
◦ δ Delta: Hz 0.5÷3 related to instinct, deep sleep
◦ θ Theta: Hz 3÷8 related to emotions
◦ α Alpha: Hz 8÷12 related to consciousness
◦ β Beta: Hz 12÷38 related to concentration, stress
◦ γ Gamma: Hz 38÷42 related to information processing
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
S. La Bua 6
7. Methodology
The math behind
Entropy:
𝐸 𝑠𝑖 = 𝑠𝑖
2
log (𝑠𝑖
2
); 𝐸 𝑠 = − σ𝑖 𝐸 𝑠𝑖
Energy:
𝐸𝑠 =
𝑛=−∞
∞
𝑥(𝑛) 2
Mental Workload:
𝛽 𝑝𝑠𝑑
𝛼 𝑝𝑠𝑑 + 𝜃 𝑝𝑠𝑑
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
S. La Bua 7
8. The Proposed Solution
Architecture Structure
▪Action Selection
◦ Direct interface with the user
◦ Acquisition of bio-signals
◦ Acquisition of eye gaze coordinates
◦ Selection of the Base action
▪Feature Extraction and Analysis
◦ Bio-signals analysis
◦ Features extraction
◦ Features analysis
◦ Computation of Intention, Attention,
Stress indices
▪Response Modulation
◦ Threshold of the Base action by means of the Intention index
◦ Modulation of the resulting action by means of Attention and Stress indices
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
S. La Bua 8
9. The Proposed Solution
Class Diagram
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
S. La Bua 9
10. The Proposed Solution
Functional Blocks
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
Action Selection
◦ Eye-Tracking module
◦ Screen coordinates acquisition
◦ Weighing module
◦ Weighing of the BCI classifier response
precision and the Eye-Tracking module
response precision, by means of the
user’s skill level
◦ ID Selection module
◦ Action selection by means of the weighted BCI classifier and Eye-Tracking module
precisions
S. La Bua 10
11. The Proposed Solution
Functional Blocks
Feature Extraction
and Analysis
◦ It makes use of external calls
to the MATLAB engine
◦ Features extracted and analysed
◦ Correlation Factor: related to the Intention index
◦ Energy: related to the Attention index
◦ Entropy: related to the Stress index
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
S. La Bua 11
12. The Proposed Solution
Functional Blocks
Response Modulation
◦ Threshold module
◦ ID Selection validation by
means of Intention index
thresholding
◦ Modulation module
◦ In the case the selected ID has passed the validation step,
the resulting action is modulated by means of the Attention and Stress indices
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
S. La Bua 12
13. The Proposed Solution
Robotic Controller
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
S. La Bua 13
14. The Proposed Solution
Utilisation Modes
Basic Mode
◦ Simplest mode
◦ Minimal number of
modules involved
◦ Classical BCI approach
◦ P300 paradigm
classification
◦ Direct Behaviour
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
S. La Bua 14
15. The Proposed Solution
Utilisation Modes
Hybrid Mode
◦ Advanced mode
◦ Eye-Tracking module
◦ Combination of brain
signals and eye gaze
◦ User skill level as
weighting parameter
◦ Composite Behaviour
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
S. La Bua 15
16. The Proposed Solution
Utilisation Modes
Bio-Hybrid Mode
◦ Complete mode
◦ Feature Extraction
and Analysis
functional block
◦ Response Modulation
functional block
◦ Intention, Attention and
Stress indices computation
◦ Modulated Behaviour
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
S. La Bua 16
17. Architecture
Eye-Tracking module
P300 6x6 spelling matrix 3x3 spelling window areas
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
S. La Bua 17
18. Architecture
Eye-Tracking module
Preliminary tests results
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
SUBCATEGORIES FOR SINGLE
ELEMENT
FOCUS % CENTRAL FOCUS % LATERAL FOCUS % EXTERNAL FOCUS %
3-BY-3, 700X700PX 100 99.9000 0.1000 0
3-BY-3, 300X300PX 98.4562 93.2697 6.7303 1.5438
6-BY-6, 700X700PX 100 84.7408 2.7592 0
6-BY-6, 300X300PX 99.5997 75.9943 24.0057 0.4003
SUBCATEGORIES FOR ROW SPAN
SELECTION
FOCUS % CENTRAL FOCUS % LATERAL FOCUS % EXTERNAL FOCUS %
3-BY-3, 700X700PX 74.2632 93.9192 6.0808 25.7368
3-BY-3, 300X300PX 77.1340 89.9075 10.0925 22.8660
6-BY-6, 700X700PX 69.5037 96.3287 3.6713 30.4963
6-BY-6, 300X300PX 75.0674 71.7202 28.2798 24.9326
AVERAGE BY PARAMETER FOCUS % CENTRAL FOCUS %
700X700PX 85.9417 93.7222
300X300PX 87.5643 82.7229
GAIN WITH LARGER WINDOW -1.8530% +13.2966%
AVERAGE BY PARAMETER FOCUS % CENTRAL FOCUS %
3-BY-3 87.4634 94.2491
6-BY-6 86.0427 82.1960
GAIN WITH LESS DENSE MATRIX +1.6512% +14.6639%
S. La Bua 18
19. Architecture
Data Structures
Generic signal data structure fields
N fields dedicated to the brain signals acquisition
◦ Ch 1 – Ch 16
3 auxiliary fields to carry peculiar information
◦ A, B, C
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
CH 1 CH 2 · · · CH N A B C
S. La Bua 19
20. Architecture
Data Structures
Baseline Calibration signal
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
A (RED) B (CYAN) C (MAGENTA)
BASELINE CALIBRATION -2 EYES STATUS 0
S. La Bua 20
21. Architecture
Data Structures
Game Session signal
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
A (RED) B (CYAN) C (MAGENTA)
GAME SESSION TRIAL STATUS TRIAL SUB-PHASE GAZE TRACKING
S. La Bua 21
22. Architecture
Data Structures
P300 Calibration signal
P300 Spelling signal
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
A B C
P300 Spelling -1 Flashing tag 0
A B C
P300 Calibration Calibration target Flashing tag 0
S. La Bua 22
23. The Framework
Main Interface
1. Basic settings
◦ P300-related settings
◦ Preset modes
2. Main functionalities
◦ Signal quality check
◦ P300 Calibration and
Recognition
◦ Game session control
3. Interface modality
◦ Alphabetic or Symbolic
4. Devices
◦ Eye-Tracker settings
5. Plots and Indicators
◦ Signals and Indices
visualisation
6. Output panel
◦ Feedback for the operator
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
1
2
3
4
5
6
S. La Bua 24
24. The Framework
Baseline Acquisition Interface
Control dialog window User dialog window
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
S. La Bua 25
25. The Framework
Game Session Interface
1. Game modality
◦ Fair
◦ Cheat-to-Win/Lose
2. Trials number per session
◦ Initial Fair sub-session
◦ Middle Cheating sub-session
◦ Terminal Fair sub-session
3. Devices
◦ BCI signal acquisition
◦ Kinect gesture recognition
◦ Play against a robotic agent
4. Session panel
◦ Moves selection
◦ Trial temporal progress
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
1 2
3
4
S. La Bua 27
26. Experiments
Introduction
▪Purpose
◦ Investigate the effects of the interaction with a cheating robotic
agent on the mental status of the human player
◦ Rock-Paper-Scissors game session
▪Scenarios
◦ The robot behaves according to the game’s rules
◦ The robot exhibits a cheat-to-win behaviour
◦ The robot exhibits a cheat-to-lose behaviour
▪Game Session
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
Initial Fair
sub-session
Cheating
sub-session
Terminal Fair
sub-session
S. La Bua 28
27. Experiments
Set-up
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
Subjects
◦ 16 Subjects
◦ Aged 18-51
Hardware
◦ g.tec g.USBamp
◦ g.tec g.GAMMAbox
◦ g.tec g.GAMMAcap2
◦ Secondary standard PC screen
◦ Tobii EyeX eye tracker
◦ Kinect for Xbox One
◦ Telenoid
◦ Camera(s)
S. La Bua 29
28. Experiments
EEG Electrodes configuration
Channels-Electrodes
correspondence
L R
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
Ch 01 F7
Ch 02 F3
Ch 03 FZ
Ch 04 T3
Ch 05 C3
Ch 06 T5
Ch 07 P3
Ch 08 O1
Ch 09 F8
Ch 10 F4
Ch 11 T4
Ch 12 C4
Ch 13 T6
Ch 14 P4
Ch 15 PZ
Ch 16 O2
S. La Bua 30
31. Experiments
Subcategories
▪Sub-Session Analysis
◦ Analysis of the Baseline signal, Fair and Cheating sub-sessions
▪Trials Analysis
◦ Single trial analysis for each subject
▪Intra-Class Comparison
◦ Comparison between Cheat-to-Win and Cheat-to-Lose classes
▪Average Analysis
◦ Average over all subjects, by class and by sub-sessions
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
S. La Bua 33
38. Experiments
Trials Analysis
Focus %: Cheat-to-Win Cheat-to-Lose
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
S. La Bua 40
43. Experiments
Average Analysis
Entropy
The entropy values do not show any particular evidence of stress
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
ENTROPY FAIR 1 CHEAT FAIR 2
MEAN STD DEV MEAN STD DEV MEAN STD DEV
CHEAT WIN 3.8584 0.2191 3.8998 0.2540 3.8742 0.1891
CHEAT LOSE 3.7420 0.0850 3.7632 0.1177 3.7304 0.1074
S. La Bua 45
44. Experiments
Average Analysis
Energy
The energy values show higher concentration level for the Cheat-to-Win class
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
ENERGY FAIR 1 CHEAT FAIR 2
MEAN STD DEV MEAN STD DEV MEAN STD DEV
CHEAT WIN 0.2572 0.2141 0.3032 0.2267 0.2254 0.1951
CHEAT LOSE 0.1498 0.0596 0.1720 0.0948 0.1143 0.0447
S. La Bua 46
45. Experiments
Average Analysis
Mental Workload
The mental workload values show a slightly lower engagement level for the
Cheat-to-Win class
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
MENTAL WL FAIR 1 CHEAT FAIR 2
MEAN STD DEV MEAN STD DEV MEAN STD DEV
CHEAT WIN 1.3798 1.1625 0.8988 0.4215 0.9437 0.4570
CHEAT LOSE 1.0923 0.2716 1.0382 0.3229 1.0777 0.3936
S. La Bua 47
46. Experiments
Average Analysis
Visual Focus
The visual focus values show higher visual attention level for the Cheat-to-Win
class
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
FOCUS % FAIR 1 CHEAT FAIR 2
MEAN STD DEV MEAN STD DEV MEAN STD DEV
CHEAT WIN 7.89100 8.93670 9.13020 11.3344 12.1404 20.1567
CHEAT LOSE 4.59710 9.91690 3.24540 7.09430 2.20110 4.79480
S. La Bua 48
48. Conclusions and Future Works
▪A robotic agent that cheats to win is perceived as more
agentic and human-like than a robot that cheats to lose
▪Some of the Questionnaire results
▪Trust related improvement
◦ Biometric features to mitigate or amplify the effects of the
robotic agent behaviour on the subject’s emotional response
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
Unusual Behaviour Fair Play Intelligence
Strongly
Disagree
Strongly
Agree
S. La Bua 50
49. Future Works
Framework Extension
Sensor Aggregation functional block
◦ Galvanic Skin Response (GSR) sensor
◦ Heart Rate (HR) sensor
◦ Other physiological sensors
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
S. La Bua 51
50. Future Works
Extended Framework
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
S. La Bua 52
51. Thank you for
your attention
Salvatore La Bua
slabua@gmail.com
WWW.SLBLABS.COM