Presented at a small group seminar (Music and acoustics research group, Graduate School for Convergence Science and Technology, Seoul National University, Suwon, South Korea)

1. / 451
Seung-Goo KIM (김승구)
Department of Psychology & Neuroscience
Duke University, USA
2020-07-02
MARG, SNU, South Korea
Seminar
Predicting the Neural Encoding of
Musical Structures

2. / 452

3. / 45
[1]
3
[1] clipartbest.com [2] freepik.com
[2]

4. / 45
Bridging the gap
4
[1] clipartbest.com [2] freepik.com
Cognitive computational
neuroscience
Com
putational
m
odeling
Cognitive
neuroscience
[2]
[1]

5. / 45
Overview
• Classical cognitive neuroscience (behavior-brain association)
- Dissonance preference associated with brainstem response
• Model-based cognitive neuroscience (feature-brain association)
- Cortical tracking of tonal stability in naturalistic music
• General discussion
5

6. / 45
PLEASE INTERRUPT ME ANYTIME
WHEN SOMETHING IS UNCLEAR!
AND PLEASE SAVE IN-DEPTH
QUESTIONS FOR DISCUSSION
AFTER PRESENTATION :)
6

7. / 457
Kim. (2017). Thesis.
Human auditory system
Apparatus Brainstem + thalamus
PT: Planum temporale, HG: Heschl’s gyrus
PP: Planum polare, LSTG: lateral superior temporal gyrus
STS: superior temporal sulcus
Cortex

8. / 45
In-vivo neuroimaging methods
• Magneto/electro-encephalogram (M/EEG)
- Source: neuronal currents
spread by volume conduction and filtered through
skull and scalp
- Sampling resolution:
good temporal (≤ 1 msec), poor spatial (≥ 2 cm)
• Functional magnetic resonance imaging (fMRI)
- Source: blood deoxygenation level
slowly (~6 sec to the peak in humans) coupled with
neural activities [2]
- Sampling resolution (3-T):
poor temporal (≥ 1 sec), good spatial (≤ 2 mm)
8
[1] https://www.humanconnectome.org/study/hcp-young-adult/project-protocol/meg-eeg
[2] Lee et al. (2010). Nature.
[2]
[1]

9. / 45
Kim S-G, Lepsien J, Fritz TH, Mildner T, & Mueller K. (2017). Scientific Reports.
Dissonance preference associated
with brainstem BOLD response
Task-based functional MRI
9

10. / 45
What is sensory dissonance?
• Lack of overlaps of harmonics
• Beating (wave interference)
Small differences in frequencies
creates an amplitude modulation
(e.g., fbeat = f1 - f2)
10
(cc) Adjwilley.
https://en.wikipedia.org/wiki/Beat_(acoustics)
(cc) Hyacinth.
https://en.wikipedia.org/wiki/Inharmonicity
Harmonic spectrum Inharmonic spectrum

11. / 45
Dissonance disfavor & IC
• Neural correlates of subjective
preference for consonant over dissonant
harmony in inferior colliculus (IC)
- Gray matter density (GDM) estimated
from T1-weighted MRI [1]
Positive correlation with consonance
preference
- Neural consonance index in
frequency-following response (FFR) [2]
Positive correlation with consonance
preference
11
[1] Fritz et al. (2013). E J Neurosci. [2] Bones et al. (2014). [2]
[1]

12. / 45
Motivation
• Given evidence, IC seems to play an important role in
encoding sensory dissonance & individual preference.
- However, the source of FFR may not be only IC [1].
- Also, it has not been shown in naturalistic music.
• Here, we used naturalistic stimuli and fMRI (data from
a previous publication [2]) to confirm that.
12
[1] Coffey et al. (2016). Nat Comm. [2] Mueller et al. (2015). NeuroImage.

13. / 45
Stimulus
• Twenty 30-s musical excerpts
(classical, fork, swing, tango) in
their original forms and dissonant
counterparts
Mixing with ones transposed by major
2nd up & augmented 4th down
13
SOUND DEMO: J. S. Bach Prelude No.3 in C# major (BWV 848)
Kim et al. (2016). Sci Rep.
Original
(“consonant”)
Manipulated
(“dissonant”)

14. / 45
Participants & protocol
• Participants: twenty-three healthy participants (non-musicians;
13 females; mean age: 25.9 ± 2.9)
• Functional MRI: 2.5 x 2.5 x 4 mm3, one 3-D image/sec,
over the ventral part (auditory + language areas) of the brain
• Protocol: listening to an excerpt (30 sec) + subjective rating of
unpleasantness (6 sec; 4-point) in the scanner (48 min)
14

15. / 45
fMRI data (4-D images)
15
Time
...
fMRI signal at one voxel
Modeled response to sound stimulation

16. / 45
fMRI preprocessing in nutshell
16
Kim. (2017). Thesis.
Time 1 Time 2 Time 3
Functional images (4D) Structural image (3D)
...

17. / 45
fMRI preprocessing in a nutshell*
17
Kim. (2017). Thesis.
Func Struct
...Subj 1
...Subj 2
...Subj 3
Subject (native) space Template space
Template
(MNI)
Linear
(6-DOF)
Nonlinear
(DCT)
Norm. Func
...
...
...
L+NL
* Yes, I’m skipping spatiotemporal realignment, temporal filtering, noise modeling, etc.

18. / 45
• First-level: does fMRI time-series correlate with the model?
Voxel-based* GLMs with autoregressive noise
18
GLM
GLM
GLM
:
y(v) = (X ⇤ h).b + AR(1) + ✏<latexit sha1_base64="k21BUkHB2Zv/1IATeQlEa+r98rE=">AAAGU3icrVRNb9NAEHVLEkqgtIUjl1WrlKR1IzttBRekFhCCW4C0jcgm0XqzcVb1F7vrSpa1/4IfhpA49I9w4QDrOHaSJhVCqqNIzzPzZubNztoKHMqFYVyvrN4rFEv31x6UHz5af7yxufXknPshw+QM+47P2hbixKEeORNUOKQdMIJcyyEX1uWbxH9xRRinvtcSUUC6LrI9OqQYCWXqbxVoBVrEpl6MHGp7exKOkIhj6CIxsobAkrKKaweLxkENvAIxHDfQYbbVjU3d0A2pkgnUNyTYB5AEnDq+V678T85JAsVflj0OE2qoQqGsp6HmfK08Y7sfN5sfpExyOmQoOiATyhiKZIwxlgBeIRaMKNgFUx7EA18A5dzN3RBThkF1GgOyoJoq7Q0mOQFk1B6J7kwXLKk/Q1QxiAswqqUJZvQrFaefqmZtuZpxnuwlAgc5/gLjao6f56jWiw9MOQ3KmFCm/aaHPVMgql7V5Nz8J+MFyCb5WwPYik5YbjgERIw8iqmgUW48AqfNz8t13E2ZWxdvXPmWhbhThanAW/s4Bu/etmQefjzb0UJDc+uxp3aj/q+1aClmomTIEI7HF2lKqONeS8aQf2XKyqntol4DV9u9VjvdicStDh4rfsfQ05+pVradGEw9GYY+mYGeS4/0ZLCzi20lnXcms9SzUWagkYHDDBx1+5s7Rt0YP2ARmBOwc7IN979dn0TN/uYPOPBx6BJPYAdx3jGNQHTVXRMUO0SWYchJgPCl6rijoIdcwrvx+EQkqCjLAAx9pv6euqmJdZYRI5fzyFUyKokiftOXGJf5OqEYvuzG1AtCQTycFhqGDhA+SD6wYEAZwcKJFECYqa3BAI+QOiihPsNlNQTzpuRFcN6om4f1xkc1jdda+qxpz7RtraqZ2gvtRHuvNbUzDRe+F34V/hS14s/i79JqqZCGrq5MOE+1uae0/hd07wlb</latexit>
⇤<latexit sha1_base64="NQekAT9OTGnhOyKsKD9yfhsG+ug=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tFEA8lEUGPRS8eW7Af0Iay2U7atZtN2N0IJfQXePGgiFd/kjf/jds2B219YeHhnRl25g0SwbVx3W+nsLa+sblV3C7t7O7tH5QPj1o6ThXDJotFrDoB1Si4xKbhRmAnUUijQGA7GN/N6u0nVJrH8sFMEvQjOpQ85IwaazUu+uWKW3XnIqvg5VCBXPV++as3iFkaoTRMUK27npsYP6PKcCZwWuqlGhPKxnSIXYuSRqj9bL7olJxZZ0DCWNknDZm7vycyGmk9iQLbGVEz0su1mflfrZua8MbPuExSg5ItPgpTQUxMZleTAVfIjJhYoExxuythI6ooMzabkg3BWz55FVqXVc9y46pSu83jKMIJnMI5eHANNbiHOjSBAcIzvMKb8+i8OO/Ox6K14OQzx/BHzucPcAOMrg==</latexit><latexit sha1_base64="NQekAT9OTGnhOyKsKD9yfhsG+ug=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tFEA8lEUGPRS8eW7Af0Iay2U7atZtN2N0IJfQXePGgiFd/kjf/jds2B219YeHhnRl25g0SwbVx3W+nsLa+sblV3C7t7O7tH5QPj1o6ThXDJotFrDoB1Si4xKbhRmAnUUijQGA7GN/N6u0nVJrH8sFMEvQjOpQ85IwaazUu+uWKW3XnIqvg5VCBXPV++as3iFkaoTRMUK27npsYP6PKcCZwWuqlGhPKxnSIXYuSRqj9bL7olJxZZ0DCWNknDZm7vycyGmk9iQLbGVEz0su1mflfrZua8MbPuExSg5ItPgpTQUxMZleTAVfIjJhYoExxuythI6ooMzabkg3BWz55FVqXVc9y46pSu83jKMIJnMI5eHANNbiHOjSBAcIzvMKb8+i8OO/Ox6K14OQzx/BHzucPcAOMrg==</latexit><latexit sha1_base64="NQekAT9OTGnhOyKsKD9yfhsG+ug=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tFEA8lEUGPRS8eW7Af0Iay2U7atZtN2N0IJfQXePGgiFd/kjf/jds2B219YeHhnRl25g0SwbVx3W+nsLa+sblV3C7t7O7tH5QPj1o6ThXDJotFrDoB1Si4xKbhRmAnUUijQGA7GN/N6u0nVJrH8sFMEvQjOpQ85IwaazUu+uWKW3XnIqvg5VCBXPV++as3iFkaoTRMUK27npsYP6PKcCZwWuqlGhPKxnSIXYuSRqj9bL7olJxZZ0DCWNknDZm7vycyGmk9iQLbGVEz0su1mflfrZua8MbPuExSg5ItPgpTQUxMZleTAVfIjJhYoExxuythI6ooMzabkg3BWz55FVqXVc9y46pSu83jKMIJnMI5eHANNbiHOjSBAcIzvMKb8+i8OO/Ox6K14OQzx/BHzucPcAOMrg==</latexit><latexit sha1_base64="NQekAT9OTGnhOyKsKD9yfhsG+ug=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tFEA8lEUGPRS8eW7Af0Iay2U7atZtN2N0IJfQXePGgiFd/kjf/jds2B219YeHhnRl25g0SwbVx3W+nsLa+sblV3C7t7O7tH5QPj1o6ThXDJotFrDoB1Si4xKbhRmAnUUijQGA7GN/N6u0nVJrH8sFMEvQjOpQ85IwaazUu+uWKW3XnIqvg5VCBXPV++as3iFkaoTRMUK27npsYP6PKcCZwWuqlGhPKxnSIXYuSRqj9bL7olJxZZ0DCWNknDZm7vycyGmk9iQLbGVEz0su1mflfrZua8MbPuExSg5ItPgpTQUxMZleTAVfIjJhYoExxuythI6ooMzabkg3BWz55FVqXVc9y46pSu83jKMIJnMI5eHANNbiHOjSBAcIzvMKb8+i8OO/Ox6K14OQzx/BHzucPcAOMrg==</latexit>
=<latexit sha1_base64="8uDF8DVeQnsAQmi3YnN5BydvDus=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfQiFL14bMF+QBvKZjtp1242YXcjlNBf4MWDIl79Sd78N27bHLT1hYWHd2bYmTdIBNfGdb+dwtr6xuZWcbu0s7u3f1A+PGrpOFUMmywWseoEVKPgEpuGG4GdRCGNAoHtYHw3q7efUGkeywczSdCP6FDykDNqrNW46ZcrbtWdi6yCl0MFctX75a/eIGZphNIwQbXuem5i/Iwqw5nAaamXakwoG9Mhdi1KGqH2s/miU3JmnQEJY2WfNGTu/p7IaKT1JApsZ0TNSC/XZuZ/tW5qwms/4zJJDUq2+ChMBTExmV1NBlwhM2JigTLF7a6EjaiizNhsSjYEb/nkVWhdVD3LjctK7TaPowgncArn4MEV1OAe6tAEBgjP8ApvzqPz4rw7H4vWgpPPHMMfOZ8/jM+MwQ==</latexit><latexit sha1_base64="8uDF8DVeQnsAQmi3YnN5BydvDus=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfQiFL14bMF+QBvKZjtp1242YXcjlNBf4MWDIl79Sd78N27bHLT1hYWHd2bYmTdIBNfGdb+dwtr6xuZWcbu0s7u3f1A+PGrpOFUMmywWseoEVKPgEpuGG4GdRCGNAoHtYHw3q7efUGkeywczSdCP6FDykDNqrNW46ZcrbtWdi6yCl0MFctX75a/eIGZphNIwQbXuem5i/Iwqw5nAaamXakwoG9Mhdi1KGqH2s/miU3JmnQEJY2WfNGTu/p7IaKT1JApsZ0TNSC/XZuZ/tW5qwms/4zJJDUq2+ChMBTExmV1NBlwhM2JigTLF7a6EjaiizNhsSjYEb/nkVWhdVD3LjctK7TaPowgncArn4MEV1OAe6tAEBgjP8ApvzqPz4rw7H4vWgpPPHMMfOZ8/jM+MwQ==</latexit><latexit sha1_base64="8uDF8DVeQnsAQmi3YnN5BydvDus=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfQiFL14bMF+QBvKZjtp1242YXcjlNBf4MWDIl79Sd78N27bHLT1hYWHd2bYmTdIBNfGdb+dwtr6xuZWcbu0s7u3f1A+PGrpOFUMmywWseoEVKPgEpuGG4GdRCGNAoHtYHw3q7efUGkeywczSdCP6FDykDNqrNW46ZcrbtWdi6yCl0MFctX75a/eIGZphNIwQbXuem5i/Iwqw5nAaamXakwoG9Mhdi1KGqH2s/miU3JmnQEJY2WfNGTu/p7IaKT1JApsZ0TNSC/XZuZ/tW5qwms/4zJJDUq2+ChMBTExmV1NBlwhM2JigTLF7a6EjaiizNhsSjYEb/nkVWhdVD3LjctK7TaPowgncArn4MEV1OAe6tAEBgjP8ApvzqPz4rw7H4vWgpPPHMMfOZ8/jM+MwQ==</latexit><latexit sha1_base64="8uDF8DVeQnsAQmi3YnN5BydvDus=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfQiFL14bMF+QBvKZjtp1242YXcjlNBf4MWDIl79Sd78N27bHLT1hYWHd2bYmTdIBNfGdb+dwtr6xuZWcbu0s7u3f1A+PGrpOFUMmywWseoEVKPgEpuGG4GdRCGNAoHtYHw3q7efUGkeywczSdCP6FDykDNqrNW46ZcrbtWdi6yCl0MFctX75a/eIGZphNIwQbXuem5i/Iwqw5nAaamXakwoG9Mhdi1KGqH2s/miU3JmnQEJY2WfNGTu/p7IaKT1JApsZ0TNSC/XZuZ/tW5qwms/4zJJDUq2+ChMBTExmV1NBlwhM2JigTLF7a6EjaiizNhsSjYEb/nkVWhdVD3LjctK7TaPowgncArn4MEV1OAe6tAEBgjP8ApvzqPz4rw7H4vWgpPPHMMfOZ8/jM+MwQ==</latexit>
Cons Diss DissCons
• Second-level: is that effect systematic across subjects?
- Cons. vs. diss
- Correlation w/ ΔRating ˆb(c) ˆb(d) = 0 + {u(c) u(d)}. 1 + ✏<latexit sha1_base64="zMF3tHpWX3TbMM3Jv8GFRRu4K4Q=">AAADd3icbVLLjtMwFHVTHkN5dWAHCyxKS6KZVgkaadggDbCBXUF0pqIOleM4jTWJE9nOSJXlT+Dn2PEfbNjh9M10LEU6vveccx9xVGZMKt//3XCat27fuXtwr3X/wcNHj9uHT85lUQlCR6TICjGOsKQZ43SkmMrouBQU51FGL6LLj3X+4ooKyQr+Tc1LGuZ4xlnCCFY2ND1s/EQpVlqjHKs0SmBkjEu8/n4w9uA7iCKq8NSHR1CjRe2JmEWhDo79Y98gXdXSylKRGSypgbFcREvJsoK3uhvH8VQPh5+NqT0zmqhJbT1jXGMh8NxoQoiB6AqLMmWwB7c6ROJCQZvsbdKIMEGgu+XANcmzpXm88oRIsFmqwp0uRF1/R2g5WCqYekuDnfntFO+/uoF38zQLn/VlDvsb/B1pd4Nfb5D3Q/cDsyWtlchM2x1/4C8O3AfBCnTA6gyn7V8oLkiVU65IhqWcBH6pQjuyYiSjpoUqSUtMLvGMTizkOKcy1It/Z2DXRmKYFMJ+3C6sju4qNM6lnOd2/G7doryeq4M35SaVSt6GmvGyUpSTZaGkyqAqYP0IYcwEJSqbW4CJYLZXSFIsMFH2qbbsEoLrI++D8zeDwOIvJ52zD6t1HIDn4CVwQQBOwRn4BIZgBEjjj/PM6TivnL/NF81e011SncZK8xT8d5rBP3zBFL8=</latexit><latexit sha1_base64="zMF3tHpWX3TbMM3Jv8GFRRu4K4Q=">AAADd3icbVLLjtMwFHVTHkN5dWAHCyxKS6KZVgkaadggDbCBXUF0pqIOleM4jTWJE9nOSJXlT+Dn2PEfbNjh9M10LEU6vveccx9xVGZMKt//3XCat27fuXtwr3X/wcNHj9uHT85lUQlCR6TICjGOsKQZ43SkmMrouBQU51FGL6LLj3X+4ooKyQr+Tc1LGuZ4xlnCCFY2ND1s/EQpVlqjHKs0SmBkjEu8/n4w9uA7iCKq8NSHR1CjRe2JmEWhDo79Y98gXdXSylKRGSypgbFcREvJsoK3uhvH8VQPh5+NqT0zmqhJbT1jXGMh8NxoQoiB6AqLMmWwB7c6ROJCQZvsbdKIMEGgu+XANcmzpXm88oRIsFmqwp0uRF1/R2g5WCqYekuDnfntFO+/uoF38zQLn/VlDvsb/B1pd4Nfb5D3Q/cDsyWtlchM2x1/4C8O3AfBCnTA6gyn7V8oLkiVU65IhqWcBH6pQjuyYiSjpoUqSUtMLvGMTizkOKcy1It/Z2DXRmKYFMJ+3C6sju4qNM6lnOd2/G7doryeq4M35SaVSt6GmvGyUpSTZaGkyqAqYP0IYcwEJSqbW4CJYLZXSFIsMFH2qbbsEoLrI++D8zeDwOIvJ52zD6t1HIDn4CVwQQBOwRn4BIZgBEjjj/PM6TivnL/NF81e011SncZK8xT8d5rBP3zBFL8=</latexit><latexit sha1_base64="zMF3tHpWX3TbMM3Jv8GFRRu4K4Q=">AAADd3icbVLLjtMwFHVTHkN5dWAHCyxKS6KZVgkaadggDbCBXUF0pqIOleM4jTWJE9nOSJXlT+Dn2PEfbNjh9M10LEU6vveccx9xVGZMKt//3XCat27fuXtwr3X/wcNHj9uHT85lUQlCR6TICjGOsKQZ43SkmMrouBQU51FGL6LLj3X+4ooKyQr+Tc1LGuZ4xlnCCFY2ND1s/EQpVlqjHKs0SmBkjEu8/n4w9uA7iCKq8NSHR1CjRe2JmEWhDo79Y98gXdXSylKRGSypgbFcREvJsoK3uhvH8VQPh5+NqT0zmqhJbT1jXGMh8NxoQoiB6AqLMmWwB7c6ROJCQZvsbdKIMEGgu+XANcmzpXm88oRIsFmqwp0uRF1/R2g5WCqYekuDnfntFO+/uoF38zQLn/VlDvsb/B1pd4Nfb5D3Q/cDsyWtlchM2x1/4C8O3AfBCnTA6gyn7V8oLkiVU65IhqWcBH6pQjuyYiSjpoUqSUtMLvGMTizkOKcy1It/Z2DXRmKYFMJ+3C6sju4qNM6lnOd2/G7doryeq4M35SaVSt6GmvGyUpSTZaGkyqAqYP0IYcwEJSqbW4CJYLZXSFIsMFH2qbbsEoLrI++D8zeDwOIvJ52zD6t1HIDn4CVwQQBOwRn4BIZgBEjjj/PM6TivnL/NF81e011SncZK8xT8d5rBP3zBFL8=</latexit><latexit sha1_base64="zMF3tHpWX3TbMM3Jv8GFRRu4K4Q=">AAADd3icbVLLjtMwFHVTHkN5dWAHCyxKS6KZVgkaadggDbCBXUF0pqIOleM4jTWJE9nOSJXlT+Dn2PEfbNjh9M10LEU6vveccx9xVGZMKt//3XCat27fuXtwr3X/wcNHj9uHT85lUQlCR6TICjGOsKQZ43SkmMrouBQU51FGL6LLj3X+4ooKyQr+Tc1LGuZ4xlnCCFY2ND1s/EQpVlqjHKs0SmBkjEu8/n4w9uA7iCKq8NSHR1CjRe2JmEWhDo79Y98gXdXSylKRGSypgbFcREvJsoK3uhvH8VQPh5+NqT0zmqhJbT1jXGMh8NxoQoiB6AqLMmWwB7c6ROJCQZvsbdKIMEGgu+XANcmzpXm88oRIsFmqwp0uRF1/R2g5WCqYekuDnfntFO+/uoF38zQLn/VlDvsb/B1pd4Nfb5D3Q/cDsyWtlchM2x1/4C8O3AfBCnTA6gyn7V8oLkiVU65IhqWcBH6pQjuyYiSjpoUqSUtMLvGMTizkOKcy1It/Z2DXRmKYFMJ+3C6sju4qNM6lnOd2/G7doryeq4M35SaVSt6GmvGyUpSTZaGkyqAqYP0IYcwEJSqbW4CJYLZXSFIsMFH2qbbsEoLrI++D8zeDwOIvJ52zD6t1HIDn4CVwQQBOwRn4BIZgBEjjj/PM6TivnL/NF81e011SncZK8xT8d5rBP3zBFL8=</latexit>
ˆb(c) ˆb(d) = 0 + ✏<latexit sha1_base64="oDvtLy3w079Ih2ZBiY6GwHQNRYg=">AAADy3iclVJNb9MwGHYbPkb5WAdHLhZVRyPWKkFIcEEacBkHpILoVlGHynGcxlriRLYzKMZH/iA3jvwTnDb92LoLliI9ft/ned6POCxSJpXn/Wk0nRs3b93eu9O6e+/+g/32wcNTmZeC0BHJ01yMQyxpyjgdKaZSOi4ExVmY0rPw/F2VP7ugQrKcf1bzggYZnnEWM4KVDU0PGn9RgpXWKMMqCWMYGtMjbn83GLnwNdRoUXEiZmGg/SPvyDMopApPPQOfQUQLydKct7r/41kbWP117rqspKWlIjNYUv3LtdaO46keDt8bU3mmNFaTynrGuMZC4LnRhBAD0QUWRcLgIdzoEIlyBW3ycJ1GhAkCexsOXJFcW5pHtSdEgs0SFWx1Iar6W0LLwVLBxF0abM1vp3jzqee710+z8Fld5rC/xl+Q7q3x0zVyv+q+bzaklRKZabvjDbzFgbvAr0EH1Gc4bf9GUU7KjHJFUizlxPcKFdiRFSMpNS1USlpgco5ndGIhxxmVgV78OwO7NhLBOBf243ZhVXRboXEm5Tyz43erFuXVXBW8LjcpVfwq0IwXpaKcLAvFZQpVDquHDSMmKFHp3AJMBLO9QpJggYmyz79ll+BfHXkXnD4f+BZ/fNE5fluvYw88Bk9AD/jgJTgGJ2AIRoA0T5q8+a353fngSOeH83NJbTZqzSNw6Ti//gGX0DOM</latexit><latexit sha1_base64="oDvtLy3w079Ih2ZBiY6GwHQNRYg=">AAADy3iclVJNb9MwGHYbPkb5WAdHLhZVRyPWKkFIcEEacBkHpILoVlGHynGcxlriRLYzKMZH/iA3jvwTnDb92LoLliI9ft/ned6POCxSJpXn/Wk0nRs3b93eu9O6e+/+g/32wcNTmZeC0BHJ01yMQyxpyjgdKaZSOi4ExVmY0rPw/F2VP7ugQrKcf1bzggYZnnEWM4KVDU0PGn9RgpXWKMMqCWMYGtMjbn83GLnwNdRoUXEiZmGg/SPvyDMopApPPQOfQUQLydKct7r/41kbWP117rqspKWlIjNYUv3LtdaO46keDt8bU3mmNFaTynrGuMZC4LnRhBAD0QUWRcLgIdzoEIlyBW3ycJ1GhAkCexsOXJFcW5pHtSdEgs0SFWx1Iar6W0LLwVLBxF0abM1vp3jzqee710+z8Fld5rC/xl+Q7q3x0zVyv+q+bzaklRKZabvjDbzFgbvAr0EH1Gc4bf9GUU7KjHJFUizlxPcKFdiRFSMpNS1USlpgco5ndGIhxxmVgV78OwO7NhLBOBf243ZhVXRboXEm5Tyz43erFuXVXBW8LjcpVfwq0IwXpaKcLAvFZQpVDquHDSMmKFHp3AJMBLO9QpJggYmyz79ll+BfHXkXnD4f+BZ/fNE5fluvYw88Bk9AD/jgJTgGJ2AIRoA0T5q8+a353fngSOeH83NJbTZqzSNw6Ti//gGX0DOM</latexit><latexit sha1_base64="oDvtLy3w079Ih2ZBiY6GwHQNRYg=">AAADy3iclVJNb9MwGHYbPkb5WAdHLhZVRyPWKkFIcEEacBkHpILoVlGHynGcxlriRLYzKMZH/iA3jvwTnDb92LoLliI9ft/ned6POCxSJpXn/Wk0nRs3b93eu9O6e+/+g/32wcNTmZeC0BHJ01yMQyxpyjgdKaZSOi4ExVmY0rPw/F2VP7ugQrKcf1bzggYZnnEWM4KVDU0PGn9RgpXWKMMqCWMYGtMjbn83GLnwNdRoUXEiZmGg/SPvyDMopApPPQOfQUQLydKct7r/41kbWP117rqspKWlIjNYUv3LtdaO46keDt8bU3mmNFaTynrGuMZC4LnRhBAD0QUWRcLgIdzoEIlyBW3ycJ1GhAkCexsOXJFcW5pHtSdEgs0SFWx1Iar6W0LLwVLBxF0abM1vp3jzqee710+z8Fld5rC/xl+Q7q3x0zVyv+q+bzaklRKZabvjDbzFgbvAr0EH1Gc4bf9GUU7KjHJFUizlxPcKFdiRFSMpNS1USlpgco5ndGIhxxmVgV78OwO7NhLBOBf243ZhVXRboXEm5Tyz43erFuXVXBW8LjcpVfwq0IwXpaKcLAvFZQpVDquHDSMmKFHp3AJMBLO9QpJggYmyz79ll+BfHXkXnD4f+BZ/fNE5fluvYw88Bk9AD/jgJTgGJ2AIRoA0T5q8+a353fngSOeH83NJbTZqzSNw6Ti//gGX0DOM</latexit><latexit sha1_base64="oDvtLy3w079Ih2ZBiY6GwHQNRYg=">AAADy3iclVJNb9MwGHYbPkb5WAdHLhZVRyPWKkFIcEEacBkHpILoVlGHynGcxlriRLYzKMZH/iA3jvwTnDb92LoLliI9ft/ned6POCxSJpXn/Wk0nRs3b93eu9O6e+/+g/32wcNTmZeC0BHJ01yMQyxpyjgdKaZSOi4ExVmY0rPw/F2VP7ugQrKcf1bzggYZnnEWM4KVDU0PGn9RgpXWKMMqCWMYGtMjbn83GLnwNdRoUXEiZmGg/SPvyDMopApPPQOfQUQLydKct7r/41kbWP117rqspKWlIjNYUv3LtdaO46keDt8bU3mmNFaTynrGuMZC4LnRhBAD0QUWRcLgIdzoEIlyBW3ycJ1GhAkCexsOXJFcW5pHtSdEgs0SFWx1Iar6W0LLwVLBxF0abM1vp3jzqee710+z8Fld5rC/xl+Q7q3x0zVyv+q+bzaklRKZabvjDbzFgbvAr0EH1Gc4bf9GUU7KjHJFUizlxPcKFdiRFSMpNS1USlpgco5ndGIhxxmVgV78OwO7NhLBOBf243ZhVXRboXEm5Tyz43erFuXVXBW8LjcpVfwq0IwXpaKcLAvFZQpVDquHDSMmKFHp3AJMBLO9QpJggYmyz79ll+BfHXkXnD4f+BZ/fNE5fluvYw88Bk9AD/jgJTgGJ2AIRoA0T5q8+a353fngSOeH83NJbTZqzSNw6Ti//gGX0DOM</latexit>
* Family-wise error rate is controlled by a topological approach based on Gaussian Random Field Theory

19. / 45
• Condition-dependent functional connectivity
- Psychophysiological interaction: seed vs. whole-brain
• Cross-correlation analysis between specific two regions
Functional connectivity analysis
19
r = (XP P I ⇤ h) · b + AR(1) + ✏<latexit sha1_base64="PtogYxIMn4VvnJIBlc9nS9IKSpI=">AAADCnicbVJNb9QwEHXCVxs+uoUjF4vVVruiRAlCggtSgQvcFsS2K8XRynGcxKrjRLZTaWXlzIW/woUDCPXKL+DGv8HZDXTbZSRLz2/ezDyPnNScKR0Evx332vUbN2/t7Hq379y9tzfYv3+sqkYSOiMVr+Q8wYpyJuhMM83pvJYUlwmnJ8npmy5/ckalYpX4qJc1jUucC5YxgrWlFvsOHBlUYl0kGUzaMZk82bymE/gSooRqvAjgY2jQamAk8yQ24WFwGLTINF1RY6UGtf5aG7ZWjGitGK+EdzFgvjDT6bu27Zpymumo650zYbCUeNkaQkgL0RmWdcHgAbyo8y1/8DfjX+YRFWnfACLJ8kLH3j+F7GaNtw0grDQsJoiklYYbL7a+X30Yh5MN/4vBMPCDVcBtEPZgCPqYLga/UFqRpqRCE46VisKg1rG1qBnhtPVQo2iNySnOaWShwCVVsVlttoUjy6Qwq6Q9QsMVu1lhcKnUsrROR51pdTXXkf/LRY3OXsSGibrRVJD1oKzhUFew+xcwZZISzZcWYCKZ9QpJgSUm2v4ezy4hvPrkbXD81A8tfv9sePS6X8cOeAgegTEIwXNwBN6CKZgB4nxyvjjfnO/uZ/er+8M9X0tdp695AC6F+/MPy4zxVQ==</latexit><latexit sha1_base64="PtogYxIMn4VvnJIBlc9nS9IKSpI=">AAADCnicbVJNb9QwEHXCVxs+uoUjF4vVVruiRAlCggtSgQvcFsS2K8XRynGcxKrjRLZTaWXlzIW/woUDCPXKL+DGv8HZDXTbZSRLz2/ezDyPnNScKR0Evx332vUbN2/t7Hq379y9tzfYv3+sqkYSOiMVr+Q8wYpyJuhMM83pvJYUlwmnJ8npmy5/ckalYpX4qJc1jUucC5YxgrWlFvsOHBlUYl0kGUzaMZk82bymE/gSooRqvAjgY2jQamAk8yQ24WFwGLTINF1RY6UGtf5aG7ZWjGitGK+EdzFgvjDT6bu27Zpymumo650zYbCUeNkaQkgL0RmWdcHgAbyo8y1/8DfjX+YRFWnfACLJ8kLH3j+F7GaNtw0grDQsJoiklYYbL7a+X30Yh5MN/4vBMPCDVcBtEPZgCPqYLga/UFqRpqRCE46VisKg1rG1qBnhtPVQo2iNySnOaWShwCVVsVlttoUjy6Qwq6Q9QsMVu1lhcKnUsrROR51pdTXXkf/LRY3OXsSGibrRVJD1oKzhUFew+xcwZZISzZcWYCKZ9QpJgSUm2v4ezy4hvPrkbXD81A8tfv9sePS6X8cOeAgegTEIwXNwBN6CKZgB4nxyvjjfnO/uZ/er+8M9X0tdp695AC6F+/MPy4zxVQ==</latexit><latexit sha1_base64="PtogYxIMn4VvnJIBlc9nS9IKSpI=">AAADCnicbVJNb9QwEHXCVxs+uoUjF4vVVruiRAlCggtSgQvcFsS2K8XRynGcxKrjRLZTaWXlzIW/woUDCPXKL+DGv8HZDXTbZSRLz2/ezDyPnNScKR0Evx332vUbN2/t7Hq379y9tzfYv3+sqkYSOiMVr+Q8wYpyJuhMM83pvJYUlwmnJ8npmy5/ckalYpX4qJc1jUucC5YxgrWlFvsOHBlUYl0kGUzaMZk82bymE/gSooRqvAjgY2jQamAk8yQ24WFwGLTINF1RY6UGtf5aG7ZWjGitGK+EdzFgvjDT6bu27Zpymumo650zYbCUeNkaQkgL0RmWdcHgAbyo8y1/8DfjX+YRFWnfACLJ8kLH3j+F7GaNtw0grDQsJoiklYYbL7a+X30Yh5MN/4vBMPCDVcBtEPZgCPqYLga/UFqRpqRCE46VisKg1rG1qBnhtPVQo2iNySnOaWShwCVVsVlttoUjy6Qwq6Q9QsMVu1lhcKnUsrROR51pdTXXkf/LRY3OXsSGibrRVJD1oKzhUFew+xcwZZISzZcWYCKZ9QpJgSUm2v4ezy4hvPrkbXD81A8tfv9sePS6X8cOeAgegTEIwXNwBN6CKZgB4nxyvjjfnO/uZ/er+8M9X0tdp695AC6F+/MPy4zxVQ==</latexit><latexit sha1_base64="PtogYxIMn4VvnJIBlc9nS9IKSpI=">AAADCnicbVJNb9QwEHXCVxs+uoUjF4vVVruiRAlCggtSgQvcFsS2K8XRynGcxKrjRLZTaWXlzIW/woUDCPXKL+DGv8HZDXTbZSRLz2/ezDyPnNScKR0Evx332vUbN2/t7Hq379y9tzfYv3+sqkYSOiMVr+Q8wYpyJuhMM83pvJYUlwmnJ8npmy5/ckalYpX4qJc1jUucC5YxgrWlFvsOHBlUYl0kGUzaMZk82bymE/gSooRqvAjgY2jQamAk8yQ24WFwGLTINF1RY6UGtf5aG7ZWjGitGK+EdzFgvjDT6bu27Zpymumo650zYbCUeNkaQkgL0RmWdcHgAbyo8y1/8DfjX+YRFWnfACLJ8kLH3j+F7GaNtw0grDQsJoiklYYbL7a+X30Yh5MN/4vBMPCDVcBtEPZgCPqYLga/UFqRpqRCE46VisKg1rG1qBnhtPVQo2iNySnOaWShwCVVsVlttoUjy6Qwq6Q9QsMVu1lhcKnUsrROR51pdTXXkf/LRY3OXsSGibrRVJD1oKzhUFew+xcwZZISzZcWYCKZ9QpJgSUm2v4ezy4hvPrkbXD81A8tfv9sePS6X8cOeAgegTEIwXNwBN6CKZgB4nxyvjjfnO/uZ/er+8M9X0tdp695AC6F+/MPy4zxVQ==</latexit>
XP P I =
⇥
' X · c ' (X · c)
⇤
<latexit sha1_base64="vNoqdozoIwfhAhixbLB5a/PtF4U=">AAADbHicbVLLjtMwFHVTHkN5TAdYgEZIFlVLo5lWCUKCDdIAG9gVRGcq6lA5jtNa4ziR7YxUWd7wiez4BDZ8A07fdLhSpON7z7nX58ZxwZnSQfCr5tVv3Lx1++BO4+69+w8Om0cPz1VeSkKHJOe5HMVYUc4EHWqmOR0VkuIs5vQivvxQ1S+uqFQsF1/1vKBRhqeCpYxg7VKTo9qPtkEZ1rM4hbHtEr+3e0x8+BaimGo8CeAJNGgxcCyncWTC0+A0sMiUlah0VINsf8kNrSMjWijGc9HYNBxNzGDwydqqJ6epHletp0wYLCWeW0MIsRBdYVnMGOzArQ6RJNfQFTubMiJMEtjdcuCa5LvJIln1hEiy6UxHja1LWc3fEToOVhrO/GWDHfvOxLsv3dDfNbPXZ32Yw94Gf0Omu8EvNsj/bnqh3ZLWSmQnzVbQDxYBr4NwBVpgFYNJ8ydKclJmVGjCsVLjMCh05CxrRji1DVQqWmByiad07KDAGVWRWfw7C9suk8A0l+4TbmFVdldhcKbUPHP229UV1X6tSv6vNi51+iYyTBSlpoIsB6UlhzqH1cuDCZOUaD53ABPJ3F0hmWGJiXbvs+GWEO5bvg7OX/ZDhz+/ap29X63jAByD56ALQvAanIGPYACGgNR+e4feE++p96f+uH5cf7akerWV5hH4J+qdv5M2D6o=</latexit><latexit sha1_base64="vNoqdozoIwfhAhixbLB5a/PtF4U=">AAADbHicbVLLjtMwFHVTHkN5TAdYgEZIFlVLo5lWCUKCDdIAG9gVRGcq6lA5jtNa4ziR7YxUWd7wiez4BDZ8A07fdLhSpON7z7nX58ZxwZnSQfCr5tVv3Lx1++BO4+69+w8Om0cPz1VeSkKHJOe5HMVYUc4EHWqmOR0VkuIs5vQivvxQ1S+uqFQsF1/1vKBRhqeCpYxg7VKTo9qPtkEZ1rM4hbHtEr+3e0x8+BaimGo8CeAJNGgxcCyncWTC0+A0sMiUlah0VINsf8kNrSMjWijGc9HYNBxNzGDwydqqJ6epHletp0wYLCWeW0MIsRBdYVnMGOzArQ6RJNfQFTubMiJMEtjdcuCa5LvJIln1hEiy6UxHja1LWc3fEToOVhrO/GWDHfvOxLsv3dDfNbPXZ32Yw94Gf0Omu8EvNsj/bnqh3ZLWSmQnzVbQDxYBr4NwBVpgFYNJ8ydKclJmVGjCsVLjMCh05CxrRji1DVQqWmByiad07KDAGVWRWfw7C9suk8A0l+4TbmFVdldhcKbUPHP229UV1X6tSv6vNi51+iYyTBSlpoIsB6UlhzqH1cuDCZOUaD53ABPJ3F0hmWGJiXbvs+GWEO5bvg7OX/ZDhz+/ap29X63jAByD56ALQvAanIGPYACGgNR+e4feE++p96f+uH5cf7akerWV5hH4J+qdv5M2D6o=</latexit><latexit sha1_base64="vNoqdozoIwfhAhixbLB5a/PtF4U=">AAADbHicbVLLjtMwFHVTHkN5TAdYgEZIFlVLo5lWCUKCDdIAG9gVRGcq6lA5jtNa4ziR7YxUWd7wiez4BDZ8A07fdLhSpON7z7nX58ZxwZnSQfCr5tVv3Lx1++BO4+69+w8Om0cPz1VeSkKHJOe5HMVYUc4EHWqmOR0VkuIs5vQivvxQ1S+uqFQsF1/1vKBRhqeCpYxg7VKTo9qPtkEZ1rM4hbHtEr+3e0x8+BaimGo8CeAJNGgxcCyncWTC0+A0sMiUlah0VINsf8kNrSMjWijGc9HYNBxNzGDwydqqJ6epHletp0wYLCWeW0MIsRBdYVnMGOzArQ6RJNfQFTubMiJMEtjdcuCa5LvJIln1hEiy6UxHja1LWc3fEToOVhrO/GWDHfvOxLsv3dDfNbPXZ32Yw94Gf0Omu8EvNsj/bnqh3ZLWSmQnzVbQDxYBr4NwBVpgFYNJ8ydKclJmVGjCsVLjMCh05CxrRji1DVQqWmByiad07KDAGVWRWfw7C9suk8A0l+4TbmFVdldhcKbUPHP229UV1X6tSv6vNi51+iYyTBSlpoIsB6UlhzqH1cuDCZOUaD53ABPJ3F0hmWGJiXbvs+GWEO5bvg7OX/ZDhz+/ap29X63jAByD56ALQvAanIGPYACGgNR+e4feE++p96f+uH5cf7akerWV5hH4J+qdv5M2D6o=</latexit><latexit sha1_base64="vNoqdozoIwfhAhixbLB5a/PtF4U=">AAADbHicbVLLjtMwFHVTHkN5TAdYgEZIFlVLo5lWCUKCDdIAG9gVRGcq6lA5jtNa4ziR7YxUWd7wiez4BDZ8A07fdLhSpON7z7nX58ZxwZnSQfCr5tVv3Lx1++BO4+69+w8Om0cPz1VeSkKHJOe5HMVYUc4EHWqmOR0VkuIs5vQivvxQ1S+uqFQsF1/1vKBRhqeCpYxg7VKTo9qPtkEZ1rM4hbHtEr+3e0x8+BaimGo8CeAJNGgxcCyncWTC0+A0sMiUlah0VINsf8kNrSMjWijGc9HYNBxNzGDwydqqJ6epHletp0wYLCWeW0MIsRBdYVnMGOzArQ6RJNfQFTubMiJMEtjdcuCa5LvJIln1hEiy6UxHja1LWc3fEToOVhrO/GWDHfvOxLsv3dDfNbPXZ32Yw94Gf0Omu8EvNsj/bnqh3ZLWSmQnzVbQDxYBr4NwBVpgFYNJ8ydKclJmVGjCsVLjMCh05CxrRji1DVQqWmByiad07KDAGVWRWfw7C9suk8A0l+4TbmFVdldhcKbUPHP229UV1X6tSv6vNi51+iYyTBSlpoIsB6UlhzqH1cuDCZOUaD53ABPJ3F0hmWGJiXbvs+GWEO5bvg7OX/ZDhz+/ap29X63jAByD56ALQvAanIGPYACGgNR+e4feE++p96f+uH5cf7akerWV5hH4J+qdv5M2D6o=</latexit>

20. / 45
• All participants rated dissonant music
unpleasant compared to consonant
music.
• Rating difference
Individual disfavor against dissonance
20
Kim et al. (2016). Sci Rep.
Behavioral results

21. / 45
Consonant vs. dissonant harmony
21
• Decreased BOLD signal in superior temporal
cortices, ventromedial frontal cortex, thalamus, and
inferior colliculus [IC] for dissonant music
Kim et al. (2016). Sci Rep.

22. / 45
Correlates of dissonance preference
• Negative correlation between differences in BOLD (cons. - diss.) in the IC
(inferior colliculus in the brainstem) and differences in rating (cons. - diss.)
• The more one rated dissonant music unpleasant, the more one’s IC was
deactivated by dissonance.
22
Kim et al. (2016). Sci Rep.

23. / 45
Functional connectivity of IC
modulated by dissonance
• Decreased functional connectivity between the IC &
the left anterior STG for dissonant music
23
Kim et al. (2016). Sci Rep.

24. / 45
Cross-correlation correlates with
dissonance preference
• Cross-correlation changes in positive lags (aSTG before IC; corticofugal)
correlated with rating differences
24
Kim et al. (2016). Sci Rep.

25. / 45
Decrease of BOLD activation & functional
connectivity in IC by dissonance harmony
• Corticofugal connections that
modifies IC neurons tuning
curves [1]
e.g., disengagement of adverse
sounds
25
[1] Suga. (2008). J Comp Physiol.
[1]

26. / 45
Correlation with subjective rating of
unpleasantness
• Possible contribution of subcortical sensitivity to
sensory information to aesthetical judgement [1,2]
- Strong disfavor against dissonant harmony due to vivid
representation of sensory dissonance in the IC
- Possibly, influencing the preference of particular musical style
in part
26
[1] Koelsch et al. (2006). Hum Brain Map. [2] Sammler et al. (2007). PsychPhysiol.

27. / 45
Interim summary
• Sensory dissonance deactivated auditory brain
regions.
• The degree of deactivation in the brainstem was
correlated with the degree of individual disfavor
against dissonance.
• Such lower-level encoding might be one of the
foundations of musical preference.
27

28. / 45
Overview
• Classical cognitive neuroscience (behavior-neural association)
- Dissonance preference associated with brainstem response
• Model-based cognitive neuroscience (feature-neural
association)
- Cortical tracking of tonal stability in naturalistic music
• General discussion
28

29. / 45
Cortical tracking of tonal
stability in naturalistic music
Encoding analysis on EEG
29
Leahy J*., Kim S-G*, Wan J., Overath T. (in prep).

30. / 45
Motivation
• Epoch-based experiments to isolate and disambiguate related
processes
- How the individual processes interact with each other remains
unknown.
- Often manipulations are disruptive (analogous to lesions) and
unrealistic.
• On the other hand, all responses are intermixed in naturalistic
experiments. How can we decompose them?
- Temporal response function (TRF) in natural speech analysis.
30

31. / 45
Cortical encoding of melodic
expectation in monophonic music in EEG
31
[1] Di Liberto et al. (2020). eLife. [2] Pearce. (2005). Thesis.
• Melody expectation
surprisal and entropy based on musical corpus [2]
• Better predicted EEG signal when adding musical features
AcousticsExpectation
[1]

32. / 45
Temporal response function (TRF)
• Brain (or whatever
between stimuli and
neural measures) as
a linear filter
32
Crosse et al. (2016). Front Hum Neurosci.

33. / 45
Ridge regression
• Why do you solve something else?
- to solve the unsolvable (λI makes
covariance matrix invertable)
- to prevent overfitting (less sensitive to
noise in training sets)
• How do you determine λ?
- Cross-validation (commonly leave-one-
trial-out)
33
s(t)<latexit sha1_base64="/eM9EEOZtdxnzHYz0Wypr6mCGHc=">AAAD23iclVPLjtMwFHUbHkN5TAeWbCyqDomYVglCgg3SABvYFURnKupQOY7TWJOXbGekyjIbFiDElh9jx2fwBzhtmnams8FSpJN7zzn3kTgoEiak6/5pta1r12/c3LvVuX3n7r397sH9E5GXnNAxyZOcTwIsaMIyOpZMJnRScIrTIKGnwdmbKn96TrlgefZRLgrqp3iesYgRLE1odtD620cxlkqhFMs4iGCgtU2cwW4wdOBLqNCy5JTPA195R+6Rq1FAJZ65Gj6BiBaCJXnW+S/P2sDor3JXZSUtDRXp4YrqXazVOE5majR6p3XlmdBITivrOcsU5hwvtCKEaIjOMS9iBg/hRodImEtokodNGhHGCbQ3HLgmOaZ0FtaeEHE2j6W/1QWv6m8JDQcLCWNnZbA1v5ni1Qfbc66eZumzflnAQYM/IWU3+HGDnM9q4OkNaa1EutN4Cls6etbtuUN3eeAu8GrQA/UZzbq/UZiTMqWZJAkWYuq5hfTNBiQjCdUdVApaYHKG53RqYIZTKny1/JQa9k0khFHOzZOZ/VXRbYXCqRCL1GyjXzUpLueq4FW5aSmjF75iWVFKmpFVoahMoMxh9aPDkHFKZLIwABPOTK+QxJhjIs116JgleJdH3gUnT4eewe+f9Y5f1+vYAw/BI2ADDzwHx+AtGIExIO1J+0v7W/u75VtfrR/WzxW13ao1D8CFY/36B1JjOXk=</latexit><latexit sha1_base64="/eM9EEOZtdxnzHYz0Wypr6mCGHc=">AAAD23iclVPLjtMwFHUbHkN5TAeWbCyqDomYVglCgg3SABvYFURnKupQOY7TWJOXbGekyjIbFiDElh9jx2fwBzhtmnams8FSpJN7zzn3kTgoEiak6/5pta1r12/c3LvVuX3n7r397sH9E5GXnNAxyZOcTwIsaMIyOpZMJnRScIrTIKGnwdmbKn96TrlgefZRLgrqp3iesYgRLE1odtD620cxlkqhFMs4iGCgtU2cwW4wdOBLqNCy5JTPA195R+6Rq1FAJZ65Gj6BiBaCJXnW+S/P2sDor3JXZSUtDRXp4YrqXazVOE5majR6p3XlmdBITivrOcsU5hwvtCKEaIjOMS9iBg/hRodImEtokodNGhHGCbQ3HLgmOaZ0FtaeEHE2j6W/1QWv6m8JDQcLCWNnZbA1v5ni1Qfbc66eZumzflnAQYM/IWU3+HGDnM9q4OkNaa1EutN4Cls6etbtuUN3eeAu8GrQA/UZzbq/UZiTMqWZJAkWYuq5hfTNBiQjCdUdVApaYHKG53RqYIZTKny1/JQa9k0khFHOzZOZ/VXRbYXCqRCL1GyjXzUpLueq4FW5aSmjF75iWVFKmpFVoahMoMxh9aPDkHFKZLIwABPOTK+QxJhjIs116JgleJdH3gUnT4eewe+f9Y5f1+vYAw/BI2ADDzwHx+AtGIExIO1J+0v7W/u75VtfrR/WzxW13ao1D8CFY/36B1JjOXk=</latexit><latexit sha1_base64="/eM9EEOZtdxnzHYz0Wypr6mCGHc=">AAAD23iclVPLjtMwFHUbHkN5TAeWbCyqDomYVglCgg3SABvYFURnKupQOY7TWJOXbGekyjIbFiDElh9jx2fwBzhtmnams8FSpJN7zzn3kTgoEiak6/5pta1r12/c3LvVuX3n7r397sH9E5GXnNAxyZOcTwIsaMIyOpZMJnRScIrTIKGnwdmbKn96TrlgefZRLgrqp3iesYgRLE1odtD620cxlkqhFMs4iGCgtU2cwW4wdOBLqNCy5JTPA195R+6Rq1FAJZ65Gj6BiBaCJXnW+S/P2sDor3JXZSUtDRXp4YrqXazVOE5majR6p3XlmdBITivrOcsU5hwvtCKEaIjOMS9iBg/hRodImEtokodNGhHGCbQ3HLgmOaZ0FtaeEHE2j6W/1QWv6m8JDQcLCWNnZbA1v5ni1Qfbc66eZumzflnAQYM/IWU3+HGDnM9q4OkNaa1EutN4Cls6etbtuUN3eeAu8GrQA/UZzbq/UZiTMqWZJAkWYuq5hfTNBiQjCdUdVApaYHKG53RqYIZTKny1/JQa9k0khFHOzZOZ/VXRbYXCqRCL1GyjXzUpLueq4FW5aSmjF75iWVFKmpFVoahMoMxh9aPDkHFKZLIwABPOTK+QxJhjIs116JgleJdH3gUnT4eewe+f9Y5f1+vYAw/BI2ADDzwHx+AtGIExIO1J+0v7W/u75VtfrR/WzxW13ao1D8CFY/36B1JjOXk=</latexit><latexit sha1_base64="/eM9EEOZtdxnzHYz0Wypr6mCGHc=">AAAD23iclVPLjtMwFHUbHkN5TAeWbCyqDomYVglCgg3SABvYFURnKupQOY7TWJOXbGekyjIbFiDElh9jx2fwBzhtmnams8FSpJN7zzn3kTgoEiak6/5pta1r12/c3LvVuX3n7r397sH9E5GXnNAxyZOcTwIsaMIyOpZMJnRScIrTIKGnwdmbKn96TrlgefZRLgrqp3iesYgRLE1odtD620cxlkqhFMs4iGCgtU2cwW4wdOBLqNCy5JTPA195R+6Rq1FAJZ65Gj6BiBaCJXnW+S/P2sDor3JXZSUtDRXp4YrqXazVOE5majR6p3XlmdBITivrOcsU5hwvtCKEaIjOMS9iBg/hRodImEtokodNGhHGCbQ3HLgmOaZ0FtaeEHE2j6W/1QWv6m8JDQcLCWNnZbA1v5ni1Qfbc66eZumzflnAQYM/IWU3+HGDnM9q4OkNaa1EutN4Cls6etbtuUN3eeAu8GrQA/UZzbq/UZiTMqWZJAkWYuq5hfTNBiQjCdUdVApaYHKG53RqYIZTKny1/JQa9k0khFHOzZOZ/VXRbYXCqRCL1GyjXzUpLueq4FW5aSmjF75iWVFKmpFVoahMoMxh9aPDkHFKZLIwABPOTK+QxJhjIs116JgleJdH3gUnT4eewe+f9Y5f1+vYAw/BI2ADDzwHx+AtGIExIO1J+0v7W/u75VtfrR/WzxW13ao1D8CFY/36B1JjOXk=</latexit>
S<latexit sha1_base64="YDKtlfshPfFsxqL7YUmMn5n+cVs=">AAAD6HiclVPLjtMwFE0THkN5TAeWbCyqDomYVglCgg3SABvYlUdnKupQOY7TmMlLtjNSZfkL2LAAIbZ8Ejt+BuG0adqZdoOlSMf3nnPuI0lQJJQL1/3TMq0rV69d37vRvnnr9p39zsHdE56XDJMRzpOcjQPESUIzMhJUJGRcMILSICGnwdmrKn96ThinefZBzAvip2iW0YhiJHRoetD624MxElLCFIk4iECglI2d/nYwdMBzIOGi5ITNAl96R+6Rq2BABJq6CjwCkBScJnnW/i/P2kDrd7nLspKWmgrVYEn1LtZqHMdTORy+UaryTEgkJpX1jGYSMYbmSmKMFYDniBUxBYdgrYM4zAXQycMmDTFlGNhrDliRHF06C2tPABmdxcLf6IJV9TeEmoO4ALGzNNiYX0/x4p3tObunWfisLnPQb/BHKO0GP2yQ80n2PbUmrZRQbZhyWziq3Vzfq2mn6w7cxQHbwKtB16jPcNr5DcMclynJBE4Q5xPPLYSv1yEoTohqw5KTAuEzNCMTDTOUEu7LxXtVoKcjIYhypp9ML7OKbiokSjmfp3o1vapDfjlXBXflJqWInvmSZkUpSIaXhaIyASIH1VcPQsoIFslcA4QZ1b0CHCOGsND/Rlsvwbs88jY4eTzwNH77pHv8sl7HnnHfeGDYhmc8NY6N18bQGBnYJOYX85v53fpsfbV+WD+XVLNVa+4ZF4716x/cTz5j</latexit><latexit sha1_base64="YDKtlfshPfFsxqL7YUmMn5n+cVs=">AAAD6HiclVPLjtMwFE0THkN5TAeWbCyqDomYVglCgg3SABvYlUdnKupQOY7TmMlLtjNSZfkL2LAAIbZ8Ejt+BuG0adqZdoOlSMf3nnPuI0lQJJQL1/3TMq0rV69d37vRvnnr9p39zsHdE56XDJMRzpOcjQPESUIzMhJUJGRcMILSICGnwdmrKn96ThinefZBzAvip2iW0YhiJHRoetD624MxElLCFIk4iECglI2d/nYwdMBzIOGi5ITNAl96R+6Rq2BABJq6CjwCkBScJnnW/i/P2kDrd7nLspKWmgrVYEn1LtZqHMdTORy+UaryTEgkJpX1jGYSMYbmSmKMFYDniBUxBYdgrYM4zAXQycMmDTFlGNhrDliRHF06C2tPABmdxcLf6IJV9TeEmoO4ALGzNNiYX0/x4p3tObunWfisLnPQb/BHKO0GP2yQ80n2PbUmrZRQbZhyWziq3Vzfq2mn6w7cxQHbwKtB16jPcNr5DcMclynJBE4Q5xPPLYSv1yEoTohqw5KTAuEzNCMTDTOUEu7LxXtVoKcjIYhypp9ML7OKbiokSjmfp3o1vapDfjlXBXflJqWInvmSZkUpSIaXhaIyASIH1VcPQsoIFslcA4QZ1b0CHCOGsND/Rlsvwbs88jY4eTzwNH77pHv8sl7HnnHfeGDYhmc8NY6N18bQGBnYJOYX85v53fpsfbV+WD+XVLNVa+4ZF4716x/cTz5j</latexit><latexit sha1_base64="YDKtlfshPfFsxqL7YUmMn5n+cVs=">AAAD6HiclVPLjtMwFE0THkN5TAeWbCyqDomYVglCgg3SABvYlUdnKupQOY7TmMlLtjNSZfkL2LAAIbZ8Ejt+BuG0adqZdoOlSMf3nnPuI0lQJJQL1/3TMq0rV69d37vRvnnr9p39zsHdE56XDJMRzpOcjQPESUIzMhJUJGRcMILSICGnwdmrKn96ThinefZBzAvip2iW0YhiJHRoetD624MxElLCFIk4iECglI2d/nYwdMBzIOGi5ITNAl96R+6Rq2BABJq6CjwCkBScJnnW/i/P2kDrd7nLspKWmgrVYEn1LtZqHMdTORy+UaryTEgkJpX1jGYSMYbmSmKMFYDniBUxBYdgrYM4zAXQycMmDTFlGNhrDliRHF06C2tPABmdxcLf6IJV9TeEmoO4ALGzNNiYX0/x4p3tObunWfisLnPQb/BHKO0GP2yQ80n2PbUmrZRQbZhyWziq3Vzfq2mn6w7cxQHbwKtB16jPcNr5DcMclynJBE4Q5xPPLYSv1yEoTohqw5KTAuEzNCMTDTOUEu7LxXtVoKcjIYhypp9ML7OKbiokSjmfp3o1vapDfjlXBXflJqWInvmSZkUpSIaXhaIyASIH1VcPQsoIFslcA4QZ1b0CHCOGsND/Rlsvwbs88jY4eTzwNH77pHv8sl7HnnHfeGDYhmc8NY6N18bQGBnYJOYX85v53fpsfbV+WD+XVLNVa+4ZF4716x/cTz5j</latexit><latexit sha1_base64="YDKtlfshPfFsxqL7YUmMn5n+cVs=">AAAD6HiclVPLjtMwFE0THkN5TAeWbCyqDomYVglCgg3SABvYlUdnKupQOY7TmMlLtjNSZfkL2LAAIbZ8Ejt+BuG0adqZdoOlSMf3nnPuI0lQJJQL1/3TMq0rV69d37vRvnnr9p39zsHdE56XDJMRzpOcjQPESUIzMhJUJGRcMILSICGnwdmrKn96ThinefZBzAvip2iW0YhiJHRoetD624MxElLCFIk4iECglI2d/nYwdMBzIOGi5ITNAl96R+6Rq2BABJq6CjwCkBScJnnW/i/P2kDrd7nLspKWmgrVYEn1LtZqHMdTORy+UaryTEgkJpX1jGYSMYbmSmKMFYDniBUxBYdgrYM4zAXQycMmDTFlGNhrDliRHF06C2tPABmdxcLf6IJV9TeEmoO4ALGzNNiYX0/x4p3tObunWfisLnPQb/BHKO0GP2yQ80n2PbUmrZRQbZhyWziq3Vzfq2mn6w7cxQHbwKtB16jPcNr5DcMclynJBE4Q5xPPLYSv1yEoTohqw5KTAuEzNCMTDTOUEu7LxXtVoKcjIYhypp9ML7OKbiokSjmfp3o1vapDfjlXBXflJqWInvmSZkUpSIaXhaIyASIH1VcPQsoIFslcA4QZ1b0CHCOGsND/Rlsvwbs88jY4eTzwNH77pHv8sl7HnnHfeGDYhmc8NY6N18bQGBnYJOYX85v53fpsfbV+WD+XVLNVa+4ZF4716x/cTz5j</latexit>
ST
S + I<latexit sha1_base64="AosgbpEP4hUJJ8fRO7veWyTg668=">AAAEFXiclVPLbhMxFHUnPEp4NIUlG4soZUZpohmEBBukAhu6C9C0EfE08ng8idV5yfZUiiz/BBt+hQ0LEGKLxI6/wZPntMkGS5aO7z3nXN8rO8hjJqTr/t2xajdu3rq9e6d+9979B3uN/YenIis4oX2SxRkfBFjQmKW0L5mM6SDnFCdBTM+Ci7dl/uyScsGy9EROc+oneJyyiBEsTWi0b7VbaIKlUijBchJEMNDaJk5nMxg68BVUaFZyyMeBr7xD99DVKKASj1wN2xDRXLA4S+v/5bkwMPpt7qoopYWhIt2dU72rtVaOg5Hq9Y61Lj1jGslhaT1mqcKc46lWhBAN0SXm+YTBA7jWIRJmEprkwSqNCOME2msOXJIcUzoNF54QcTaeSL9yC17WrwgNBwsJJ87coNK/6eL1B9tztncz81keprCzwp+Qslf46Qo556rj6TVpqUS6Yips6VTPH3V9jc9PdCXRRrF5RiFehY71qNF0u+5swU3gLUATLFZv1PiDwowUCU0libEQQ8/NpW9mJxmJqa6jQtAckws8pkMDU5xQ4avZI9CwZSIhjDJudmomX0arCoUTIaaJmWOrvKG4niuD23LDQkYvfcXSvJA0JfNCURFDmcHyi8CQcUpkPDUAE87MXSGZYI6JNB+pbobgXW95E5w+63oGv3/ePHqzGMcueAyeABt44AU4Au9AD/QBsT5bX63v1o/al9q32s/arznV2lloHoErq/b7H45iT5A=</latexit><latexit sha1_base64="AosgbpEP4hUJJ8fRO7veWyTg668=">AAAEFXiclVPLbhMxFHUnPEp4NIUlG4soZUZpohmEBBukAhu6C9C0EfE08ng8idV5yfZUiiz/BBt+hQ0LEGKLxI6/wZPntMkGS5aO7z3nXN8rO8hjJqTr/t2xajdu3rq9e6d+9979B3uN/YenIis4oX2SxRkfBFjQmKW0L5mM6SDnFCdBTM+Ci7dl/uyScsGy9EROc+oneJyyiBEsTWi0b7VbaIKlUijBchJEMNDaJk5nMxg68BVUaFZyyMeBr7xD99DVKKASj1wN2xDRXLA4S+v/5bkwMPpt7qoopYWhIt2dU72rtVaOg5Hq9Y61Lj1jGslhaT1mqcKc46lWhBAN0SXm+YTBA7jWIRJmEprkwSqNCOME2msOXJIcUzoNF54QcTaeSL9yC17WrwgNBwsJJ87coNK/6eL1B9tztncz81keprCzwp+Qslf46Qo556rj6TVpqUS6Yips6VTPH3V9jc9PdCXRRrF5RiFehY71qNF0u+5swU3gLUATLFZv1PiDwowUCU0libEQQ8/NpW9mJxmJqa6jQtAckws8pkMDU5xQ4avZI9CwZSIhjDJudmomX0arCoUTIaaJmWOrvKG4niuD23LDQkYvfcXSvJA0JfNCURFDmcHyi8CQcUpkPDUAE87MXSGZYI6JNB+pbobgXW95E5w+63oGv3/ePHqzGMcueAyeABt44AU4Au9AD/QBsT5bX63v1o/al9q32s/arznV2lloHoErq/b7H45iT5A=</latexit><latexit sha1_base64="AosgbpEP4hUJJ8fRO7veWyTg668=">AAAEFXiclVPLbhMxFHUnPEp4NIUlG4soZUZpohmEBBukAhu6C9C0EfE08ng8idV5yfZUiiz/BBt+hQ0LEGKLxI6/wZPntMkGS5aO7z3nXN8rO8hjJqTr/t2xajdu3rq9e6d+9979B3uN/YenIis4oX2SxRkfBFjQmKW0L5mM6SDnFCdBTM+Ci7dl/uyScsGy9EROc+oneJyyiBEsTWi0b7VbaIKlUijBchJEMNDaJk5nMxg68BVUaFZyyMeBr7xD99DVKKASj1wN2xDRXLA4S+v/5bkwMPpt7qoopYWhIt2dU72rtVaOg5Hq9Y61Lj1jGslhaT1mqcKc46lWhBAN0SXm+YTBA7jWIRJmEprkwSqNCOME2msOXJIcUzoNF54QcTaeSL9yC17WrwgNBwsJJ87coNK/6eL1B9tztncz81keprCzwp+Qslf46Qo556rj6TVpqUS6Yips6VTPH3V9jc9PdCXRRrF5RiFehY71qNF0u+5swU3gLUATLFZv1PiDwowUCU0libEQQ8/NpW9mJxmJqa6jQtAckws8pkMDU5xQ4avZI9CwZSIhjDJudmomX0arCoUTIaaJmWOrvKG4niuD23LDQkYvfcXSvJA0JfNCURFDmcHyi8CQcUpkPDUAE87MXSGZYI6JNB+pbobgXW95E5w+63oGv3/ePHqzGMcueAyeABt44AU4Au9AD/QBsT5bX63v1o/al9q32s/arznV2lloHoErq/b7H45iT5A=</latexit><latexit sha1_base64="AosgbpEP4hUJJ8fRO7veWyTg668=">AAAEFXiclVPLbhMxFHUnPEp4NIUlG4soZUZpohmEBBukAhu6C9C0EfE08ng8idV5yfZUiiz/BBt+hQ0LEGKLxI6/wZPntMkGS5aO7z3nXN8rO8hjJqTr/t2xajdu3rq9e6d+9979B3uN/YenIis4oX2SxRkfBFjQmKW0L5mM6SDnFCdBTM+Ci7dl/uyScsGy9EROc+oneJyyiBEsTWi0b7VbaIKlUijBchJEMNDaJk5nMxg68BVUaFZyyMeBr7xD99DVKKASj1wN2xDRXLA4S+v/5bkwMPpt7qoopYWhIt2dU72rtVaOg5Hq9Y61Lj1jGslhaT1mqcKc46lWhBAN0SXm+YTBA7jWIRJmEprkwSqNCOME2msOXJIcUzoNF54QcTaeSL9yC17WrwgNBwsJJ87coNK/6eL1B9tztncz81keprCzwp+Qslf46Qo556rj6TVpqUS6Yips6VTPH3V9jc9PdCXRRrF5RiFehY71qNF0u+5swU3gLUATLFZv1PiDwowUCU0libEQQ8/NpW9mJxmJqa6jQtAckws8pkMDU5xQ4avZI9CwZSIhjDJudmomX0arCoUTIaaJmWOrvKG4niuD23LDQkYvfcXSvJA0JfNCURFDmcHyi8CQcUpkPDUAE87MXSGZYI6JNB+pbobgXW95E5w+63oGv3/ePHqzGMcueAyeABt44AU4Au9AD/QBsT5bX63v1o/al9q32s/arznV2lloHoErq/b7H45iT5A=</latexit>
OLS
Ridge
Crosse et al. (2016). Front Hum Neurosci.
Example:

34. / 45
Data acquisition
• Stimulus: same 20 musical excerpts from Kim et al. 2017
• Subjects: three amateur musicians (1 female) and COVID19
happened...
• EEG recording: 64-ch BrainVision actiChamp (ref: right mastoid;
sampled at 1 kHz)
• Sound presentation: Etymotic Research ER-2 (air-tube earbuds)
• Task: emotional ratings (20 excepts x 4 repetitions)
34

35. / 45
Measure of tonal structures
• Krumhansl’s “tonal hierarchy” [1]
• Key strength measure in MIR
Toolbox [2, 3]
35
[1] Krumhansl. (1990). Mus Percept. [2] Gómez. (2006) INFORMS J Com. [3] Lartillot et al. (2008). Data Analysis, Machine Learning and Applications.

36. / 45
Interim summary
• An entropy model detected the neural encoding of
melodic surprisal via ridge regression.
• A measure of ‘tonal stability’ is proposed to detect
the cortical encoding of tonal hierarchy in listeners
36

37. / 45
Overview
• Classical cognitive neuroscience (behavior-neural association)
- Dissonance preference associated with brainstem response
• Model-based cognitive neuroscience (feature-neural
association)
- Cortical tracking of tonal stability in naturalistic music
• General discussion
37

38. / 45
Overall summary
• In-vivo human auditory neuroimaging
studies revealed neural correlates of the
perception of musical elements
Sensory dissonance and inferior colliculus
• Computational modeling of musical
structures enables studies on the
appreciation of natural music
Melodic surprisal, tonal stability, ...
38

39. / 45
...and beyond
39
[1] clipartbest.com [2] freepik.com
Cognitive computational
neuroscience
Com
putational
m
odeling
Cognitive
neuroscience
[2]
[1][2]
• Machine composition
to evoke specific
brain-states?
• Individual/population-
tuned models?

40. / 45
Acknowledgement
40
Prof. Tobias Overath
@ Duke University
Prof. Tom H. Fritz
@ Max Planck Inst.
for CBS, Leipzig
Prof. Karsten Müller
@ Max Planck Inst.
for CBS, Leipzig
Jasmine Leahy
(B.Sc. student)
@ Duke University

41. / 45
Thank you for attention!
Now time for discussion! :D
seunggoo.kim@duke.edu
41