This document discusses MuSe2023, a competition to predict emotional state from multimodal data including audio, video, and biosignals. It describes the input data and features, hardware resources including several servers with GPUs, and workflow improvements using Wandb Sweep to automatically tune hyperparameters in parallel across resources. Previous methods of manually generating and distributing experiment scripts led to challenges in restarting crashed runs; Wandb Sweep addresses this by dynamically assigning tasks and monitoring progress.

1. MuSe2023 from practical view
Ganghyun Kim
2023 / 08 / 18

2. 2/30

3. 3/30

4. MuSe2023 overview
• Goal: Predict subject’s emotional state using multimodal data
• Subject is under stressed condition (simulated job interview)
• Emotional state can be represented in 2 axis, Arousal and Valence
4/30
https://www.mdpi.com/1424-8220/23/3/1598

5. Input data (feature)
• Audio: DeepSpectrum, eGeMAPS, Wav2vec2.0
• Video: ViT, FaceNet, FAU, Pose
• Other: BERT, Biosignals
• All of them tabular data extracted with corresponding feature extractor.
5/30

6. 6/30

7. 2022
7/30

8. 2023
8/30

9. Big Titan server: Titan Xp X2
Small Titan server: Titan Xp X2
Student server 1: RTX 1080 X2
Student server 2: RTX2080 X2
Student server 3: A5000 X2
TA server: RTX3090 X3
Code Management

10. 6F
6F
IGC IT CENTER
IGC IT CENTER
IGC IT CENTER
IGC IT CENTER
Execute commands via
6F

11. Tmux
11/30

12. Termius
12/30

13. Termius
13/30

14. Wandb Sweep
Generates HP tuning task list
Start signal
Check remaining task,
assign 1 HP combination run to server,
repeat when server finishes

15. Wandb Sweep
• wandb sweep sweep.yaml # returns your SWEEP_ID
• wandb agent mcg70107
• Done!
15/30

16. Previously..
• PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python CUDA_VISIBLE_DEVICES=1 python main_rnn.py --cache --task stress --emo_dim valence --d_rnn 64 --rnn_n_layers 2 --
lr 0.005 --feature mfcc_combined --win_len 200 --rnn_bi --n_seeds 20 --seed 101 --batch_size 256 --early_stopping_patience 15 --reduce_lr_patience 5 --project_name
Muse_20221026_early_fusion_RNN_valence_mfcc_combined --use_gpu
• PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python CUDA_VISIBLE_DEVICES=1 python main_rnn.py --cache --task stress --emo_dim valence --d_rnn 64 --rnn_n_layers 2 --
lr 0.005 --feature mfcc_combined --win_len 300 --rnn_bi --n_seeds 20 --seed 101 --batch_size 256 --early_stopping_patience 15 --reduce_lr_patience 5 --project_name
Muse_20221026_early_fusion_RNN_valence_mfcc_combined --use_gpu
• PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python CUDA_VISIBLE_DEVICES=1 python main_rnn.py --cache --task stress --emo_dim valence --d_rnn 64 --rnn_n_layers 2 --
lr 0.005 --feature mfcc_combined --win_len 400 --rnn_bi --n_seeds 20 --seed 101 --batch_size 256 --early_stopping_patience 15 --reduce_lr_patience 5 --project_name
Muse_20221026_early_fusion_RNN_valence_mfcc_combined --use_gpu
• PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python CUDA_VISIBLE_DEVICES=1 python main_rnn.py --cache --task stress --emo_dim valence --d_rnn 64 --rnn_n_layers 4 --
lr 0.001 --feature mfcc_combined --win_len 200 --rnn_bi --n_seeds 20 --seed 101 --batch_size 256 --early_stopping_patience 15 --reduce_lr_patience 5 --project_name
Muse_20221026_early_fusion_RNN_valence_mfcc_combined --use_gpu
• PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python CUDA_VISIBLE_DEVICES=1 python main_rnn.py --cache --task stress --emo_dim valence --d_rnn 64 --rnn_n_layers 4 --
lr 0.001 --feature mfcc_combined --win_len 300 --rnn_bi --n_seeds 20 --seed 101 --batch_size 256 --early_stopping_patience 15 --reduce_lr_patience 5 --project_name
Muse_20221026_early_fusion_RNN_valence_mfcc_combined --use_gpu
• PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python CUDA_VISIBLE_DEVICES=1 python main_rnn.py --cache --task stress --emo_dim valence --d_rnn 64 --rnn_n_layers 4 --
lr 0.001 --feature mfcc_combined --win_len 400 --rnn_bi --n_seeds 20 --seed 101 --batch_size 256 --early_stopping_patience 15 --reduce_lr_patience 5 --project_name
Muse_20221026_early_fusion_RNN_valence_mfcc_combined --use_gpu
• PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python CUDA_VISIBLE_DEVICES=1 python main_rnn.py --cache --task stress --emo_dim valence --d_rnn 64 --rnn_n_layers 4 --
lr 0.002 --feature mfcc_combined --win_len 200 --rnn_bi --n_seeds 20 --seed 101 --batch_size 256 --early_stopping_patience 15 --reduce_lr_patience 5 --project_name
Muse_20221026_early_fusion_RNN_valence_mfcc_combined --use_gpu
• PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python CUDA_VISIBLE_DEVICES=1 python main_rnn.py --cache --task stress --emo_dim valence --d_rnn 64 --rnn_n_layers 4 --
lr 0.002 --feature mfcc_combined --win_len 300 --rnn_bi --n_seeds 20 --seed 101 --batch_size 256 --early_stopping_patience 15 --reduce_lr_patience 5 --project_name
Muse_20221026_early_fusion_RNN_valence_mfcc_combined --use_gpu
• PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python CUDA_VISIBLE_DEVICES=1 python main_rnn.py --cache --task stress --emo_dim valence --d_rnn 64 --rnn_n_layers 4 --
lr 0.002 --feature mfcc_combined --win_len 400 --rnn_bi --n_seeds 20 --seed 101 --batch_size 256 --early_stopping_patience 15 --reduce_lr_patience 5 --project_name
Muse_20221026_early_fusion_RNN_valence_mfcc_combined --use_gpu
16/30

17. Previously..
• Command generator writes combinations of HP tuning runs
• Split them in .sh file, in appropriate length considering GPU power of server, # of GPU
• Upload .sh file to each server, run them
Problem
• Hard to restart from crashed run. Needed to locate which run crashed, edit .sh file to start from
there
• Tedious work, considering number of experiments I had to run
17/30

18. Previously..

19. 19/30

20. 20/30

21. Parallel coordinates plot
21/30

22. 22/30

23. 23/30

24. Result
24/30

25. Result
25/30