Review ppt of Interpolative Distillation for Unifying Biased and Debiased Recommendation

1. Interpolative Distillation for Unifying Biased
and Debiased Recommendation
SIGIR’22, Sihao Ding(USTC) et al.
POSTECH DI Lab
Presenter: Changsoo Kwak
2022.5.24
1

2. Motivation
2
▪ Most recommender system’s test set for evaluate
▪ Normal biased test set(𝐷𝑏)
▪ Debiased test set (𝐷𝑑)
[1] Self-supervised Graph Learning for Recommendation, Jiancan Wu(USTC) et al, SIGIR’21
Existing models didn’t perform well on both test set
Biased or Unbiased model
Only reflect part of whole picture

3. Intuitive solution?
3
▪ Unifying 𝐷𝑏, 𝐷𝑑
▪ Usually 𝐷𝑏 ≫ |𝐷𝑑|
▪ Train two models for 𝐷𝑏, 𝐷𝑑 respectively, and ensemble
▪ Unclear that each models are strong/weak at which type of users/items
▪ Existing ensemble strategies are not tailored for win-win recommendation scenario
▪ Possible solution?
▪ Distillation!
▪ Aggregate two models at the level of user-item pair
Determine coefficient automatically for distillation

4. Proposed model(InterD)
4
Environment 𝐸 ∈ {𝑒𝑏, 𝑒𝑑}
Probability of environment given user-item pair
Existing models only consider one environment
- Only achieve good performance on one of 𝐷𝑏 or 𝐷𝑑
Predicted rating with given environment assumption
Let student model learns predicted ratings generated by
fine-grained weighted sum of prediction of pre-trained
models, considering environment

5. Proposed model(InterD)
5
𝑓𝑏, 𝑓𝑑: Pre-trained biased/unbiased model
▪ Estimate 𝑃(𝑅|𝑈, 𝐼, 𝐸)
▪ Directly use prediction of 𝑓𝑏, 𝑓𝑑
▪ Estimate 𝑃 𝐸 𝑈, 𝐼
𝑤𝑏 =
𝐿𝑏(𝑟𝑏, 𝑟)𝛾1
𝐿𝑏(𝑟𝑏, 𝑟)𝛾1+𝐿𝑑(𝑟𝑑, 𝑟)𝛾1
, 𝑤𝑑 =
𝐿𝑑(𝑟𝑑, 𝑟)𝛾1
𝐿𝑏(𝑟𝑏, 𝑟)𝛾1+𝐿𝑑(𝑟𝑑, 𝑟)𝛾1
𝐿𝑏: MSE, 𝐿𝑑: IPS weighted MSE, 𝛾1: Negative hyperparameter
𝑃 𝑅 𝑈, 𝐼 =
𝐸
𝑃 𝑅 𝑈, 𝐼, 𝐸 𝑃 𝐸 𝑈, 𝐼 = 𝑟∗ = 𝑤𝑏 𝑟𝑏 + 𝑤𝑑𝑟𝑑
▪ Training student model
Distillation loss 𝐿𝑂 =
1
|𝐷𝑏| + |𝐷𝑑|
(𝑢,𝑖,𝑟)∈𝐷𝑏∪𝐷𝑑
𝐿(𝑟, 𝑟∗ )

6. Proposed model(InterD)
6
▪ Incorporate unobserved data 𝐷𝑛 = 𝑈 × 𝐼 − 𝐷𝑏 ∪ 𝐷𝑑
𝑤𝑏
′
=
𝐿𝑏(𝑟𝑏, 𝑟)𝛾2
𝐿𝑏(𝑟𝑏, 𝑟)𝛾2+𝐿𝑑(𝑟𝑑, 𝑟)𝛾2
, 𝑤𝑑
′
=
𝐿𝑑(𝑟𝑑, 𝑟)𝛾2
𝐿𝑏(𝑟𝑏, 𝑟)𝛾2+𝐿𝑑(𝑟𝑑, 𝑟)𝛾2
𝑟∗
′
= 𝑤𝑏
′
𝑟𝑏 + 𝑤𝑑
′
𝑟𝑑
Imputation distillation loss 𝐿𝑁 =
1
|𝐷𝑛|
(𝑢,𝑖)∈𝐷𝑛
𝐿(𝑟, 𝑟∗
′)
Student model learn more from closer teacher over unobserved data

7. Experiments
7

8. Experiments
8