Summary of MAST

Sangjun Son (SNU) 1
Multi-Aspect Streaming
Tensor Completion
(Qingquan Song et al., KDD 2017)
Sangjun Son
Data Mining Lab
Dept. of CSE
Seoul National University

Sangjun Son (SNU) 2
Keywords
◼ Multi-Aspect Streaming Tensor (MAST)
❑ Incomplete, Multiple-mode streaming
◼ MAST Completion
❑ Dynamic Tensor Decomposition (DTD)
❑ Low Rank Tensor Completion (LRTC)
◼ Temporal MAST (T-MAST)

Sangjun Son (SNU) 3
Outline
◼ Introduction
◼ Preliminaries
◼ Proposed Method
◼ Experiments
◼ Conclusion

Sangjun Son (SNU) 4
Tensor
◼ An N-way array which is a generalization of
vectors and matrices.

Sangjun Son (SNU) 5
Real-world Tensor
◼ Recommendation
❑ A 3rd order tensor
Index: (user, movie, time)
❑ Value: movie rating
◼ Link Prediction
❑ A 3rd order tensor
Index: (user, user, interaction)
❑ Value: connectivity
Missing entries of partially observed tensors

Sangjun Son (SNU) 6
Tensor Completion (1/2)
◼ Real-world tensors are often incomplete.
❑ Due to missing at random,
limited permissions and maloperations.
◼ Tensor Decomposition (CP)
user
topic
𝒖𝑖
≈ ෍
𝑖=1
𝑅
𝒕𝑖
=
𝐔
𝐓
𝐄
𝓣 ෍
𝑖=1
𝑅
𝒖𝑖 ∘ 𝒕𝑖 ∘ 𝒆𝑖≈ = 𝐔, 𝐄, 𝐓

Sangjun Son (SNU) 7
user
topic
≈
𝐔
𝐓
𝐄
user
topic
=

Sangjun Son (SNU) 8
user
topic
≈
𝐔
𝐓
𝐄
user
topic
=

Sangjun Son (SNU) 9
user
topic
≈
𝐔
𝐓
𝐄
user
topic
=

Sangjun Son (SNU) 10
user
topic
≈
𝐔
𝐓
𝐄
user
topic
=
◼ Fill missing entries of partially observed tensors.
❑ Recommender systems
❑ Image recovery
❑ Clinical data analysis

Streaming Tensor
◼ High velocity streaming tensors in real world.
❑ Due to popularity of online information systems.
❑ Develop in one temporal mode.
user
item
𝓣 𝑡
user
item
user
item
𝓣 𝑡+1 𝓣 𝑡+2

Multi-Aspect Streaming Tensor
◼ Existing methods ignore that,
❑ A tensor may develop in multiple dimensions.
user
item
𝓣 𝑡 𝓣 𝑡+1 𝓣 𝑡+2
item
user
item
user

Multi-Aspect Streaming Tensor
◼ Existing methods ignore that,
❑ A tensor may develop in multiple dimensions.
user
item
𝓣 𝑡 𝓣 𝑡+1 𝓣 𝑡+2
item
user
item
user
⊆ ⊆

Outline
◼ Introduction
◼ Preliminaries
◼ Proposed Method
◼ Experiments
◼ Conclusion

Notation for Tensor

Khatri-Rao Product ⨀
◼ Given matrices 𝐀 ∈ ℝ𝐼×𝐾 and 𝐁 ∈ ℝ 𝐽×𝐾,
Khatri–Rao product is denoted by 𝐀 ⨀ 𝐁.
❑ 𝐀 ⨀ 𝐁 = [𝐚1 ⊗ 𝐛1, ⋯ , 𝐚 𝐾 ⊗ 𝐛 𝐾] ∈ ℝ𝐼𝐽×𝐾
◼ where ⊗ is Kronecker product
𝐚 ⊗ 𝐛 =
𝑎1
⋮
𝑎𝐼
⊗
𝑏1
⋮
𝑏𝐽
=
𝑎1 𝐛
𝑎2 𝐛
⋮
𝑎𝐼 𝐛
=
𝑎1 𝑏1
𝑎1 𝑏2
⋮
𝑎𝐼 𝑏𝐽
◼ while ∘ is outer product
𝐚 ∘ 𝐛 = 𝐚 ⊗ 𝐛T =
𝑎1
⋮
𝑎𝐼
⊗ 𝑏1 ⋯ 𝑏𝐽 =
𝑎1 𝐛T
𝑎2 𝐛T
⋮
𝑎𝐼 𝐛T
=
𝑎1 𝑏1
𝑎2 𝑏1
⋮
𝑎𝐼 𝑏1
⋯
𝑎1 𝑏𝐽
𝑎2 𝑏𝐽
⋮
𝑎𝐼 𝑏𝐽

Hadamard Product ⊛
◼ Given matrices in the same dimension
𝐀, 𝐁 ∈ ℝ𝐼×𝐽
, Hadamard product is denoted
by 𝐀 ⊛ 𝐁 ∈ ℝ𝐼×𝐽
.
❑ 𝐀 ⊛ 𝐁 𝑖𝑗 = 𝐀 𝑖𝑗 𝐁 𝑖𝑗
❑ 𝐀 ⊛ 𝐁 =
𝑎11 𝑏11
𝑎21 𝑏22
⋮
𝑎𝐼1 𝑏𝐼1
⋯
𝑎1𝐽 𝑏1𝐽
𝑎2𝐽 𝑏2𝐽
⋮
𝑎𝐼𝐽 𝑏𝐼𝐽

Unfolding
Figure from Lieven De Lathauwer et al,
A Multi-Linear Singular Value Decomposition,
𝓧
𝓧
𝓧
𝐗(1)
𝐗(2)
𝐗(3)
𝓧 ∈ ℝ𝐼1×𝐼2×𝐼3
❑ 𝐗(1) ∈ ℝ𝐼1×𝐼2 𝐼3
❑ 𝐗(2) ∈ ℝ𝐼2×𝐼3 𝐼1
❑ 𝐗(3) ∈ ℝ𝐼3×𝐼1 𝐼2
Given a tensor
𝓧 ∈ ℝ𝐼1×⋯×𝐼 𝑁,
its mode-𝑛 unfolding matrix
𝐗(𝑛) ∈ ℝ𝐼 𝑛×ς 𝑖≠𝑛
𝑁
𝐼 𝑖.

CP Decomposition (CPD)
◼ Given a 𝑁th order tensor 𝓧 ∈ ℝ𝐼1×⋯×𝐼 𝑁,
CPD is an approximation form of 𝑁 matrices
𝐀 𝑛 ∈ ℝ𝐼 𝑛×𝑅
.
❑ where 𝑛 = 1, … , 𝑁 and 𝑅 is the rank of 𝓧.
◼ 𝓧 ≈ 𝐀1, … , 𝐀 𝑁
❑ 𝓧
unfold
𝐗(𝑛) ≈ 𝐀 𝑛 𝐀 𝑁 ⨀ … 𝐀 𝑛+1 ⨀ 𝐀 𝑛−1 … ⨀ 𝐀1
T
= 𝐀 𝑛 𝐀 𝑘
⊙ 𝑘≠𝑛
T
𝓧 ≈ 𝐀, 𝐁, 𝐂
𝐗(1) ≈ 𝐀 𝐂 ⨀ 𝐁 T
𝐗(2) ≈ 𝐁 𝐂 ⨀ 𝐀 T
𝐗(3) ≈ 𝐂 𝐁 ⨀ 𝐀 T
𝐀
𝐁
𝐂

Alternating Least Square Update
𝐀 ≈ 𝐗(1) 𝐂 ⨀ 𝐁 T
†
𝐁 ≈ 𝐗(2) 𝐂 ⨀ 𝐀 T
†
𝐂 ≈ 𝐗(3) 𝐁 ⨀ 𝐀 T
†
= 𝐗(1) 𝐂 ⨀ 𝐁 𝐂T 𝐂 ⊛ 𝐁T 𝐁
†
= 𝐗(2) 𝐂 ⨀ 𝐀 𝐂T 𝐂 ⊛ 𝐀T 𝐀
†
= 𝐗(3) 𝐁 ⨀ 𝐀 𝐁T 𝐁 ⊛ 𝐀T 𝐀
†
◼ Find factor sequence 𝐀 𝑛 that minimizes
𝓛 𝐀1, … , 𝐀 𝑁 = 𝓧 − 𝐀1, … , 𝐀 𝑁 F
2
❑ Optimize factor matrix 𝐀 𝑛 while others 𝐀 𝑘≠𝑛 are fixed.
❑ And repeat until convergence.

◼ Find factor sequence 𝐀 𝑛 that minimizes
𝓛 𝐀1, … , 𝐀 𝑁 = 𝓧 − 𝐀1, … , 𝐀 𝑁 F
2
❑ Optimize factor matrix 𝐀 𝑛 while others 𝐀 𝑘≠𝑛 are fixed.
❑ And repeat until convergence.
Alternating Least Square Update
𝐀 ≈ 𝐗(1) 𝐂 ⨀ 𝐁 T
†
𝐁 ≈ 𝐗(2) 𝐂 ⨀ 𝐀 T
†
𝐂 ≈ 𝐗(3) 𝐁 ⨀ 𝐀 T
†
= 𝐗(1) 𝐂 ⨀ 𝐁 𝐂T 𝐂 ⊛ 𝐁T 𝐁
†
= 𝐗(2) 𝐂 ⨀ 𝐀 𝐂T 𝐂 ⊛ 𝐀T 𝐀
†
= 𝐗(3) 𝐁 ⨀ 𝐀 𝐁T 𝐁 ⊛ 𝐀T 𝐀
†
𝐀 𝑛 ≈ 𝐗(𝑛) 𝐀 𝑘
⊙ 𝑘≠𝑛
T †
= 𝐗(𝑛) 𝐀 𝑘
⊙ 𝑘≠𝑛 𝐀 𝑘
T
𝐀 𝑘
⊛ 𝑘≠𝑛
†

Problem Definition
◼ Multi-Aspect Steaming Tensor Completion
❑ Given MAST sequence 𝓧(𝑇) with missing entries,
recover the missing data in current snapshot 𝓧(𝑇).
❑ Input
◼ Previously recovered MAST
𝓧(𝑇−1)
⊆ 𝓧 𝑇
❑ Output
◼ Relative complement of
𝓧(𝑇−1)
in 𝓧 𝑇
, 𝓧(𝑇)
𝓧 𝑇−1
❑ Goal
◼ Maximize the effectiveness
and efficiency
user
item
item
user
𝓧(𝑇−1)
𝓧(𝑇)

Outline
◼ Introduction
◼ Preliminaries
◼ Proposed Method
◼ Experiments
◼ Conclusion

Challenges
◼ Uncertainty of tensor mode changes
◼ Incompleteness: missing data problem
◼ Higher time and space complexity

MAST Framework
◼ Dynamic Tensor Decomposition (DTD)
❑ Model the incremental pattern of MAST
◼ Low Rank Tensor Completion (LRTC)
❑ Track its low-rank subspace

Dynamic Tensor Decomposition
◼ Given the recovered tensor in previous step,
෩𝓧
෩𝓧
𝓧
𝐼1
𝐼2
𝐼3
𝐼2 + 𝑑2
𝐼1+𝑑1
Time 𝑇 − 1 Time 𝑇
≈
෩𝐀
෨𝐂
𝐀0
𝐂0
𝐂1
𝐀1

Partition → Substitution → Re-decomposition
෩𝓧
𝓧
𝐼2 + 𝑑2
𝐼1+𝑑1
෩𝓧

෩𝓧
𝓧
𝐼2 + 𝑑2
𝐼1+𝑑1
𝓧1,1,1
𝓧1,0,1
𝓧1,1,0
𝓧0,1,1
𝓧0,0,1
𝓧1,0,0
෩𝓧 = 𝓧0,0,0
𝓧0,1,0

𝓧1,1,1
𝓧1,0,1
𝓧1,1,0
𝓧0,1,1
𝓧0,0,1
𝓧1,0,0
𝓧0,1,0
෩𝐀
෨𝐂
𝓧1,1,1
𝓧1,0,1
𝓧1,1,0
𝓧0,1,1
𝓧0,0,1
𝓧1,0,0
෩𝓧 = 𝓧0,0,0
𝓧0,1,0

𝓧1,1,1
𝓧1,0,1
𝓧1,1,0
𝓧0,1,1
𝓧0,0,1
𝓧1,0,0
𝓧0,1,0
෩𝐀
෨𝐂
𝐀0
𝐂0
𝐂1
𝐀1

◼ Loss function
❑ 𝓛 𝐀, 𝐁, 𝐂 = 𝓧 − 𝐀, 𝐁, 𝐂 F
2
= ෍
(𝑖,𝑗,𝑘)∈Θ
𝓧𝑖,𝑗,𝑘
− 𝐀 𝑖, 𝐁𝑗, 𝐂 𝑘 F
2
= 𝓧0,0,0
− 𝐀0, 𝐁0, 𝐂0 F
2
+ 𝓛0
≈ 𝜇 ෩𝐀, ෩𝐁, ෨𝐂 − 𝐀0, 𝐁0, 𝐂0 F
2
+ 𝓛0
❑ Forgetting factor 𝜇 ∈ 0, 1
to alleviate the influence of the previous decomposition error.
𝓧1,1,1𝓧1,0,1
𝓧1,1,0
𝓧0,1,1
𝓧0,0,1
𝓧1,0,0
𝓧0,1,0
෩𝐀
෨𝐂
𝐀0
𝐂0
𝐂1
𝐀1
෩𝓧
Θ ≜ 0,1 3
𝐀
𝐁
𝐂
≈ ≈

◼ ALS update
❑ Static
◼ 𝓛 𝐀, 𝐁, 𝐂 = 𝓧0,0,0
− 𝐀0, 𝐁0, 𝐂0 F
2
+ 𝓛0
𝐀0 ←
σ 𝑗,𝑘 𝐗(1)
0,𝑗,𝑘
𝐂 𝑘 ⨀ 𝐁𝑗
σ 𝑘=0
1
𝐂 𝑘
T
𝐂 𝑘 ⊛ σ 𝑗=0
1
𝐁𝑗
T
𝐁𝑗
❑ DTD
◼ 𝓛 𝐀, 𝐁, 𝐂 ≈ 𝜇 ෩𝐀, ෩𝐁, ෨𝐂 − 𝐀0, 𝐁0, 𝐂0 F
2
+ 𝓛0
𝐀0 ←
𝜇෩𝐀 ෨𝐂T
𝐂0 ⊛ ෩𝐁T
𝐁0 + σ(𝑗,𝑘)≠(0,0) 𝐗(1)
0,𝑗,𝑘
σ 𝑘=0
1
𝐂 𝑘
T
𝐂 𝑘 ⊛ σ 𝑗=0
1
𝐁𝑗
T
𝐁𝑗 − (1 − 𝜇) 𝐂 𝑘
T
𝐂 𝑘 ⊛ 𝐁𝑗
T
𝐁𝑗

◼ ALS update
❑ Static
◼ 𝓛 𝐀, 𝐁, 𝐂 = 𝓧0,0,0
− 𝐀0, 𝐁0, 𝐂0 F
2
+ 𝓛0
𝐀0 ←
𝐗(1)
0,0,0
𝐂0 ⨀ 𝐁0 + σ(𝑗,𝑘)≠(0,0) 𝐗(1)
0,𝑗,𝑘
σ 𝑘=0
1
𝐂 𝑘
T
𝐂 𝑘 ⊛ σ 𝑗=0
1
𝐁𝑗
T
𝐁𝑗
❑ DTD
◼ 𝓛 𝐀, 𝐁, 𝐂 ≈ 𝜇 ෩𝐀, ෩𝐁, ෨𝐂 − 𝐀0, 𝐁0, 𝐂0 F
2
+ 𝓛0
𝐀0 ←
𝜇෩𝐀 ෨𝐂T
𝐂0 ⊛ ෩𝐁T
𝐁0 + σ(𝑗,𝑘)≠(0,0) 𝐗(1)
0,𝑗,𝑘
σ 𝑘=0
1
𝐂 𝑘
T
𝐂 𝑘 ⊛ σ 𝑗=0
1
𝐁𝑗
T
𝐁𝑗 − (1 − 𝜇) 𝐂 𝑘
T
𝐂 𝑘 ⊛ 𝐁𝑗
T
𝐁𝑗
Reduction of time complexity, 𝒪 𝑅𝐼1 𝐼2 𝐼3 → 𝒪 𝑅2 𝐼1 + 𝐼2 + 𝐼3

Low Rank Tensor Completion
◼ Generalized from matrix completion problem.
minimize
𝓧
𝑟𝑎𝑛𝑘(𝓧) subject to 𝛀 ⊛ 𝓧 = 𝓣
❑ 𝓧 is the complete tensor and
𝓣 denotes the practical observations of 𝓧.
𝛀 is a binary tensor indicating whether
each corresponding entry is observed or not.
❑ Rank Calculation is NP-hard!

Low Rank Tensor Completion
◼ Relaxation objective function
minimize
𝓧
𝑟𝑎𝑛𝑘(𝓧) subject to 𝛀 ⊛ 𝓧 = 𝓣
minimize
𝓧, 𝐀1,𝐀2,… ,𝐀 𝑁
෍
𝑛=1
𝑁
𝛼 𝑛 𝐀 𝑛 ∗ subject to 𝛀 ⊛ 𝓧 = 𝓣
where 𝛼 𝑛 are trade-offs to balance the significance of each mode
and ∙ ∗ is a nuclear norm of a matrix.

MAST Framework
◼ Loss function
❑ 𝓛DTD = 𝜇 ෪𝐀1, ෪𝐀2, … , ෪𝐀 𝑁 − 𝐀1
(0)
, 𝐀2
(0)
, … , 𝐀 𝑁
(0)
F
2
+ 𝓛0
❑ 𝓛LRTC = σ 𝑛=1
𝑁
𝛼 𝑛 𝐀 𝑛 ∗
❑ 𝓛 = 𝓛DTD + 𝓛LRTC
◼ Optimization method
❑ Alternating Direction Method of Multipliers
❑ Recover missing entries with an EM-like approach.

MAST Framework
Auxiliary variables for ADMM
Termination Criterion
← 𝓣 + 𝛀C ⊛ 𝐀1, … , 𝐀 𝑁

T-MAST vs MAST
user
𝑡 + 2
𝑡 + 1
𝑡
item
user
𝑡 + 2
𝑡 + 1
𝑡
𝑶
𝑶
𝑶
item
𝐀3
(𝑡)
𝐀1
(𝑡)
𝐀1
(𝑡+1)
𝐀3
(𝑡+1)
𝐀3
(𝑡)
𝐀1
(𝑡+1)
𝐀3
(𝑡+1)
𝐀3
(𝑡)
𝐀1
(𝑡)
◼ T-MAST alleviates the substitution on missing entries.

T-MAST
user
𝑡 + 2
𝑡 + 1
𝑡
item
𝐀3
(𝑡)
𝐀1
(𝑡+1)
𝐀3
(𝑡+1)
𝐀3
(𝑡)
𝐀1
(𝑡)
Given recovered slices 𝓧(1:𝑇−1) ,
update current CPD at time step 𝑇.
𝓧(𝑇) ≈ { 𝐀1
(𝑇)
,
𝐀2
(1)
⋮
𝐀2
(𝑇)
, … ,
𝐀 𝑁
(1)
⋮
𝐀 𝑁
(𝑇)
}
Optimize w. ADMM
𝓛 = 𝓧(𝑇)
− 𝐀1
(𝑇)
, 𝐀2, 𝐀3, … , 𝐀 𝑁
F
2
+ ෍
𝑛=1
𝑁
𝛼 𝑛 𝐀 𝑛 ∗
+ ෍
𝑡=1
𝑇−1
𝜇 𝑡
෩𝐀1
(𝑡)
,
෩𝐀2
(1)
⋮
෩𝐀2
(𝑡)
, … ,
෩𝐀 𝑁
(1)
⋮
෩𝐀 𝑁
(𝑡)
− 𝐀1
(𝑡)
,
𝐀2
(1)
⋮
𝐀2
(𝑡)
, … ,
𝐀 𝑁
(1)
⋮
𝐀 𝑁
(𝑡)
F
2

Outline
◼ Introduction
◼ Preliminaries
◼ Proposed Method
◼ Experiments
◼ Conclusion

Experimental Setup
◼ Datasets
◼ Baselines
❑ Static CP-ALS: CPD with ALS
❑ TNCP: trace norm based CPD using ADMM
❑ OLSTEC: online CPD with recursive least square

Evaluation Metric
◼ How effective?
❑ Running Average Area Under Curve, RA-AUC
❑
1
𝑇
σ 𝑡=1
𝑇
AUC 𝑡
◼ How efficient?
❑ Average Running Time
❑
1
𝑇
σ 𝑡=1
𝑇
RT𝑡
Figure from GLASS BOX
Machine Learning and Medicine,
by Rachel Lea Ballantyne Draelos

Evaluation of Effectiveness
◼ MAST has commensurate performance w. static CPD
method.
◼ T-MAST outperforms MAST on temporal tensor
completion.

Evaluation of Efficiency
◼ MAST outperforms all the others.
◼ Computation of T-MAST is faster than static methods,
but slower than MAST.

Outline
◼ Introduction
◼ Preliminaries
◼ Proposed Method
◼ Experiments
◼ Conclusion

Conclusion
◼ Define the problem of MAST completion.
❑ Propose a CP-based general algorithm MAST.
❑ Propose a modified model T-MAST for a special case.
◼ Empirically validate the effectiveness
and efficiency on real-world datasets.
❑ MAST, T-MAST outperforms in speed,
maintaining decomposition accuracy.

Relevance to My Research
◼ This is the first work on streaming analysis
which solved multi-aspect streaming problem.
◼ I’m also working on streaming tensor.
❑ Online tensor analysis when drastic data incomes.
◼ I will implement this approach.
❑ Get intuitions to improve my model.
❑ Experiment with real world streaming datasets.

Thank you !

Summary of MAST

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Summary of MAST

Similar to Summary of MAST (20)

Recently uploaded

Recently uploaded (20)

Summary of MAST