Summary: At Stitch Fix, Hierarchical models are one of the core machine learning frameworks used in our recommender systems technology. Hierarchical models allow for estimation on clustered data, when classical assumptions of identically distributed random variables break down. Traditional likelihood-based methods for fitting hierarchical models often struggle with the scale of data found in industry, which has prompted recent research into moment-based procedures for parameter estimation. Spark doesn’t have a native library for fitting these models, and to our knowledge, no moment-based estimation software has been developed previously utilizing a distributed computational system. This talk will review our development of spark software utilizing these new estimation methods, detail the theory behind the approach, and compare our software to similar open source packages in Spark and other popular languages.

1. Kyle Schmaus, Stitch Fix
Moment-Based Estimation
for Hierarchical Models in
Apache Spark
#DSSAIS17

2. Overview
• About Stitch Fix
• Hierarchical (Mixed Effects) Models
• Moment Based Estimation
• Spark Implementation & Application
2#DSSAIS17

3. Stitch Fix
3#DSSAIS17

4. Stitch Fix
4#DSSAIS17

5. Stitch Fix
5#DSSAIS17

6. Stitch Fix
6#DSSAIS17

7. Stitch Fix
7#DSSAIS17
Machine learning Human curation
Algorithmic
recommendations

8. Stitch Fix
• Algorithms Team: 80+ Data Scientists and Data
Engineers
• Data Integrates into every aspect of the
business
• Our Blog: multithreaded.stitchfix.com
8#DSSAIS17

9. Hierarchical Models Motivation
9#DSSAIS17
ℙ(sale) = ?

10. Hierarchical Models Motivation
10#DSSAIS17
Color
Cut
Size
Brand
Material
…

11. Shared Model
ℙ(sale) = "#$
%$
&
'
11#DSSAIS17
ℙ(sale) = "#$
%(
&
'
" ) = log
)
1 − )

12. Individual Model Per Group
12#DSSAIS17

13. Individual Model Per Group
13#DSSAIS17
ℙ(sale) = "#$
%&$
ℙ(sale) = "#$
%&'
" ( = log
(
1 − (

14. Hierarchical Models
14#DSSAIS17
• ![#$] = ()* +$, + .$/$
• /$ ~ 1 0, Σ
• 5 ∈ 1, … , 9
• ,, /$, and Σ are unknown

15. Simulation Results
15#DSSAIS17
• ! ∈ 1, … , & = 100
• )* = +*, + .*/* + 0*
• dim , = dim /* = 11 × 1
• /* ~ 7 0, Σ
• 0* ~ 7 0, 9
• Σ ~ :;< <
<<
9, 11
●
● ● ● ● ● ●
●
●
●
●
● ●
●
●
● ●
●
● ●
●
1.00
1.25
1.50
1.75
3 4 5 6 7
log(n_obs)
rmse
model_type
●
●
●
individual model
mixed model
shared model
RMSE By N Observations Per Group

16. Software Implementations
16#DSSAIS17
lme4, nlme, mbest, …
statsmodels
MixedModels

17. Software Implementations
17#DSSAIS17
lme4, nlme, mbest, …
statsmodels
MixedModels

18. Likelihood Based Methods
Expectation-Maximization, Variational Approximations, or
Likelihood Maximization require an ! "#$
initial cost, then
a series of iterations costing ! %#& , where
• " is the number of total observations
• # the number of fixed and random effects
• % the number of groups
18#DSSAIS17

19. Moment Based Methods
Using a moment-based approach laid out in Perry (2015)
and implemented in the mbest package, we can achieve
a non-iterative fit in ! "#$ + ! &#' steps. This can be
trivially spread across ( processors.
19#DSSAIS17

20. Moment Based Methods
Using a moment-based approach laid out in Perry (2015)
and implemented in the mbest package, we can achieve
a non-iterative fit in ! "#$ + ! &#' steps. This can be
trivially spread across ( processors.
This improvement in computational efficiency is paid for by
sacrificing some statistical efficiency.
20#DSSAIS17

21. Moment Based Setup
!" = $"% + '"(" + )" for * ∈ {1, … , 0}
(" ∼ 3(0, Σ)
)" ∼ 3(0, 89)
(" and )" independent
21#DSSAIS17

22. Moment Based Setup
• Define !" ≡ $" %" and &"
'
≡ ('
)"
'
• *" = $"( + %")" + -" ⟹
• *" = !"&" + -"
• Estimate /&" = !"
'
!"
0
!"
'
1", for each 3 ∈ 1, … , 7
• Note when !" is rank deficient, /&" is not an
unbiased estimator
22#DSSAIS17

23. Moment Based Setup
• !" = $"%"&"
'
• (" = $"%"&")
'
• *" = $"%"&"+
'
• ,ϕ = ./+
=
)
012
∑"4)
5
6" − ,!" ̂9"
+
• : ≡ ∑"4)
5
<"
23#DSSAIS17

24. Estimate !
Ω ≡ $
%&'
(
)%'*%)%'
+
,!- = Ω/'
$
%&'
(
)%'*%)%
+
01%
is an unbiased estimator for !.
24#DSSAIS17

25. Estiamte !
Say we knew *, not just an estimate. Define
• 6 ≡ ∑9:;
<
=9>?9(=9
A
BC9 − =9;
A
*) VG
H
BC9 − =9;
A
*
A
?9=9>
A
• I ≡ ∑9:;
<
=9>?9J9
K>
?9=9>
A
• Ω> ≡ ∑9:;
<
=9>?9=9>
A
⨂=9>?9=9>
A
• vec{ QΣS} = Ω>
K;
vec{6 − VI}
• QΣS is an unbiased estimator for Σ
25#DSSAIS17

26. Estiamte !
• "Σ$ is an unbiased estimator for Σ if you know 6
a priori. We don’t …
• Instead, we use >?. This mean "Σ$ is not an
unbiased estimator in practice.
• It can be shown the bias is often negligible.
• We can project "Σ$ to be Positive Semidefinite
26#DSSAIS17

27. Estimate !"| $%"
• Using a gaussian approximation for p($67|87) and
p(87), compute posterior distribution
– p(87|$67) ∝ p($67|87) p(87)
• There exists a formula for C7(Σ) such that
– EF(87 G = C7I7J (I7
K
$67 − I7M
K
N)
– OVar(87 G = OQ7C7
27#DSSAIS17

28. That was a lot of math! Read
Fast moment-based estimation for hierarchical models
on arXiv, it’s a great paper
28#DSSAIS17

29. Moment Based Estimation Summary
29#DSSAIS17
• Estimate !"#

30. Moment Based Estimation Summary
30#DSSAIS17
• Estimate !"#
• Use !"# to estimate $%

31. Moment Based Estimation Summary
31#DSSAIS17
• Estimate !"#
• Use !"# to estimate $%
• Use !"# to estimate &'(

32. Moment Based Estimation Summary
32#DSSAIS17
• Estimate !"#
• Use !"# to estimate $%
• Use !"# to estimate &'(
• Use !"#, $%, &'( to estimate &Σ(

33. Moment Based Estimation Summary
33#DSSAIS17
• Estimate !"#
• Use !"# to estimate $%
• Use !"# to estimate &'(
• Use !"#, $%, &'( to estimate &Σ(
• Use !"#, $%, &'(, &Σ( to estimate +#|!"#

34. lme4 vs mbest
34#DSSAIS17
●
● ●
●
●
●
●
●
●
●
●
●
1.02
1.04
1.06
1.08
1.10
3 4 5 6
log(n_obs)
rmse
model_type
●
●
lmer
mhglm
RMSE By N Observations Per Group
● ● ● ●
●
●
●
●
●
●
●
●
0
2
4
6
3 4 5 6
log(n_obs)
log(seconds)
model_type
●
●
lmer
mhglm
Seconds To Train By N Observations Per Group

35. Memory Constrained For Large N, M
• Current software is memory constrained on a
single machine
• My macbook pro (2015) starts to run out of
memory for
– M = 100,000
– N = 10,000,000
35#DSSAIS17

36. Spark Implementation
36#DSSAIS17

37. Spark Implementation
37#DSSAIS17

38. Application
Logistic Model with
• N = 120,000,000
• M = 4,000,000
• p = q = 7
• Executors = 8, 16gb memory each
• Running time approximately 40 minutes
38#DSSAIS17

39. Thank You
github.com/stitchfix/MomentMixedModels
39#DSSAIS17