Ilya Trofimov - Distributed Coordinate Descent for L1-regularized Logistic Regression

Ðàñïðåäåëåííûé áëî÷íî-êîîðäèíàòíûé ñïóñê äëÿ
îáó÷åíèÿ ëîãèñòè÷åñêîé ðåãðåññèè ñ
L1-ðåãóëÿðèçàöèåé
Èëüÿ Òðîôèìîâ (Yandex Data Factory),
Àëåêñàíäð Ãåíêèí (AVG Consulting)
4-ÿ Ìåæäóíàðîäíàÿ êîíôåðåíöèÿ ïî àíàëèçó èçîáðàæåíèé,
ñîöèàëüíûõ ñåòåé è òåêñòîâ (ÀÈÑÒ)
Åêàòåðèíáóðã, 09.04.2015

Îáîáùåííûå ëèíåéíûå ìîäåëè
Çàäà÷à ìàøèííîãî îáó÷åíèÿ ïî ïðåöåäåíòàì.
Äàíî: îáó÷àþùàÿ âûáîðêà (xi , yi )n
i=1, xi ∈ Rp
Íóæíî ïîñòðîèòü çàâèñèìîñòü y(x).

i=1, xi ∈ Rp
Ìîäåëèðóåì çàâèñèìîñòü â âèäå y ∼ f (βT
x), íóæíî ïîäîáðàòü
β ∈ Rp

i=1, xi ∈ Rp
β ∈ Rp
Ïðèìåðû: ëèíåéíàÿ ðåãðåññèÿ, ëîãèñòè÷åñêàÿ ðåãðåññèÿ,
ïóàññîíîâñêàÿ ðåãðåññèÿ, ïðîáèò-ðåãðåññèÿ.

i=1, xi ∈ Rp
β ∈ Rp
Ëîãèñòè÷åñêàÿ ðåãðåññèÿ: yi ∈ {−1, +1}
P(y = +1|x) =
1
1 + exp(−βT
x)

i=1, xi ∈ Rp
β ∈ Rp
Ëîãèñòè÷åñêàÿ ðåãðåññèÿ: yi ∈ {−1, +1}
P(y = +1|x) =
1
1 + exp(−βT
x)
Ìèíóñ ëîã-ïðàâäîïîäîáèå (ýìïèðè÷åñêèé ðèñê) L(β)
L(β) =
n
i=1
log(1 + exp(−yi βT
xi ))
argmin
β
L(β)

Îáîáùåííûå ëèíåéíûå ìîäåëè, ðåãóëÿðèçàöèÿ
L2-ðåãóëÿðèçàöèÿ
argmin
β
L(β) +
λ2
2
||β||2

argmin
β
L(β) +
λ2
2
||β||2
L1-ðåãóëÿðèçàöèÿ (îäíîâðåìåííàÿ ðåãóëÿðèçàöèÿ + îòáîð
ïðèçíàêîâ)
argmin
β
(L(β) + λ1||β||1)

Big Data
Áîëüøèå îáó÷àþùèå âûáîðêè
n, p > 106, ðàçìåð > 10 Gb.

Big Data
Áîëüøèå îáó÷àþùèå âûáîðêè
n, p > 106, ðàçìåð > 10 Gb.
Íóæíû áûñòðûå àëãîðèòìû, êîòîðûå ðàñïàðàëëåëèâàþòñÿ
ïî íåñêîëüêèì ïðîöåññîðàì/ÿäðàì
ïî íåñêîëüêèì ñåðâåðàì

argmin
β
L(β) +
λ2
2
||β||2
Ìèíèìèçàöèÿ ãëàäêîé âûïóêëîé ôóíêöèè.

argmin
β
L(β) +
λ2
2
||β||2
Êàê îïòèìèçèðîâàòü?
Ìåòîä SGD
Ìåòîä ñîïðÿæåííûõ ãðàäèåíòîâ
Ìåòîä L-BFGS

argmin
β
L(β) +
λ2
2
||β||2
Ìåòîä SGD ïëîõî ïàðàëëåëèòñÿ
Ìåòîä ñîïðÿæåííûõ ãðàäèåíòîâ õîðîøî ïàðàëëåëèòñÿ
Ìåòîä L-BFGS õîðîøî ïàðàëëåëèòñÿ

ïðèçíàêîâ)
argmin
β
(L(β) + λ1||β||1)
Ìèíèìèçàöèÿ íåãëàäêîé âûïóêëîé ôóíêöèè.

ïðèçíàêîâ)
argmin
β
(L(β) + λ1||β||1)
Ìåòîä ñóáãðàäèåíòà
Ìåòîä online learning via truncated gradient
Ìåòîäû ïîêîîðäèíàòíîãî ñïóñêà (GLMNET, BBR)

ïðèçíàêîâ)
argmin
β
(L(β) + λ1||β||1)
Ìåòîä ñóáãðàäèåíòà ïëîõî ðàáîòàåò
Ìåòîä online learning via truncated gradient ïëîõî
ïàðàëëåëèòñÿ
Ìåòîäû ïîêîîðäèíàòíîãî ñïóñêà (GLMNET, BBR) ?

Öåëü
Íàéòè ñàìûé ëó÷øèé àëãîðèòì äëÿ ìèíèìèçàöèè öåëåâîé
ôóíêöèè çàäà÷è ëîãèñòè÷åñêîé ðåãðåññèè ñ L1-ðåãóëÿðèçàöèåé
íà îäíîé ìàøèíå

Öåëü
Íàéòè ñàìûé ëó÷øèé àëãîðèòì äëÿ ìèíèìèçàöèè öåëåâîé
ôóíêöèè çàäà÷è ëîãèñòè÷åñêîé ðåãðåññèè ñ L1-ðåãóëÿðèçàöèåé
íà îäíîé ìàøèíå
...è ðàñïàðàëëåëèòü åãî

Àëãîðèòì GLMNET
Íóæíî íàéòè: argminβ (L(β) + λ1||β||1)

L(β + ∆β) + λ1||β + ∆β||1 ≈
≈ L(β) + L (β)T
∆β +
1
2
∆βT 2
L(β)∆β + λ1||β + ∆β||1
=
1
2
n
i=1
wi (zi − ∆βT
xi )2
+ C(β) + λ1||β + ∆β||1

L(β + ∆β) + λ1||β + ∆β||1 ≈
≈ L(β) + L (β)T
∆β +
1
2
∆βT 2
L(β)∆β + λ1||β + ∆β||1
=
1
2
n
i=1
wi (zi − ∆βT
xi )2
+ C(β) + λ1||β + ∆β||1
ãäå
zi =
(yi + 1)/2 − p(xi )
p(xi )(1 − p(xi ))
wi = p(xi )(1 − p(xi ))
p(xi ) =
1
1 + e−βT
xi

Âõîä: îáó÷àþùàÿ âûáîðêà {xi , yi }n
i=1, íà÷àëüíîå ïðèáëèæåíèå
β, ïàðàìåòð ðåãóëÿðèçàöèè λ1
Ïîâòîðÿòü, ïîêà íå âûïîëåíî óñëîâèå îñòàíîâà:
1 Äëÿ k = 1 ... p
2 Ïîêà íå âûïîëíåíî óñëîâèå îñòàíîâà:
∆βk ← argmin
∆βk
1
2
n
i=1
wi (zi − ∆βT
xi )2
+ λ1||β + ∆β||1
3 β ← β + ∆β
Âåðíóòü β

Âõîä: îáó÷àþùàÿ âûáîðêà {xi , yi }n
i=1, íà÷àëüíîå ïðèáëèæåíèå
β, ïàðàìåòð ðåãóëÿðèçàöèè λ1
Ïîâòîðÿòü, ïîêà íå âûïîëåíî óñëîâèå îñòàíîâà:
1 Äëÿ k = 1 ... p
2 Ïîêà íå âûïîëíåíî óñëîâèå îñòàíîâà:
∆βk ←
S
n
i=1 wi xik qi , λ1
n
i=1 wi x2
ik
− βk
qi = zi − ∆βT
xi + (βk + ∆βk )xik
S(x, a) = sgn(x) max(|x| − a, 0)
3 β ← β + ∆β
Âåðíóòü β

Äëÿ ýôôåêòèâíîé ðåàëèçàöèè íóæíî ïîääåðæèâàòü â RAM
âåêòîðà (βT
xi ), (∆βT
xi ) (ðàçìåð - n)

Êàê ðàñïàðàëëåëèòü GLMNET?
Èñïîëüçóåì íåñêîëüêî ìàøèí (êëàñòåð).

Åñòåñòâåííî, ÷òîáû êàæäàÿ ìàøèíà îòâå÷àëà çà ñâîå
ïîäìíîæåñòâî ïåðåìåííûõ.

S1 ∪ . . . ∪ SM = {1, ..., p}
Sm ∩ Sk = ∅, k = m

S1 ∪ . . . ∪ SM = {1, ..., p}
Sm ∩ Sk = ∅, k = m
Èäåÿ: êàæäàÿ ìàøèíà ïàðàëëåëüíî âûïîëíÿåò øàãè ïî ñâîåìó
ïîäìíîæåñòâó ïåðåìåííûõ ∆βm
∆βm
← argmin
∆βm
1
2
n
i=1
wi (zi − ∆βT
xi )2
+ λ1||β + ∆β||1 ∆βm
j = 0 åñëè j /∈ Sm

Êàê ðàñïàðàëëåëèòü ìåòîäû ïîêîîðäèíàòíîãî ñïóñêà?
Àëãîðèòì d-GLMNET
Âõîä: Îáó÷àþùàÿ âûáîðêà {xi , yi }n
i=1, ðàçäåëåííàÿ íà M
÷àñòåé ïî ïåðåìåííûì.
β ← 0, ∆β ← 0, ãäå m - íîìåð ìàøèíû
Ïîêà íå âûïîëíåíî óñëîâèå îñòàíîâà:
1 Âûïîëíèòü ïàðàëëåëüíî íà M ìàøèíàõ:
2 Âûïîëíèòü øàãè ïî ïåðåìåííûì, ñîõðàíèòü ∆βm
,
(∆(βm
)T xi ))
3 Ñóììèðîâàòü âåêòîðà ∆βm
, (∆(βm
)T xi ) ñ ïîìîùüþ
MPI_AllReduce
4 ∆β ← M
m=1 ∆βm
5 (∆βT
xi ) ← M
m=1(∆(βm
)T xi )
6 Íàéòè α ñ ïîìîùüþ àëãîðèòìà ëèíåéíîãî ïîèñêà (ïðàâèëî
Armijo)
7 β ← β + α∆β,
8 (exp(βT
xi )) ← (exp(βT
xi + α∆βT
xi ))

Òåîðåòè÷åñêèå ðåçóëüòàòû
Òåîðåìà 1. Èòåðàöèÿ àëãîðèòìà d-GLMNET ñîîòâåòñòâóåò
îïòèìèçàöèè
argmin
∆β
L(β) + L (β)T
∆β +
1
2
∆βT
H∆β + λ1||β + ∆β||1
ãäå H - áëî÷íî-äèàãîíàëüíîå ïðèáëèæåíèå ê Ãåññèàíó 2L(β)

Òåîðåòè÷åñêèå ðåçóëüòàòû
Òåîðåìà 1. Èòåðàöèÿ àëãîðèòìà d-GLMNET ñîîòâåòñòâóåò
îïòèìèçàöèè
argmin
∆β
L(β) + L (β)T
∆β +
1
2
∆βT
H∆β + λ1||β + ∆β||1
ãäå H - áëî÷íî-äèàãîíàëüíîå ïðèáëèæåíèå ê Ãåññèàíó 2L(β)
Òåîðåìà 2. Àëãîðèìò d-GLMNET îáëàäàåò êàê ìèíèìóì
ëèíåéíîé ñêîðîñòüþ ñõîäèìîñòè.

×èñëåííûå ýêñïåðèìåíòû
dataset size #examples (train/test) #features nnz
epsilon 12 Gb 0.4× 106 / 0.1× 106 2000 8.0× 108
webspam 21 Gb 0.315× 106 / 0.035× 106 16.6× 106 1.2× 109
dna 71 Gb 45× 106 / 5× 106 800 9.0× 109

epsilon 12 Gb 0.4× 106 / 0.1× 106 2000 8.0× 108
webspam 21 Gb 0.315× 106 / 0.035× 106 16.6× 106 1.2× 109
dna 71 Gb 45× 106 / 5× 106 800 9.0× 109
16 ìàøèí ñ Intel(R) Xeon(R) CPU E5-2660 2.20GHz, 32 GB
RAM, ãèãàáèòíûé Ethernet.

epsilon 12 Gb 0.4× 106 / 0.1× 106 2000 8.0× 108
webspam 21 Gb 0.315× 106 / 0.035× 106 16.6× 106 1.2× 109
dna 71 Gb 45× 106 / 5× 106 800 9.0× 109
Ñðàâíèâàëèñü àëãîðèòìû
d-GLMNET
Online learning via truncated gradient (Vowpal Wabbit)

epsilon 12 Gb 0.4× 106 / 0.1× 106 2000 8.0× 108
webspam 21 Gb 0.315× 106 / 0.035× 106 16.6× 106 1.2× 109
dna 71 Gb 45× 106 / 5× 106 800 9.0× 109
Ñðàâíèâàëèñü àëãîðèòìû
d-GLMNET
Online learning via truncated gradient (Vowpal Wabbit)
Íà êàæäîé ìàøèíå çàïóñêàëñÿ îäèí ïðîöåññ d-GLMNET èëè
Vowpal Wabbit.

1 Ñ ïîìîùüþ d-GLMNET âû÷èñëÿëñÿ ïóòü ðåãóëÿðèçàöèè
äëÿ 20 çíà÷åíèé λ1. Äëÿ êàæäîãî ðåøåíèÿ âû÷èñëÿëîñü
êîëè÷åñòâî íåíóëåâûõ âåñîâ è òî÷íîñòü íà òåñòîâîì
ìíîæåñòâå.
2 Äëÿ âñåõ çíà÷åíèé λ ∈ [λmax 2−1, λmax 2−2, ..., λmax 2−20]
ïåðåáèðàëèñü ãèïåðïàðàìåòðû îíëàéí-îáó÷åíèÿ ñîâìåñòíî
â äèàïàçîíàõ η ∈ [0.1, 0.5], p ∈ [0.5, 0.9] è âûïîëíÿëîñü 50
ïðîõîäîâ îíëàéí-îáó÷åíèÿ.
Äëÿ êàæäîé êîìáèíàöèè (η, p, íîìåð ïðîõîäà)
âû÷èñëÿëîñü êîëè÷åñòâî íåíóëåâûõ âåñîâ è òî÷íîñòü íà
òåñòîâîì ìíîæåñòâå.

Äàòàñåò ¾epsilon¿
Êà÷åñòâî êëàññèôèêàöèè äëÿ ðàçíûõ λ1

Äàòàñåò ¾epsilon¿
0.93
0.935
0.94
0.945
0.95
0.955
0.96
0 200 400 600 800 1000 1200 1400
auPRC
Time, sec
d-GLMNET
VW
Ñêîðîñòü àëãîðèòìîâ äëÿ ëó÷øåãî λ1 è ëó÷øèõ ïàðàìåòðîâ
îíëàéí-îáó÷åíèÿ

d-GLMNET
Ðåàëèçàöèÿ d-GLMNET äîñòóïíà ïî àäðåñó
https://github.com/IlyaTrofimov/dlr

d-GLMNET
ïðåïðèíò http://arxiv.org/abs/1411.6520

d-GLMNET
ïðåïðèíò http://arxiv.org/abs/1411.6520
Äàëüíåéøåå ðàçâèòèå:
L2-ðåãóëÿðèçàöèÿ, elastic net
èñïîëüçîâàíèå íåñêîëüêèõ ÿäåð
ðåàëèçàöèÿ LASSO

Ñïàñèáî çà âíèìàíèå :)
Âîïðîñû ?

Ilya Trofimov - Distributed Coordinate Descent for L1-regularized Logistic Regression

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

More from AIST

More from AIST (20)

Ilya Trofimov - Distributed Coordinate Descent for L1-regularized Logistic Regression