テーマ ・Spark MLlib ML Pipelines の概要 ・pysparkからML Pipelinesを扱う方法 内容 ・EstimatorとTransformerの概要 ・ML Pipelinesの概要・使い方 ・pysparkから、Scalaで実装した処理を呼び出す ・先行stageの副作用を利用する (Pipelineの拡張)

1. ) / ) / . ) / ) /: / . () ./ / ) / 11 / .
2:2 @ 01 00 0 . .
92 ()

2. ) 2/ )2/ . ) 2/ )2/ 2
•M
• ( Pf
• L a
•
• E Pf
• Pf
• ) L
• S TeLge
: / . () ./ / )2/ / .
b i c

3. ) / ) / 3 . ) / ) /: / . () ./ / ) / / .

4. ) / ) / 4 . ) / ) / 44 : / . () ./ / ) / / .
) L M
O
( S R
AC U

5. : 2. 12 1 5 :. :: : 2. 12 1
• 5 7 f a
( 1/ : ( 2. 1 )12 1 1/ : . . .
i
5
8 # 6 #
9 6
6 6 6 6 6 6
) ( 6 (
) 6
6 6 6 6 6 6
# 6 #
5
8 # 6 #
9 6
6 6 6 6 6 6
) ( 6 (
5
) 6
6 6 6 6 6 6
# 6 #
f
5
) 6
6 6 6 6 6 6
# 6 ##
4
f
42 03 12/
5
5
5
5 f5 f

6. : 2. 12 1 6:. :: : 2. 12 1 6
• 6D9E? 32 :T eIi S F 9 EU3 ObN
(6 1/ : ( 2. 1 )12 1 1/ : . . .
F 9 EU3 d
L # #
M
) ( (
)
#
)
# ##
f
i a
F 9 E
3
E9 F E
14 9 90E9
587 3
14 9 90E9 587 9 90E9

7. : 2. 12 1 :. ::7 : 2. 12 1
• 5 7 ?: U 7D7 7?:Tl en
• ioTl H4 :0 D 8 9: 5 : E: ?7 E: I a
( 7 1/ : ( 2. 1 )12 1 1/ : . . .
5 7 ?: T m O
)# ( ,
,
## ) ,
)# (
## )
k F d F
5 7 ?:
D 7 ?
1 7D7 7?: 4 5 7D7 7?:
2 9: U5 7 ?: T c
D ?7D M fN 5 7 ?: 2 9:

8. 8) / ) / 8 .88 8) / ) /: / .8 () ./ / ) / / .8

9. ) / ) / . ) / ) /
• L C M
•
S P V C
: / . () ./ / ) / / . 9
S
A
P V

10. : 2. 12 1 :. 0:: : 2. 12 1
• / 05947584 L CMCL
ls P k V V
( 1/0: ( 2. 01 )12 1 1/0: . . .
193 6 / 05947584 C
#
A #
#
a
#
ls
0 V
nr
12/
0 V
nr
12/
kk ? e i p V M
S C

11. 1 1 / 1 1
• / 06 58695 PM M
d li ? l n s eL S
./ ) 1 / ( 1 22 ./ :
1 3 7 / 06 58695 C P
A #
#
#
rk
#
d ?
li ?
rk
06 58695
0 S
a
12/L
Vo
06 58695/ 458
0 S
a
12/L
Vo06 58695 M
pLn s e
06 58695 M
pLn s e

12. 1 1 / 1 1./ ) 1 / ( 1 22 ./ :

13. C 3 : .
• 3.- 452M39 8 9 8
7 70 7 8L MPCL D F
:A 3 3 ) 3 /
21 39 8 9 8 M 7 70 7 8Me M Sa
1 :. ( / . : / .
, 6, ,, 6 ( 6 , (
, 6 6 ( , 6 ((, ,
A A A A A
, 6 , 6 , 6 (( ,
3.-LP
[
452 M
V i

14. : 2.412 41 :. :: : 2.412 41 4( 1/ : ( 2. 1 )12 41 1/ : . . .
4925472 P a LE
A # #
461
A # #
17 3 62
461
17 3 62
TS e
A # # A # #
17 3 62 17 3 62
A # #
V
4925472 M C

15. : 2. 12 1 5 :. :: : 2. 12 1( 1/ : ( 2. 1 )12 1 1/ : . . .
16 214 L
5
# #
5
# #
7 4 573 7
913 957
7 4 573 7
MT
5
# #
5
# #
7 4 573 7 7 4 573 7
5
# #
MT
L 9 L
P a
9 E 913 957

16. 1 1 / 6 1 16 ./ ) 1 / ( 1 ./ :

17. 1 1 / 1 1
( )
7 ./ ) 1 / ( 1 7 ./ 7 7:
<: : ilW K: KF>K[p]hW
I:<D: > < F B A ; :D>FBD:FB > B<: F >: K>
BFI K < F : B BD: D K F B BI:=B< T D> " D> BS>KU
BFI K K :I:<A> I:KD F B T,> : :K:F KB :; >" .=> B B:; >U
BFI K K :I:<A> I:KD F :K K: KF>K
BFI K K :I:<A> I:KD J I> T KK: I>" ,: : I>" KB I>U
< : :I: > > D> BS>K O>KKB=> O: B= KB
> > = :K K: KF>K KB " >J KB " :I: > > D> BS>K PB A ,> : :K:F KB :; > T
=> AB ( AB .=> B B:; > K: = F ., > B<: I D
O>KKB=> IK >< >= => <K>: > K: KF- < KB () >J KB ( T :I: > > D> BS>K D> BS> U
O>KKB=> IK >< >= => I ,: : I> ,: : I> ( >P KK: I> KB I>" K >
U
; >< :I: > > D> BS>K T
O: D> BS>K ( >P D> BS>K B =>K ; B =
=> D> BS> > > <> KB KK: KB ( T
O: D> ( D> BS>K D> BS> > > <> KK:
O: P K= ( K D> D> ( : . : <> D> B> = D> > K :<>
P K=
U
U
B BD: K F B [
emkdj]PK:IW p
KB oq[ _nb
emkdj[ig fc[oqa

18. 1 1 8 / 1 1
12/
./ ) 1 8/ ( 1 ./ 8 8 :
4: C T b S H I= H HW O T cT _RdW R
=H OI H @ HJ 4 H ) J NJ OM H CO
=H OI H C MH H @ HJ , L H I= H H
=H OI H C KJ@C @ HJ , L . C , L . H@J C
=H OI H C H I H @ HJ * I KJ) C * I KJ KJ) C
:C II 4 C@: KH @ @P H , L H I= H H * I KJ) C * I KJ KJ) C , L . C , L . H@J C
( OM H CO
= @ @J I C= @ KJ) C KJ KJ) C
IK H 4 C@: KH @ @P H I C= @ @J
I C= L I C= M L : @J K J @ @ I C@: C = JKH , I @P H I C= K@
M H I I C= @ KJ M H I
I C= I J H I "" M H I
( OM H CO
= I J H I I C= @ KJ) C KJ KJ) C
M H I I C= @ KJ M H I
H JKH I C= I J "" M H I
4: C T b S W O Ta

19. 9 1 1 9 /99 9 1 1
• q ki
/ 2
./9 ) 1 / ( 1 ./9 :
4 D ]os [ FK = h P( ]jt]lap whn b
F = F )
PK C K D L 4 C4=KK F 4 + FL=OL
K C 4 C4=KK F : D = = K D= " =L1 + = L= "
C C=F Q= 4=D . C=F Q= F L+ D K=FL=F = L L+ D N K "
K D+ FL=OL 4 + FL=OL K C"
K D+ FL=OL = L=, L - = ' eeeeeeee " K=FL=F = "
L C=F Q= C C=F Q= L FK "
L C=F Q= K N "
K=FL=F = N K
' eeeeeeee ee e ee e e
PK C fL FK hlap
T4 D ]os [q eS PK C =D F=_rcmd pug

20. 0 0 / 0 0
( )
0./ ) /0 (0 0 22 0./ :

21. 1 1 / 1 1./ ) 1 / ( 1 22 ./ :
27 325 9E 9 1 L M LT H
# #
9 24 6
# #
59 6 4
9 24 6
59 6 4
# # # #
59 6 4 59 6 4
# #
P
• 9 1 P E L M
9 1 E L
9 24 6 7 4

22. ) 2/ )2/ . ) 2/ )2/ 2: / . () ./ / )2/ / .
aC : kbl mlO T LN. = di ] cLSM
> C <#>= > . = . = = )C > C >
= CC ( == <. = . =
C = C : C , == <C
CE ( == <. = C = # C : C C : C
C = # == <C == <C
C = C
C : C C = #: : C
fn
C : E> C : C
C > C
= C = # == <C C = # == <C
C = # == <C C : C > C
C )C >
fn
aC : C > P [M
( == <_g eM

23. 3 3 / 3 3 3./ ) / ( 3 22 ./ :
=. 2 a b cb C :2 2 cb [ B
: :2 2 ). .0 :2 2 =. 2 ,2 = .= 2 = .=
0. .0 , 2
. 1. = 2=((( =, ....
). .0
=. 2
P N ] B