Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
G
A
A
Q
Q R D
Google Kubernetes Engine
MR
N S C T O
D
2 #
9 @2 0 A 0 / # :. 01 0 / #
/ # . / #
19 @2:0 A 20 7 1:2 #
)0
• A 0 J i s o d pl8D CC A C J ot l J E )
• J C l s ber4 J E J B A I
• R R M- c u l s dr s T
n kh S R l/ 1 / IJ D CA ACA...
( ) 3 1
? @ 0 00 0
: 7
/
> > -
2
5 3 015 38
• ? ?
• ? C ? CT
• U A ? A / ? C UWL
• ? / ? C p
• ? / ? C
• D
• U D
4
• k lGcWtrw Ff e ibl
• G W
• G T A ? A / ? C
• P ys...
•
•
•
• ?A
• E
• -
• -
Argoproj is a collection of tools
for getting work done with Kubernetes.
• Argo Workflows - Container-native Workflow Engi...
O D CR W G
7 K
D 7 A
L + - - -
• arfi
- - - - 8- + -, +?
• arfi 8 gc or
Wb n c M ehW
• kp cfel arfi E
cfel cfels w V kp tEd ca h v
)G E EGA ELH - 8 G H
• ) EG FH 8H 8G8 ED E LEGA ELH
• )G 8 H FFEG
• )G 8 EG / G8L
• F K DF E F H
• 8G 8 H F8G8C GH
• 1EEFH...
1 -
- 1
H
E
- 12 E
1 3
• A F E
• c d
• c - - - - -
• We a
•
• c We a
5 /
1 /
2
6
C B C 2
C 2
11 Co B
7 1
1
• c P
• S a E
•
• A I
• c
•
8 - - -1
- 1- -1
• e U D
• ca k nt D
• g c g c L
• a EU D T
• D C g g cP
• p i r i L S I
I
• a EU
• g T IP a
- 1- -1
• DI ...
:9- 8. - 2 / 02 10
• 9 9 1
0
:9- 8. - 2 / 02 10
• 0 2
•
8 L l I m k 8 S m g l
j d j o
• j
• j
j j iy l
k b l
j a j
C j k s j G C j k
e a ui g
• L 1 2 fa
• 1 1 p c 1 1 1
• 1 1 D n
• ( ) (
• e 1 og 1L 1 S m
sm
Csm
s LI
lknt t
( ) ( (
y
) sa
p
l o
8 l s l l
8
R h y
D S
3 2
3 2
• /
• E D
• IC
• L
L I ) 3 2L
5 E
r C reh l C jc
L
• ro_m C
• ( r in p C
• 0:: 9 /. 1: :/.0 .6 2 / 9 ) (- ) 0: 3
reh
P 1/
V
jc
/2
(
& (& &( )
3
2
• steps.{step-name}.outputs.parameters
• steps.{step-name}.outputs.artifacts
8 4
8 4
1 2 4 / 8 4
• .99 8 ::: 9 71/9 5 0 /9 79/ 2 8 ( 4 :8 ) .9 2
• .99 8 ::: 9 71/9 5 0 /9 79/ 2 8 4 :8 .9 2
2 /4
T 3E
/ 0
P S
P 4
C G 3 1 / 4
RC A F
E O .
PI
:
AWS GCS Minio
: : / .
3 2 W
/ 1
2 W / 1
/
%
%
3 2/ 2
A
B
/
4 /
H
S H
B A
R E
R
/
3
AC ( 3 3
(
(
)
5 /
2
1
5 (
6 /
2 2 3
• DC MO C z AC CK
• D g mbR V rP
D BK
• Snv S S vR C
E DC DC DCBK
• A DB K
• lI kIS R & C
BK
• D A C B gc b f b
...
3 .
• )/
• ( )/
• )/
• )/
• )/
3 )3 138 (/
( 2 /32
• ( 2 /32 )3 138 A
• ( ( ES -1 g t w t
• l AdeE ) yf yf A Rn T kpAES
• o DP ES
uBr h uBr h
uBr h
h h
h
3 3 / 39
2 T
/ 3 A 3 W P
( ( ))
2 T
E /0 G P - / Ec A
• / -/ E T a C A PE S
• 00 4 / 4 4 / / /- E UG TC
/ 3 1
/ 3 1 4 / 1 A
/ 3
/ 3 4 2 / A
•
• S G G
• C M G
3 4 4
• h E A N
• /. GH C H
3. T c ge ./ 3 /. i
/. a
• / 4 i
•
. 4 /.
21 /1
G G A io s
h e W g h ea ei
P b h
m rlE d g
k nu P bg
h E p ht
argo submit -f ci/hello.yaml
git clone
spec:
templates...
2 /
/ n /
D w P p t
/
ID C r a c e
o d d / Ck
d i hm / Csp DC
Cn
- name: build
sidecars:
- name: dind
image: docker:18-din...
1 26
D C
SO e D f
g d
PR R ia Be
cCe A C i
a Be
Argo CD
Sealed Secrets
Controller
Secrets
Sealed Secrets
Custom Resource
K...
/
cg S A l
n 2 .
S 2 RFR
voS t
production
develop
spec:
destination:
server:
https://kubernetes.default.svc
source:
target...
5 4 .
A 9 /
/ I 4/ /4 1 31
w s A / i g l
/ 5 32 i l l
/ 5 32 / / e t
• 5 32 r vo k n
• l e
• TMG c E M
gd G E
c G MiN M M Ma
fk 5 2 / 0t
• E E / 5 32...
u Wo P 5 hEe la
/ 9 : C l iDat v R P PESE
•
kD cjD m W Ps SE
• -
•
•
/ 9 : C.3 : 3 T P r S PE
• 57 A1 2 5 5 7 A3
• 57 A1 2...
1 0
r @
A i ? 7
e d o h Q
ng 0
4 + 5
55 4 RS g u c
g 4 o CTA
T
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: my-project
spec:
descrip...
Argoによる機械学習実行基盤の構築・運用からみえてきたこと
Argoによる機械学習実行基盤の構築・運用からみえてきたこと
Argoによる機械学習実行基盤の構築・運用からみえてきたこと
Upcoming SlideShare
Loading in …5
×

Argoによる機械学習実行基盤の構築・運用からみえてきたこと

4,494 views

Published on

機械学習は大量のリソースを必要とするためマネージドな環境をスケーラブルに使えることが重要です.また,機械学習システムは専門性が高く属人化してしまうことも事実です.これらの課題を解決するためにリクルートテクノロジーズではArgoワークフローを導入し,属人化や環境依存性が高いといった問題に取り組んでいます.Argoはコンテナベースのワークフローエンジンで,ワークフローの各ステップをコンテナとして実装することを可能にします.また, 弊社ではArgoプロジェクトに含まれるArgoCDと組み合わせることで,CI/CD パイプラインを構築し,GitOps を実現しています.Argo Workflowsによる機械学習実行基盤を運用していく上で見えてきた,使用者側の視点・運用する側の視点両方からの知見を共有します!
[1C1] 13:20 - 14:00
河野 晋策 (Shinsaku Kono) / 機械学習エンジニア/データサイエンティスト/検索エンジニア, リクルートテクノロジーズ
稲村 秀人 (Hideto Inamura) / システムアーキテクト, リクルートテクノロジーズ
Cloud Native Days Tokyo 2019 / OpenStack Days Tokyo 2019
#CNDT2019 / #OSDT2019
https://cloudnativedays.jp/cndt2019/

cndt2019 / osdt2019 / cndt / osdt / argo / workflow engine / workflow / gitpos / argo workflows / argo cd / 機械学習 / 機械学習実行基盤 / リクルート / リクルートテクノロジーズ / mlops / data science / machine learning / deep learning / google cloud platform / 深層学習 / cloud native days tokyo 2019 / cloud native days / open stack days tokyo 2019 / open stack days / digdag / argo events

Published in: Technology
  • ACCESS that WEBSITE Over for All Ebooks (Unlimited) ......................................................................................................................... DOWNLOAD FULL PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... DOWNLOAD FULL EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m6jJ5M }
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THAT BOOKS/FILE INTO AVAILABLE FORMAT - (Unlimited) ......................................................................................................................... ......................................................................................................................... Download FULL PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... accessibility Books Library allowing access to top content, including thousands of title from favorite author, plus the ability to read or download a huge selection of books for your pc or smartphone within minutes Christian, Classics, Comics, Contemporary, Cookbooks, Art, Biography, Business, Chick Lit, Children's, Manga, Memoir, Music, Science, Science Fiction, Self Help, History, Horror, Humor And Comedy, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • accessibility Books Library allowing access to top content, including thousands of title from favorite author, plus the ability to read or download a huge selection of books for your pc or smartphone within minutes.........ACCESS WEBSITE Over for All Ebooks ..... (Unlimited) ......................................................................................................................... Download FULL PDF EBOOK here { http://bit.ly/2m6jJ5M } .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • accessibility Books Library allowing access to top content, including thousands of title from favorite author, plus the ability to read or download a huge selection of books for your pc or smartphone within minutes DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ...................................ALL FOR EBOOKS................................................. Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • accessibility Books Library allowing access to top content, including thousands of title from favorite author, plus the ability to read or download a huge selection of books for your pc or smartphone within minutes DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ...................................ALL FOR EBOOKS................................................. Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Argoによる機械学習実行基盤の構築・運用からみえてきたこと

  1. 1. G A A Q Q R D Google Kubernetes Engine MR N S C T O D 2 # 9 @2 0 A 0 / # :. 01 0 / # / # . / # 19 @2:0 A 20 7 1:2 #
  2. 2. )0 • A 0 J i s o d pl8D CC A C J ot l J E ) • J C l s ber4 J E J B A I • R R M- c u l s dr s T n kh S R l/ 1 / IJ D CA ACAJ 1E AE AE R i k S 2 22 0 ) ( 0 ? @9 : u u R u R R u R R ) ( & R P R R P s g a dP m l sch P y y P
  3. 3. ( ) 3 1 ? @ 0 00 0 : 7 / > > - 2 5 3 015 38
  4. 4. • ? ? • ? C ? CT • U A ? A / ? C UWL • ? / ? C p • ? / ? C • D • U D 4 • k lGcWtrw Ff e ibl • G W • G T A ? A / ? C • P ys W • ? xv I • G W ys ( E I • - T • m jGdn • / F gka c o F m • I A 23 . • ? / ? C - • ? D ) • ? A - • ? / ? C - • ? - 4 3 5 1
  5. 5. • • • • ?A • E • - • -
  6. 6. Argoproj is a collection of tools for getting work done with Kubernetes. • Argo Workflows - Container-native Workflow Engine • Argo CD - Declarative GitOps Continuous Delivery • Argo Events - Event-based Dependency Manager • Argo Rollouts(NEW!) - additional deployment strategies https://argoproj.github.io/argo/
  7. 7. O D CR W G 7 K D 7 A
  8. 8. L + - - - • arfi - - - - 8- + -, +? • arfi 8 gc or Wb n c M ehW • kp cfel arfi E cfel cfels w V kp tEd ca h v
  9. 9. )G E EGA ELH - 8 G H • ) EG FH 8H 8G8 ED E LEGA ELH • )G 8 H FFEG • )G 8 EG / G8L • F K DF E F H • 8G 8 H F8G8C GH • 1EEFH • 8G8C G 8 ED • ED ED8 H • C E H H F LEGA EL K • 3 G H F LEGA EL K • 3 H C C CE • HF D 3 H C & • 8D 8 ED • 0(H G HE G EG H G8 ED • , /EEAH DE 8 EDH 8D F • 8G 8 E ED E ECF LEGA EL • D • 8 D E G8 EDH DE H EGH • E C H F C G8 H D • 8G8 HC C H • 8 CED H FH • D E A G D E A G • G F H FH
  10. 10. 1 - - 1 H E
  11. 11. - 12 E
  12. 12. 1 3
  13. 13. • A F E • c d • c - - - - - • We a • • c We a
  14. 14. 5 / 1 /
  15. 15. 2 6 C B C 2 C 2 11 Co B
  16. 16. 7 1 1 • c P • S a E • • A I • c •
  17. 17. 8 - - -1 - 1- -1 • e U D • ca k nt D • g c g c L • a EU D T • D C g g cP • p i r i L S I I • a EU • g T IP a - 1- -1 • DI g c • g - -- - L E
  18. 18. :9- 8. - 2 / 02 10 • 9 9 1
  19. 19. 0 :9- 8. - 2 / 02 10 • 0 2 • 8 L l I m k 8 S m g l j d j o • j • j j j iy l k b l j a j C j k s j G C j k
  20. 20. e a ui g • L 1 2 fa • 1 1 p c 1 1 1 • 1 1 D n • ( ) ( • e 1 og 1L 1 S m
  21. 21. sm Csm s LI lknt t ( ) ( ( y ) sa p l o 8 l s l l 8 R h y D S
  22. 22. 3 2 3 2
  23. 23. • / • E D • IC • L
  24. 24. L I ) 3 2L 5 E r C reh l C jc L • ro_m C • ( r in p C • 0:: 9 /. 1: :/.0 .6 2 / 9 ) (- ) 0: 3 reh P 1/ V jc
  25. 25. /2 ( & (& &( ) 3 2 • steps.{step-name}.outputs.parameters • steps.{step-name}.outputs.artifacts
  26. 26. 8 4 8 4 1 2 4 / 8 4 • .99 8 ::: 9 71/9 5 0 /9 79/ 2 8 ( 4 :8 ) .9 2 • .99 8 ::: 9 71/9 5 0 /9 79/ 2 8 4 :8 .9 2
  27. 27. 2 /4
  28. 28. T 3E / 0 P S P 4
  29. 29. C G 3 1 / 4 RC A F E O . PI : AWS GCS Minio : : / .
  30. 30. 3 2 W / 1 2 W / 1 / % %
  31. 31. 3 2/ 2 A B /
  32. 32. 4 / H S H B A R E R / 3
  33. 33. AC ( 3 3 ( ( ) 5 / 2 1 5 (
  34. 34. 6 / 2 2 3 • DC MO C z AC CK • D g mbR V rP D BK • Snv S S vR C E DC DC DCBK • A DB K • lI kIS R & C BK • D A C B gc b f b SYhI uRMH / 2 • C f b RL st D A C B i IRo Pp • DBC G R , S x • i I SaekY • p S x • /
  35. 35. 3 . • )/ • ( )/ • )/ • )/ • )/
  36. 36. 3 )3 138 (/ ( 2 /32 • ( 2 /32 )3 138 A • ( ( ES -1 g t w t • l AdeE ) yf yf A Rn T kpAES • o DP ES uBr h uBr h uBr h h h h
  37. 37. 3 3 / 39 2 T / 3 A 3 W P ( ( )) 2 T
  38. 38. E /0 G P - / Ec A • / -/ E T a C A PE S • 00 4 / 4 4 / / /- E UG TC
  39. 39. / 3 1 / 3 1 4 / 1 A
  40. 40. / 3 / 3 4 2 / A • • S G G • C M G
  41. 41. 3 4 4 • h E A N • /. GH C H 3. T c ge ./ 3 /. i /. a • / 4 i • . 4 /.
  42. 42. 21 /1 G G A io s h e W g h ea ei P b h m rlE d g k nu P bg h E p ht argo submit -f ci/hello.yaml git clone spec: templates: - name: approve suspend: {} 2 4
  43. 43. 2 / / n / D w P p t / ID C r a c e o d d / Ck d i hm / Csp DC Cn - name: build sidecars: - name: dind image: docker:18-dind securityContext: privileged: true mirrorVolumeMounts: true container: image: docker:18 env: - name: DOCKER_HOST value: 127.0.0.1 command: - sh - -c args: [" until docker ps; do sleep 3; done; ¥ docker build -t example . && ¥ docker save example > example.tar "] I 4 C 5 DA
  44. 44. 1 26 D C SO e D f g d PR R ia Be cCe A C i a Be Argo CD Sealed Secrets Controller Secrets Sealed Secrets Custom Resource Kubernetes API Server apply sync D 4 1241 / 24 : 4 / A 1 C A
  45. 45. / cg S A l n 2 . S 2 RFR voS t production develop spec: destination: server: https://kubernetes.default.svc source: targetRevision: master spec: destination: server: https://kubernetes.default.svc source: targetRevision: develop kunm Svo a e bgSrh P 2 1 py KJSr 2 dd o si o apiVersion: batch/v1 kind: Job metadata: generateName: post-sync- annotations: argocd.argoproj.io/hook: PostSync 47 7 4 2 4
  46. 46. 5 4 .
  47. 47. A 9 / / I 4/ /4 1 31
  48. 48. w s A / i g l / 5 32 i l l / 5 32 / / e t • 5 32 r vo k n • l e • TMG c E M gd G E c G MiN M M Ma fk 5 2 / 0t • E E / 5 32 u fk o r • TMG M e TA E / 5 32 p o
  49. 49. u Wo P 5 hEe la / 9 : C l iDat v R P PESE • kD cjD m W Ps SE • - • • / 9 : C.3 : 3 T P r S PE • 57 A1 2 5 5 7 A3 • 57 A1 2 5 5 A:: kD cjD bhgD W s E p . W :A 3f la bhgD E E w W s SE
  50. 50. 1 0 r @ A i ? 7 e d o h Q ng 0
  51. 51. 4 + 5 55 4 RS g u c g 4 o CTA T apiVersion: argoproj.io/v1alpha1 kind: AppProject metadata: name: my-project spec: description: Example Project sourceRepos: - '*' destinations: - namespace: guestbook server: https://kubernetes.default.svc clusterResourceWhitelist: - group: '' kind: Namespace namespaceResourceBlacklist: - group: '' kind: ResourceQuota 4 i P 4 mCTA R G p P tl m 44 g h s BG T D 44 4 5 O e r j apiVersion: v1 kind: ConfigMap metadata: name: argocd-cm namespace: argocd labels: app.kubernetes.io/name: argocd-cm data: dex.config: | connectors: - type: oidc id: google name: Google config: {} helm.repositories: | - url: https://kubernetes- charts.storage.googleapis.com name: sealed-secrets

×