Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Kubernetes Jobによるバッチシステムのリソース最適化 / AbemaTV DevCon 2018 TrackB Session B6

2,843 views

Published on

AbemaTVの動画トランスコードを支えるバッチシステムが抱えていた問題と、Kubernetes Jobを使った改善策についてお話します。

Published in: Technology
  • Be the first to comment

Kubernetes Jobによるバッチシステムのリソース最適化 / AbemaTV DevCon 2018 TrackB Session B6

  1. 1. 

  2. 2. ! "
  3. 3. 
 

  4. 4. ! ! 

  5. 5. 1080p! 720p! 480p" 240p"360p" 180p" HLS MPEG2-TS 1080p! 720p! 480p" 240p"360p" 180p" MPEG-DASH Fragmented MP4 # $ Thumbnail % DRM packaging
  6. 6. Worker 1 Worker 2 Worker 3 Queue ∠
  7. 7. Task1Task2Task3 Worker Pod1 Worker Pod2 Worker Pod3 Node (GCE VM Instance) Enqueue task ID Store task details Pull tasks Transcode Request Kubernetes Task1 Task2 program: XXX type: master status: inqueue program: YYY type: mbr status: inqueue ∠
  8. 8. Task1Task2Task3 Worker Pod1 Worker Pod2 Worker Pod3 Node (GCE VM Instance) Enqueue task ID Store task details Pull tasks Transcode Request Kubernetes Task1 Task2 program: XXX type: master status: inqueue program: YYY type: mbr status: inqueue ∠
  9. 9. Worker Pod 1 Worker Pod 2 Worker Pod 3 vCPUx5, Mem 8GB vCPUx5, Mem 8GB vCPUx5, Mem 8GB メタデータ生成 必要リソース小 240pトランスコード 必要リソース中 1080pトランスコード 必要リソース大 Kubernetes Node (vCPUx16, Mem 25GB)
  10. 10. kind: Pod spec: containers: - name: transcoder image: transcoder resources: requests: memory: “500Mi" cpu: "1" limits: memory: “8Gi" cpu: "5"
  11. 11. Kubernetes Worker Pod 1 use: vCPUx1, Mem “500Mi” Worker Pod 2 use: vCPUx5, Mem “8Gi” Node (vCPUx16, Mem 25GB) … Kubernetes Worker Pod 2 use: vCPUx1, Mem “500Mi” Worker Pod 1 use: vCPUx5, Mem “8Gi” Node (vCPUx16, Mem 25GB) …
  12. 12. Kubernetes Worker Pod 1 use: vCPUx8, Mem 10GB Worker Pod 2 use: vCPUx4, Mem 8GB Worker Pod 3 use: vCPUx4, Mem 8GB Worker Pod 4 use: vCPUx8, Mem 10GB Node (vCPUx16, Mem 32GB) Node Evict
  13. 13. Kubernetes Worker Pod 1 use: vCPUx8, Mem 10GB Worker Pod 2 use: vCPUx4, Mem 8GB Worker Pod 3 use: vCPUx4, Mem 8GB Worker Pod 4 use: vCPUx8, Mem 10GB Node (vCPUx16, Mem 32GB) Node Evict
  14. 14. High-perf Workers vCPUx4 Mem 8GB. Low-perf Workers vCPUx1 Mem 2GB. Kubernetes ex) metadata, 180p, 240p ex) 720p, 1080p Trancode ∠ ∠Transcode Request
  15. 15.
  16. 16. MBR Queue ∠ Transcode Queue ∠ DRM Queue ∠ Metadata Generator vCPUx1 Mem 2GB. MBR Transcoder vCPUx4 Mem 8GB. DRM Packager vCPUx2 Mem 4GB. Kubernetes
  17. 17.
  18. 18. Job Pod
  19. 19. Task1 Task2 program: XXX type: master status: complete program: YYY type: mbr status: running Create JobStore task details Kubernetes Transcode 1080p vCPUx8, Mem 10GB DRM packaging vCPUx1, Mem 1GB Transcode 240p vCPUx2, Mem 2GB Transcode 480p vCPUx4, Mem 4GB Transcode Manager
  20. 20. Task1 Task2 program: XXX type: master status: complete program: YYY type: mbr status: running Create JobStore task details Kubernetes Transcode 1080p vCPUx8, Mem 10GB DRM packaging vCPUx1, Mem 1GB Transcode 240p vCPUx2, Mem 2GB Transcode 480p vCPUx4, Mem 4GB Transcode Manager
  21. 21.
  22. 22. ! Video Metadata " Thumbnail ! MPEG2-TS MBR ! FMP4 MBR # DRM packaging 1080p 1080p 720p 720p 480p 480p 360p 360p 240p 240p 180p 180p MPEG2-TS MBR FMP4 MBR
  23. 23. ! Video Metadata " Thumbnail ! MPEG2-TS MBR ! FMP4 MBR # DRM packaging 1080p 1080p 720p 720p 480p 480p 360p 360p 240p 240p 180p 180p MPEG2-TS MBR FMP4 MBR
  24. 24. kind: Job spec: template: spec: containers: - name: transcoder image: abema-transcoder command: [“transcoder”, …] restartPolicy: OnFailure backOffLimit: 3
  25. 25. ! Video Metadata " Thumbnail ! MPEG2-TS MBR ! FMP4 MBR # DRM packaging 1080p 1080p 720p 720p 480p 480p 360p 360p 240p 240p 180p 180p MPEG2-TS MBR FMP4 MBR
  26. 26. kind: Job spec: template: spec: containers: - name: transcoder image: abema-transcoder command: [“transcoder”, …] parallelism: 3 completions: 3
  27. 27. Job MBR Manager Job 1080p Job 720p Job 480p Job 360p Job 240p Job 180p
  28. 28. import ( “k8s.io/client-go/kubernetes" "k8s.io/client-go/pkg/watch" ) wi, err := client.BatchV1().Jobs(..).Watch(..) defer wi.Stop() for { r, _ := <- wi.ResultChan() switch r.Type { case watch.Modified: // ... case watch.Deleted: // ... case watch.Error: // … } (k8s.io/client-go/kubernetes)
 Clientset.BatchV1().Jobs().Watch()
  29. 29. https://kubernetes.io/docs/concepts/workloads/controllers/ jobs-run-to-completion/#job-patterns A single Job to create a pod which then creates other pods, acting as a sort of custom controller for those pods. This allows the most flexibility, but may be somewhat complicated to get started with and offers less integration with Kubernetes. $ %
  30. 30. https://kubernetes.io/docs/concepts/workloads/controllers/ jobs-run-to-completion/#job-patterns A single Job to create a pod which then creates other pods, acting as a sort of custom controller for those pods. This allows the most flexibility, but may be somewhat complicated to get started with and offers less integration with Kubernetes. $ % 

  31. 31.
  32. 32. THANK YOU

×