In this presentation, we will share one of the platforms we have built to support video processing based on Airflow. Through this platform, we can measure the utilization of resources, check the status of batch tasks, quantify and analyze the efficiency of each pipeline and thus adjust the workflow flexibly at run time and in an intuitive way.

1. Airflow based Video Encoding Platform
Hotstar

2. About Hotstar
Hotstar is an online video streaming platform wholly owned by Fox (Disney).
We offers over 120,000 hours of TV content and movies. In this July we
created a record with 100 Million DAU and 25.3 concurrent viewers.
1. Huge amount of videos and heavy encoding service.
2. Various genres and studios, needs ML to classify & optimize.
3. Long-tail distribution, head contents need ultimate encoding efficiency.

3. Our Challenges
How to manage and visualize vast encoding jobs？
How to optimize pod scheduling with limited resources？
How to get the statistics of all tasks?
How to create tasks dynamically？

4. Our Platform
Kubernetes
…
AWS S3
VOD-API
Ceph
Airflow
PostgreSQL
CMS
KubernetePod
KubernetePod
KubernetePod

5. Basic Concepts
• DAG (Dag run)
• Operator（Task）
Bash, Docker, SSH, KubernetesPod...

6. How manage global variables？
Airflow Variable
1. Key-Value format
2. Can be configured in Web UI，can
also be modified in DAG
3. Can save token and pass it into DAG,
avoid hardcoding passwords in codes
4. Can be used to set environment
specific variables.

7. How to exchange info between operators？
XCOM
Xcom: “cross-communication”

8. How to control Kubernetes?
KubernetesPodOperator
Support most KubernetesPod operations. Can limit pod resource allocation
using resources param.
Using the KubernetesPodOperator （Google）
Resources
request & limit
Command to run in Pod
Docker image to use

9. Build an Airflow encoding work flow
Download
source and
metadata
Encode and
package
CMS Deploy
Verification and
generate
configuration
Scene analysis
and choose
proper params
Backup and
cleanup
Create
thumbnails
QA
More Codecs
Skip Intro
Check and
upload
⌛️

10. Build an Airflow encoding work flow

11. Deployment
CI/CD
Build
Docker
Airflow Kubernetes
CI/CD
Update
DAG File
Encoder，Analysis SDK，
Operator scripts

12. How to get the statistics of all tasks?
Gantt & Task Duration View

13. How to optimize pod scheduling
with limited resources？
Time Consuming
Statistics
Task Duration view
Dependency
Analysis
Gantt View
Estimate proper
CPU and mem
limits
Task Level
Resource Map
A1CPU + 2GB
C3CPU + 3GB
D5CPU + 4GB
E1CPU + 2GB
B1CPU + 1GB

14. How to create tasks dynamically？
Operator Placeholder+BranchOperator
Solution
Create redundant operators at buildtime，and place a BranchOperator before them.
At runtime use BranchOperator to control how many tasks to launch，and set
remaining tasks to skip state.
DAG topology is
built before
workflow start and
can not be
changed during
runtime.
Different contents
have different
output layers. So
we need to
launch different
num of tasks at
runtime.

15. Traps
1. When Dag file is updated, all old (completed) Dags will be displayed in
latest graph too (this is incorrect).
2. DAG file is executed before DagRun workflow start，only commands /
scripts in Operator works at Runtime.

16. Traps
3. Scheduler will be quite slow when the amount of DAG Run is large.
4. Developing KubernetesPodOperator Dags takes much more time than
writing python/shell scripts.

17. Summary
1. Decoupling development work and operation work
2. Communities are maturing
3. Flexible and easy to customize
4. Powerful analysis tool

18. Quick start
1. Install Docker Desktop and enable Kubernetes
2. brew install kubernetes-helm
3. helm install --namespace "airflow" --name "airflow" stable/airflow
4. export POD_NAME=$(kubectl get pods --namespace airflow -l
"component=web,app=airflow" -o jsonpath="{.items[0].metadata.name}")
echo http://127.0.0.1:8080
kubectl port-forward --namespace airflow $POD_NAME 8080:8080
5. Open http://127.0.0.1:8080

19. Future Improvements
1. Only one scheduler allowed in airflow
2. Batch analysis (Task Duration、Tree View) is inefficient
3. Xcom、Log fulltext search
4. Airflow scheduling of workers across different hardware platforms

20. Thank you