The document discusses an orchestration service built using Apache Airflow on Kubernetes. It provides the following key points: 1) The service aims to provide a standardized workflow management system for multiple teams at Adobe to author, manage, and schedule multi-step workflows in a scalable and multi-tenant manner. 2) Apache Airflow was selected as the underlying execution engine due to its active community, extensibility, rich UI, and support for distributed execution using Kubernetes. 3) The service abstracts Apache Airflow using a JSON DSL for authoring workflows and provides CRUD APIs for management to overcome challenges with Airflow like scheduling latency and lack of scalability. 4) The architecture supports

1. Orchestration Service with
Apache Airflow on Kubernetes
Presented By:- Raman Gupta
Sr Computer Scientist, ADOBE

2. Agenda:-
• Overview
• Orchestration Service: Requirements
• Why we selected Apache Airflow as underlying execution Engine
• Problems Faced with Apache Airflow
• Orchestration Service: Overcoming Challenges
• JSON DSL for workflow Authoring
• Architecture: Highly Scalable and Multi-Tenant Orchestration Service with Apache Airflow
on Kubernetes
 Orchestration Service Data Flow & Storage on Azure
 Running Apache Airflow on Kubernetes
 Orchestration Service support for multi-cluster.
• Key Learnings

3. Overview
• There are different teams in Adobe which need a workflow management system
to meet variety of use cases like Machine Learning, ETL, Scheduled Reporting etc..
• Each Team has set up its own scheduler like cron or workflow management
system. And they are facing similar kind of challenges while maintaining these
crons or workflow managers in production.
• There is no standardization and lot of duplication of work and effort in learning
and managing these workflow management systems.
• We started working n providing Orchestration Service as a fully managed &
Scalable, Multi-tenant service to help different Adobe teams to author / manage
& schedule multi step workflows.

4. Orchestration Service: Requirements
• Providing User Friendly APIs to author and Manage workflows lifecycle. Should
not expose underlying Workflow execution Engine.
• Should be multi-tenant and should provide resource guarantees
• Support for different workflow triggers like scheduled based trigger, ad-hoc
trigger, event based trigger
• Should be highly scalable. Should support 1000(s) of concurrent workflow
execution.
• Should manage different nature of workflows like workflow can run from few
minutes to few days.
• Scheduling latency should be within a threshold.
• It should support both static and dynamic workflows.

5. Why we selected Apache Airflow as underlying
execution Engine
• It has very active community.
• Its easily extensible through plugins.
• Very Rich UI to provide workflows Insights.
• Support for Distributed Executor like Kubernetes.
• Webserver can be extended support RESTful APIs for workflow
operations.

6. Problems Faced with Apache Airflow
• Workflow Scheduling Latency increases with increase in number of
workflow files.
• Airflow Releases are not backward compatible and might need
downtime.
• WorkflowTask is Memory intensive. Running single task can take
~150 MB of memory.
• Airflow scheduler is recommended to run as singleton so it can scale
vertically and not horizontally
• Kubernetes Executor is still evolving.

7. Orchestration Service: Overcoming Challenges
• Abstracts Apache Airflow by providing JSON based DSL to author workflow.
At the backend it transforms JSON to Airflow compatible DAG.
• Provides CRUD APIs to manage workflows
• It tracks Airflow Cluster Usage and provides necessary alerts
• To achieve high scalability it supports multiple Apache Airflow installation
and does all the routing.
• Support for custom Airflow config like “dag_dir_list_interval” for different
users.
• Support for Plugin interface to add and expose Airflow Operators as
reusable components.
• Contributions to Apache Airflow to make it more resilient and functionality
rich

8. As we have seen DAGS are python code comprising of airflow operators and various arguments of the DAGS. Adobe Orchestrator don't expect its
users to write a Python code for their DAGs, instead it provide python generation of DAGS using JSON where they can provide all the properties
and operators of the DAGs.
JSON DSL for workflow Authoring
JSON
validator
Parser and
Python code
generator
passes
Property Type
Id String
Description String
Type properties JSON
Connections List
Retries Number
dependsOn JSON
Using JSON properties
REST API
Create DAG
using JSON
Properties
for
JSON(Table
)
DAG
validator
Checks all
imports,
cycles,
code
syntax of
the
generated
Python
code
Adobe Orchestrator is like a service environment over Airflow which is used by the user. If any issue occurs then users does not get affected by that
and the service takes care of that. e.g. Scheduler goes down if there is any database connection issue, for which Orchestrator provides the AUTO-
RESTART which restarts the scheduler

9. Service Data Flow &
Storage

10. Running Apache Airflow on Kubernetes

11. Orchestration Service support for Multi-Cluster

12. Key Learnings
• Apache Airflow is not exposed to users so its possible to support different
or multiple Scheduling engine without Impacting usage.
• With JSON based DSL users can construct workflow without knowing
Airflow Primitive(s).
• HTTP interface between Orchestration Service & Airflow Webserver helped
in keeping separation of logic.
• Airflow metastore can become a bottleneck so provided a cleanup Api.
• Monitoring tools to check Airflow scheduler Health and restart it if
required.
• Monitoring & Alerts on number of Workflow files on Airflow Scheduler
otherwise scheduling latency can increase.

13. Thank You !!