In Data Engineer's Lunch #47, we will use Kubernetes to deploy airflow Accompanying Blog: https://blog.anant.us/data-engineers-lunch-47-airflow-on-kubernetes/ Accompanying YouTube: https://youtu.be/jaDykaEFops Sign Up For Our Newsletter: http://eepurl.com/grdMkn Join Data Engineer’s Lunch Weekly at 12 PM EST Every Monday: https://www.meetup.com/Data-Wranglers-DC/events/ Cassandra.Link: https://cassandra.link/ Follow Us and Reach Us At: Anant: https://www.anant.us/ Awesome Cassandra: https://github.com/Anant/awesome-cassandra Email: solutions@anant.us LinkedIn: https://www.linkedin.com/company/anant/ Twitter: https://twitter.com/anantcorp Eventbrite: https://www.eventbrite.com/o/anant-1072927283 Facebook: https://www.facebook.com/AnantCorp/ Join The Anant Team: https://www.careers.anant.us

1. Airflow on Kubernetes
An Anant Corporation Story.
Using kubernetes to deploy airflow

2. What Is Airflow?
Apache Airflow is one realization of the
DevOps philosophy of "Configuration As
Code." Airflow allows users to launch multi-
step pipelines using a simple Python object
DAG (Directed Acyclic Graph). You can
define dependencies, programmatically
construct complex workflows, and monitor
scheduled jobs in an easy to read UI.

3. Why Airflow on Kubernetes?
Since its inception, Airflow's greatest strength has been its flexibility. Airflow offers a wide range of integrations for services
ranging from Spark and HBase, to services on various cloud providers. Airflow also offers easy extensibility through its plug-
in framework. However, one limitation of the project is that Airflow users are confined to the frameworks and clients that
exist on the Airflow worker at the moment of execution. A single organization can have varied Airflow workflows ranging
from data science pipelines to application deployments. This difference in use-case creates issues in dependency
management as both teams might use vastly different libraries for their workflows.
To address this issue, we've utilized Kubernetes to allow users to launch arbitrary Kubernetes pods and configurations.
Airflow users can now have full power over their run-time environments, resources, and secrets, basically turning Airflow into
an "any job you want" workflow orchestrator.

4. The Kubernetes Operator
Before we move any further, we should clarify that an Operator in Airflow is a task definition. When a
user creates a DAG, they would use an operator like the "SparkSubmitOperator" or the
"PythonOperator" to submit/monitor a Spark job or a Python function respectively. Airflow comes with
built-in operators for frameworks like Apache Spark, BigQuery, Hive, and EMR. It also offers a Plugins
entrypoint that allows DevOps engineers to develop their own connectors.
Airflow users are always looking for
ways to make deployments and
ETL pipelines simpler to manage.
Any opportunity to decouple
pipeline steps, while increasing
monitoring, can reduce future
outages and fire-fights. The
following is a list of benefits
provided by the Airflow Kubernetes
Operator:
● Increased flexibility for deployments
● Flexibility of configurations and dependencies
● Usage of kubernetes secrets for added
security

5. Architecture
The Kubernetes Operator uses the
Kubernetes Python Client to generate a
request that is processed by the
APIServer (1). Kubernetes will then
launch your pod with whatever specs
you've defined (2). Images will be loaded
with all the necessary environment
variables, secrets and dependencies,
enacting a single command. Once the job
is launched, the operator only needs to
monitor the health of track logs (3). Users
will have the choice of gathering logs
locally to the scheduler or to any
distributed logging service currently in
their Kubernetes cluster.

6. Demo