This document provides an overview of Apache Airflow, an open-source workflow management platform. It describes Airflow as a tool for scheduling and running jobs and data pipelines, ensuring correct ordering based on dependencies and recovering from failures. The key benefits of Airflow are that it is easy to use with Python knowledge, open source, supports many platforms and systems through integrations, uses Python flexibly for workflows, and enables visualization of workflows. The document outlines Airflow's architecture, core concepts including DAGs (directed acyclic graphs), tasks, and operators, and how to create a workflow by defining a DAG as a Python file with tasks and their dependencies and order.

1. Introduction
Airflow – an Automation Tool
Metfone 2022

2. AGENDA
What is Airflow
Benefit of Airflow
Airflow Structure
Core Concepts
How to create workflow
Demo

3. WHAT IS AIRFLOW
Airflow is the workflow orchestration tools:
 Manage scheduling and running jobs and data pipelines
 Ensure jobs are ordered correctly based on dependencies
 Provides mechanisms for tracking and monitoring state of jobs and recovering from
failure

4. BENEFITS OF AIRFLOW
 Easy to Use: User need a bit of Python knowledge to use.
 Open Source: Free and open source with a lot of contributing
 Robust Integrations: Ready to use with various platforms and systems: BashShell, SFTP,
MySQL, Posgres, ORACLE, Python …
 Standard Python: User can use Python to create workflow very flexibilities.
 Visualization: Can monitor and manage workflows by web interface.

5. AIRFLOW ARCHITECTURE

6. CORE CONCEPTS
DAG: Directed Acyclic Graph - A DAG is a series of tasks that you want to run as part of
your workflow. This might include something like execute bashshell, performing some
checklist by Python script or Database script ... In Airflow each of these steps would be
written as individual tasks in a DAG.
Airflow enables you to also specify the relationship between the tasks, any dependencies
(e.g. data having loaded in a table before a task is run) and the order in which the tasks
should be run.

7. CORE CONCEPTS
TASK: represent each node of a defined DAG. They are visual representations of the work
being done at each step of the workflow, with the actual work that they represent being
defined by operators

8. CORE CONCEPTS
Operators: An operator encapsulates the operation to be performed in each task in a DAG.
Airflow has a wide range of built-in operators that can perform specific tasks some of which are
platform-specific. Additionally, it is possible to create your own custom operators.

9. HOW TO CREATE WORKFLOW
DAG will be saved as a .py file in the dags directory. The steps to create a dag:
Define DAG
•Define dag_id, start_date
and how often the tasks
should be run.
•Airflow uses a CRON
expression to define the
schedule
CREATE TASKS
•Airflow provides a range
of operators to perform
most functions:
BashShell, Database
(Oracle, Postgres, MySQL
…), Python, FTP … to
create tasks.
ORDER TASKS
•Define dependency and
order to perform tasks

10. References:
https://airflow.apache.org/docs/apache-airflow/stable/tutorial.html
https://www.youtube.com/watch?v=CLkzXrjrFKg

11. DEMO
http://10.79.9.131:8080/home

12. THANKS