In the session, we discussed the End-to-end working of Apache Airflow that mainly focused on "Why What and How" factors. It includes the DAG creation/implementation, Architecture, pros & cons. It also includes how the DAG is created for scheduling the Job and what all steps are required to create the DAG using python script & finally with the working demo.
Orchestrating workflows Apache Airflow on GCP & AWSDerrick Qin
Working in a cloud or on-premises environment, we all somehow move data from A to B on-demand or on schedule. It is essential to have a tool that can automate recurring workflows. This can be anything from an ETL(Extract, Transform, and Load) job for a regular analytics report all the way to automatically re-training a machine learning model.
In this talk, we will introduce Apache Airflow and how it can help orchestrate your workflows. We will cover key concepts, features, and use cases of Apache Airflow, as well as how you can enjoy Apache Airflow on GCP and AWS by demo-ing a few practical workflows.
Introduction to Apache Airflow, it's main concepts and features and an example of a DAG. Afterwards some lessons and best practices learned by from the 3 years I have been using Airflow to power workflows in production.
Orchestrating workflows Apache Airflow on GCP & AWSDerrick Qin
Working in a cloud or on-premises environment, we all somehow move data from A to B on-demand or on schedule. It is essential to have a tool that can automate recurring workflows. This can be anything from an ETL(Extract, Transform, and Load) job for a regular analytics report all the way to automatically re-training a machine learning model.
In this talk, we will introduce Apache Airflow and how it can help orchestrate your workflows. We will cover key concepts, features, and use cases of Apache Airflow, as well as how you can enjoy Apache Airflow on GCP and AWS by demo-ing a few practical workflows.
Introduction to Apache Airflow, it's main concepts and features and an example of a DAG. Afterwards some lessons and best practices learned by from the 3 years I have been using Airflow to power workflows in production.
Slide deck for the fourth data engineering lunch, presented by guest speaker Will Angel. It covered the topic of using Airflow for data engineering. Airflow is a scheduling tool for managing data pipelines.
Presentation given at Coolblue B.V. demonstrating Apache Airflow (incubating), what we learned from the underlying design principles and how an implementation of these principles reduce the amount of ETL effort. Why choose Airflow? Because it makes your engineering life easier, more people can contribute to how data flows through the organization, so that you can spend more time applying your brain to more difficult problems like Machine Learning, Deep Learning and higher level analysis.
In the session, we discussed the End-to-end working of Apache Airflow that mainly focused on "Why What and How" factors. It includes the DAG creation/implementation, Architecture, pros & cons. It also includes how the DAG is created for scheduling the Job and what all steps are required to create the DAG using python script & finally with the working demo.
A 20 minute talk about how WePay runs airflow. Discusses usage and operations. Also covers running Airflow in Google cloud.
Video of the talk is available here:
https://wepayinc.box.com/s/hf1chwmthuet29ux2a83f5quc8o5q18k
Building a Data Pipeline using Apache Airflow (on AWS / GCP)Yohei Onishi
This is the slide I presented at PyCon SG 2019. I talked about overview of Airflow and how we can use Airflow and the other data engineering services on AWS and GCP to build data pipelines.
Running Airflow Workflows as ETL Processes on Hadoopclairvoyantllc
While working with Hadoop, you'll eventually encounter the need to schedule and run workflows to perform various operations like ingesting data or performing ETL. There are a number of tools available to assist you with this type of requirement and one such tool that we at Clairvoyant have been looking to use is Apache Airflow. Apache Airflow is an Apache Incubator project that allows you to programmatically create workflows through a python script. This provides a flexible and effective way to design your workflows with little code and setup. In this talk, we will discuss Apache Airflow and how we at Clairvoyant have utilized it for ETL pipelines on Hadoop.
From AWS Data Pipeline to Airflow - managing data pipelines in Nielsen Market...Itai Yaffe
Tal Sharon (Software Architect), Aviel Buskila (DevOps Engineer) and Max Peres (Data Engineer) @ Nielsen:
At the Nielsen Marketing Cloud, we used to manage our data pipelines via AWS Data Pipeline. Over the years, we’ve encountered several issues with this tool, and a year ago we decided to embark on a journey to replace it with a tool more suitable for our needs.
In this session, we’ll discuss how we actually migrated to Airflow, what challenges we faced and how we mitigated them (and even contributed to the open-source project along the way). We’ll also provide some helpful tips for Airflow users
We will introduce Airflow, an Apache Project for scheduling and workflow orchestration. We will discuss use cases, applicability and how best to use Airflow, mainly in the context of building data engineering pipelines. We have been running Airflow in production for about 2 years, we will also go over some learnings, best practices and some tools we have built around it.
Speakers: Robert Sanders, Shekhar Vemuri
Finally available on SlideShare, Dan Goodinson's SAPinsider presentation: How to pinpoint and fix sources of performance problems in your SAP BusinessObjects BI reports and dashboards.
Slide deck for the fourth data engineering lunch, presented by guest speaker Will Angel. It covered the topic of using Airflow for data engineering. Airflow is a scheduling tool for managing data pipelines.
Presentation given at Coolblue B.V. demonstrating Apache Airflow (incubating), what we learned from the underlying design principles and how an implementation of these principles reduce the amount of ETL effort. Why choose Airflow? Because it makes your engineering life easier, more people can contribute to how data flows through the organization, so that you can spend more time applying your brain to more difficult problems like Machine Learning, Deep Learning and higher level analysis.
In the session, we discussed the End-to-end working of Apache Airflow that mainly focused on "Why What and How" factors. It includes the DAG creation/implementation, Architecture, pros & cons. It also includes how the DAG is created for scheduling the Job and what all steps are required to create the DAG using python script & finally with the working demo.
A 20 minute talk about how WePay runs airflow. Discusses usage and operations. Also covers running Airflow in Google cloud.
Video of the talk is available here:
https://wepayinc.box.com/s/hf1chwmthuet29ux2a83f5quc8o5q18k
Building a Data Pipeline using Apache Airflow (on AWS / GCP)Yohei Onishi
This is the slide I presented at PyCon SG 2019. I talked about overview of Airflow and how we can use Airflow and the other data engineering services on AWS and GCP to build data pipelines.
Running Airflow Workflows as ETL Processes on Hadoopclairvoyantllc
While working with Hadoop, you'll eventually encounter the need to schedule and run workflows to perform various operations like ingesting data or performing ETL. There are a number of tools available to assist you with this type of requirement and one such tool that we at Clairvoyant have been looking to use is Apache Airflow. Apache Airflow is an Apache Incubator project that allows you to programmatically create workflows through a python script. This provides a flexible and effective way to design your workflows with little code and setup. In this talk, we will discuss Apache Airflow and how we at Clairvoyant have utilized it for ETL pipelines on Hadoop.
From AWS Data Pipeline to Airflow - managing data pipelines in Nielsen Market...Itai Yaffe
Tal Sharon (Software Architect), Aviel Buskila (DevOps Engineer) and Max Peres (Data Engineer) @ Nielsen:
At the Nielsen Marketing Cloud, we used to manage our data pipelines via AWS Data Pipeline. Over the years, we’ve encountered several issues with this tool, and a year ago we decided to embark on a journey to replace it with a tool more suitable for our needs.
In this session, we’ll discuss how we actually migrated to Airflow, what challenges we faced and how we mitigated them (and even contributed to the open-source project along the way). We’ll also provide some helpful tips for Airflow users
We will introduce Airflow, an Apache Project for scheduling and workflow orchestration. We will discuss use cases, applicability and how best to use Airflow, mainly in the context of building data engineering pipelines. We have been running Airflow in production for about 2 years, we will also go over some learnings, best practices and some tools we have built around it.
Speakers: Robert Sanders, Shekhar Vemuri
Finally available on SlideShare, Dan Goodinson's SAPinsider presentation: How to pinpoint and fix sources of performance problems in your SAP BusinessObjects BI reports and dashboards.
www.magnifictraining.com - "sap basis" Online Training contact us:info@magnifictraining.com or+1-6786933994,+1-6786933475, +919052666559,+919052666558 By Real Time Experts from Hyderabad, Bangalore,India,USA,Canada,UK, Australia,South Africa,Sweden,Denmark.
Sap basis training demo basis online training in usa,uk and indiamagnificsmile
www.Magnifictraining.com-sap basis ONLINE TRAINING. contact us: info@magnifictraining.com
or call us: +919052666559,919052666558 sap technologies like sap basis,sap ewm,sap basis,sap basis ,sap basis,
sap hcm ,sap bi/bw, sap is banking,sap srm,
sap gts online training by industrail hands on training on sap basis online TRAINING.
SAP BASIS Online Training Course Contents :
What is SAP?
An introduction to ERP
An introduction to SAP
SAP AG: Evolution & Strategy
SAP Product Evolution
SAP Services Overview ( OSS )
An introduction to BASIS
Basics to startup with BASIS Administration
An introduction to Operating Systems
An introduction to Database Systems
Overview of computer Networks
Network types & devices
Protocol & IP Address concepts
An introduction to Kernel Software
Description of R/3
Client / Server Solution
Overview of 3 layer interface
Sap basis training demo basis online training in usa,uk and indiamagnifics
www.Magnifictraining.com-sap basis ONLINE TRAINING. contact us: info@magnifictraining.com
or call us: +919052666559,919052666558 sap technologies like sap basis,sap ewm,sap basis,sap basis ,sap basis,
sap hcm ,sap bi/bw, sap is banking,sap srm,
sap gts online training by industrail hands on training on sap basis online TRAINING.
SAP BASIS Online Training Course Contents :
What is SAP?
An introduction to ERP
An introduction to SAP
SAP AG: Evolution & Strategy
SAP Product Evolution
SAP Services Overview ( OSS )
An introduction to BASIS
Basics to startup with BASIS Administration
An introduction to Operating Systems
An introduction to Database Systems
Overview of computer Networks
Network types & devices
Protocol & IP Address concepts
An introduction to Kernel Software
Description of R/3
Client / Server Solution
Overview of 3 layer interface
Precima data scientist and architect discussing their data science and big data tech stack at the Toronto Data Science & Big Data meetup on Jan 30, 2019 hosted by WeCloudData https://weclouddata.com and sponsored by precima and loyaltyone.
Delivering Insights from 20M+ Smart Homes with 500M+ DevicesDatabricks
We started out processing big data using AWS S3, EMR clusters, and Athena to serve Analytics data extracts to Tableau BI.
However as our data and teams sizes increased, Avro schemas from source data evolved, and we attempted to serve analytics data through Web apps, we hit a number of limitations in the AWS EMR, Glue/Athena approach.
This is a story of how we scaled out our data processing and boosted team productivity to meet our current demand for insights from 20M+ Smart Homes and 500M+ devices across the globe, from numerous internal business teams and our 150+ CSP partners.
We will describe lessons learnt and best practices established as we enabled our teams with DataBricks autoscaling Job clusters and Notebooks and migrated our Avro/Parquet data to use MetaStore, SQL Endpoints and SQLA Console, while charting the path to the Delta lake…
How I learned to time travel, or, data pipelining and scheduling with AirflowLaura Lorenz
****UPDATE: Project is now open sourced at https://www.github.com/industrydive/fileflow****
From Pydata DC 2016
Description
Data warehousing and analytics projects can, like ours, start out small - and fragile. With an organically growing mess of scripts glued together and triggered by cron jobs hiding on different servers, we needed better plumbing. After perusing the data pipelining landscape, we landed on Airflow, an Apache incubating batch processing pipelining and scheduler tool from Airbnb.
Abstract
The power of any reporting tool breaks based on the data behind it, so when our data warehousing process got too big for its humble origins, we searched for something better. After testing out several options such as Drake, Pydoit, Luigi, AWS Data Pipeline, and Pinball, we landed on Airflow, an Apache incubating batch processing pipelining and scheduler tool originating from Airbnb, that provides the benefits of pipeline construction as directed acyclic graphs (DAGs), along with a scheduler that can handle alerting, retries, callbacks and more to make your pipeline robust. This talk will discuss the value of DAG based pipelines for data processing workflows, highlight useful features in all of the pipelining projects we tested, and dive into some of the specific challenges (like time travel) and successes (like time travel!) we’ve experienced using Airflow to productionize our data engineering tasks. By the end of this talk, you will learn
- pros and cons of several Python-based/Python-supporting data pipelining libraries
- the design paradigm behind Airflow, an Apache incubating data pipelining and scheduling service, and what it is good for
- some epic fails to avoid and some epic wins to emulate from our experience porting our data engineering tasks to a more robust system
- some quick-start tips for implementing Airflow at your organization.
Slides from Oracle's ADF Architecture TV series covering the Design phase of ADF projects, looking into advanced task flow concepts in ADF applications.
Like to know more? Check out:
- Subscribe to the YouTube channel - http://bit.ly/adftvsub
- Design Playlist - http://www.youtube.com/playlist?list=PLJz3HAsCPVaSemIjFk4lfokNynzp5Euet
- Read the episode index on the ADF Architecture Square - http://bit.ly/adfarchsquare
Using InfluxDB for real-time monitoring in JmeterKnoldus Inc.
Explore the integration of InfluxDB with JMeter for real-time performance monitoring. This session will cover setting up InfluxDB to capture JMeter metrics, configuring JMeter to send data to InfluxDB, and visualizing the results using Grafana. Learn how to leverage this powerful combination to gain real-time insights into your application's performance, enabling proactive issue detection and faster resolution.
Intoduction to KubeVela Presentation (DevOps)Knoldus Inc.
KubeVela is an open-source platform for modern application delivery and operation on Kubernetes. It is designed to simplify the deployment and management of applications in a Kubernetes environment. KubeVela is a modern software delivery platform that makes deploying and operating applications across today's hybrid, multi-cloud environments easier, faster and more reliable. KubeVela is infrastructure agnostic, programmable, yet most importantly, application-centric. It allows you to build powerful software, and deliver them anywhere!
Stakeholder Management (Project Management) PresentationKnoldus Inc.
A stakeholder is someone who has an interest in or who is affected by your project and its outcome. This may include both internal and external entities such as the members of the project team, project sponsors, executives, customers, suppliers, partners and the government. Stakeholder management is the process of managing the expectations and the requirements of these stakeholders.
Introduction To Kaniko (DevOps) PresentationKnoldus Inc.
Kaniko is an open-source tool developed by Google that enables building container images from a Dockerfile inside a Kubernetes cluster without requiring a Docker daemon. Kaniko executes each command in the Dockerfile in the user space using an executor image, which runs inside a container, such as a Kubernetes pod. This allows building container images in environments where the user doesn’t have root access, like a Kubernetes cluster.
Efficient Test Environments with Infrastructure as Code (IaC)Knoldus Inc.
In the rapidly evolving landscape of software development, the need for efficient and scalable test environments has become more critical than ever. This session, "Streamlining Development: Unlocking Efficiency through Infrastructure as Code (IaC) in Test Environments," is designed to provide an in-depth exploration of how leveraging IaC can revolutionize your testing processes and enhance overall development productivity.
Exploring Terramate DevOps (Presentation)Knoldus Inc.
Terramate is a code generator and orchestrator for Terraform that enhances Terraform's capabilities by adding features such as code generation, stacks, orchestration, change detection, globals, and more . It's primarily designed to help manage Terraform code at scale more efficiently . Terramate is particularly useful for managing multiple Terraform stacks, providing support for change detection and code generation 2. It allows you to create relationships between stacks to improve your understanding and control over your infrastructure . One of the key features of Terramate is its ability to detect changes at both the stack and module level. This capability allows you to identify which stacks and resources have been altered and selectively determine where you should execute commands.
Clean Code in Test Automation Differentiating Between the Good and the BadKnoldus Inc.
This session focuses on the principles of writing clean, maintainable, and efficient code in the context of test automation. The session will highlight the characteristics that distinguish good test automation code from bad, ultimately leading to more reliable and scalable testing frameworks.
Integrating AI Capabilities in Test AutomationKnoldus Inc.
Explore the integration of artificial intelligence in test automation. Understand how AI can enhance test planning, execution, and analysis, leading to more efficient and reliable testing processes. Explore the cutting-edge integration of Artificial Intelligence (AI) capabilities in Test Automation, a transformative approach shaping the future of software testing. This session will delve into practical applications, benefits, and considerations associated with infusing AI into test automation workflows.
State Management with NGXS in Angular.pptxKnoldus Inc.
NGXS is a state management pattern and library for Angular. NGXS acts as a single source of truth for your application's state - providing simple rules for predictable state mutations. In this session we will go through the main for components of NGXS -Store, Actions, State, and Select.
Authentication in Svelte using cookies.pptxKnoldus Inc.
Svelte streamlines authentication with cookies, offering a secure and seamless user experience. Effortlessly manage sessions by storing tokens in cookies, ensuring persistent logins. With Svelte's simplicity, implement robust authentication mechanisms, enhancing user security and interaction.
OAuth2 Implementation Presentation (Java)Knoldus Inc.
The OAuth 2.0 authorization framework is a protocol that allows a user to grant a third-party web site or application access to the user's protected resources, without necessarily revealing their long-term credentials or even their identity. It is commonly used in scenarios such as user authentication in web and mobile applications and enables a more secure and user-friendly authorization process.
Supply chain security with Kubeclarity.pptxKnoldus Inc.
Kube clarity is a comprehensive solution designed to enhance supply chain security within Kubernetes environments. Kube clarity enables organizations to identify and mitigate potential security threats throughout the software development and deployment process.
Mastering Web Scraping with JSoup Unlocking the Secrets of HTML ParsingKnoldus Inc.
In this session, we will delve into the world of web scraping with JSoup, an open-source Java library. Here we are going to learn how to parse HTML effectively, extract meaningful data, and navigate the Document Object Model (DOM) for powerful web scraping capabilities.
Akka gRPC Essentials A Hands-On IntroductionKnoldus Inc.
Dive into the fundamental aspects of Akka gRPC and learn to leverage its power in building compact and efficient distributed systems. This session aims to equip attendees with the essential skills and knowledge to leverage Akka and gRPC effectively in building robust, scalable, and distributed applications.
Entity Core with Core Microservices.pptxKnoldus Inc.
How Developers can use Entity framework(ORM) which provides a structured and consistent way for microservices to interact with their respective database, prompting independence, scaliblity and maintainiblity in a distributed system, and also provide a high-level abstraction for data access.
Introduction to Redis and its features.pptxKnoldus Inc.
Join us for an interactive session where we'll cover the fundamentals of Redis, practical use cases, and best practices for incorporating Redis into your projects. Whether you're a developer, architect, or system administrator, this session will equip you with the knowledge to harness the full potential of Redis for your applications. Get ready to elevate your understanding of in-memory data storage and revolutionize the way you handle data in your projects with Redis
GraphQL with .NET Core Microservices.pdfKnoldus Inc.
In this Webinar, will talk on GraphQL with .NET, that provides a modern and flexible approach to building APIs. It empowers developers to create efficient and tailored APIs that meet the specific needs of their applications and clients.
NuGet Packages Presentation (DoT NeT).pptxKnoldus Inc.
These packages and topics cover various aspects of .NET development, offering solutions for common needs in software development, including logging, database interaction, API communication, testing, security, and more. Depending on the requirements of your project, incorporating these packages can significantly enhance the development process.
Data Quality in Test Automation Navigating the Path to Reliable TestingKnoldus Inc.
Data Quality in Test Automation: Navigating the Path to Reliable Testing" delves into the crucial role of data quality within the realm of test automation. It explores strategies and methodologies for ensuring reliable testing outcomes by addressing challenges related to the accuracy, completeness, and consistency of test data. The discussion encompasses techniques for managing, validating, and optimizing data sets to enhance the effectiveness and efficiency of automated testing processes, ultimately fostering confidence in the reliability of software systems.
K8sGPTThe AI way to diagnose KubernetesKnoldus Inc.
K8sGPT is a tool for scanning your Kubernetes clusters, diagnosing and triaging issues in simple english. It has SRE experience codified into its analyzers and helps to pull out the most relevant information to enrich it with AI. In the session we will integrate OpenAI with K8sGPT to diagnose our Kubernetes Cluster.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
"Impact of front-end architecture on development cost", Viktor TurskyiFwdays
I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
3. c
Airflow is a platform to
programmatically author, schedule
and monitor workflows (a.k.a DAGs or
Directed Acyclic Graph)
What is Apache
Airflow?
4. Incubating
- Created @ Airbnb in 2015 by Maxime
Beauchemin
- 6100+ Forks
- 16k+ Github Stars
- 1k+ Contributors
- 150+ companies officially using it
- More than 8k committers
5. What is Airflow?
Pipelines are configured as
code, allowing for dynamic
pipeline generation
A platform to schedule
data pipelines
Easy to define our own
operators, executors
It’s all about DAGs
A platform to monitor and
control data pipelines
6. Why to use Airflow ?
When would you use a Workflow Scheduler like
Airflow?
- ETL Pipelines
- Machine Learning Projects
- Predictive Data Pipelines such as Fraud Detection etc.
- Job Scheduling
7. Why to use Airflow ?
What should a Workflow do well?
- Schedule your workflow plan
- Handle task failures
- Monitor Performance
8. Very Flexible
- Rich User Interface
- Easily Extensible
- Allow communication between task
- Backfill Control
- Efficient CLI
10. How the Job initiates?
Web Server
Scheduler
Worker Worker
Metadata
User
- A user manages/ schedules
DAGs using Airflow
- Airflow webserver stores
scheduling metadata in
metadata DB
- Scheduler picks up new
schedule and distributes work
over Celery or RabbitMQ
- Workers picks up Airflow Task
Note- Celery is used to manage
node and Redis or RabbitMQ for
communication
11. Pros
Easy-to-use user interface - Independent scheduler Active open-source
community
● can easily view task logs
● can easily view hierarchy,
statuses, and code runs
● allows you to easily
change task statuses,
rerun historical tasks
● comes with its own
scheduler
● supports multiple DAGs
● chatroom itself is very
active
● newbies can get their
questions answered within
a span of few hours.
12. Cons
Task optimization - No direct dealing with
- tasks
Lesser flexibility
● sometimes unclear of
how to organize multiple
tasks into the massive
pipeline
● doesn't deal with data sets
or files as inputs of tasks
directly
● If database is lost, harder
to restore
● Workers cannot start the
task independently or pick
tasks flexibly.