GKE, Helm, Terraform and Airflow Integration
•Download as PPTX, PDF•
0 likes•8 views
In this talk would be sharing the knowledge of how Airflow is hosted in the Gitlab and how Pod scheduling have been leveraged at scale. Also would be talking about how we have leveraged Terraform for GKE setup and used helm for airflow upgrade.
Recommended
Recommended
More Related Content
Similar to GKE, Helm, Terraform and Airflow Integration
Similar to GKE, Helm, Terraform and Airflow Integration (20)
More from DataScienceConferenc1
More from DataScienceConferenc1 (20)
Recently uploaded
Recently uploaded (20)
GKE, Helm, Terraform and Airflow Integration
- 1. © 2023 GitLab Inc. Ved Prakash Staff Data Engineer GitLab Intro to GKE setup of Airflow with helm and Terraform
- 2. © 2023 GitLab Inc. GitLab company Our mission: Everyone Can Contribute
- 3. © 2023 GitLab Inc. GitLab CREDIT values Collaboration Results Iteration Transparency Efficiency inclusion & belonging Diversity,
- 4. © 2023 GitLab Inc. Plan Create Verify Package Secure Deploy Monitor Govern What GitLab is doing? The DevSecOps Platform delivered as a single application to help you iterate faster and innovate together
- 5. © 2023 GitLab Inc. Introduction to Google Kubernetes Engine(GKE),Apache Airflow,Helm and Terraform. How all Components fit together. Agenda Best Practices and Considerations Wins of Data platform Team. Installation within Gitlab Data platform.
- 6. © 2023 GitLab Inc. Introduction to Google Kubernetes Engine (GKE) Google Kubernetes Engine (GKE) is a managed Kubernetes service that simplifies containerized application deployment, scaling, and management on Google Cloud, offering a robust and efficient platform for container orchestration. What is GKE? Key Features ● Managed Kubernetes ● Automatic Scaling ● Security and Compliance ● Integrations Use Cases: ● Microservices Deployment ● Continuous Integration/Continuous Deployment (CI/CD) ● Scalable Applications
- 7. © 2023 GitLab Inc. Overview of Apache Airflow Apache Airflow is an open-source platform designed to programmatically author, schedule, and monitor workflows. What is Apache Airflow? Key Features ● Directed Acyclic Graphs (DAGs) ● Extensibility ● Dynamic Workflow Execution ● Rich UI and Logging ● Scalability Use Cases: ● Data pipeline orchestration ● ETL (Extract, Transform, Load) processes ● Workflow automation in diverse industries
- 8. © 2023 GitLab Inc. Terraform: Infrastructure as Code for GKE Terraform is an open-source Infrastructure as Code (IaC) tool that enables users to define and provision infrastructure using a declarative configuration language. What is Terraform? Key Features ● Infrastructure as Code (IaC) ● Multi-Cloud Provisioning ● Declarative Configuration Language ● Plan and Apply Workflow ● State Management Use Cases: ● Provisioning Servers ● Network Infrastructure ● Application Deployments
- 9. © 2023 GitLab Inc. Helm: A Package Manager for Kubernetes Helm is a package manager for Kubernetes applications, simplifying the deployment and management of containerized applications. What is Helm? Key Features ● Standardized Packaging ● Simplified Configuration ● Version Control and Rollbacks ● Dependency Management Use Case ● Microservices Deployment
- 10. © 2023 GitLab Inc. Integration of Airflow with GKE using Helm and Terraform Why Use GKE for Airflow? Why Helm Charts for Airflow? Terraform Modules for GKE
- 11. © 2023 GitLab Inc. Infrastructure Provisioning with Terraform Connecting the Dots: GKE + Helm + Terraform + Airflow Terraform Sets the Foundation Kubernetes Cluster Orchestration with GKE GKE Ensures Seamless Operations Package Management with Helm Helm Charts Define Airflow Configurations: Helm Charts Deployed on GKE Smooth Deployment on GKE Integrated Workflows Orchestration with Airflow
- 12. © 2023 GitLab Inc. GKE cluster provisioned through Terraform Installation within Gitlab Data Platform Team ● Two Namespace (Prod and testing) ● Seven Nodepools (Different Machine type for each type load) ● Remote state file for any changes required for GKE cluster. ● Gitlab CI/CD pipeline to validate the Changes done to terraform script.(This ensure the changes will not break the terraform apply) Airflow Installed using helm chart ● Airflow Version 2.5.3 using Helm Chart for Apache Airflow which will bootstrap an Airflow deployment on a Kubernetes cluster using the Helm package manager. ● Overridden with Cloud SQL Postgres instance. ● Git sync with the analytics repository. ● Modified Web server secret key and fernet key
- 13. © 2023 GitLab Inc. ● 88 Active Airflow Dags ● 1200+ Task run every 24 hours ● Empowering Workflows:Task Dynamism with Airflow ● K8s Pods operator to schedule dynamic workload ● Cost effective solution ● On-Demand Node Provisioning with Terraform ● Minimal downtime , typically under 45 min, in the event of a disaster recovery scenario. How this benefits the data platform team managing data pipeline?
- 14. © 2023 GitLab Inc. Security Best Practices ● Private Cluster Configuration ● VPC Peering ● Identity and Access Management (IAM) Controls ● Node Pool Isolation ● Securing Secrets Best Practices and Considerations Scalability Considerations ● Horizontal Pod Autoscaling (HPA) ● Database Scaling ● Task Parallelism ● Resource Requests and Limits ● Persistent Storage Considerations ● GKE Node Pools Monitoring and Logging Strategies ● Leverage K8s-native monitoring solutions Prometheus and Grafana ● Alerting and Notification Channels. ● Airflow Metrics.
- 15. © 2023 GitLab Inc. About me Find me, ping me, ask me
- 16. © 2023 GitLab Inc. Additional Resources ● Gitlab Handbook for information about nodepool and namespace. ● Airflow Infrastructure ● Gitlab Data Analytics or our Dag bags
- 17. © 2023 GitLab Inc. Thank you
Editor's Notes
- Introduction 1
- Introduction 4
- And Off course Q&A
- Key Features: Managed Kubernetes: Leverage the power of Kubernetes without the operational overhead. Automatic Scaling: Seamlessly scale your applications with automated load balancing. Security and Compliance: Built-in security features and compliance standards for peace of mind. Integrated Developer Tools: Tight integration with Google Cloud's developer tools and services. Benefits: Efficiency: Simplifies container orchestration, enabling efficient deployment and scaling. Reliability: Google's infrastructure ensures high availability and reliability. Flexibility: Run containerized applications anywhere, on-premises or in the cloud. Use Cases: Microservices Deployment: Ideal for deploying and managing microservices architectures. Continuous Integration/Continuous Deployment (CI/CD): Streamlines CI/CD pipelines with Kubernetes. Scalable Applications: Easily scale applications based on demand. At gitlab within data platform team it is being used for Scalable airflow , Gitlab CI/CD pipeline for our analytics repo.
- Key Features: Directed Acyclic Graphs (DAGs): Represent workflows as code, defining the sequence and dependencies of tasks. Extensibility: Easily extend functionality with custom operators, sensors, and hooks. Dynamic Workflow Execution: Dynamically generate workflows based on external parameters. Rich UI and Logging: User-friendly interface for monitoring, logging, and visualizing workflow runs. Scalability: Scales horizontally to handle large-scale data processing and orchestration. Apache Airflow empowers organizations to streamline complex data workflows with flexibility and reliability.
- Few Key features for Terraform Infrastructure as Code (IaC): Terraform allows users to define and manage infrastructure using a declarative configuration language, enabling version control, collaboration, and the ability to treat infrastructure as code. Multi-Cloud Provisioning: Terraform supports various cloud providers (AWS, Azure, Google Cloud, etc.) and on-premises environments, providing a consistent approach to provisioning and managing infrastructure across different platforms. Declarative Configuration Language: The HashiCorp Configuration Language (HCL) used by Terraform is designed for readability and ease of use, making it straightforward to express infrastructure configurations. Plan and Apply Workflow: Terraform follows a workflow of planning and applying changes. The terraform plan command previews the changes before execution, and terraform apply implements the changes, ensuring safety and control over infrastructure modifications. State Management: Terraform maintains a state file that records the current state of the infrastructure. This state allows Terraform to determine what changes are necessary and provides a basis for understanding the existing infrastructure. Use Cases: Provisioning Servers: Create and manage virtual machines or containers. Network Infrastructure: Define and configure networks, subnets, and security groups. Application Deployments: Deploy and manage applications and their dependencies.
- What is Helm? Helm is a package manager for Kubernetes applications, simplifying the deployment and management of containerized applications. Key Concepts: Charts: Helm packages are called charts, which encapsulate all the resources needed for an application—services, deployments, and more. Values: Parameterized configurations allow customization of charts for different environments. Repositories: Share and discover charts through Helm repositories, fostering a vibrant ecosystem. Benefits of Helm: Reusability: Easily share and reuse application configurations across teams and projects. Versioning: Charts can be versioned, enabling precise control over application deployments. Templating: Helm uses Go templating to generate Kubernetes manifests dynamically. Workflow: helm install: Deploy a chart to a Kubernetes cluster with a single command. helm upgrade: Seamlessly update a deployed application with new configurations or versions. helm rollback: Roll back to a previous version of an application in case of issues. Community and Adoption: Helm has a thriving community and is widely adopted in the Kubernetes ecosystem. Many popular applications and services provide Helm charts for easy integration. Conclusion: Helm simplifies Kubernetes application deployment and management, offering a standardized and efficient way to package, version, and share applications in the Kubernetes environment. Use Case: Microservices Deployment: Step 1: Chart Creation: Package each microservice with its associated Kubernetes resources (Deployments, Services, ConfigMaps) into a Helm chart. Step 2: Chart Sharing: Share Helm charts across your development team or with the broader community via Helm Hub. Step 3: Consistent Deployments: Developers can use the same Helm chart to deploy the microservice consistently across different environments. Step 4: Versioning: Version your Helm charts to track changes, ensuring consistency and repeatability in deployments.
- Integration of Airflow with GKE using Helm and Terraform Why Use GKE for Airflow? Lots of advantages and reason but to summary we can call . Managed Kubernetes Service: Effortless Orchestration: GKE provides a fully managed Kubernetes service, eliminating the operational burden of setting up and maintaining Kubernetes clusters. This allows users to focus more on Airflow configurations and workflows. 2. Scalability: Dynamic Scaling: GKE allows for easy horizontal scaling, enabling Airflow to adapt to varying workloads by dynamically adjusting the number of pods based on demand. This ensures optimal resource utilization. 3. Automated Operations: Built-in Automation: GKE automates routine operational tasks like patching, updates, and cluster scaling. This reduces manual intervention and ensures that the Airflow environment is consistently up-to-date and secure. 4. Integrated Developer Tools: Seamless Integration: GKE integrates seamlessly with other Google Cloud services and developer tools. This includes integration with Cloud Monitoring, Logging, and Identity and Access Management (IAM), enhancing the overall management experience. 5. Google Cloud Ecosystem: Interoperability: Leveraging GKE within the broader Google Cloud ecosystem provides opportunities for integration with various services such as BigQuery, Cloud Storage, and Pub/Sub, enhancing the capabilities and data processing options for Airflow workflows. 6. High Availability and Reliability: Built-in Redundancy: GKE ensures high availability and reliability through multi-zone deployments, distributing Airflow components across multiple availability zones to mitigate the risk of single points of failure. Cost Efficiency: Pay-as-You-Go Model: GKE operates on a pay-as-you-go pricing model, providing cost efficiency by dynamically scaling resources based on demand. Users only pay for the resources consumed during active workflows. Helm Charts for Airflow. Due to below reason Standardized Packaging: Consistent Deployment: Helm Charts provide a standardized way to package, version, and deploy applications. Using Helm for Airflow ensures consistency across different environments, making it easier to reproduce deployments. 2. Simplified Configuration: Templating Engine: Helm uses Go templating to parameterize Kubernetes manifests. This allows users to customize Airflow configurations easily, adapting them to specific deployment scenarios without manual editing of YAML files. 3. Version Control and Rollbacks: Built-in Versioning: Helm Charts support versioning, allowing users to roll back to a previous state in case of issues. This ensures that changes to the Airflow deployment can be tracked, managed, and reverted when necessary. 4. Reusability: Shareable Configurations: Helm Charts can be shared and reused across teams and projects. This promotes collaboration and standardizes the deployment process, as the same Helm Chart can be used across different Airflow instances. In conclusion, Helm Charts offer a robust and flexible solution for deploying Apache Airflow by providing a standardized packaging format, streamlined configuration management, and a vibrant community ecosystem. The use of Helm simplifies the deployment and management of Airflow in Kubernetes environments. Terraform Modules for GKE Using Terraform modules for GKE provides a structured, reusable, and scalable approach to managing Kubernetes clusters, promoting consistency and best practices across your infrastructure deployments.
- Terraform Sets the Foundation: Initiate the process by using Terraform to provision a robust GKE cluster. Define infrastructure as code to establish the underlying Kubernetes environment for Apache Airflow. GKE Ensures Seamless Operations: Google Kubernetes Engine manages the Kubernetes cluster, providing automated operations, scalability, and integration with Google Cloud services. The GKE cluster becomes the orchestration backbone for deploying and managing applications. The synergy of GKE, Helm, Terraform, and Airflow provides a comprehensive solution for deploying, managing, and orchestrating data workflows in a cloud-native environment. This integrated approach combines infrastructure provisioning, application deployment, and workflow orchestration, offering a scalable, efficient, and maintainable solution for complex data processing scenarios.
- The helm chart of airflow creates 4 pods in the cluster for managing airflow, below: airflow-scheduler airflow-webserver airflow-pgbouncer: supplemental DB component which provides additional DB security and connection management. airflow-statsd: enables reading and monitoring of airflow metrics in prometheus (still to be implemented) The scheduler, webserver, and any workers created also include cloud-sql-proxy side car container which connects the containers to the external DB using service account credentials. Additionally, the scheduler and webserver also include: git-sync side car container which updates the DAGs repo with any changes detected in the repository. The install also requires an external postgres DB, which needs to be created manually.
- Private Cluster Configuration: Recommendation: Deploy GKE clusters as private clusters to limit exposure to the public internet. Rationale: Private clusters minimize the attack surface by restricting external access to the cluster. VPC Peering or VPN Setup: Recommendation: Establish VPC peering or set up a VPN connection between GKE clusters and other relevant networks. Rationale: Securely connect GKE clusters to other resources while maintaining network isolation and encryption. Identity and Access Management (IAM) Controls: Recommendation: Implement the principle of least privilege by assigning minimal necessary permissions to service accounts and users. Rationale: Reducing unnecessary access minimizes the risk of unauthorized actions. Node Pool Isolation: Recommendation: Utilize separate node pools for Airflow components and user workloads. Rationale: Isolating node pools ensures that Airflow components run independently from user applications, enhancing security and resource management. Securing Secrets: Recommendation: Utilize Kubernetes Secrets or external secret management tools for storing sensitive information such as database credentials and API keys. Rationale: Protecting secrets is crucial for preventing unauthorized access to critical resources. Adhering to these security best practices helps fortify your Apache Airflow installation on Google Kubernetes Engine, fostering a secure and resilient orchestration environment. Scalability Considerations When running Apache Airflow on Google Kubernetes Engine (GKE), several scalability considerations should be taken into account to ensure optimal performance and resource utilization: Horizontal Pod Autoscaling (HPA): Utilize Kubernetes Horizontal Pod Autoscaling to automatically adjust the number of Airflow worker pods based on CPU or memory utilization. This ensures that resources are allocated efficiently to meet the demands of running workflows. Database Scaling: Consider the scalability of the database backend used by Airflow (e.g., PostgreSQL). Ensure that the database is appropriately provisioned and tuned to handle the increasing metadata storage requirements as the number of tasks and workflows grows. Task Parallelism: Design Airflow DAGs with parallelism in mind. Break down workflows into smaller tasks to enable better parallel execution, taking advantage of the scalability features in GKE. Resource Requests and Limits: Set appropriate resource requests and limits for Airflow pods to ensure they receive the necessary resources and prevent resource contention within the cluster. GKE Node Pools: Utilize GKE node pools to segregate workloads with varying resource requirements. This allows for better resource isolation and scaling based on specific task characteristics. By addressing these scalability considerations, you can create a robust and scalable Apache Airflow deployment on GKE, ensuring efficient utilization of resources and accommodating the evolving demands of your data workflows. Monitoring and Logging Strategies When setting up Apache Airflow on Google Kubernetes Engine (GKE) using Terraform, it's crucial to establish effective monitoring and logging strategies to ensure the stability, performance, and security of your deployment. Here are key considerations for monitoring and logging: 1. Kubernetes Monitoring: Leverage Kubernetes-native monitoring solutions like Prometheus and Grafana. Set up Prometheus to collect metrics from the Airflow pods and use Grafana dashboards for visualization. 2. Airflow Metrics:Enable Airflow's built-in metrics exporter to expose key performance metrics. This includes metrics related to DAG execution, task durations, and scheduler performance. 3. Alerting and Notification Channels:- Configure alerting channels such as email, Slack, or PagerDuty to receive notifications when predefined thresholds are breached. Ensure timely responses to critical issues. By incorporating these monitoring and logging strategies into your Airflow deployment on GKE with Terraform, you can create a robust observability framework, allowing for proactive issue detection, efficient debugging, and continuous improvement of your orchestration environment.
- Now the favorite part of Q&A/ For any additional questions or info needed, looking forward to hearing from you. Do not hesitate to contact me with any questions.