Cloud Infrastructure automation with Python-3.pdf

•

1 like•57 views

This document discusses automating cloud infrastructure using Python. It provides examples of automating tasks like OpenVPN user management and Gitlab access management. It then describes a case study of automating VPN access requests using a Slack bot and Python. Key aspects of the system are utilizing the Slack and Gitlab APIs, Python libraries, Gitlab CI for testing, and deploying the Slack bot and services on Kubernetes. The automation helps reduce the time and effort for non-technical users to request access and improves security and cost management. Testing is done manually through smoke testing and tracking scenarios in a spreadsheet.

Technology

CLOUD INFRASTRUCTURE
AUTOMATION WITH
PYTHON
By Ridwan Fadjar Septian
Cloud Infrastructure Engineer at DKatalis

MINIMAL FOOTPRINT
Minimal footprint
A lot of libraries
Multiparadigm
Open source
Community supports
Cloud provider
supports

CLOUD INFRASTRUCTURE
AUTOMATION
EXAMPLES

OpenVPN User Management Pritunl VPN User Management
Gitlab Repo Access Management Gitlab Projects Audit
Ansible Module for Sentry API Gitlab Repo Access Cleanup (cronjob)
VPN Access Cleanup OpsGenie On-Call Analysis
MORE WITH EXAMPLES

SAMPLE CASE:
VPN REQUESTS WITH
SLACK BOT AND PYTHON

MINIMAL FOOTPRINT
Slack Bot API
Gitlab API
Standard Python
Library
Python3
Gitlab CI
Requests
STACKS

PAIN POINTS
Cloud infra team spend too
much time to create MR for
non-tech. It could take
around 15 minutes. Just
multiply it with the num of
requests
Techies person spend time
for creating MR to the repo
Increased of Gitlab users
caused by Non-Techies
that should access the vpn
repo.
More unnecessary
subscription cost caused
by Non-Techies access
Non-Techies person should
learn Git and Gitlab just to
make MR to the repo
MR sometimes fail due to
YAML format. Non-
standardized changes

Peer review,
Flake8, Hadolint,
Bandit, Build-
and-push Docker
image
GITLAB CI
host Docker
Image generated
by Gitlab CI
DOCKER
REGISTRY
standardized
Kubernetes
manifest
HELM
CHARTS
Synchronize
between Helm
Charts spec and
actual
deployment in
Kubernetes
ARGO CD
Host the Slack
Bot as
Deployment
KUBERNETES
DEPLOYMENT

HOW WE TEST IT (AS NON-QA GUY)?
Functional testing (manual) after major release
Smoke testing (manual) after minor release
Spreadsheet
Current fields
No
Scenarios
Parameters
Expected Results
Actual Results
Status
Artifacts

AFTER SLACKBOT IMPLEMENTATION
Cloud infra team rarely
spend time to create MR for
non-tech
Techies person use
SlackBot for submit the
request
Reduced of Gitlab users
caused by Non-Techies
that should access the vpn
repo.
Reduced Gitlab
subscription cost caused
by Non-Techies access
Non-Techies don't need to
learn Git and Gitlab
anymore. Just use SlackBot
Rarely fail due to YAML
format. Standardized
changes

Rundeck Ansible Tower
Ansible Terraform
Github Action
OTHER TOOLS
Jenkins

REFERENCES
More detail check here -> https://medium.com/dkatalis/vpn-access-
request-automation-with-slackbot-and-terraform-a6b9975fed0d
Slack Bolt Framework: https://slack.dev/bolt-python/concepts
Block Kit: https://api.slack.com/block-kit/building
Gitlab API: https://docs.gitlab.com/ee/api/api_resources.html
Gitlab CI: https://docs.gitlab.com/ee/ci/
Requests: https://requests.readthedocs.io/en/latest/
Flake8: https://flake8.pycqa.org/en/latest/
Bandit: https://bandit.readthedocs.io/en/latest/
Hadolint: https://github.com/hadolint/hadolint

Emergent Open Source IoT Ecosystem There is a vibrant open source ecosystem developing around all layers of the IoT software stack. These technologies, when woven together, have the potential of propelling the Internet of things forward exponentially. Open source provides a trusted space where device vendors and software companies can reliably share components essential to interconnect the currently splintered IoT ecosystem. Come see what is happening and how you can leverage open source IoT software right now. Ian Skerrett, VP of Marketing, Eclipse Foundation Param Singh, CEO, iotracks; IoT Advisor, City of San Francisco https://iotworldevent.com/iot-open-source-summit/ This work is licensed under a Creative Commons Attribution 4.0 International License.

Continuous Lifecycle London 2018 Event Keynote

Weaveworks

Today it’s all about delivering velocity without compromising on quality, yet it’s becoming increasingly difficult for organisations to keep up with the challenges of current release management and traditional operations. The demand for developers to own the end-to-end delivery, including operational ownership, is increasing. A “you build it, you own it” development process requires tools that developers know and understand. So I’d like to introduce “GitOps”- an agile software lifecycle for modern applications. In this session, I will discuss these industry challenges, including current CICD trends and how they’re converging with operations and monitoring. I’ll also illustrate the GitOps model, identify best practices and tools to use, and explain how you can benefit from adopting this methodology inherited from best practices going back 10-15 years.

Google Cloud Next '22 Recap: Serverless & Data edition

Daniel Zivkovic

See what's new in #Serverless and #Data at GCP. Our guest, Guillaume Blaquiere - Stack Overflow contributor & #GCP #Developer Expert from France, covered the best #GoogleCloudNext announcements, practically demoed how to benefit from #BigQuery Remote Functions and answered many questions. The meetup recording with TOC for easy navigation is at https://youtu.be/AuZZTwHIcdY P.S. For more interactive lectures like this, go to http://youtube.serverlesstoronto.org/ or sign up for our upcoming live events at https://www.meetup.com/Serverless-Toronto/events/

Slide share device to iot solution – a blueprint

Guy Vinograd ☁

Creating an IoT Cloud service for a connected product presents a huge challenge. Why? Because the tasks of serving millions, responding to events in near real-time, securing the solution from ambitious IoT hackers, AND generating a monthly bill that doesn't collapse the business model, resemble attempts to solve Rubik's Cube, but are far more difficult. Commercial IoT platforms are irrelevant because of the vendor-lock, so we must use basic building blocks to accomplish all this. This session will illustrate the architecture of an IoT service on top of the AWS Cloud.

Microservices & API Gateways

Kong Inc.

Introducing Pico - Object Detection & Analytics using Docker, IoT & Amazon Re...

Ajeet Singh Raina

Cloud Native Transformation (Alexis Richardson) - Continuous Lifecycle 2018 ...

Weaveworks

OPENING KEYNOTE: The Cloud Native Computing Foundation (CNCF) is an open source software foundation dedicated to making cloud native computing universal and sustainable. With over 300 members including the world’s largest public cloud and enterprise software companies, Alexis Richardson, CEO of Weaveworks and chair of the CNCF Technical Oversight Committee will walk you through some success stories, and why cloud native is the way forward. You’ll learn why Kubernetes and other CNCF projects have some of the fastest adoption rates in the history of open source, and how this is only the beginning. Alexis will then show how you can increase speed and reliability in your development workflows even further by using the GitOps model, which has been developed at Weaveworks. You’ll learn about the core concepts of GitOps, including customer success stories, and how you can benefit from using this model.

The document discusses the challenges of monitoring microservices architectures. It notes that microservices are deployed much more frequently than traditional applications, sometimes in seconds or milliseconds. This rapid deployment rate requires monitoring tools that can measure and report metrics at an equally fast pace. It also explains that microservices architectures involve large numbers of loosely coupled services distributed across multiple regions, zones, and instances. Monitoring tools must be able to handle the scale and complexity of these distributed systems. Other challenges discussed include visualizing request flows, understanding common failure scenarios, and testing monitoring systems at a large scale. The document advocates for using simulations to model microservices architectures and stress test monitoring tools to help address these challenges.

Gluecon 2013 - NetflixOSS Cloud Native Tutorial Introduction

Adrian Cockcroft

Automate Workflows With The Open-source Cloud-native Tool Boomerang Flow

Konveyor Community

In the cloud-native workflow automation world, there are many great open-source projects such as Argo, Airflow, and Brigade. These tools require technical expertise to be used by business users, or they are aligned with a specific use case such as CI/CD. Boomerang Flow is an alternative tool that is usable by a business user but also provides an extensible framework developers can use for new use cases. It aims to replace Robotic Process Automation flows, which don’t necessarily need to be full-blown bots. And it doesn’t try to tackle use cases such as screen-scraping legacy green text applications. Instead, it tries to solve the problem where enterprises are using bots to perform tasks such as receiving a service now ticket and automatically moving it to a special status or workflow queue. These types of scenarios can be automated using Boomerang Flow. And the cost of running these workflows in Kubernetes is fractions of the cost of running an RPA bot for licensing. Presenters: Tyson W Lawrie, head of engineering - IBM, Marcus D Roy, Senior Lead Software engineer - IBM, and Benjamin Ruby, Analytics Consultant - IBM Recording link: https://youtu.be/-lOninwMoz4

Google Cloud Platform Solutions for DevOps Engineers

Márton Kodok

learn the DevOps essentials about cloud components, FaaS, PaaS architectural patterns that make use of Cloud Functions, Pub/Sub, Dataflow, Kubernetes and how we develop and deploy cloud software. You will get hands on information how to build, run, monitor highly scalable and flexible applications optimized to run on GCP. We will discuss cloud concepts and highlights various design patterns and best practices.

Gilmore, Palani [InfluxData] | Use Case: Monitoring / Observability | InfluxD...

InfluxData

Stanislav Kolenkin & Igor Khoroshchenko - Knock Knock: Security threats with ...

NoNameCon

Talk byStanislav Kolenkin & Igor Khoroshchenko at NoNameCon 2019. https://nonamecon.org https://cfp.nonamecon.org/nnc2019/talk/3EXNKX/ We will try to describe the most interesting security problems with Kubernetes environments from a DevOps and Security side. We'll discuss the actual cloud security threats and trends for 2019. Look behind the curtain of modern data breaches, weak identity and access management and incident response flaws. The rise of Serverless and Kubernetes as Enterprise solutions and lack of related security expertise during SDLC. Summarize the analytics and practical researches on adversaries techniques and tactics, a mass scan of cloud services and the uncertainty of business impacts behind them. Provide materials for further education.

Pulumi iac on gcp

Vishwas N

Pulumi IAC on GCP allows defining infrastructure as code using a DSL and JSON in a way that transitions infrastructure from dev to ops. It provides a centralized resource for container image discovery, distribution, and change management. Typical development involves testing changes, review by DevOps engineers, collecting and transforming data to build and deploy models. Infrastructure as code manages infrastructure through code instead of manually, creating reusable abstractions that can be shared and previewed before deployment through flexible CLIs integrated with CI/CD workflows while auditing all changes and implementing policies and controls for complex managed environments. Pulumi has components for resource definition using imperative programming, testing, and secret management.

Keptn - Automated Operations & Continuous Delivery for k8s

Andreas Grabner

Centralizing Kubernetes Management in Restrictive Environments

Kublr

Centralizing Kubernetes Management in Highly Restrictive Environments discusses managing Kubernetes in enterprise environments with multiple complex environments and constraints. It introduces Kublr, an enterprise Kubernetes management platform that provides centralized management, automation, security, and governance to address these challenges. Kublr abstracts away infrastructure details and enables operations, security, and application teams to work together through the platform.

Alan Pope [InfluxData] | Data Collectors | InfluxDays 2022

InfluxData

Challenges In Modern Application

Rahul Kumar Gupta

How Redis powers next-gen of API platform at millions RPS scale

Guanlan Dai

Kong is the most popular open-source API Platform. It just launched its 1.0 version with 20,000 stars, 38,000 community members and 100+ enterprise customers. Under the hood, Kong uses Redis for some of its key functions including rate limiting, HTTP caching, and others. In his talk, Guanlan will draw on years of practical experience, to share stories and knowledge about using Redis in API Platform design, which enables rate limiting and caching in a scalable and performant way. He will also cover the microservices architecture and how Redis fits into it.

How Redis Powers Next-Gen Of API Platform At Millions RPS Scale: Guanlan Dai

Redis Labs

Kong is a popular open-source API management platform that uses Redis to power API platforms at millions of requests per second scale. Redis allows Kong to perform global rate limiting and efficient caching across multiple layers using techniques like layered caching and twemproxy. OpenResty enables non-blocking access to Redis from Nginx through its cosocket library, avoiding callback hell. Kong has demonstrated its ability to handle high throughput through benchmarks showing its performance.

Hands on-intro to Node-RED

Pooja Mistry

Node-RED is a programming tool for wiring together hardware devices, APIs and online services. It provides a browser-based drag-and-drop interface for developing flows of events between these different technologies. The lightweight runtime can run on edge devices or in the cloud. Node-RED makes it easy for developers of all levels to integrate different data streams and quickly prototype ideas without needing to be an expert programmer.

Docs Python Org Howto Webservers Html

AkramWaseem

This document discusses how to use Python for web development. It presents several methods for integrating Python with a web server, including the Common Gateway Interface (CGI) which allows Python programs to communicate with the server. CGI is the oldest interface but has performance limitations. Newer interfaces like WSGI allow Python programs to emulate CGI while avoiding its downsides. The document provides a simple CGI script example for testing integration and discusses common issues with CGI scripts. It also introduces several popular Python web frameworks.

Can i service this from my raspberry pi

Thoughtworks

Infrastructure-related skills are essential for developers in cross-functional teams who build microservices for the cloud. Becoming proficient in infrastructure development is not just about understanding the hardware and software components on top of which applications run in the cloud. It's also about being able to use the tools that provide virtual access to this infrastructure and enable us to provision, configure, monitor it, and deploy applications to it. In this talk Gesa shares how building a Kubernetes cluster of Raspberry Pis and serving applications from it can help in acquiring fundamental infrastructure skills.

Simplifying Real Time Data Analytics with Docker, IoT & Cloud

Ajeet Singh Raina

Meetup Devops-Geneva-19.10.2019

Hidora

Bonjour à tous, Pour ce meetup, nous avons la chance d'être reçu dans les locaux de Richemont. Je remercie particulièrement Cédric Georg ainsi que l'équipe de Richemont pour leur accueil. A ce meetup DevOps, nous aurons 2 Retours d'Expérience, voici l'agenda de la soirée: 18:30 - Ouverture des portes (il faudra donner votre nom et prénom ainsi que votre numéro de plaque d'immatriculation si vous êtes venu en voiture, c'est pour la sécurité, et oui, on ne rigole pas ici :-)) 18:50 - Introduction de Matthieu et de Cédric 19:00 - Richemont et sa transformation DevOps Richemont, fort de sa transformation digitale, a dû s'adapter afin de faire travailler ensemble, avec des outils d'automatisation et de communication, les équipes de développeurs et les équipes opérationnelles. Squad, DevOps, Tests, Sécurité, Agile et Scrum, comment tous ces termes ont sû devenir le quotidien de Richemont en seulement quelques années. Nous verrons comment nous avons mis cela en place, quels ont été les points positifs et négatifs de cette transformation. 19:40 - SixSq et l'automatisation du docker sur des edge points (DEMO) Edge computing is gaining in popularity to address the explosion of data produced by IoT sensors, and the need to better manage AI both in the cloud and at the edge. To address this paradigm shift, SixSq has launched two open source projects: Nuvla for managing applications, and NuvlaBox, a cloud-in-a-box edge solution. Using these open source projects, in this session we'll demonstrate how edge computing can now be integrated to agnostically operate containerized applications on CaaS infrastructures anywhere, using a Raspberry Pi-based platform.

Ship code like a keptn

Rob Jahn

Keptn is an open-source project that provides tools to enable continuous delivery and automation for modern applications using Kubernetes. It allows developers to focus on code and DevOps teams to focus on tools rather than building custom pipelines. Keptn provides automated multi-stage delivery pipelines, automated quality gates, self-healing deployments, and enables zero-touch toolchain integration and updates. It also supports automated problem remediation in production for continuous operations. Keptn follows cloud-native design principles and provides a common way for organizations to achieve autonomous delivery and operations.

Which Application Modernization Pattern Is Right For You?

Apigee | Google Cloud

App modernization projects are hard. Enterprises are looking to cloud-native platforms like Pivotal Cloud Foundry to run their applications, but they’re worried about the risks inherent to any replatforming effort. Fortunately, several repeatable patterns of successful incremental migration have emerged. In this webcast, Google Cloud’s Prithpal Bhogill and Pivotal’s Shaun Anderson will discuss best practices for app modernization and securely and seamlessly routing traffic between legacy stacks and Pivotal Cloud Foundry.

My Hashitalk Indonesia April 2024 Presentation

Cloud Infrastructure automation with Python-3.pdf

Recommended

Recommended

More Related Content

Similar to Cloud Infrastructure automation with Python-3.pdf

Similar to Cloud Infrastructure automation with Python-3.pdf (20)

More from Ridwan Fadjar

More from Ridwan Fadjar (20)

Recently uploaded

Recently uploaded (20)

Cloud Infrastructure automation with Python-3.pdf