Highlighting problems of the current tools in IaC space and showing better paths and alternatives for those kinds of issues. As a solution to the current state of affairs, I would like to present AWS CDK, which in my opinion shows the direction of the future Infrastructure as Code.
Boost your AWS Infrastructure with CDKphilippgarbe
Infrastructure as Code has been adopted by many teams in the last years. It makes provisioning of your infrastructure easy and helps to keep your environments consistent. But by using declarative templates we still miss many practices that we are used for ìnormalî code. Youíve probably already felt the pain that each CloudFormation template is just a copy & paste of your last projects or from StackOverflow. Can you trust these snippets? How can you align improvements or even security fixes through your codebase? How can you share best practices within your company? This talk gives an introduction to the new AWS Cloud Development Kit (CDK) which is an imperative way to write CloudFormation templates in the language of your choice (even in Java). Philipp also explains why CDK is such a game changer and how your teams can spend less time with writing CloudFormation templates while even increasing the quality.
Initially presented at AWS User Group Meetup Surabaya, Indonesia.
The AWS Cloud Development Kit is an open source software development framework to model and provision your cloud application resources using familiar programming languages. In this session, we will start with why IaC (Infrastructure as Code) is a good practice and explore various ways on its implementation. Then, we'll do a live coding as a step-by-step guidance on how you can use AWS CDK to programmatically provision a serverless APIs system.
Infrastructure is code with the AWS CDK - MAD312 - New York AWS SummitAmazon Web Services
The AWS Cloud Development Kit (AWS CDK) is a multi-language, open-source framework from AWS that enables developers to harness the full power of modern programming languages to define reusable cloud components and provision applications built from those components using AWS CloudFormation. In this session, we quickly cover the basic concepts of the AWS CDK. We then develop an application using the AWS CDK. We show you how to use the AWS CDK to quickly assemble your AWS infrastructure using the new Python CDK that launched earlier this year. We explore the AWS Construct Library and show you how easy it is configure your cloud resources, manage permissions, connect event sources, and even build and publish your own constructs. Join us for a fun session with a heavy emphasis on live coding.
Amazon EC2 Container Service is a new AWS service that makes it easy to run and manage Docker-enabled applications across a cluster of Amazon EC2 instances. Amazon EC2 Container Service lets you define, schedule, and stop sets of containers. You have access to the state of your resources, making it easy to confirm that tasks are running or view the utilization of Amazon EC2 instances in your cluster. This session will describe the benefits of containers, introduce the Amazon EC2 Container Service, and demonstrate how to use Amazon EC2 Container Service for your applications.
Speakers:
Ian Massingham, AWS Technical Evangelist and
Boyan Dimitrov, Platform Automation Lead, Hailo Cabs
Infrastructure Is Code with the AWS Cloud Development Kit (DEV372) - AWS re:I...Amazon Web Services
The AWS Cloud Development Kit (AWS CDK) is a new open-source framework from AWS that enables developers to harness the full power of modern programming languages to define reusable cloud components and applications and provision them through AWS CloudFormation. The AWS CDK is shipped with a rich class library that encapsulates the details-defining infrastructure on AWS and enables you to focus on your application. In this session, we discuss why we decided to build the AWS CDK; we describe some of the high-level concepts; and we write some code on stage to demonstrate why we think the AWS CDK is going to be your best friend.
Improving Infrastructure Governance on AWS - AWS June 2016 Webinar SeriesAmazon Web Services
As your teams and infrastructure grow, it becomes more difficult to track IT resource changes as well as identify who made changes and when. It also becomes harder to enforce standards for your infrastructure resources, resulting in configuration drift and potential security issues. On AWS, you can easily standardize infrastructure configurations for commonly used IT services while also enabling self-service provisioning for your company. Once these resources are provisioned, you can then track how these resources are connected and monitor configuration changes and drift. In this session, we will discuss how you can achieve a sophisticated level of standardization, configuration compliance, and monitoring using a combination of AWS Service Catalog, AWS Config, and AWS CloudTrail.
Learning Objectives:
Understand how to use AWS services to enable governance while providing self-service
Learn to codify your business policies to promote compliance
How to improve security without sacrificing developer productivity
Discuss the basics of the AWS CDK with its pros and cons. Including how the Cloud Development Kit (CDK) helped overcome the challenges faced in their previous serverless IaC solution.
Github repo for the PoC Source Code: https://github.com/dtl-open/cdkpoc
Boost your AWS Infrastructure with CDKphilippgarbe
Infrastructure as Code has been adopted by many teams in the last years. It makes provisioning of your infrastructure easy and helps to keep your environments consistent. But by using declarative templates we still miss many practices that we are used for ìnormalî code. Youíve probably already felt the pain that each CloudFormation template is just a copy & paste of your last projects or from StackOverflow. Can you trust these snippets? How can you align improvements or even security fixes through your codebase? How can you share best practices within your company? This talk gives an introduction to the new AWS Cloud Development Kit (CDK) which is an imperative way to write CloudFormation templates in the language of your choice (even in Java). Philipp also explains why CDK is such a game changer and how your teams can spend less time with writing CloudFormation templates while even increasing the quality.
Initially presented at AWS User Group Meetup Surabaya, Indonesia.
The AWS Cloud Development Kit is an open source software development framework to model and provision your cloud application resources using familiar programming languages. In this session, we will start with why IaC (Infrastructure as Code) is a good practice and explore various ways on its implementation. Then, we'll do a live coding as a step-by-step guidance on how you can use AWS CDK to programmatically provision a serverless APIs system.
Infrastructure is code with the AWS CDK - MAD312 - New York AWS SummitAmazon Web Services
The AWS Cloud Development Kit (AWS CDK) is a multi-language, open-source framework from AWS that enables developers to harness the full power of modern programming languages to define reusable cloud components and provision applications built from those components using AWS CloudFormation. In this session, we quickly cover the basic concepts of the AWS CDK. We then develop an application using the AWS CDK. We show you how to use the AWS CDK to quickly assemble your AWS infrastructure using the new Python CDK that launched earlier this year. We explore the AWS Construct Library and show you how easy it is configure your cloud resources, manage permissions, connect event sources, and even build and publish your own constructs. Join us for a fun session with a heavy emphasis on live coding.
Amazon EC2 Container Service is a new AWS service that makes it easy to run and manage Docker-enabled applications across a cluster of Amazon EC2 instances. Amazon EC2 Container Service lets you define, schedule, and stop sets of containers. You have access to the state of your resources, making it easy to confirm that tasks are running or view the utilization of Amazon EC2 instances in your cluster. This session will describe the benefits of containers, introduce the Amazon EC2 Container Service, and demonstrate how to use Amazon EC2 Container Service for your applications.
Speakers:
Ian Massingham, AWS Technical Evangelist and
Boyan Dimitrov, Platform Automation Lead, Hailo Cabs
Infrastructure Is Code with the AWS Cloud Development Kit (DEV372) - AWS re:I...Amazon Web Services
The AWS Cloud Development Kit (AWS CDK) is a new open-source framework from AWS that enables developers to harness the full power of modern programming languages to define reusable cloud components and applications and provision them through AWS CloudFormation. The AWS CDK is shipped with a rich class library that encapsulates the details-defining infrastructure on AWS and enables you to focus on your application. In this session, we discuss why we decided to build the AWS CDK; we describe some of the high-level concepts; and we write some code on stage to demonstrate why we think the AWS CDK is going to be your best friend.
Improving Infrastructure Governance on AWS - AWS June 2016 Webinar SeriesAmazon Web Services
As your teams and infrastructure grow, it becomes more difficult to track IT resource changes as well as identify who made changes and when. It also becomes harder to enforce standards for your infrastructure resources, resulting in configuration drift and potential security issues. On AWS, you can easily standardize infrastructure configurations for commonly used IT services while also enabling self-service provisioning for your company. Once these resources are provisioned, you can then track how these resources are connected and monitor configuration changes and drift. In this session, we will discuss how you can achieve a sophisticated level of standardization, configuration compliance, and monitoring using a combination of AWS Service Catalog, AWS Config, and AWS CloudTrail.
Learning Objectives:
Understand how to use AWS services to enable governance while providing self-service
Learn to codify your business policies to promote compliance
How to improve security without sacrificing developer productivity
Discuss the basics of the AWS CDK with its pros and cons. Including how the Cloud Development Kit (CDK) helped overcome the challenges faced in their previous serverless IaC solution.
Github repo for the PoC Source Code: https://github.com/dtl-open/cdkpoc
In this session, we cover all options for running containers on AWS. This includes an introduction of container concepts and an overview of the different services: Amazon Elastic Container Service, AWS Fargate, and Amazon Elastic Container Service for Kubernetes. We also cover best practices for how to choose the right orchestration platform for your workload, the different tools for making this process easier, and ways to find more information and support as you work.
by Omar Lari, Partner Solutions Architect, AWS
Amazon Elastic Container Service for Kubernetes (Amazon EKS) is a new managed service for running Kubernetes on AWS. This session will provide an overview of Amazon EKS, why we built it, and how it works.
AWS Kubernetes 서비스 자세히 살펴보기
Kubernetes와 Amazon Elastic Container Service for Kubernetes(EKS)의 주요 개념과 로깅/모니터링, 보안, 스토리지, 오토스케일링을 상세히 다룹니다. 주요 핵심 개념과 함께 Fluentd, Prometheus, Grafana를 활용한 로깅 및 모니터링, 보안 정책, Dynamic provisioning, Headless Service, StatefulSet을 활용한 stateful apps, Horizontal Pod AutoScaler에 대해 설명해 드립니다.
Amazon EKS를 통한 빠르고 편리한 컨테이너 플랫폼 활용 – 이일구 AWS 솔루션즈 아키텍트:: AWS Cloud Week - Ind...Amazon Web Services Korea
컨테이너를 활용하고자 하는 고객은 많이 있지만, 일정 규모 이상의 서비스를 하려면 오케스트레이션 플랫폼이 필수적 입니다. 직접 물리/가상 서버를 이용하여 컨테이너 플랫폼을 설치하는 방법도 있지만 이 경우 설치, 모니터링, 용량관리, 트래픽 처리 등 다양한 문제들을 마주하게 됩니다. AWS의 완전 관리형 쿠버네티스 서비스인 EKS를 통해 클러스터에 운영 시 고려해야하는 다양한 문제를 보다 쉽게 해결할 수 있습니다. 또한 다양한 에코 시스템과 연동하여 탄력적이고 비용 효율적인 모델을 서비스할 수 있도록 소개해 드립니다.
서비스 가용성을 높이기 위해 Amazon EKS를 멀티 AZ로 사용할 경우, 노드의 위치에 따라 데이터 전송 비용이 추가로 과금됩니다. 본 세션에서는 쿠버네티스 내에서 같은 서비스를 하는 포드의 경우 동일 AZ에서 통신을 하도록 로컬리티 설정을 통해 비용 절감한 사례를 소개합니다.
Amazon EKS 환경에서 오토스케일링을 위해 Karpenter를 쓰는 경우, 노드 그룹이 죽거나 DB 연결에 문제가 생기는 등 장애 상황을 미리 검증하기 어렵습니다. 본 세션에서는 카오스 엔지니어링에 사용되는 AWS Fault Injection을 활용하여 EKS 장애 검증 사례를 소개합니다.
AWS Infrastructure as Code - September 2016 Webinar SeriesAmazon Web Services
AWS CloudFormation lets you model, provision, and update a collection of AWS resources with JSON templates. You can manage your Infrastructure as Code and deploy stacks from a single Amazon EC2 instance to multi-tier applications. In this session, we will explore CloudFormation best practices in planning and provisioning your AWS infrastructure. We will cover recent product updates that will help users to make the most of this service and demonstrate new features. This session will benefit both new and experienced users of CloudFormation.
Learning Objectives:
• Learn best practices for managing your infrastructure as code using CloudFormation
• Discover new techniques for making the most of CloudFormation
• Hear about the latest product updates and new features released
Who Should Attend:
• Developers, DevOps, IT Operations, Systems Administrators, Solutions Architects
Running Microservices and Docker on AWS Elastic Beanstalk - August 2016 Month...Amazon Web Services
In this session, we introduce you to a solution for easily running a Docker-powered microservices architecture on AWS using Elastic Beanstalk. We will also cover the fundamentals of Elastic Beanstalk and how it benefits developers looking for a quick and scalable way to get their applications running on AWS with no infrastructure work required.
Building a microservices architecture using Docker can require a lot of work, from launching and operating the underlying infrastructure to installing and maintaining cluster management software. With AWS Elastic Beanstalk’s multicontainer support feature, many of these tasks are simplified and abstracted away so you can focus on your application code. AWS Elastic Beanstalk is an easy-to-use service for deploying and scaling web applications and services developed with Java, .NET, PHP, Node.js, Python, Ruby, Go, and Docker."
Learning Objectives:
• Learn the basics of AWS Elastic Beanstalk
• Understand how to use Elastic Beanstalk to run containerized applications
• Learn how to use Elastic Beanstalk to start architecting microservices-based applications
AWS Landing Zone Deep Dive (ENT350-R2) - AWS re:Invent 2018Amazon Web Services
In this session, we discuss how to deploy a scalable environment that considers the AWS account structure, security services, network architecture, and user access. We present an overview of the AWS Landing Zone solution, an automated solution for setting up a robust and flexible AWS environment designed from the collective experience of AWS and our customers. The AWS Landing Zone helps automate the setup of a flexible account structure, security baseline, network structure, and user access based on best practices. Future growth is facilitated by an account vending machine component that simplifies the creation of additional accounts. Learn how the AWS Landing Zone can ensure that you start your AWS journey with the right foundation. We encourage you to attend the full AWS Landing Zone track, including SEC303. Search for #awslandingzone in the session catalog.
Learn how the Blue/Green Deployment methodology combined with AWS tools and services can help reduce the risks associated with software deployment. We will illustrate common patterns and highlight ways deployment risks are mitigated by each pattern. Topics will include how services like AWS CloudFormation, AWS Elastic Beanstalk, Amazon EC2 Container Service, Amazon Route53, Auto Scaling and Elastic Load Balancing can help automate deployment. We will also address how to effectively manage deployments in the context of data model and schema changes. Learn how you can adopt blue/green for your software release processes in a cost-effective and low-risk way.
AWS Summit Seoul 2023 | AWS의 개발자를 위한 신규 서비스 소개 Amazon CodeCatalyst & Amazon C...Amazon Web Services Korea
AWS에서 개발자를 위한 두 가지 새로운 서비스를 출시했습니다. 첫 번째는 클라우드 애플리케이션의 통합 소프트웨어 개발 서비스 Amazon CodeCatalyst입니다. 제공되는 다양한 프로젝트 청사진을 사용하여 개발 사이클을 시작하는 데모를 선보입니다. 두 번째는 ML 기반으로 개발자가 필요한 코드를 즉시 완성시켜주는 Amazon CodeWhisperer입니다. 개발 효율성을 극대화할 수 있는 다양한 유즈케이스들을 소개합니다.
AWS January 2016 Webinar Series - Managing your Infrastructure as CodeAmazon Web Services
In this session, you will learn how you can provision, configure, and manage your infrastructure using code and treat it just like your application code. We will discuss the AWS services that enable these practices (AWS CloudFormation, AWS OpsWorks, and AWS CodeDeploy) and that allow you to control everything from Amazon VPCs and AWS Identity and Access Management to the configuration of individual applications on a single host. We’ll also talk about on-going management, how to best update your resources, and which tools are best suited for AWS resource management and host-based configuration management.
Learning Objectives:
Understand Infrastructure as Code
Understand the AWS services that help you manage your infrastructure as code
Discover best practices for managing your AWS infrastructure, host configuration, and applications
Who Should Attend:
DevOps Engineers, IT Professionals, Systems Administrators, Architects, Operations Professionals, Developers
Cluster-as-code. The Many Ways towards KubernetesQAware GmbH
CloudLand, Juni/Juli 2022, Mario-Leander Reimer (@LeanderReimer, Principal Software Architect bei QAware).
== Dokument bitte herunterladen, falls unscharf! Please download slides if blurred! ==
Kubernetes is the de-facto standard when it comes to container orchestration. But why is there is no established, standard and uniform way to spin-up and manage a single or even a whole farm of Kubernetes clusters yet? Instead, a whole bunch of different and mostly incompatible ways towards Kubernetes exist today. Each with its own pros and cons in regards to ease of use, flexibility and many other requirements. In this session we will have a closer look at the different available options to create, manage and operate Kubernetes clusters at scale.
Cluster-as-code. The Many Ways towards KubernetesQAware GmbH
iSAQB Software Architecture Gathering – Digital 2022, November 2022, Mario-Leander Reimer (@LeanderReimer, Principal Software Architect bei QAware).
== Dokument bitte herunterladen, falls unscharf! Please download slides if blurred! ==
Kubernetes is the de-facto standard when it comes to container orchestration. But why is there is no established, standard and uniform way to spin-up and manage a single or even a whole farm of Kubernetes clusters yet? Instead, a whole bunch of different and mostly incompatible ways towards Kubernetes exist today. Each with its own pros and cons in regards to ease of use, flexibility and many other requirements. In this session we will have a closer look at the different available options to create, manage and operate Kubernetes clusters at scale.
In this session, we cover all options for running containers on AWS. This includes an introduction of container concepts and an overview of the different services: Amazon Elastic Container Service, AWS Fargate, and Amazon Elastic Container Service for Kubernetes. We also cover best practices for how to choose the right orchestration platform for your workload, the different tools for making this process easier, and ways to find more information and support as you work.
by Omar Lari, Partner Solutions Architect, AWS
Amazon Elastic Container Service for Kubernetes (Amazon EKS) is a new managed service for running Kubernetes on AWS. This session will provide an overview of Amazon EKS, why we built it, and how it works.
AWS Kubernetes 서비스 자세히 살펴보기
Kubernetes와 Amazon Elastic Container Service for Kubernetes(EKS)의 주요 개념과 로깅/모니터링, 보안, 스토리지, 오토스케일링을 상세히 다룹니다. 주요 핵심 개념과 함께 Fluentd, Prometheus, Grafana를 활용한 로깅 및 모니터링, 보안 정책, Dynamic provisioning, Headless Service, StatefulSet을 활용한 stateful apps, Horizontal Pod AutoScaler에 대해 설명해 드립니다.
Amazon EKS를 통한 빠르고 편리한 컨테이너 플랫폼 활용 – 이일구 AWS 솔루션즈 아키텍트:: AWS Cloud Week - Ind...Amazon Web Services Korea
컨테이너를 활용하고자 하는 고객은 많이 있지만, 일정 규모 이상의 서비스를 하려면 오케스트레이션 플랫폼이 필수적 입니다. 직접 물리/가상 서버를 이용하여 컨테이너 플랫폼을 설치하는 방법도 있지만 이 경우 설치, 모니터링, 용량관리, 트래픽 처리 등 다양한 문제들을 마주하게 됩니다. AWS의 완전 관리형 쿠버네티스 서비스인 EKS를 통해 클러스터에 운영 시 고려해야하는 다양한 문제를 보다 쉽게 해결할 수 있습니다. 또한 다양한 에코 시스템과 연동하여 탄력적이고 비용 효율적인 모델을 서비스할 수 있도록 소개해 드립니다.
서비스 가용성을 높이기 위해 Amazon EKS를 멀티 AZ로 사용할 경우, 노드의 위치에 따라 데이터 전송 비용이 추가로 과금됩니다. 본 세션에서는 쿠버네티스 내에서 같은 서비스를 하는 포드의 경우 동일 AZ에서 통신을 하도록 로컬리티 설정을 통해 비용 절감한 사례를 소개합니다.
Amazon EKS 환경에서 오토스케일링을 위해 Karpenter를 쓰는 경우, 노드 그룹이 죽거나 DB 연결에 문제가 생기는 등 장애 상황을 미리 검증하기 어렵습니다. 본 세션에서는 카오스 엔지니어링에 사용되는 AWS Fault Injection을 활용하여 EKS 장애 검증 사례를 소개합니다.
AWS Infrastructure as Code - September 2016 Webinar SeriesAmazon Web Services
AWS CloudFormation lets you model, provision, and update a collection of AWS resources with JSON templates. You can manage your Infrastructure as Code and deploy stacks from a single Amazon EC2 instance to multi-tier applications. In this session, we will explore CloudFormation best practices in planning and provisioning your AWS infrastructure. We will cover recent product updates that will help users to make the most of this service and demonstrate new features. This session will benefit both new and experienced users of CloudFormation.
Learning Objectives:
• Learn best practices for managing your infrastructure as code using CloudFormation
• Discover new techniques for making the most of CloudFormation
• Hear about the latest product updates and new features released
Who Should Attend:
• Developers, DevOps, IT Operations, Systems Administrators, Solutions Architects
Running Microservices and Docker on AWS Elastic Beanstalk - August 2016 Month...Amazon Web Services
In this session, we introduce you to a solution for easily running a Docker-powered microservices architecture on AWS using Elastic Beanstalk. We will also cover the fundamentals of Elastic Beanstalk and how it benefits developers looking for a quick and scalable way to get their applications running on AWS with no infrastructure work required.
Building a microservices architecture using Docker can require a lot of work, from launching and operating the underlying infrastructure to installing and maintaining cluster management software. With AWS Elastic Beanstalk’s multicontainer support feature, many of these tasks are simplified and abstracted away so you can focus on your application code. AWS Elastic Beanstalk is an easy-to-use service for deploying and scaling web applications and services developed with Java, .NET, PHP, Node.js, Python, Ruby, Go, and Docker."
Learning Objectives:
• Learn the basics of AWS Elastic Beanstalk
• Understand how to use Elastic Beanstalk to run containerized applications
• Learn how to use Elastic Beanstalk to start architecting microservices-based applications
AWS Landing Zone Deep Dive (ENT350-R2) - AWS re:Invent 2018Amazon Web Services
In this session, we discuss how to deploy a scalable environment that considers the AWS account structure, security services, network architecture, and user access. We present an overview of the AWS Landing Zone solution, an automated solution for setting up a robust and flexible AWS environment designed from the collective experience of AWS and our customers. The AWS Landing Zone helps automate the setup of a flexible account structure, security baseline, network structure, and user access based on best practices. Future growth is facilitated by an account vending machine component that simplifies the creation of additional accounts. Learn how the AWS Landing Zone can ensure that you start your AWS journey with the right foundation. We encourage you to attend the full AWS Landing Zone track, including SEC303. Search for #awslandingzone in the session catalog.
Learn how the Blue/Green Deployment methodology combined with AWS tools and services can help reduce the risks associated with software deployment. We will illustrate common patterns and highlight ways deployment risks are mitigated by each pattern. Topics will include how services like AWS CloudFormation, AWS Elastic Beanstalk, Amazon EC2 Container Service, Amazon Route53, Auto Scaling and Elastic Load Balancing can help automate deployment. We will also address how to effectively manage deployments in the context of data model and schema changes. Learn how you can adopt blue/green for your software release processes in a cost-effective and low-risk way.
AWS Summit Seoul 2023 | AWS의 개발자를 위한 신규 서비스 소개 Amazon CodeCatalyst & Amazon C...Amazon Web Services Korea
AWS에서 개발자를 위한 두 가지 새로운 서비스를 출시했습니다. 첫 번째는 클라우드 애플리케이션의 통합 소프트웨어 개발 서비스 Amazon CodeCatalyst입니다. 제공되는 다양한 프로젝트 청사진을 사용하여 개발 사이클을 시작하는 데모를 선보입니다. 두 번째는 ML 기반으로 개발자가 필요한 코드를 즉시 완성시켜주는 Amazon CodeWhisperer입니다. 개발 효율성을 극대화할 수 있는 다양한 유즈케이스들을 소개합니다.
AWS January 2016 Webinar Series - Managing your Infrastructure as CodeAmazon Web Services
In this session, you will learn how you can provision, configure, and manage your infrastructure using code and treat it just like your application code. We will discuss the AWS services that enable these practices (AWS CloudFormation, AWS OpsWorks, and AWS CodeDeploy) and that allow you to control everything from Amazon VPCs and AWS Identity and Access Management to the configuration of individual applications on a single host. We’ll also talk about on-going management, how to best update your resources, and which tools are best suited for AWS resource management and host-based configuration management.
Learning Objectives:
Understand Infrastructure as Code
Understand the AWS services that help you manage your infrastructure as code
Discover best practices for managing your AWS infrastructure, host configuration, and applications
Who Should Attend:
DevOps Engineers, IT Professionals, Systems Administrators, Architects, Operations Professionals, Developers
Cluster-as-code. The Many Ways towards KubernetesQAware GmbH
CloudLand, Juni/Juli 2022, Mario-Leander Reimer (@LeanderReimer, Principal Software Architect bei QAware).
== Dokument bitte herunterladen, falls unscharf! Please download slides if blurred! ==
Kubernetes is the de-facto standard when it comes to container orchestration. But why is there is no established, standard and uniform way to spin-up and manage a single or even a whole farm of Kubernetes clusters yet? Instead, a whole bunch of different and mostly incompatible ways towards Kubernetes exist today. Each with its own pros and cons in regards to ease of use, flexibility and many other requirements. In this session we will have a closer look at the different available options to create, manage and operate Kubernetes clusters at scale.
Cluster-as-code. The Many Ways towards KubernetesQAware GmbH
iSAQB Software Architecture Gathering – Digital 2022, November 2022, Mario-Leander Reimer (@LeanderReimer, Principal Software Architect bei QAware).
== Dokument bitte herunterladen, falls unscharf! Please download slides if blurred! ==
Kubernetes is the de-facto standard when it comes to container orchestration. But why is there is no established, standard and uniform way to spin-up and manage a single or even a whole farm of Kubernetes clusters yet? Instead, a whole bunch of different and mostly incompatible ways towards Kubernetes exist today. Each with its own pros and cons in regards to ease of use, flexibility and many other requirements. In this session we will have a closer look at the different available options to create, manage and operate Kubernetes clusters at scale.
Hybrid and multicloud deployments are critical approaches for bridging the gap between legacy and modern architectures. Sandeep Parikh discusses common patterns for creating scalable cross-environment deployments using Kubernetes and explores best practices and repeatable patterns for leveraging Kubernetes as a consistent abstraction layer across multiple environments.
Understand how to architect an infrastructure to handle going from zero to millions of users. From leveraging highly scalable AWS services to making smart decisions on building out your application, you'll learn a number of best practices for scaling your infrastructure in the cloud.
This talk give you an overview of the new AWS Managed Kubernetes Service. Why do we want to use an managed service and most importend is this a good idea with EKS.
What does programming without servers look like? What are the possibilities? And how does it work? Wojciech Gawroński (Pattern Match) told us about it during the third meeting of Serverless User Group Poland, which took place on 27/09/2018 in Warsaw.
Wojtek's social media:
LinkedIN https://www.linkedin.com/in/afronski/
www https://pattern-match.com/
Serverless UG Poland
Facebook https://bit.ly/2zHuJeo
Apache Spark Streaming in K8s with ArgoCD & Spark OperatorDatabricks
Over the last year, we have been moving from a batch processing jobs setup with Airflow using EC2s to a powerful & scalable setup using Airflow & Spark in K8s.
The increasing need of moving forward with all the technology changes, the new community advances, and multidisciplinary teams, forced us to design a solution where we were able to run multiple Spark versions at the same time by avoiding duplicating infrastructure and simplifying its deployment, maintenance, and development.
Matt Chung (Independent) - Serverless application with AWS Lambda Outlyer
The talk will focus on how we are utilizing AWS Lambda for certain applications and the advantages/disadvantages, and the challenges we discovered along the way. It would help those who are looking to reduce technical debt with the infrastructure and costs.
Previously a Director of technical operations at fox networks (21st Century Fox/News Corporation) responsible for infrastructure and building deployment pipelines. Currently a Python programmer / DevOps engineer with roots in systems/networks administration. Focus is on infrastructure and application automation. Worked as an engineer for Cisco Systems with emphasis on video conferencing. Built microwave networks at Bel Air Internet. Find me on github and twitter @itsmemattchung
Video: https://www.youtube.com/watch?v=BLcElBUhfrQ
Join DevOps Exchange London here: http://www.meetup.com/DevOps-Exchange-London
Follow DOXLON on twitter http://www.twitter.com/doxlon
Zenko & MetalK8s @ Dublin Docker Meetup, June 2018Laure Vergeron
A presentation of the Zenko project, that gives you a single namespace and a single API (the S3 API) across all of your clouds (AWS S3, MS Azure, GCP, Wasabi, Digital Ocean, and private clouds), and that lets you perform metadata seach across all these backends.
We also cover MetalK8s, an opinionated release of Kubernetes committed to bare metal environments, and growing on Kuberspray.
Ever since the “CloudNative revolution” took over our development environment (devenv), we have never been more challenged (or more excited). With Kubernetes, Docker (Containerd) & many other microservice-related technologies, we have a handful of technologies to master before we write the first line of code.
Confluent On Azure: Why you should add Confluent to your Azure toolkit | Alic...HostedbyConfluent
As a data professional, you are the glue that makes cross-platform integrations possible. With the increase in adoption of hybrid cloud architectures, Kafka is an increasingly relevant tool for building data pipelines between platforms and accelerating delivery on cloud projects. Early exposure to Kafka on Azure capabilities gives you an edge to build better mousetraps at the design phase.
Customers already running Kafka on premises and are looking to extend Kafka systems to Azure can get started quickly with Confluent Cloud. Additionally, DevOps for self-managed options can be easily scalable with Ansible for Virtual Machines or containers via Azure Kubernetes Services or Azure Container Instances.
This session is presented from the Microsoft Solution Architect perspective by Israel Ekpo, Microsoft Cloud Solution Architect and Alicia Moniz, Microsoft MVP. They will cover use cases and scenarios, along with key Azure integration points and architecture patterns.
Building a Global-Scale Multi-Tenant Cloud Platform on AWS and Docker: Lesson...Felix Gessert
In this talk we share the lessons learned while building out the Baqend Cloud platform on AWS and Docker. Baqend’s AWS-hosted architecture consists of a caching CDN-Layer, global and local load balancing, a group of REST and Node.js servers and a database cluster with Redis and MongoDB. As customers have their own set of containerized REST and Node servers, we needed a cluster that on the one hand is horizontally scalable and on the other hand easily manageable and fault-tolerant from an operational perspective. Today there are at least 4 popular systems that claim to support this:
- Kubernetes
- Apache Mesos
- Docker Swarm
- AWS Elastic Container Service (ECS)
Thinking that ECS would certainly be the easiest option on AWS, we started building our cluster on it. We quickly came to realize that while ECS was astoundingly stable and easy to use there were inherent limitations that could not be worked around. An old Docker version, missing network isolation, no means of parameterizing task and forced memory constraints are major limitations of ECS we will talk about. Seeing the daunting operational overhead of running Kubernetes or Mesos in practice we turned to Docker’s native clustering solution Swarm. We will present how Swarm works with both Docker and AWS and highlight the advantages and downsides compared to Amazon’s ECS.
Serverless is getting more and more attention in today's world. Ease of use, the promise of infinite scale and reducing operational complexity to the minimal level - those are only a few of selling points of that particular "technology". Unfortunately, we have limited choice when it comes to the languages and runtimes available in that environment. Service providers support only a fraction of platforms (to be perfectly honest - the most popular). By default, there are no functional languages available out of the box - assuming that we will not perform any additional work. But I did not surrender! If you are interested in how much yak shaving, frustration, and unnecessary practice is required to bring our favorite functional flavors to the Serverless world - that talk is for you! And I can promise that in the end, I will not leave you without a reproducible and reliable solution.
Similar to AWS CDK: Your Infrastructure is Code! (20)
Serverless is coming. However, it transformed the IT landscape already. Advocates of that kind of solutions are claiming that those solutions are cost-efficient, less problematic to maintain (again - cheaper, smaller OPEX), and modern (we like those arguments in IT, don't we?).
The truth is that there is a much more significant advantage of embracing serverless. Money is not everything, as there is at least one more precious resource: time. I want to present an interesting case study, how to significantly reduce time to market with using serverless powered by AWS ecosystem.
As always, context is king, and when it comes to building an MVP, serverless provides a tremendous advantage in focusing on business value first. The goal of the talk is to show pitfalls and fallacies related to developing serverless applications but still delivering an MVP (minimum viable product) without sacrificing quality or further enhancements.
Amazon SageMaker in Practice - Workshop at Big Data Moscow 2018 (10.10.2018)Wojciech Gawroński
At re:Invent in 2017 Amazon announced new service – SageMaker, that allows you to build, train and deploy ML models at any scale. Sounds fantastic, but is it true?
We challenged the marketing taglines and decided to test a specific use case which is familiar to us: ad targeting.
During the workshop, which is an adventure itself, we build an approximation of an ad targeting system with the use of Amazon SageMaker – from development to deployment. We show how with use of such service you can start small and be pretty confident about scaling up and expanding the system later.
We start from the ground up, so no worries – there is no need for previous experience with advertising systems and neither AWS, although general knowledge about machine learning or data crunching is a plus. If you were curious after the last year’s re Invent announcement, that is a workshop for you!
Slides from Wojtek Gawroński's talk at 4Developers 2018 in Katowice about how serverless approach can help you maximize delivered business value, how API mashup enables you to create an MVP as soon as possible, without cutting edges and ignoring maintainability aspects.
Slides from the workshop that I conducted for community announced as a devwarsztaty.pl (Link to the event: devwarsztaty.pl/warsztaty/2018-04-07).
It is a workshop from zero to hero, when at we are deploying a working application to the App Engine on Google Cloud Platform.
Recently functional programming gets a lot of traction. In most cases, arguments are related with distributed systems, also with increased and inherent concurrency. Maybe because of near end of Moore’s law, perhaps because of the new domains which are inherently parallel and distributed (IoT is just a single example).
But it is nothing new under the sun (e.g., Lisp was invented almost 60 years ago). You cannot talk about that paradigm, in isolation – without real and concrete examples.
I will do my best to present differences from the current state of the art for a demanding domain and its implementation in the form of a significant, distributed and concurrent system.
Why docker@localhost is not even remotely near DevOps?Wojciech Gawroński
"Hey, I've just installed this docker-thing on my laptop! This DevOps looks like real fun." - *sigh*, it happened again.
In the world of buzzwords, devaluation of words and rapidly evolving requirements when it comes to tech we did that again. We did that again. *DevOps* became a new *Agile*.
If you are a newbie in that topic or someone already presented you the idea, and you are not a strong believer of that culture and that it may change anything for you, I would love to convince you that it makes difference.
I would like you to show how we are doing *DevOps*. How it enhanced our capabilities and enabled us - a team of 10 people - to share knowledge, cooperate with other teams and care about the huge distributed system, which handles billions of transactions every month.
How to move a mission critical system to 4 AWS regions in one year?Wojciech Gawroński
A year ago our team was challenged to enhance the scope and scale of an existing platform, that is providing significant revenue for our client. As the designers and maintainers of that solution, we decided to leverage AWS cloud during that transition. In the presentation, I would like to discuss how we have tackled that migration - with the assumption that we had to move in a limited resource, hybrid cloud environment - working in close cooperation with teams responsible for other parts of the system. As I stated previously - it was a challenge - and I would like to talk what problems we have solved during that process. Also, what services we have leveraged to smooth the transition. And last, but not least - I would like to present how we have maintained the delivery pipeline, automation and massive pile of CloudFormation templates and why AWS Lambda is an excellent glue for any operational work you have to do in the cloud. Our hard work paid off. In October 2017 we have deployed our system into 4th AWS region. Bare with me during the talk, and you will learn how we achieved that
Abusing Erlang compilation pipeline for Fun and ProfitWojciech Gawroński
Erlang VM is a brilliant piece of engineering. One of the exciting places is compilation pipeline - it is organized in the stages. One of them relates to something called Core Erlang representation. In the talk, I will describe that part of Erlang VM: what it is, why it is organized in such way and how Core Erlang relates to the whole pipeline. Last but not least - how it can be used (or abused) by showing very simple Brainfuck to Core Erlang compiler (which is fun) and other community projects (which are much more profitable than the previous example).
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024Neo4j
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!SOFTTECHHUB
As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
UiPath Test Automation using UiPath Test Suite series, part 6DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
UiPath Test Automation using UiPath Test Suite series, part 5DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
Communications Mining Series - Zero to Hero - Session 1DianaGray10
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIVladimir Iglovikov, Ph.D.
Presented by Vladimir Iglovikov:
- https://www.linkedin.com/in/iglovikov/
- https://x.com/viglovikov
- https://www.instagram.com/ternaus/
This presentation delves into the journey of Albumentations.ai, a highly successful open-source library for data augmentation.
Created out of a necessity for superior performance in Kaggle competitions, Albumentations has grown to become a widely used tool among data scientists and machine learning practitioners.
This case study covers various aspects, including:
People: The contributors and community that have supported Albumentations.
Metrics: The success indicators such as downloads, daily active users, GitHub stars, and financial contributions.
Challenges: The hurdles in monetizing open-source projects and measuring user engagement.
Development Practices: Best practices for creating, maintaining, and scaling open-source libraries, including code hygiene, CI/CD, and fast iteration.
Community Building: Strategies for making adoption easy, iterating quickly, and fostering a vibrant, engaged community.
Marketing: Both online and offline marketing tactics, focusing on real, impactful interactions and collaborations.
Mental Health: Maintaining balance and not feeling pressured by user demands.
Key insights include the importance of automation, making the adoption process seamless, and leveraging offline interactions for marketing. The presentation also emphasizes the need for continuous small improvements and building a friendly, inclusive community that contributes to the project's growth.
Vladimir Iglovikov brings his extensive experience as a Kaggle Grandmaster, ex-Staff ML Engineer at Lyft, sharing valuable lessons and practical advice for anyone looking to enhance the adoption of their open-source projects.
Explore more about Albumentations and join the community at:
GitHub: https://github.com/albumentations-team/albumentations
Website: https://albumentations.ai/
LinkedIn: https://www.linkedin.com/company/100504475
Twitter: https://x.com/albumentations
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
How to Get CNIC Information System with Paksim Ga.pptxdanishmna97
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
2. ~ # whoami
(afronski)
✓ Co-founder and Cloud Architect
at Pattern Match
Erlang, Elixir, Java, Node.js
Python, DevOps, AWS
✓ Co-organizer of Natywna Chmura
✓ Program member at Cloudyna
10. 1. As a developer, I am tired of using a DSL/YAML.
a. Lack of expressiveness.
b. Limited reusability and modularity.
c. Learning curve in the shared codebase.
2. Configuration Drift.
3. State Management and Stateful Resources.
4. Sensible Testing.
5. Sensible Local Development.
What’s missing?
11. Alternatives?
1. Ansible - YAML strikes back.
2. Sceptre - Jinja-based CloudFormation templates.
3. AWS SAM - Again, YAML on top of YAML.
4. Stack Deployment Tool - Perun-like case.
5. Terraform - Do I have to explain myself again?
6. Troposphere *
7. Pulumi *
8. AWS CDK
16. Features
1. Multiple platforms and programming languages support:
a. TypeScript, Python, JVM (Java), .NET (C#)
2. CloudFormation-native workflow (rollbacks).
a. If something is not supported, you are able
to use so called L1 constructs on your own.
3. Testability and built-in support for local development.
4. Flexibility (AWS CDK Constructs).
5. Easy migration from pure CloudFormation templates.
17.
18. let participant = new User (this, ...);
let userBucket = new Bucket(this, ...);
userBucket.grantReadWrite(participant);
userBucket.addDependsOn(participant);
20. Problems?
1. Documentation.
2. Learning curve (AWS CDK Constructs).
3. No support for passing CloudFormation parameters.
a. AWS CDK introduces Context instead.
4. No support for StackPolicies so far.
5. Some things may need override in the resulting CloudFormation:
a. e.g. not supported services.
6. Generated logical IDs are not exactly user-friendly (debugging).
a. Such template is hard to analyze (assembly language).
21. What’s next?
1. Documentation!
2. Infrastructure as Code as a first-class source code citizen.
a. e.g. Jenkins Pipelines or Jenkins X.
b. Sidenote: Kubernetes and co do not invalidate the need for IaC.
3. Opening door for abstracting the cloud vendors.
4. Type-safe AWS Infrastructure?
a. punchcard/punchcard
5. Even better support for serverless applications.
22. References
1. Our company - Pattern Match and my talks.
2. Natywna Chmura (facebook, website).
3. Cloudyna (website) - 13.11.2019, Katowice.
4. Source Code: patternmatch/aws-cdk-playground.
5. Another simple example: patternmatch/amazon-sagemaker-in-practice.
6. AWS Cloud Development Kit - Official Page.
7. AWS CDK Workshop.
8. Python and TypeScript officially supported in AWS CDK.
9. CDK All The Things: A Whirlwind Tour.
10. Building serverless apps with AWS CDK and testing them locally.
11. The Last Thing I missed in the CloudFormation.
12. punchcard/punchcard - Type-safe AWS Infrastructure.
13. Why We Built Ludwig? - Fugue Blog.
14. AWS CloudFormation is an infrastructure graph management service.
15. The Definitive Guide to using Terraform with Serverless Framework.
16. Thoughtworks Technology Radar - HOLD: Handwritten CloudFormation.
17. Terraform vs CloudFormation.
Editor's Notes
Hello!
Thank you for being here!
I am energized, because I would like to share some exciting news with you.
I really think that it provides a fresh and novel ideas to the Infrastructure as Code space.
My name is Wojtek, afronski on the internet.
Cloud Architect and Co-founder of the Pattern Match.
That lovely avatar is made by my friends from Pattern Match after I bragged about that I can write a bash loop from memory (and obviously failed). ;)
In PM we’re doers - experienced and trusted advisors in best practices of software delivery, efficiency, performance and availability.
We do that by employing successfully cloud computing, cloud native approach and destroying walls between business and technical teams.
Why me? Am I qualified to whine about this topic?
That is just a snapshot from one of the projects that I bootstrapped and maintained for more than a 2 years.
YAML here is representing a pile of AWS CloudFormation.
So I know a thing or two about Infrastructure as Code from design, implementation and maintenance points of view.
Nowadays I am writing more Python than bash, but pile of text files is not smaller surprisingly.
The amount of YAML and CloudFormation was so overwhelming that I thought it would be a great idea to write a separate tool that will perform an opinionated linting, validation, and other helpful operations on it.
Name of this tool is Perun and its written in Go.
I am the creator of this tool, most of us from Pattern Match contributed to this tool - it stayed in the good hands of our colleagues at Appliscale.
At this point of time I think it was a Stockholm Syndrome.
Reasoning and justification sounded perfectly fine to me.
From the perspective of couple more projects I think we did a huge mistake there to not employ better tools at that time.
Knowing a bit about me, allow me to show you the current landscape of the Infrastructure as Code.
And I will start with a rant about the current state of IaC.
WARNING: bear with me, when I will hyperventilate.
Oh boy there are a lot of yaks to be shaved here.
YAML anyone? I have a bad news to you - if you are throwing YAML files back and forth you are not an engineer, nor a programmer.
Anyone recognizes this cute error up in the corner?
Yup, that is Terraform that blown up and if you will have luck, you will be able to restore state from the remote backend without nitpicking JSON via scissors and glue.
Those who work with CloudFormation on a daily basis and are old enough to remember the JSON representation of it will have a goosebumps right now.
I know it sounds, but YAML is a better part of the CloudFormation.
But here it comes the best part!
This YAML is a subset of an original YAML specification.
What is missing? E.g. no anchors (very useful), no variables, no way to reuse fragments.
Why am I talking about this?
Because more smarter people than me are seeing the shortcomings of those methods.
Here you have a screenshot from the latest Technology Radar by Thoughtworks where they’ve marked Handwritten CloudFormation as a Hold (so you should avoid that).
I am surprised tho, that they are naming Terraform as a sensible default - as it has its downfalls too and does not provide a sensible rollback, where CloudFormation does.
Current state of IaC sucks, and it’s covered by a lot of DSLs or YAML files that eventually will blow up in our faces.
Nowadays most of the decisions are made between Terraform (supposedly free of vendor lock-in) and Cloud Vendor specific tools.
Which is represented by the new and official logo of AWS CloudFormation, but you can use any logo e.g. Google Deployment Manager.
Our friends that pray to the Kubernetes god are smiling in the corner - they will powerful and superior.
They are supposedly agnostic of “this crap” and … they use tool specific YAML format.
Those people are sharing the smirk with the guys doing serverless apps with use Serverless Framework.
Guess what? This YAML is problematic too, and maintaining a significant serverless system causes a lot of pain.
At least it is much easier and more maintainable than using Terraform for that. God forbid!
All those techniques are susceptible on main issues:
Configuration Drift.
State Management.
Dealing with stateful resources.
How to sensibly test this crap?
All those problems are showing clearly:
Is it a proper battle to fight in a first place?
Let’s see what is missing and how it can be improved.
For sure we can benefit from having a real programming language and its expressiveness.
You may ask: what with my declarativeness?
Well, as a functional programmer I can tell you that in most cases YAML is as declarative as the API allows for it.
Declarative programming has roots in functional programming and is about telling what we want to do, without describing how we want to do it.
We express the logic of a computation without describing its control flow, which is perfectly possible with regular programming languages.
How to handle configuration drift? Assuming we are dealing with distributed environments, on-calls and so on.
Managing state when provisioning is also problematic, as you have to deal with many shortcomings, and people that are using Terraform and nested CloudFormation stacks are having goosebumps now.
Sensible testing mechanism to shorten the feedback loop and maintainability.
Sensible local development - again for shortening the feedback loop and experimentation.
Let’s investigate alternatives and how they map with the deficiencies that we have described previously:
So you can go with Ansible for orchestration and its wrappers - we are using that, and it works sensibly, although it’s not a real code, and you are moving above the original abstraction.
You can use Sceptre (provided by CloudReach), but still it is a Jinja2 templating smacked on top of YAML with added orchestration.
AWS SAM has the same deficiencies and its serverless specific.
Stack Deployment Tool is basically the same story I have described with Perun - it’s a Stockholm Syndrome.
Terraform was covered previously - I do not treat it as a viable alternative here. No rollback, share most of the issues with CloudFormation, just has different syntax.
Last three options are the closest to the real code and they can be considered a viable alternative.
To be precise only AWS CDK, because Troposphere and Pulumi for now are not providing a sensible testability.
Also Pulumi is serverless specific.
With Troposphere it is possible, although it’s not available out of the box.
Let’s talk about our “savior” now.
AWS CDK was originally created around September 2018 and released as a beta around that time.
I have used the beta version in Autumn 2018, where it was available just for TypeScript and at that time it already looked promising.
I have gathered my thoughts and observation in the blog post attached to this slide.
In June 2019 AWS announced that Cloud Development Kit is generally available for TypeScript and Python.
Yet, it supports additionally Java and C# too.
How it works?
It basically allows you to model your infrastructure with use of code and programming languages using a library of constructs.
With use of this library you are create, manage, implement, and maintain infrastructure as code definitions that at the end are translated directly to the AWS CloudFormation.
It allows you to build on top of well-known patterns and behaviors of this service - including rollback, configuration drift, state management.
Yet, you have capabilities of a first-class programming languages.
I deliberately said translates.
If you will treat CloudFormation templates as an intermediate language (e.g. assembly), it works exactly like the compiler.
For us, programmers, it is a bread and butter - we work on a higher level of abstraction, reusing higher level constructs and modules that are compiling to the low-level elements delivered by the platform or runtime.
In that case CloudFormation service is our runtime or execution engine.
Let’s discuss briefly its the most interesting features before we will dive into the details during demo session.
I’ve already said that they are supporting multiple programming languages - the best and official support is for TypeScript and Python.
Java and C# are still in beta, but perfectly usable.
It’s fully based on CloudFormation service and templates, so you have all benefits and traits of those built-in.
That includes rollbacks, configuration drifts, full support on the AWS side, and many others.
It means, that even if the library maintainers are a little bit behind, you are able to spit our regular CloudFormation with use of L1 constructs and fill out the gaps with pure CloudFormation syntax.
It is fully testable locally, and even encourages local development - with checking configuration drift, diffs, generating synthesized template, etc.
It gives you the flexibility of the programming language and ability to create modular, reusable abstractions with use of constructs.
Which is a game changer for the most of the alternatives mentioned previously.
It allows for an easy migration from the pure CloudFormation templates with use of many techniques:
You can rewrite your templates into CDK using overrides for simpler templates.
You can include your existing more complicated templates into CDK and rewrite them bit by bit.
You can use built-in disassembler that creates a raw AWS CDK code in a given language from AWS CloudFormation.
Even if it does not fully use the power of CDK (it’s of course a limited tool), it really helps with kickstarting the effort and starting the project.
Here you have just one example of the diff between the current managed state and the CDK definition stored locally.
It shows you the exact difference, and you can use the same mechanism doing local tests (e.g. with unit testing).
Speaking about abstractions and helpers:
AWS CDK provides a lot of existing abstractions and constructs, including permissions and dependency management as written on the slide.
Behind the line with grant we have already impemented the IAM and permissions management for many resources.
Let’s now pray to the demo gods to not screw up with me!
Let me recap the demo, by unveiling the current problems - no pain, no gain:
Documentation sucks currently - most of the examples are written and prepared for TypeScript only.
It has some learning curve especially if you used to CloudFormation.
Outside of constructs, the most surprising change is related to parameters.
CloudFormation parameters are resolved when deploying, so you can generate a CloudFormation template with parameters, but you cannot deploy it with use of CDK, there is no way to type check/validate that on the code level - instead CDK recommends to use contexts. Again, it is the controversial decision that is a tradeoff. They did that because of compile-time guarantees. You can read more about that here.
Even it is generally available, it has no support for StackPolicies.
Also on the same note, some things requires override in the resulting CloudFormation, if they are not supported.
When it comes to debugging - it is not so user-friendly as handwritten template (which is kinda obvious if we are using it like assembly language), and it may be harder to analyze those.
However, it is possible and it just requires practice, nothing else.
Let me wrap up the presentation by talking about possible future enhancements and direction where it can drive us:
First they need to work on the documentation, but I think it is a matter of time.
The more important thing is that we finally treating Infrastructure as Code.
That gives us interesting capabilities and opens new doors for future.
A few years back nobody thought that we will configure CI/CD deployment pipelines alongside with the code - we may observe similar trend thanks to such solutions.
Additionally starting the discussion about abstraction allows us to think about real solutions for abstracting cloud providers.
I think that such frameworks are the first step for that.
In future we may observe trends and moves related with the type-safe AWS Infrastructure or describing infrastructure needed for serverless applications in the same language we write the compute layer.
Here you can find most important resources that I have used for preparing the presentation.