Azure Zürich User Group: Azure Kubernetes Service – more than just a managed Kubernetes

Azure Kubernetes Service – more than just a
managed Kubernetes
Microsoft Azure Zürich User Group, March 2022

Nico Meisenzahl
• Cloud Solution Architect at white duck
• Microsoft MVP, GitLab Hero
• Cloud Native, Kubernetes & Azure
© white duck GmbH 2022
Phone: +49 8031 230159 0
Email: nico.meisenzahl@whiteduck.de
Twitter: @nmeisenzahl
LinkedIn: https://www.linkedin.com/in/nicomeisenzahl
Blog: https://meisenzahl.org

Agenda
• Azure Kubernetes Service – a managed K8s
• AKS features (my picks)
• AKS add-ons & extensions
• further resources

AKS – A MANAGED K8S

Azure Kubernetes Service
“Deploy and scale containers on managed Kubernetes”
“Deploy and manage containerized applications more easily
with a fully managed Kubernetes service”
“Build on an enterprise-grade, more secure foundation”
https://azure.microsoft.com/services/kubernetes-service

A managed K8s, but …
• what you will get out of the box
• Kubernetes à great flexibility also introduces complexity!
• a fully managed control plane
• worker nodes you need to care about
• fully managed Kubernetes is possible
• not enabled by
• can cause issues (you must be ahead of all changes)
• addons / integrations required

Fast changing world
• AKS/Kubernetes is a fast changing world
• integrations/features evolve quickly and need to be
implemented on an ongoing basis
• fire and forget is not an option
• you will need a team to operate your clusters
• if you are not able to provide this, AKS/Kubernetes is not an
option for you à Azure Container Apps (preview) might help

That said, AKS …
• is the best choice if you require Kubernetes
• can help you a lot and make your life much easier
• perfectly integrated with other Azure services
• provides you with useful open-source integrations

AKS FEATURES (MY PICKS)

Private AKS
• expose API Server via Private Link into an internal subnet
• expose services into an internal subnet using internal Load
Balancer
• access Azure PaaS services via Private Link endpoints
• Container Registry
• Storage services (Storage Account, Databases, …)
• can introduce some complexity due to networking and DNS
• there will be an updated version (v2) in the future which reduces the
complexity

Azure AD integration
• assign IAM to Azure AD user's identity or directory group
membership
• integrated with the Azure Portal and CLI
• allows to disable local cluster-admin account
• can be assigned via Azure Roles or Kubernetes
Roles/RoleBindings
• support for Group Managed Service Accounts (GMSA) for
your Windows nodes (preview)
• https://docs.microsoft.com/azure/aks/managed-aad

Azure AD Pod Identity (preview)
• assigns Azure AD identities to Pods to leverage Azure
resource that depends on AAD as an identity provider
• e.g., securely talk with databases or Storage Accounts without
injecting secrets and connection strings
• no code changes required (relies on the default credentials)
• will not leave preview!
• the successor will be Azure AD Workload Identity
• same outcome, new implementation

Azure AD Workload Identity (preview)
• successor of Azure AD Pod Identity
• implements known-issues and learnings
• removes scale and performance issues
• supports Kubernetes clusters hosted in any cloud or on-
premises
• supports both Linux and Windows workloads
• removes the need for CRDs and pods that intercept Instance
Metadata Service (IMDS) traffic

Azure AD Workload Identity

Auto-upgrade & node upgrade
• AKS can automatically upgrade clusters and nodes
• there are different upgrade channels
• none, patch, stable, rapid, node-image
• https://docs.microsoft.com/azure/aks/upgrade-cluster#set-auto-upgrade-
channel
• manifests & API calls need to stay up-to-date for stable/rapid
• do not miss to define a maintenance windows (preview, currently
best-effort only)
• node auto-repair
• AKS automatically try to fix node issues if node is “NotReady”
• steps are reboot, reimage, recreate
• https://docs.microsoft.com/azure/aks/node-auto-repair

Autoscaling & Spot instances
• Cluster Autoscaler allows node
scalling (on a node pool level)
• support for Azure Spot VMs
(on a node pool level)
• take advantage of unused
capacity at a significant cost
savings
• Virtual Node interation via ACI

Integrated Storage
• AKS integrates with Azure Disk (incl. Ultra Disk) and
Azure Files
• REST and network based storage should be prefered
where possible
• stateless workload will make your life much easier
• Azure HPC Cache and NFS (Storage Account) can be
integrated via Kubernetes-native NFS
• Azure Backup for AKS PVs (private preview)

AKS and CSI
• Azure Disk and Azure Files are supported by CSI since AKS 1.21
• CSI (Container Storage Interface) is the future of storage integration and
will replace the in-tree implementation soon
• CSI brings you many advantages
• ZRS and ReadWriteMany support for Azure Disk
• Kubernetes-native integrations for Volume snapshots, resizing and cloning
• https://medium.com/01001101/azure-kubernetes-service-next-level-persistent-
storage-with-azure-disk-csi-driver-c5a04ac775c1
• you will have to migrate existing clusters to use CSI
• https://docs.microsoft.com/azure/aks/csi-storage-drivers#migrating-
custom-in-tree-storage-classes-to-csi

Azure Event Grid integration (preview)
• Azure Events Grid now supports AKS as a source
• allows to subscribing to AKS events for further integration
• preview, and early stage
• so far following events are supported
• new Kubernetes version upgrade availability
• new Node image version upgrade availability
• https://docs.microsoft.com/azure/aks/quickstart-event-grid

Microsoft Defender for Containers
• environment hardening
• provides visibility into misconfigurations and guidelines
• vulnerability assessment
• vulnerability assessment images after build, when stored in ACR
and running in AKS
• runtime protection
• threat protection for clusters and Linux nodes generates security
alerts for suspicious activities
• why?
• https://github.com/nmeisenzahl/hijack-kubernetes

• upgrade Defender if you previously used it to get the
latest features
• Microsoft Defender for Kubernetes
• Microsoft Defender for Containers Registries
• also supports non-Azure environments (via Azure Arc)
• Amazon Elastic Kubernetes Service (EKS)
• Google Kubernetes Engine (GKE)
• self-hosted CNCF-certified Kubernetes

Confidential computing
• allows you to protect your sensitive data while it's in use
• allow user-level as well as OS code to define/use private
regions of memory
• based on Intel SGX (Software Guard Extensions)
• requires DCsv2 VMs
• supporting confidential containers out of the box
• application is loaded in the trusted boundary (enclave)
• https://docs.microsoft.com/azure/defender-for-
cloud/defender-for-containers-introduction

Enclave aware containers
• are supported via the Open
Enclave SDK
• container development
involves untrusted and
trusted parts to the container
application

Uptime SLA
• AKS is available with two tiers
• free tier (default)
• fewer replicas and limited resources for the control plane
• paid tier packed by SLA
• guaranteeing 99.95% (99.9% for non-AZ)
• why?
• I have seen issues with free tier in “smaller” regions due to
lower prioritization of requests

AKS ADD-ONS & EXTENSIONS

Add-ons and Extenions
• add-ons and extensions allowing to extend/integrate AKS
with Azure services and open-source projects
• are integrated with the Azure Resource Manager
• easy to use

AKS Add-ons
• fully managed and supported by Azure
• fixes are applied automatically on a weekly basis
• minor/major changes are implemented via AKS updates
• part of the Azure RM AKS resource provider
• limited configuration options
• https://docs.microsoft.com/azure/aks/integrations#add-
ons

AKS Extenions
• relatively new with AKS
• still on preview
• already know concept from Azure Arc
• easy integration
• installation and lifecycle management via Azure tooling (API, CLI, …)
• build on top of Helm Charts (but abstracted)
• not managed nor automatically updated
• separate resource provider within the Azure RM
• therefore not yet supported in all IaC Tools (e.g. Terraform)
• https://docs.microsoft.com/azure/aks/cluster-extensions

Add-On: Container Insights
• entry point for logs and metrics & diagnostic data
• integrates with Azure Portal
• provides out-of-the-box workbooks and KQL queries
• supports Prometheus endpoint scrapping
• Azure Managed Grafana (currently private preview)
• integrates via AKS data source
• https://docs.microsoft.com/azure/azure-
monitor/containers/container-insights-overview

Add-On: Container Insights

Add-On: Virtual Node
• rapidly scale container workloads
• no cluster autoscaler / node
provisioning required
• can also be useful for batch/job
workload with special requirements
(e.g., GPU)
• https://docs.microsoft.com/azure/a
ks/virtual-nodes

Add-On: Azure Policy
• integrates AKS with Azure
Policies
• based on Open Policy Agent
Gatekeeper
• can be enforced or audited
• compliance across clusters

Add-On: Azure Policy
• use built-in definitions to base-level security
• pod security baseline standards for Linux-based workloads
• pod security restricted standards for Linux-based workloads
• apply custom policies for your use-cases (preview)
• https://docs.microsoft.com/azure/governance/policy/conce
pts/policy-for-kubernetes

Add-On: Application Gateway Ingress Controller
• integrates Azure Application Gateway as an ingress
controller (managed Ingress)

Add-On: Application Gateway Ingress Controller
• supports URL-based routing, cookie-based affinity, WAF,
end-to-end TLS, …
• TLS certificates can be served by Kubernetes secrets
(Cert-Manager)
• add-on is more limited than Helm deployment
• https://docs.microsoft.com/azure/application-
gateway/ingress-controller-overview

Add-On: HTTP Application Routing
• quick development option to spin up an Ingress Controller
• not intended for production
• spins up
• Nginx Ingress Controller
• External-DNS Controller (watching Ingress resources)
• Azure DNS Zone
• https://docs.microsoft.com/azure/aks/http-application-
routing

Add-On: Open Service Mesh
• managed service mesh based on Open Service Mesh
• lightweight service mesh implementing Service Mesh Interface
• helps you with
• service to service mTLS
• traffic shifting (A/B, canary)
• access control policies
• monitoring and instrumentation
• https://docs.microsoft.com/azure/aks/open-service-mesh-
about

Add-On: Azure Keyvault Secrets Provider
• inject secret, certificates and keys into
container workload without storing them
outside of Azure Key Vault
• based on Container Storage Interface
• injection is done via volumes
• can also be synced with Kubernetes
secrets (and then inject via env)
• https://docs.microsoft.com/azure/aks/csi-
secrets-store-driver

Extension: GitOps (preview)
• abstracted GitOps setup based on Flux
• already known from Azure Arc
• integrated via ARM à no need to ”talk” to K8s directly
• GitOps?
• check out Azure Rosenheim Meetup for further details
• https://github.com/whiteducksoftware/azure-meetup-rosenheim
• https://docs.microsoft.com/azure/azure-
arc/kubernetes/conceptual-gitops-flux2

Extension: Dapr (preview)
• a portable, event-driven, runtime for building distributed
applications across cloud and edge
• https://docs.microsoft.com/azure/aks/dapr

Extension: Azure ML (preview)
• use AKS to train, inference, and manage machine
learning models in Azure Machine Learning
• Azure ML extension will deploy an Azure Machine Learning
agent
• https://docs.microsoft.com/azure/machine-learning/how-
to-attach-arc-kubernetes

Extension: KEDA (preview soon)
• not yet available as extension
• Kubernetes event-driven autoscaling
• scale to zero
• scale based on various events
• scale-based on events from
• Application Insights, Azure Monitor
• Azure Blob, Azure Storage Queue
• Azure Event Hub, Azure Service Bus
• and many more

FURTHER RESOURCES

Get involved
• AKS office hours (bi-weekly call)
• https://github.com/Azure/aks-gbb-officehours
• AKS release notes
• https://github.com/Azure/AKS/releases
• AKS Roadmap
• https://github.com/Azure/AKS/projects/1
• Stack Overflow AKS tag
• https://stackoverflow.com/questions/tagged/azure-aks

More details
• AKS docs
• https://docs.microsoft.com/azure/aks
• AKS Reference Architecture
• https://docs.microsoft.com/azure/architecture/reference-
architectures/containers/aks-start-here
• AKS checklist
• https://www.the-aks-checklist.com

Questions?
• Slides: https://www.slideshare.net/nmeisenzahl
Phone: +49 8031 230159 0
Email: nico.meisenzahl@whiteduck.de
Twitter: @nmeisenzahl
LinkedIn: https://www.linkedin.com/in/nicomeisenzahl
Blog: https://meisenzahl.org

Azure Zürich User Group: Azure Kubernetes Service – more than just a managed Kubernetes

More Related Content

What's hot

Similar to Azure Zürich User Group: Azure Kubernetes Service – more than just a managed Kubernetes

More from Nico Meisenzahl

Recently uploaded

Azure Zürich User Group: Azure Kubernetes Service – more than just a managed Kubernetes