Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Azure Kubernetes Service 2019 ふりかえり

1,195 views

Published on

年末企画

Published in: Technology
  • Be the first to comment

Azure Kubernetes Service 2019 ふりかえり

  1. 1. Azure Kubernetes Service 真壁 徹 日本マイクロソフト株式会社 クラウドソリューションアーキテクト 2019/12/4 2019年を ふりかえる
  2. 2. 自己紹介 apiVersion: selfIntroduction/v1 name: “真壁 徹(まかべ とおる)” company: name: “日本マイクロソフト株式会社” role: “クラウド ソリューションアーキテクト” career: - name: “大和総研” - name: ”HP Enterprise” cert : “CNCF Certified Kubernetes Admin.”
  3. 3. AKS(Azure Kubernetes Service)の 2019年の機能拡充やトピックを おさらいしよう ※AKSの利用に大きく影響する他サービスも一部含みます (ACR、ACIなど)
  4. 4. Azure Kubernetes Service (AKS) 主なリリース 補足のないものはGAしてます API ServerのIPアドレス ホワイト リスト制御 APIサーバー 監査ログ Azure Monitor for containers Virtual Node in ACI ユーザー定義ルートのサポート Azure Arc による管理 (Preview) Kubernetes 1.12、1.13、1.14、 1.15(Preview) Availability Zones 複数Node Pool Cluster Autoscaler & VMSS Network Policy Japan East/West両リージョン対応
  5. 5. Azure Kubernetes Service (AKS) 主なリリース 補足のないものはGAしてます AKS Pod Security Policy (Preview) Azure PolicyとOpen Policy Agent の統合 (Preview) Azure Monitor for containers ラ イブデータ表示 (Preview) Azure Monitor Prometheus Metric スクレイピング (Preview) Standard Load Balancerサポート App Gateway Ingress Controller 証明書ローテーション Egress lockdown 対話型診断 Managed Identity 統合 (Preview) Windowsノード (Preview)
  6. 6. Azure Container Registry (ACR) 主なリリース 補足のないものはGAしてます リポジトリスコープ RBACサポー ト (Preview) 監査と診断ログ (Preview) Azure Security Centerによるイ メージの脆弱性スキャン (Preview) 署名済みイメージのサポート ACR Tasks スケジューリング ACR in VNet (Preview) Helm Chartリポジトリ (Preview) ACR Tasks Cloud Native Buildpack サポート (Preview)
  7. 7. 今後のプランは? 公開ロードマップをご覧下さい GitHubに公開してます プライベートクラスター ノード自己修復 ノード自動アップグレード Low Priority ノードプール などなど
  8. 8. そのほかリリース マイクロソフトがリード/関与が強い CNCFプロジェクトなど Kubernetes Confidential computing Distributed Application Runtime (Dapr) Cloud Native Application Bundle (CNAB) with Brigade KEDA (Kubernetes-based Event Driven Autoscaling) GitHub Actions for deploying to Kubernetes service Service Mesh Interface Helm 3
  9. 9. 2019年末時点での AKS ベストプラクティス
  10. 10. 公式ベストプラクティス Azureの製品グループだけでなく、 ユーザーに接しているメンバーが その知見をコンテンツとして提供 手を動かす前に目を通してもよ し、振り返り/改善のチェックに読 むもよし まずは ここ を見よう
  11. 11. Ignite 2019 Breakout Session BRK4006 Applying best practices to Azure Kubernetes Service (AKS) AKSを本番で使うなら必見 高可用性 バックアップ&リストア マルチクラスター & マルチリー ジョン アップグレード などなど
  12. 12. 不安定な時はノードのディスク性能を疑う kubeletやtunnnelfrontなど、ノード上にはアプリ以外にも重要コンポーネントが動いている IOPSの高いディスクにすると安定することが とても とても 多い PodのResource Limitも重要 暴れん坊を抑え込もう 特にメモリ 周辺リソースと合わせてInfrastructure as Code化する アップグレードや新機能の検証など、運用しているとクラスターを作る機会は多い Kubernetesだけで完結するシステムは稀 データストアやネットワーク系サービスなど周辺リソースも合わせて作れるようにする TerraformでもARM Templateでも お好みで 俺のベストプラクティス
  13. 13. おかしいな、という時はGitHubのIssueを検索する サポート窓口ではありません が、PMがIssueを見て対応しています ズバリな解決策が見つかることも とはいえ サポート要求はサポート窓口、新機能リクエストはAzure Feedbackへ 話題の機能/OSSだからといって あれもこれも使う必要はない 運用するなら腹落ちしたもの、ビジネス目標に合うものを選んで使いましょう 華やかな 事例の裏には 生存者バイアスが みなぎらないなら その時を待つ どんどん進化するので 追従には組織としてのモチベーションが重要です 流行ってるっぽいから、誰かにやれって言われたから、では続きません AKS/Kubernetesの代替方式は多くあります (App Service、Functions、ACI、etc) 俺のベストプラクティス
  14. 14. ここからは 鮮度重視で 英語です あっ 5 7 5
  15. 15. 主要リリース/機能の 図解とポイント
  16. 16. Additional nodes needed Pods are in pending state Cluster Autoscaler Pod Pod Node Pod Pod Node Pod Pod AKS cluster Cluster AutoscalerAzure Node is granted Pending pods are scheduled The cluster autoscaler watches for pods that can't be scheduled on nodes because of resource constraints. The cluster then automatically increases the number of nodes. 1. HPA obtains resource metrics and compares them to user-specified threshold 2. HPA evaluates whether user specified threshold is met or not 3. HPA increases/decreases the replicas based on the specified threshold 4. The Deployment controller adjusts the deployment based on increase/decrease in replicas
  17. 17. • Elastically provision compute capacity in seconds • No infrastructure to manage • Built on open sourced Virtual Kubelet technology, donated to the Cloud Native Computing Foundation (CNCF) Serverless Kubernetes using AKS virtual nodes Node Pods Node Pods Kubernetes control plane Azure Container Instances (ACI) Pods Virtual node
  18. 18. Application Gateway Ingress Controller • Application Gateways as Ingress for AKS • Deployed using Helm • Utilizes pod-AAD for ARM authentication. • Tighter integration with AKS add on support upcoming • Supports URI path based, host based, SSL termination, SSL re-encryption, redirection, custom health probes, draining, cookie affinity • Support for Let’s Encrypt provided TLS certificates • WAF fully supported with custom listener policies • Support for multiple AKS as backend • Support for mixed mode – both AKS and other backend types on the same Application Gateway! Application Gateway Azure ARM Azure Key Vault Azure Kubernetes Services (AKS) API server AGIngressController Ingressresource Pods Configure routing rules
  19. 19. Accelerate containerized development Kubernetes and DevOps better together Develop • Native containers and Kubernetes support in IDE • Remote debugging and iteration for multi- containers • Effective code merge • Automatic containerization Deliver • CI/CD pipeline with automated tasks in a few clicks • Pre-configured canary deployment strategy • In depth build and delivery process review and integration testing • Private registry with Helm support Operate • Out-of-box control plane telemetry, log aggregation, and container health • Declarative resource management • Auto scaling Inner loop Test Debug Azure DevSpaces AKS dev cluster CI/CD Pipelines GitHub repos Azure Container Registry Helm chart Container image AKS production cluster Azure Monitor Scale Terraform Develop Deliver Operate Boards
  20. 20. GitHub Actions for Kubernetes on Azure 1. Authenticate and login securely to an Azure subscription 2. Set the target AKS cluster 3. Create Kubernetes secret objects to manage sensitive information 4. Connect to the Kubernetes cluster and deploy manifests, etc. Action docker-login Action k8s-create-secret Action aks-set-context Action k8s-deploy
  21. 21. Pull Request flow in Dev Spaces 1. John is working out of branch “feature-x” locally 2. John commits his code and pushes his branch to his remote GitHub repo 3. John creates a pull request before merging the changes into the application’s main branch 4. GitHub Actions workflow is triggered upon PR creation; a delta namespace for the pull request is created and the code is deployed to the namespace Source code control master namespace feature-x namespace GitHub Actions workflow builds and deploys feature-x John Developer Lisa Reviewer Open pull request, deploy feature-branch Pull request merged, master updated PR namespace created, changes deployed Azure Dev Spaces + AKS cluster 5. A team member reviews the changes in the context of the entire application 6. The pull request is approved and a GitHub workflow is triggered to update the master namespace with the merged code changes
  22. 22. AKS with RBAC Storage SQL Database Cosmos DB VNet Node Node Pod Pod AAD Pod Identity Key Vault Active Directory Active Directory Identity Use familiar tools like AAD for fine-grained identity and access control to Kubernetes resources from cluster to containers
  23. 23. 1. Kubernetes operator defines an identity map for K8s service accounts AAD Pod identity 2. Node Managed Identity (NMI) watches for mapping reaction and syncs to Managed Service Identify (MSI) 3. Developer creates a pod with a service account, and pod uses standard Azure SDK to fetch a token bound to MSI 4. Pod uses access token to consume other Azure services; services validate token Kubernetes Kubernetes controller Azure MSI Azure Identity Binding Active Directory Pod Identity NMI + EMSI Pod Token Azure SQL Server Developer <¥>
  24. 24. 1. Cloud architect assigns a deployment policy across cluster(s) 2. Developer uses standard Kubernetes API to deploy to the cluster 3. Real-time deployment enforcement (acceptance/denial) provided to developer based on policy 4. Cloud architect obtains compliance report for the entire environment and can drill down to individual pod level Azure Policy for clusters (OPA Integration) Cloud Architect Developer Cluster-1 Cluster-2 Cluster-3 AKS Azure Policy Cluster-3  Cluster-2Cluster-1 Compliance reports
  25. 25. 2. Node and cluster level security • Automatic security patching nightly • Nodes deployed in private virtual network subnet w/o public addresses • Network policy to secure communication paths between namespaces (and nodes) • Pod Security Policies using Gatekeeper • K8s RBAC and AAD for authentication • Threat protection on nodes AKS with RBAC Security overview 1. Image and container level security • AAD authenticated Container registry access • ACR image scanning and content trust for image validation 3. Pod level security • Pod level control using AAD Pod Identity • Pod Security Context 4. Workload level security • Azure Role-based Access Control (RBAC) & security policy groups • Secure access to resources & services (e.g. Azure Key Vault) via Pod Identity • Storage Encryption • App Gateway with WAF to protect against threats and intrusions Developer Azure Container Registry Kubernetes Admin Azure Storage SQL Database Cosmos DB Internal User Internal Load Balancer External User External Load Balancer Azure VNet Node Node Pod Pod AAD Pod Identity Ingress Controller Encrypted Storage Azure Key Vault Ingress Controller App Gateway External DNS Active Directory
  26. 26. AKS Support in Azure Security Center 1. For managed subscriptions, each new AKS cluster and node are discovered in ASC 2. ASC monitors AKS cluster for security misconfigurations and provides actionable recommendations for compliance with security best practices 3. ASC continuously analyzes AKS for potential threats based on: a. Raw security events such as network data and process creation b. Kubernetes log audit Azure Security Center Continuous discovery of managed AKS instances Actionable recommendations for security best practices Detect threats across AKS nodes and clusters using advanced analytics Azure Kubernetes Service AKS security configuration API Server Master Workers Node1 Container runtime Security center Node2 Container runtime Security center Node3 Container runtime Security center Verified by Security Center Audit log Raw security events …and reports any threats and malicious activity detected (e.g., “API requests to your cluster from a suspicious IP was detected”)
  27. 27. Threat protection Automated threat detection and best practices recommendation for Kubernetes clusters using advanced analytics from Azure Security Center Cluster Cluster Cluster Azure Security Center Continuous discovery of managed AKS instances Actionable recommendations for security best practices Detect threats across AKS nodes and clusters using advanced analytics Azure Kubernetes Service
  28. 28. Image Security Developer CI/CD Pipelines Azure Container Registry Azure Kubernetes ServiceImage scanning Fail Pass Your private registry, with built-in Helm chart support, only deploys validated images and can be automatically geo-replicated to the data center close to where your users are Vulnerability scanning Actionable recommendations Admin
  29. 29. Secure network communications with VNET and CNI AKS subnet Backend services subnet Azure VNet A On-premises infrastructure Enterprise system Other peered VNets VNet peering Azure Express RouteAKS cluster SQL Server 1. Uses Azure subnet for both your containers and cluster VMs 2. Allows for connectivity to existing Azure services in the same VNet AKS VNet integration works seamlessly with your existing network infrastructure 3. Use Express Route to connect to on- premises infrastructure 4. Use VNet peering to connect to other VNets Service Endpoint Azure SQL PaaS DB 5. Connect AKS cluster securely and privately to other Azure resources using VNet endpoints
  30. 30. 1. A developer authenticates to the AAD token issuance endpoint and requests an access token Identity and access management through AAD and RBAC Azure delivers a streamlined identity and access management solution with Azure Active Directory (AAD) and Azure Kubernetes Services (AKS) 2. The AAD token issuance endpoint issues the access token 3. The access token is used to authenticate to the secured resource 4. Data from the secured resource is returned to the web application AKS Azure Active Directory Token Token Developer
  31. 31. Azure Pipelines build audit & enforcement using Azure Policy 1. Cloud architect assigns a policy across clusters; policy can be set to block non- compliance (deny) or generate non- compliance warnings (audit) 2. Developer makes code change that kicks off a build on Azure Pipelines 3. Azure Pipelines evaluates the request for policy compliance 4. If policy is set to deny, Azure Pipelines rejects the build attempt if any non- compliance is identified 5. If policy is set to audit, a non-compliance event is logged and the build is allowed to proceed Cloud Architect Developer Cluster-1 Cluster-2 Cluster-3 AKS Azure Policy CI/CD Pipelines Pass Fail Deny policy </>Yes No Compliance check </>
  32. 32. Azure management experiences Azure Portal Azure CLI Azure SDK Hybrid Agent and Services: Azure Arc for Kubernetes - Components Azure Identity RBAC Policy Index Groups Etc. Azure Resource Manager Azure Container Registry Kubernetes K8s API server Cluster provisioning Cluster upgrade and patch management Cluster lifecycle management Cluster monitoring Administrative access K8s native tools Customer locations Config Service K8s Connect Service Source Repo GitOps Manager Config Agent Connect Agent
  33. 33. k8s cluster 1: security.yaml Cluster Admin 3: Arc operators 4: Kubernetes Cluster – Azure Arc 2: Policy Security Admin 6: config to cluster 8: get manifest from repo 7: Git Url 9: apply and enforce rules Azure Arc for Kubernetes - Workflow 5
  34. 34. Azure Monitor for containers Azure Monitor for containers Visualization Insights Monitor & analyze Response Native alerting with integration to issue management and ITSM tools Monitor and analyze Kubernetes and container deployment performance, events, health, and logs Provide insights with cluster health rollup view Visualize overall health and performance from cluster to containers with drilldowns and filters Cloud native experience for Azure Monitor with Prometheus integration Azure Kubernetes Service Azure Pipelines Observability Observe live container logs and Kubernetes event log on container deployment status Virtual node Prometheus 1. Get detailed insights about your workloads with Azure Monitor 3. See graphical insights about clusters 2. Filter for details about nodes, controllers, and containers 4. Pull events and logs for detailed activity analysis
  35. 35. 1. Deploy Azure Arc for Kubernetes agent 2. Azure Arc agent registers cluster with ARM 3. Cluster operator applies cluster configuration via ARM 4. Configuration agent picks up configuration and syncs state from git repo 5. Configuration agent informs Azure policy of status 6. Cluster operator or application developer pushes changes via GitHub Cluster Connect RP Cluster operator Azure Resource Manager Cluster Config RP Azure Policy GitHub Config agent Azure Arc agent Cluster operator/ Application dev Azure Monitor for containers Configuration management scenario Kubernetes on-prem
  36. 36. AKS Diagnostics Azure backend telemetry Node 1 Node 2 AKS production cluster User Azure portal Zero configuration and zero cost Intelligent detectors based on AKS-specific telemetry Cluster-specific observations Recommended actions for troubleshooting <¥>Cluster Insights Cluster Node Issues Node Issues Detected Node Insufficient Resources Detected Create, Read, Update & Delete Operations Identity and Security Management AKS Diagnostics Sample diagnostics web portal ! ! !   x An interactive and intelligent experience for self-troubleshooting your app issues Diagnose and guide you through each problem with best practices recommendations Intelligent search capabilities to help you find right answers fast Straight out-of-the box, no extra configuration necessary
  37. 37. Open-source component jointly built by Microsoft and RedHat • Event-driven container creation & scaling Allows containers to “scale to zero” until an event comes in, which will then create the container and process the event, resulting in more efficient utilization and reduced costs • Native triggers support Containers can consume events directly from the event source, instead of routing events through HTTP • Can be used in any Kubernetes service This includes in the cloud (e.g., AKS, EKS, GKE, etc.) or on-premises with OpenShift—any Kubernetes workload that requires scaling by events instead of traditional CPU or memory scaling can leverage this component. Kubernetes-based event-driven auto-scaling (KEDA) Kubernetes cluster External trigger source KEDA AKS clusterScaler Controller Metrics adapter
  38. 38. SMI defines a set of APIs that can be implemented by individual mesh providers. Service meshes and tools can either integrate directly with SMI or an adapter can consume SMI and drive native mesh APIs. • Standard interface for service mesh on Kubernetes • Basic feature set to address most common scenarios • Extensible to support new features as they become widely available Service Mesh Interface (SMI) Apps Tooling Ecosystem …and more Service Mesh Interface Routing Telemetry Policy Kubernetes
  39. 39. ユーザー事例と そのアーキテクチャー
  40. 40. Bosch Increases Vehicle Safety Using Precision GPS Algorithms and Azure Kubernetes Service Challenge: Bosch designed a software development kit (SDK) that can be used by original equipment manufacturers (OEMs) to embed driving safety information at scale. For such a service to work commercially, they had to build a real- time data ingestion and processing pipeline capable of detecting hazards and notifying drivers within seconds Solution: The solution is deployed as multiple microservices running in containers behind an Azure API Management gateway. AKS provided the simplicity a serverless Kubernetes experience that provided the elastic provisioning they wanted without the need to manage the infrastructure. Outcome: By running their solution, which has been downloaded by 12 million users, on Azure and AKS, the average time to detect driving hazards dropped to approximately 60 milliseconds. What we like about AKS is the simplified Kubernetes experience. It's click and deploy, it’s click and scale. It’s infrastructure as code too, which is quite cool for us.” — Christian Jeschke, Product Owner, Bosch Click icon to learn more “
  41. 41. Bosch: architecture 1. Sensor data is generated and streamed to Azure API Management 2. AKS cluster runs microservice that are deployed as containers behind a service mesh; containers are built using a DevOps process and stored in Azure Container Registry 3. Ingest service stores data in an Azure Cosmos DB and other data storage destinations 4. Asynchronously, the map matching service receives the data from Kafka Streams on Azure HDInsight 5. Data is processed and stored the result in Azure Database for PostgreSQL and maps are continuously updated using Azure Databricks 6. A web app running in Azure App Service is used to visualize the results VNet Security Public API Key Vault SDK Hotspots WDW Service Blob Storage Web Apps ACR AKS Service Kafka Streams on HD Insights AKS Map matching Data Explorer Clusters Cosmos DB Cache for Redis PostgreSQL Server Databricks mVISE
  42. 42. Power grid operator uses containerized software to promote smart utility initiatives for 1.5M people Challenge: Legacy systems for reading meter data needed greater capacity to process large volumes of IoT data—but implementing the necessary system enhancements was difficult and expensive Solution: Hafslund chose to develop its own software for processing meter data. The company used Microsoft Azure as its cloud platform, AKS to manage software containers, and Azure Monitor for containers to optimize container performance. Outcome: Halfslund now has a standard way to create, monitor, scale, and manage applications, which means it can respond to customer needs faster. We wanted a platform to speed development and testing but do it safely, without losing control over security and performance. That’s why Azure and AKS are the perfect fit for us.” — Ståle Heitmann, Chief Technology Officer Hafslund Nett Click icon to learn more “
  43. 43. Hafslund Nett: architecture 1. Azure Pipelines automates container image build, push and release to Azure Kubernetes service, triggered by source code updates. 2. Azure Kubernetes Services provides the always-on service for meter reading and connects with Azure managed databases to process the massive amounts of data the IoT devices generate 3. Azure API Management serves as the secure gateway that helps connect to data and services anywhere. 4. Azure network and Active Directory provides fine-grain controls for external and inter-service communication. 5. Azure Monitor provides a single pane of glass for cluster-to-container monitoring. … Terraform Infrastructure AKS … Namespace…Namespace Namespace Halsfund Nett CSS Styles Express Routes Virtual Network Table Storage GitHub VM Active Directory Key Vault … Application Insights Log Analytics Cosmos DB SQL Server Azure Search Container Monitor On-prem services VM VM VM Load Balancer Internal ACR DevOps Load Balancer External API Mgmt.
  44. 44. DNV GL scales up machine learning using Azure Kubernetes Service Challenge: Initially, the group trained machine learning models locally and deployed each application to Azure Virtual Machines. This process took up to 2 weeks and consumed more Azure resources than needed. Solution: DNV GL created a service using that builds and deploys each machine learning application as a container on AKS. They’re able to use the Kubernetes Cluster Autoscaler to add resources on demand as the need for more compute power arises. Outcome: Data scientists and developers at DNV GL can now deliver more solutions to their internal and external customers with more speed, for less money, and with a more elastic software stack. Now the data scientists and engineers at DNV GL can focus on developing new, predictive solutions and providing real business value. Click icon to learn more We decided to address the friction areas of our internal company deployment, management, and operations, and after evaluating commercial offerings, we chose to develop ML Factory based on Azure services.” — Kristian Ramsrud, Machine Learning group DNV GL Maritime “
  45. 45. DNV GL: architecture 1. Data scientists create their machine learning applications as containers using the ML Factory development tools 2. ML Apps are built using automatically using Azure Container Registry Tasks and are deployed to Azure Kubernetes Service 3. Realtime logs can be streamed directly for debugging purposes. Azure Log Analytics also provides access to historical logs within defined retention periods 4. As the data flows through the platform, multiple functions hosted in Azure Functions work together to fire alerts or trigger actions, triggered by signals from Azure Event Grid 5. Published applications are automatically added to the company’s corporate API Management gateway and the internal API catalog ML development and monitoring Support components ML Factory Event Grid Function Apps ML Factory Developer tools Active Directory Blob Storage API Management Key Vault ML Factory Web portal AKS SQL Server Storage Accounts App Service API Gateway Consuming applications ACR
  46. 46. Click icon to learn more Maersk uses AKS for a customer service process to elevate NSAT, an industry-wide challenge Needs: Get near-real-time data to provide better customer service Collect data for future Machine Learning driven features Challenges: Compute & memory intensive features Data integration difficulties Limited organisational experience in Cloud & Kubernetes Requirements: Spend less time on container software management Automation and continuous delivery Full visibility to application, container and infrastructure Fine grained security and access control Outcomes: Reduced environment provisioning time from 1+ weeks to 2.5 hours AKS and CaaS can potentially save 33% on run cost Using Kubernetes on Azure satisfies our objectives for efficient software development. It aligns well with our digital plans and our choice of open-source solutions for specific programming languages.” “ — Rasmus Hald, Head of Cloud Architecture, A.P. Moller - Maersk
  47. 47. 1. Azure Pipelines for automation and CI/CD pipelines; adding Terraform for further automation 2. Key Vault to secure secrets and for persistent configuration store 3. Azure Monitor for containers to provide better logging, troubleshooting, with no direct container access 4. RBAC control for fine grained Kubernetes resources access control Firewall App Gateway AKS w/ RBAC Azure Monitor App Insights SQL Database Cosmos DB Performance Document DB Key Vault Vault Event Hub Batch processing Event Simulation Data Factory Data Management Gateway On-premises database Express Route Service Bus Internal Queuing SQL Database Azure Pipelines Maersk: architecture
  48. 48. © Copyright Microsoft Corporation. All rights reserved.

×