Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

New features of Azure Cloud Provider in OpenShift Container Platform 3.10

66 views

Published on

These slides are presented at London OpenShift meetup in August 2018.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

New features of Azure Cloud Provider in OpenShift Container Platform 3.10

  1. 1. New features of Azure Cloud Provider in OpenShift 3.10 Takayoshi Tanaka, Software Maintenance Engineer tatanaka@redhat.com
  2. 2. Takeaways Let you know Azure Cloud Provider is getting more and more useful. Let you know some upstream project/feature can work on OpenShift (however, without support)
  3. 3. Agenda What’s Azure Cloud Provider in OpenShift Big Changes for Azure Cloud Provider Experimental Features
  4. 4. Azure Cloud Provider Concepts Underlying the Cloud Controller Manager
  5. 5. Azure Cloud Provider kubernetes: ◦integrating Cloud features ◦Azure is one of the implementations OpenShift ◦includes kubernetes = almost all Cloud Provider features could work ◦Our tested features is describes in product documents
  6. 6. OpenShift 3.10 on Azure includes kubernetes 1.10 – actually 1.10.1 + α kubernetes now supports, but OpenShift doesn’t: ◦ cluster auto scale ◦ https://github.com/feiskyer/autoscaler/blob/master/cluster- autoscaler/cloudprovider/azure/README.md ◦ Azure VM Scale Set Private Preview in OpenShift in the future ◦ Windows Container Support ◦ If you have an interest, contact Red Hat sales in charge of you, or me.
  7. 7. Red Hat OpenShift on Azure https://www.redhat.com/en/about/press-releases/red-hat-and-microsoft-co- develop-first-red-hat-openshift-jointly-managed-service-public-cloud https://azure.microsoft.com/en-gb/blog/openshift-on-azure-the-easiest-fully-managed- openshift-in-the-cloud/ https://www.youtube.com/watch?v=WDr9TSyK9DA
  8. 8. Looking back the announcment 10000 feet overview ◦ Launch OpenShift cluster with one command ◦ First-class citizen in Azure ◦ Including Open Service Broker for Azure If you have an interest, please contact me later. I'll pass your info to our team. $ az openshift create -n oscluster -g osrg ` –node-vm-size Standard_DS4_v3` --l eastus
  9. 9. Before starting… Requirements for Azure Cloud Provider Virtual Machines (VMs) must belong to a single resource group & VNET ◦ Some improvements are ongoing VM name requirement ➔ Next page Internal Load Balancer is unavailable for master public end point OCP 3.7 or later is required for Managed Disk Service Type: LoadBalancer supports only Basic Azure Load Balancer ◦ Node VMs must belong to Availability Set (* I’ll explain later) Sovereign Cloud (Azure Germany, China etc) are available. ◦ Some extra configurations are required VM scale set is not tested in OCP
  10. 10. Node Name = VM name: DO NOT include capital letters Be sure these values are same ◦ Azure VM name: az vm create –n <name> ◦ Internal DNS Name: az nic create --internal-dns-name <name> ◦ Hostname (DO not edit manually) ◦ OpenShift node name (inventory file) This name should not include capital letters ◦ OK: ocp-node1 ◦ NG: OCP-Node1
  11. 11. Big Changes
  12. 12. Big Changes Installer Improvement Upgrade Issue (3.9 to 3.10) Azure Disk/File Mount options Azure File default permission changed to 0755 Improved Azure Disk mount/unmount performance
  13. 13. Installer Improvement: Internal Hostname in Azure NIC Be sure to set internal hostname in the Azure NIC. - Create a new NIC - Apply to the existing NIC $ az network nic create -g MyResourceGroup --vnet- name MyVnet --subnet MySubnet -n MyNic --internal- dns-name node1 $ az network nic update -g MyResourceGroup - n MyNic --internal-dns-name node1
  14. 14. Installer Improvement: Support creating azure.conf Advanced installer creates azure.conf. Note: If you need more parameters such as VNET or security group, you have to add them to azure.conf and restart services manually. openshift_cloudprovider_kind=azure openshift_cloudprovider_azure_client_id=<AAD_app_id> openshift_cloudprovider_azure_client_secret=<AAD_password> openshift_cloudprovider_azure_tenant_id=<AAD_tenant_id> openshift_cloudprovider_azure_subscription_id=<subscription_id> openshift_cloudprovider_azure_resource_group=<resource_group_name> openshift_cloudprovider_azure_location=<location>
  15. 15. How to create a service principal? Execute the following three commands. If you’re using Azure Cloud Shell, try the following script. https://gist.github.com/tanaka- takayoshi/802fcfbd8d544495977c2e8660604bc3 $ az account show -o json $ az group show -n <resource_group_name> -o json $ az ad sp create-for-rbac --name <app_name> --role contributor --scopes <resource_group_id> -o json
  16. 16. Installer Improvement: Support creating storage class Advanced installer creates storage class for Azure Disk/File. openshift_storageclass_name=slow-disk openshift_storageclass_provisioner=azure-disk openshift_storageclass_parameters={'storageaccounttype' : 'Standard_LRS', 'kind': 'Managed'}
  17. 17. Upgrade Issue (3.9 to 3.10): relocate azure.conf OpenShift 3.10 doesn’t allow to locate azure.conf under /etc/azure. Use /etc/origin/cloudprovider instead of /etc/azure. Before upgrading the cluster, please copy the file and edit the master and node configuration files.
  18. 18. Azure Disk/File Mount options You can specify mount options for Azure Disk and Azure File as a Volume. Storage Class and Persistent Volume has a mountOptions parameters. apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: slow mountOptions: - dir_mode=0777 - file_mode=0777 parameters: kind: Managed storageaccounttype: Standard_LRS provisioner: kubernetes.io/azure-disk reclaimPolicy: Delete apiVersion: v1 kind: PersistentVolume metadata: name: static-pv-file spec: accessModes: - ReadWriteOnce azureFile: secretName: azure-secret secretNamespace: openshift shareName: share1 capacity: storage: 1Gi mountOptions: - dir_mode=0777 - file_mode=0777 persistentVolumeReclaimPolicy: Delete
  19. 19. Azure File default permission changed to 0755 The permission changed from 0777 to 0755. (at OCP 3.9 or newer) The owner of the directory is different from the UID of container by default. ◦ https://docs.openshift.com/container- platform/3.10/install_config/persistent_storage/persistent_storage_azure_file.html#azu re-file-before-you-begin Configure mountOptions to ensure the permission. Use SecurityContext to run a container by specific GID. (OCP 3.10) Use SecurityContext (RunAsUser) & mountOptions (uid) to run a container by specific UID.
  20. 20. Improved Azure Disk mount/unmount performance It takes huge time to unmount and mount an Azure Disk when a new version of pod is located another node. It could happen when a new version of pod is located to a different node than the node where a current version of pod is running. pod pod
  21. 21. Experimental Features RED HAT DOESN’T TESTED THESE FEATURES.
  22. 22. Experimental Features Advanced Options for ServiceType: LoadBalancer Azure REST API authentication Monitoring
  23. 23. Service Type: LoadBalancer Node Virtual Machines apiVersion: v1 kind: Service metadata: name: alb-svc spec: loadBalancerIP: 40.121.183.52 type: LoadBalancer ports: - port: 80 selector: app: my-app IP should be created in advance. Or you don’t have to specify the ip.
  24. 24. Advanced Options Internal Load Balancer Internal Load Balancer with subnet DNS Label name https://github.com/kubernetes/cloud-provider-azure/blob/master/docs/development- loadbalancer.md annotations: service.beta.kubernetes.io/azure-load-balancer-internal: "true" annotations: service.beta.kubernetes.io/azure-load-balancer-internal: "true" service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "apps-subnet" annotations: service.beta.kubernetes.io/azure-dns-label-name: ""
  25. 25. Service Type: LoadBalancer for huge cluster One Availability Set has maximum 200 VMs. If you deploy over 200 OpenShift nodes, you have to deploy two or more AvailabilitySet. One ALB can have only one AvailabilitySet. So, you have to deploy one ALB per AvailabilitySet. Node VMs – group A Node VMs – group B annotations: service.beta.kubernetes.io/azure-load-balancer-mode: “group-a,group-b"
  26. 26. Azure REST API authentication: Managed Service Identity Managed Service Identity: ◦ You don’t have to provide password to configuration file ◦ https://docs.microsoft.com/en-us/azure/active-directory/managed-service- identity/overview Steps ◦ Enable MSI on all VMs for OpenShift nodes ◦ Grant your VM access to a Resource Group ◦ Update azure.conf useManagedIdentityExtension: true aadClientId: aadClientSecret:  Remove two lines  Add this line
  27. 27. Monitoring - Prometheus Prometheus monitors Azure API Request Duration Seconds & Error. ◦ Prometheus is Technology Preview at 3.10

×