1. New features of
Azure Cloud Provider
in OpenShift 3.10
Takayoshi Tanaka, Software Maintenance Engineer
tatanaka@redhat.com

2. Takeaways
Let you know Azure Cloud Provider is getting more and
more useful.
Let you know some upstream project/feature can work on
OpenShift (however, without support)

3. Agenda
What’s Azure Cloud Provider in OpenShift
Big Changes for Azure Cloud Provider
Experimental Features

4. Azure Cloud Provider
Concepts Underlying the Cloud Controller Manager

5. Azure Cloud Provider
kubernetes:
◦integrating Cloud features
◦Azure is one of the implementations
OpenShift
◦includes kubernetes
= almost all Cloud Provider features could work
◦Our tested features is describes in product documents

6. OpenShift 3.10 on Azure
includes kubernetes 1.10 – actually 1.10.1 + α
kubernetes now supports, but OpenShift doesn’t:
◦ cluster auto scale
◦ https://github.com/feiskyer/autoscaler/blob/master/cluster-
autoscaler/cloudprovider/azure/README.md
◦ Azure VM Scale Set
Private Preview in OpenShift in the future
◦ Windows Container Support
◦ If you have an interest, contact Red Hat sales in charge of you, or me.

7. Red Hat OpenShift on Azure
https://www.redhat.com/en/about/press-releases/red-hat-and-microsoft-co-
develop-first-red-hat-openshift-jointly-managed-service-public-cloud
https://azure.microsoft.com/en-gb/blog/openshift-on-azure-the-easiest-fully-managed-
openshift-in-the-cloud/
https://www.youtube.com/watch?v=WDr9TSyK9DA

8. Looking back the announcment
10000 feet overview
◦ Launch OpenShift cluster with one command
◦ First-class citizen in Azure
◦ Including Open Service Broker for Azure
If you have an interest, please contact me later. I'll pass your info to our team.
$ az openshift create -n oscluster -g osrg `
–node-vm-size Standard_DS4_v3`
--l eastus

10. Before starting…
Requirements for Azure Cloud Provider
Virtual Machines (VMs) must belong to a single resource group & VNET
◦ Some improvements are ongoing
VM name requirement ➔ Next page
Internal Load Balancer is unavailable for master public end point
OCP 3.7 or later is required for Managed Disk
Service Type: LoadBalancer supports only Basic Azure Load Balancer
◦ Node VMs must belong to Availability Set (* I’ll explain later)
Sovereign Cloud (Azure Germany, China etc) are available.
◦ Some extra configurations are required
VM scale set is not tested in OCP

11. Node Name = VM name:
DO NOT include capital letters
Be sure these values are same
◦ Azure VM name: az vm create –n <name>
◦ Internal DNS Name: az nic create --internal-dns-name <name>
◦ Hostname (DO not edit manually)
◦ OpenShift node name (inventory file)
This name should not include capital letters
◦ OK: ocp-node1
◦ NG: OCP-Node1

12. Big Changes

13. Big Changes
Installer Improvement
Upgrade Issue (3.9 to 3.10)
Azure Disk/File Mount options
Azure File default permission changed to 0755
Improved Azure Disk mount/unmount performance

14. Installer Improvement:
Internal Hostname in Azure NIC
Be sure to set internal hostname in the Azure NIC.
- Create a new NIC
- Apply to the existing NIC
$ az network nic create -g MyResourceGroup --vnet-
name MyVnet --subnet MySubnet -n MyNic --internal-
dns-name node1
$ az network nic update -g MyResourceGroup -
n MyNic --internal-dns-name node1

15. Installer Improvement:
Support creating azure.conf
Advanced installer creates azure.conf.
Note: If you need more parameters such as VNET or security group,
you have to add them to azure.conf and restart services manually.
openshift_cloudprovider_kind=azure
openshift_cloudprovider_azure_client_id=<AAD_app_id>
openshift_cloudprovider_azure_client_secret=<AAD_password>
openshift_cloudprovider_azure_tenant_id=<AAD_tenant_id>
openshift_cloudprovider_azure_subscription_id=<subscription_id>
openshift_cloudprovider_azure_resource_group=<resource_group_name>
openshift_cloudprovider_azure_location=<location>

16. How to create a service principal?
Execute the following three commands.
If you’re using Azure Cloud Shell, try the following script.
https://gist.github.com/tanaka-
takayoshi/802fcfbd8d544495977c2e8660604bc3
$ az account show -o json
$ az group show -n <resource_group_name> -o json
$ az ad sp create-for-rbac --name <app_name> --role
contributor --scopes <resource_group_id> -o json

17. Installer Improvement:
Support creating storage class
Advanced installer creates storage class for Azure Disk/File.
openshift_storageclass_name=slow-disk
openshift_storageclass_provisioner=azure-disk
openshift_storageclass_parameters={'storageaccounttype'
: 'Standard_LRS', 'kind': 'Managed'}

18. Upgrade Issue (3.9 to 3.10):
relocate azure.conf
OpenShift 3.10 doesn’t allow to locate azure.conf under /etc/azure.
Use /etc/origin/cloudprovider instead of /etc/azure.
Before upgrading the cluster, please copy the file and edit the master and
node configuration files.

19. Azure Disk/File Mount options
You can specify mount options for Azure Disk and Azure File as a Volume.
Storage Class and Persistent Volume has a mountOptions parameters.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: slow
mountOptions:
- dir_mode=0777
- file_mode=0777
parameters:
kind: Managed
storageaccounttype: Standard_LRS
provisioner: kubernetes.io/azure-disk
reclaimPolicy: Delete
apiVersion: v1
kind: PersistentVolume
metadata:
name: static-pv-file
spec:
accessModes:
- ReadWriteOnce
azureFile:
secretName: azure-secret
secretNamespace: openshift
shareName: share1
capacity:
storage: 1Gi
mountOptions:
- dir_mode=0777
- file_mode=0777
persistentVolumeReclaimPolicy: Delete

20. Azure File default permission
changed to 0755
The permission changed from 0777 to 0755. (at OCP 3.9 or newer)
The owner of the directory is different from the UID of container by default.
◦ https://docs.openshift.com/container-
platform/3.10/install_config/persistent_storage/persistent_storage_azure_file.html#azu
re-file-before-you-begin
Configure mountOptions to ensure the permission.
Use SecurityContext to run a container by specific GID. (OCP 3.10)
Use SecurityContext (RunAsUser) & mountOptions (uid) to run a
container by specific UID.

21. Improved Azure Disk
mount/unmount performance
It takes huge time to unmount and mount an Azure Disk when a new version of pod is
located another node.
It could happen when a new version of pod is located to a different node than the node
where a current version of pod is running.
pod pod

22. Experimental
Features
RED HAT DOESN’T TESTED THESE FEATURES.

23. Experimental Features
Advanced Options for ServiceType: LoadBalancer
Azure REST API authentication
Monitoring

24. Service Type: LoadBalancer
Node Virtual Machines
apiVersion: v1
kind: Service
metadata:
name: alb-svc
spec:
loadBalancerIP: 40.121.183.52
type: LoadBalancer
ports:
- port: 80
selector:
app: my-app
IP should be created in advance.
Or you don’t have to specify the ip.

25. Advanced Options
Internal Load Balancer
Internal Load Balancer with subnet
DNS Label name
https://github.com/kubernetes/cloud-provider-azure/blob/master/docs/development-
loadbalancer.md
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "apps-subnet"
annotations:
service.beta.kubernetes.io/azure-dns-label-name: ""

26. Service Type: LoadBalancer
for huge cluster
One Availability Set has maximum 200 VMs.
If you deploy over 200 OpenShift nodes,
you have to deploy two or more
AvailabilitySet.
One ALB can have only one AvailabilitySet.
So, you have to deploy one ALB per
AvailabilitySet.
Node VMs – group A Node VMs – group B
annotations:
service.beta.kubernetes.io/azure-load-balancer-mode: “group-a,group-b"

27. Azure REST API authentication:
Managed Service Identity
Managed Service Identity:
◦ You don’t have to provide password to configuration file
◦ https://docs.microsoft.com/en-us/azure/active-directory/managed-service-
identity/overview
Steps
◦ Enable MSI on all VMs for OpenShift nodes
◦ Grant your VM access to a Resource Group
◦ Update azure.conf
useManagedIdentityExtension: true
aadClientId:
aadClientSecret:
 Remove two lines
 Add this line

28. Monitoring - Prometheus
Prometheus monitors Azure API Request Duration Seconds & Error.
◦ Prometheus is Technology Preview at 3.10