London OpenSift meetup Aug 14

1. Deep Dive
OpenShift on Azure
&
.NET Core on OpenShift
Takayoshi Tanaka @TanakaTakayoshi
Red Hat K.K. (Japan) tatanaka@redhat.com

2. Notes:
This slide is available online.
As I have tested at OCP 3.5 and .NET Core 2.0 preview2,
something will be changed at the latest OCP 3.6 and .NET Core 2.0
RTM.
If you have any question or comments, feel free to contact me:
email: tatanaka@redhat.com
Twitter: @TanakaTakayoshi

3. Background
Red Hat K.K. (Japan)
◦ Software Maintenance Engineer
◦ OpenShift
◦ Red Hat solutions on Azure
◦ .NET Core on RHEL
Personal
◦ Microsoft MVP for VSDT
◦ C# Lang, .NET Core on Linux
◦ Blogs:
◦ Red Hat Developers
◦ Personal Blog “Silver light and Blue sky”
VSDT: Visual Studio & Development Technologies

4. Goal
◦ Learn about OpenShift on Azure Reference Architecture
◦ How to integrate Azure Features with OpenShift
◦ .NET Core 2.0/integrating OpenShift features with
ASP.NET Core

6. Reference Architecture
Document is now available
◦ Deploying Red Hat OpenShift Container Platform 3 on
Microsoft Azure

8. OpenShift Ansible - Azure ARM Template

9. Installation Summary
1
• ARM Template for Azure Resources (VM, LB, NW…)
2
• Custom Script Extension with ARM
• generate config. files & execute ansible
3
• Ansible Installer for OpenShift

10. Known Issue
Available only in the Azure Marketplace VM
• duplicated billing. Custom image (.vhd) is on the roadmap.
No official Red Hat is available (self-support only)
• You should troubleshoot by yourself.
The OpenShift VM configuration is fixed
• 3 masters with etcd (same hosts), 3 infra nodes, 3+ nodes, 1 bastion

11. 構成図

12. 構成図

13. 構成図

14. 構成図

15. 構成図

16. VMs
1 bastion
3 masters with etcd
3 infra nodes (router/docker registry)
3+ nodes
Support request required for increasing cpu core limit.
This limitation is due to design of ARM template.
You can install all-in-one OpenShift on 1 host (not supported)

17. Examples: Integrating Azure Features
Availability Set
Azure Load Balancer
◦ master endpoint
◦ backend is a group of masters
◦ routing endpoint
◦ backend is a group of infra nodes (routers)
Azure VHD for Persistent Volume (PV)
◦ Virtual Hard Disk for Azure VM (VHD)
◦ Dynamic provisioning Available at OCP 3.5+

18. How does Azure VHD for PV work?
node
service
/etc/azure.conf
1. node service receives
Volume Mount request
2. Load azure.conf
(API auth etc)
3. (if dynamic provisioning)
Create an empty VHD
4. Mount VHD to Azure VM
5. Create filesystem if needed
6. Mount filesystem to container
Depending on kubernetess Azure Volume Plugin
VHD
attach

19. How to configure azure.conf
See the document for more detail.
Easy 3 steps with Azure CLI 2.0
$ az account list -o json
//Retrieve tenantID & id
$ az group show --name <ResourceGroupName> -o json
//Retrieve id & location
$ az ad sp create-for-rbac --name <ResourceGroupName> --role contributor
--scopes "<Resource Id>“ -o json
//Retrieve appId, password

20. Azure VHD for PV Notes
Managed Disk is unavailable
◦ kubernetes Azure Disk plugin is not supported Managed Disk
Be sure to confirm VM name to hostname
◦ Also specification of kubernetes plugin
Configure DNS yourself
◦ VMs can be communicate with their VM name.
◦ If not using Azure internal DNS
◦ If using VNET peering or other

21. More Azure Features
Azure Active Directory Open ID Connect
◦ authentication for master
◦ LDAP integration with AAD+AAD DS or AD is also available.
Azure Blob Storage for OpenShift internal docker registry
◦ object storage is suitable for docker registry storage
Azure File Storage
◦ File storage is also available for PV
◦ Linux kernel CIFS module with SMB 3 is still experimental
Operation Management Suite integration
◦ Log Analysis also available for containers

22. How to set up OpenID connect?
Create Azure AD App using the Microsoft Azure portal

23. How to set up LDAP auth with AD?
Option A) AAD + AAD DS + (VNET peering or VNET-to-VNET VPN)
* AAD DS only supports Classic VNET and requires private network from ARM VNET.
AAD AAD DS
classic VNET ARM VNET
OCP
master
ldap://xx.xx.xx.xx/

24. AAD DS configuration example
- name: "aad_ds_provider"
challenge: true
login: true
mappingMethod: claim
provider:
apiVersion: v1
kind: LDAPPasswordIdentityProvider
attributes:
id:
- dn
email:
- userPrincipalName
name:
- cn
preferredUsername:
- cn
bindDN: "cn=adadmin,ou=AADDC Users,DC=example,DC=onmicrosoft,DC=com"
bindPassword: "<password>"
insecure: true
url: "ldap://XXX.XX.XX.XX/OU=AADDC Users,DC=example,DC=onmicrosoft,DC=com?
userPrincipalName?sub?(memberof=CN=ocpgroup,OU=AADDC Users,DC=example,DC=onmicrosoft,DC=com)"
master-config.yaml
ou: AADDC Users
AAD default OU
userPrincipalName will be email

25. How to set up LDAP auth with AD?
Option B) on premise AD + VPN
Connect on premise Network and ARM Network with VPN.
AD
on premise NW ARM NW
OCP
master
ldap://xx.xx.xx.xx/

26. Storage Technology Comparison
Type References Notes
Azure Blob
Storage
Object
storage
Extended Registry Configuration
Microsoft Azure storage driver
Deploying Your Own Private Docker
Registry on Azure
Azure Blob Storage
Only Available for
docker registry
storage
Azure VHD Filesystem
on external
Disk
Persistent Storage Using Azure Disk
Configuring for Azure
About disks and VHDs for Azure Linux VMs
Depends on k8s
plugin
Azure File
Storage
NFS Persistent Storage Using Azure File
Configuring for Azure
How to use Azure File Storage with Linux
Depends on k8s
plugin
Experimental.
External
NFS
service
NFS N/A Should maintain
yourself or buy 3rd
party service.

27. Operation Management Suite (OMS)
Log Analysis & other features for on-premised to cloud
Containers (Preview) solution in Log Analytics now support OpenShift

28. Installing OMS Agent
Adding OMS agent directly on Linux Host
Or, install agent as a OpenShift daemonset

29. Container solution

30. Future: Windows Container?
No roadmap: Windows Container
kubernets has roadmap for working with Windows Container
“Capability” is existing.

32. .NET Application Model
35

33. .NET Core Inside
36
.NET Core App
(C#/VB)
IL Assembly (exe, dll)
Roslyn CoreFX
(.NET Core
Class
Library)
ManagedUnmanaged
OS
Native ABI
etc
(F#)
compile
r
CoreCLR
(.NET Core Runtime)
.NET
Core
SDK
Tools

34. .NET Core on OpenShift
s2i build
◦ “Source code in the Git repo” To “docker Image”
◦ can run out of OpenShift
s2i image
◦ parameters for simple customize
◦ more customization are available with s2i scripts
Template project
◦ Start .NET Core on OpenShift with few clicks at the portal
◦ All in one: deploymentconfig, service, route etc…

35. s2i build & deploy flow
builder pod
SCM(git) internal registry
deployer pod
pod
deploymentConfigbuildConfig
$ dotnet build
$ dotnet publish
$ dotnet <dll>

36. .NET Core 2.0 launch start today!
rh-dotnet supports csproj at .NET Core 2.0
◦ rpm version will be available
◦ s2i for .NET Core 2.0 & ASP.NET Core 2.0
◦ Runtime image & s2i image (s2i image only at 1.x)
More new features coming
◦ Announcing .NET Standard 2.0
◦ Announcing .NET Core 2.0
◦ Introducing ASP.NET Core 2.0
◦ Announcing Entity Framework Core 2.0

37. Use Case Examples
◦Schedule Jobs with .NET Core
◦Switching Configuration for Dev & Prod Environment
◦Razor Page & C# 7.1
◦Redis for HTTP Session storage with multi pods
Notes:
All examples are built on .NET Core 2.0 preview.
We’re actively working on it now.

38. Schedule Job with .NET Core
Run .NET Core Console App as a cron job: Cron Jobs
Example Repository
Web portal does not support cron jobs, so use the CLI.
$ oc create imagestream cronjobexample
$ oc create -f cronjob-buildconfig.yaml
$ oc create -f cronjob.yaml

39. Schedule Jobs with .NET Core
spec:
containers:
- command:
- /opt/rh/rh-dotnet20/root/bin/dotnet
- bin/Release/netcoreapp2.0/CronJobExample.dll
image: 172.30.142.2:5000/london-openshift/cronjobexample:latest
imagePullPolicy: Always
name: lipsum-cron
resources: {}
restartPolicy: Never
securityContext: {}
terminationGracePeriodSeconds: 30
schedule: '*/1 * * * *'
command to execute:
should be the full path
command to execute:
*scl should be enabled.
To be fixed in my example
image should be specified with full URL
OCP 3.6 will support imagestreamtag.
Replace 172.30.142.2:5000 with your
internal registry’s IP and port

40. Switching Configuration for Dev & Prod Environment
How to treat different environments with one code
◦ Connect to different database
◦ Use Redis as a cache - only in a production environment
◦ Integrate with a different OpenID account
Use Environment feature in ASP.NET Core
◦ Specified by environment variables.
Configuration can be injected specific to each environment.

41. Switching with Environment
Startup class
• constructor
• Configure method
• ConfigurreService method
Check IHostingEnvironment
Can’t inject IHostingEnvironment into ConfigerService method

42. Switching ConfigureXXXServices
ConfigureXXXServices

43. Switching ConfigureXXX
ConfigureXXX

44. Inject configuration
from Environment Variable

45. Loading configuration from Secret
use OpenShift secret feature.

46. Razor Page + C# 7.1
RazorPage:
◦ Simpler application than original MVC: “Page-focused scenarios”
◦ WebMatrix like easy development
◦ Razor Page is enabled with MVC
C# 7.1:
◦ available at .NET Core 2.0 & ASP.NET Core 2.0
◦ C# 7.1 in Razor page is not working at Preview 2 by bug (see issue)
◦ It should be fixed at 2.0 RTM.

47. HTTP session for multi pods
By default:
◦ Sticky session: request goes to the same pod in same user session
◦ HTTP session is stored in the memory of each pod
◦ HTTP session is encrypted by pod specific key
It means:
When a pod has died, a user session will be lost.
 How to keep HTTP session

48. IDistributedCache & IDataProtection
IDistributedCache
◦ Provide distribution cache
◦ Available for storing session
◦ ASP.NET Core team provides SQLServer and Redis
IDataProtection
◦ Provide key management for encryption
◦ Encrypt http session
◦ By default, generate machine (=pod) specific key and store in local file
◦ ASP.NET Core team provides NFS, Redis and AzureStorage (Preview)

49. machine A
pod B
pod C
ASP.NET Core
ASP.NET Core
ASP.NET Core
Session Data
A’
Each pod has a different key.
Can’t decrypt session data
When loading another pod
from a different session
load with same id.
~/.aspnet
default implementation of IDataProtection

50. machine A
machine B
machine C
ASP.NET Core
ASP.NET Core
ASP.NET Core
Session Data
Use DataProtection.Redis

51. Configuration for Redis
public void ConfigureServices(IServiceCollection services)
{
// You can retrieve this connection string from Azure Portal.
var conn = Configuration["REDIS_CONNECTION_STRING"];
var redis = ConnectionMultiplexer.Connect(conn);
services.AddDataProtection()
.PersistKeysToRedis(redis, "DataProtection-Keys");
services.AddDistributedRedisCache(option =>
{
option.Configuration = conn;
option.InstanceName = "master";
});
services.AddSession();
}
httpsession.redis.cache.windows.net:6380,password=<password>,ssl=True,abortConnect=False

52. High Level Debugging .NET Core
58
GDB/LLDB
MICore
MIDE/Engine.Impl
MIDE(*)/AD7.Impl
VSCode Debugger
AD7 Interface
*MIDE: MIDebugEngine: GitHub repository
*vsdbg can be used only in VS products and might not be distributed.
MIText
VS Debugger
VS Debugger
Engine vsdbg
(closed license*)
windbg
See more:
Architecture of MIEngine

53. Remote Debugging .NET Core
vsdbg provided by Microsoft
◦ only trusted communication is required
◦ SSH is generally available
◦ VS remote debugger tools is also available on Windows
◦ Due to the license limitation, VS products (VS, VS Code, VS for mac) are only
available for debugging.
* Low level debugger is provided by Red Hat
◦ sos
◦ Not providing graphical debugger interface
59

54. Remote debugging to
a container on OpenShift
“oc rsh” is available instead of ssh
vsdbg should be manually installed
◦ install script is unavailable as s2i image doesn’t have unzip
◦ download vsdbg on local and rsync
see more detail in my wiki

55. Remote debug from Visual Studio Code
.vscode/launch.json
61
{
"name": ".NET Core Docker Remote Attach",
"type": "coreclr",
"request": "attach",
“processId”: “1”,
"pipeTransport": {
"pipeProgram": “oc",
“pipeArgs”: [ “rsh”, “-T”, “tanaka733@centos.example.com”],
“quoteArgs”:false,
“debuggerPath”: “/opt/app-root/src/vsdbg/vsdbg”,
"pipeCwd": "${workspaceRoot}"
},
"sourceFileMap": {
"/opt/app-root/src": "${workspaceRoot}“
}
}

56. Summary
OpenShift on Azure
◦ Reference Architecture is a good place to start.
◦ More Azure features available-- Authenticating with OpenID and others
.NET Core 2.0/ASP.NET Core 2.0 on OpenShift
◦ csproj support
◦ cronjob for .NET Core console app
◦ OpenShift secret & configuration. ASP.NET Core environment
◦ Remote debugging