SlideShare a Scribd company logo
1 of 26
Download to read offline
See What Happened With Real-Time KVM
When Building Real Time Cloud?
Pei Zhang (张 培)
pezhang@redhat.com
June 20, 2017
Agenda
1. Background
2. Build real time cloud
3. Configure Open vSwitch, DPDK and vhost-user in real time environment
4. Show performance testing results
2
1. Background
3
1.1 Real-Time
Real-Time is about
determinism, constant
and predictive behaviour.
4
1.2 Network Function Virtualization(NFV)
5
Network functions will be
implemented in software that
can run on a range of industry
standard server hardware.
Dedicated network appliances
will be replaced by
virtualization and software.
1.3 FCAPS in Telecommunications
6
● Fault management
● Configuration management
● Accounting management
● Performance management
○ Throughput, network response times, packet loss rates, etc
● Security management
2. Build real time cloud
7
8
2.1 Real-Time in OpenStack
2.2 Strict pre-requisites for compute nodes
9
- Set up BIOS
- Install real time kernel/kvm
- Isolate cores
- Set hugepages
- Real time policy
- Pin vCPUs to individual cores
- Set housekeeping cores and real time cores
- Reserve hugepages
- Avoid devices which cause high latency
10
2.3 Configure Nova for launching real time VM
3. Configure Open vSwitch, DPDK and vhost-user
in real time environment
11
12
● Real-Time OS in the VNF
● Real-Time KVM in the NFVI
3.1 NFV Solution
3.2 VM with Open vSwitch, DPDK and vhost-user
13
- Poll mode driver
- Hugepage
- User space
- Strict NUMA policy
- Real time scheduling
- Individual cores pinning
14
3.2 Single queue topology
15
3.3 Multiple queues topology
4. Show performance testing results
16
4.1 Real-Time KVM latency testing
17
Versions: Red Hat Enterprise Linux 7
Testing environment:
CPU: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
vCPU 1 vCPU 2 vCPU 3 vCPU 4 vCPU 5 vCPU 6 vCPU 7 vCPU 8
Min(us) 00006 00006 00006 00006 00006 00005 00006 00006
Avg(us) 00006 00006 00006 00006 00006 00006 00006 00006
Max(us) 00012 00012 00012 00011 00016 00012 00012 00017
- Multiple vCPUs in VM
4.1 Real-Time KVM latency testing
18
Versions: Red Hat Enterprise Linux 7
Testing environment:
CPU: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
VM 1 VM 2 VM 3 VM 4
Min(us) 00005 00005 00005 00005
Avg(us) 00006 00006 00006 00006
Max(us) 00012 00011 00013 00013
- Multiple VMs
4.2 Network layer 2 latency testing
19
1 hour Min(us) Avg(us) Max(us) v_95p v_99p v_9999p
KVM / 2Q 15.987 17.077 777.990 17.555 18.547 23.692
RT-KVM /2Q 15.916 16.429 39.084 17.004 17.171 17.683
Testing environment:
CPU: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
NIC: 10-Gigabit X540-AT2
Versions: Red Hat Enterprise Linux 7
Thanks.
20
21
# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-661.rt56.579.el7.x86_64 ...default_hugepagesz=1G iommu=pt
intel_iommu=on isolcpus=1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,30,28,26,24,22,20,18,16 nohz=on
nohz_full=1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,30,28,26,24,22,20,18,16
rcu_nocbs=1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,30,28,26,24,22,20,18,16 intel_pstate=disable
nosoftlockup
# tuned-adm active
Current active profile: realtime-virtual-host
# cat /usr/lib/tuned/realtime-virtual-host/lapic_timer_adv_ns
3500
# cat /sys/module/kvm/parameters/lapic_timer_advance_ns
3500
# cat /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages
20
- Configuring compute node
Appendix
[root@compute1 ~]# cat /etc/nova/nova.conf
vcpu_pin_set=2,4,6,8,10,12,14,16,18
…
[root@compute1 ~]# source admin-openrc
[root@compute1 ~]# nova flavor-key 1 set hw:cpu_policy=dedicated
[root@compute1 ~]# nova flavor-key 1 set hw:cpu_realtime=yes
[root@compute1 ~]# nova flavor-key 1 set hw:cpu_realtime_mask="^0-1"
[root@compute1 ~]# nova flavor-key 1 set hw:mem_page_size=1GB
22
- configuring nova
23
vCPU pin
Real time
Hugepage
<memoryBacking>
<hugepages>
<page size='1048576' unit='KiB' nodeset='0'/>
</hugepages>
<locked/>
</memoryBacking>
<vcpu placement='static'>2</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='19'/>
<vcpupin vcpu='1' cpuset='18'/>
<emulatorpin cpuset='1,3,5,7,9'/>
<vcpusched vcpus='0-1' scheduler='fifo' priority='1'/>
</cputune>
<cpu mode='host-passthrough'>
<feature policy='require' name='tsc-deadline'/>
</cpu>
Note:
Remove devices which will cause high latency:
1. All USB support
2. All Spice support and QXL hardware
3. Virtio serial and guest agent sockets
4. Sound card
- booting VM
24
# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-661.rt56.579.el7.x86_64 root=/dev/mapper/rhel-root ro
crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet LANG=en_US.UTF-8
console=tty0 console=ttyS0,115200 isolcpus=1 nohz=on nohz_full=1 rcu_nocbs=1
intel_pstate=disable nosoftlockup
# tuned-adm active
Current active profile: realtime-virtual-guest
- Configuring VM
- Set up of Open vSwitch
1. Bind network devices to the driver dpdk supported, like vfio-pci
2. Open DPDK support option
# ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
3. Use hugepage
# ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem="1024,1024"
4. Create DPDK ports and vhost-user ports
#ovs-vsctl add-port ovsbr1 dpdk1 -- set Interface dpdk1 type=dpdk
# ovs-vsctl add-port ovsbr1 vhost-user1 -- set Interface vhost-user1 type=dpdkvhostuserclient
options:vhost-server-path=/tmp/vhostuser1.sock
# ovs-vsctl add-port ovsbr0 vhost-user0 -- set Interface vhost-user0 type=dpdk vhostuser
5. Pin each PMD thread to individual isolate core
Eg. 4 port, 4 pmd thread, 4 individual cores.
6. Follow strict NUMA placement policy
Should use cores and hugepage of same NUMA with network devices locate
7. Change the real time scheduling of all PMD processes.
# chrt -fp 95 $processID
25
- Set up of VM
1. Follow strict NUMA placement policy
Should use cores and hugepage of same NUMA with host network devices(or Open vSwitch)
2. Boot with enough vCPUs
3. Add network devices with vhost-user client/server mode
If Open vSwitch vhost-user ports is client mode, then guest should be server mode. Vice visa.
26

More Related Content

What's hot

4. CNCF kubernetes Comparison of-existing-cni-plugins-for-kubernetes
4. CNCF kubernetes Comparison of-existing-cni-plugins-for-kubernetes4. CNCF kubernetes Comparison of-existing-cni-plugins-for-kubernetes
4. CNCF kubernetes Comparison of-existing-cni-plugins-for-kubernetes
Juraj Hantak
 

What's hot (20)

The Open vSwitch and OVN Projects
The Open vSwitch and OVN ProjectsThe Open vSwitch and OVN Projects
The Open vSwitch and OVN Projects
 
Running Legacy Applications with Containers
Running Legacy Applications with ContainersRunning Legacy Applications with Containers
Running Legacy Applications with Containers
 
How Networking works with Data Science
How Networking works with Data Science How Networking works with Data Science
How Networking works with Data Science
 
Scale Kubernetes to support 50000 services
Scale Kubernetes to support 50000 servicesScale Kubernetes to support 50000 services
Scale Kubernetes to support 50000 services
 
Meetup 23 - 02 - OVN - The future of networking in OpenStack
Meetup 23 - 02 - OVN - The future of networking in OpenStackMeetup 23 - 02 - OVN - The future of networking in OpenStack
Meetup 23 - 02 - OVN - The future of networking in OpenStack
 
Unrevealed Story Behind Viettel Network Cloud Hotpot | Đặng Văn Đại, Hà Mạnh ...
Unrevealed Story Behind Viettel Network Cloud Hotpot | Đặng Văn Đại, Hà Mạnh ...Unrevealed Story Behind Viettel Network Cloud Hotpot | Đặng Văn Đại, Hà Mạnh ...
Unrevealed Story Behind Viettel Network Cloud Hotpot | Đặng Văn Đại, Hà Mạnh ...
 
OpenStack Korea 2015 상반기스터디(devops) 스크립트로 오픈스택 설치하기 20150728
OpenStack Korea 2015 상반기스터디(devops) 스크립트로 오픈스택 설치하기 20150728OpenStack Korea 2015 상반기스터디(devops) 스크립트로 오픈스택 설치하기 20150728
OpenStack Korea 2015 상반기스터디(devops) 스크립트로 오픈스택 설치하기 20150728
 
Policy-based Resource Placement
Policy-based Resource PlacementPolicy-based Resource Placement
Policy-based Resource Placement
 
Practical CNI
Practical CNIPractical CNI
Practical CNI
 
[OpenStack Day in Korea 2015] Track 1-6 - 갈라파고스의 이구아나, 인프라에 오픈소스를 올리다. 그래서 보이...
[OpenStack Day in Korea 2015] Track 1-6 - 갈라파고스의 이구아나, 인프라에 오픈소스를 올리다. 그래서 보이...[OpenStack Day in Korea 2015] Track 1-6 - 갈라파고스의 이구아나, 인프라에 오픈소스를 올리다. 그래서 보이...
[OpenStack Day in Korea 2015] Track 1-6 - 갈라파고스의 이구아나, 인프라에 오픈소스를 올리다. 그래서 보이...
 
Is there still room for innovation in container orchestration and scheduling
Is there still room for innovation in container orchestration and scheduling Is there still room for innovation in container orchestration and scheduling
Is there still room for innovation in container orchestration and scheduling
 
HA in OpenStack service - meetup #9
HA in OpenStack service - meetup #9HA in OpenStack service - meetup #9
HA in OpenStack service - meetup #9
 
Get a Taste of 1 k+ Nodes by a Handful of Servers
Get a Taste of 1 k+ Nodes by a Handful of Servers Get a Taste of 1 k+ Nodes by a Handful of Servers
Get a Taste of 1 k+ Nodes by a Handful of Servers
 
Orchestration tool roundup kubernetes vs. docker vs. heat vs. terra form vs...
Orchestration tool roundup   kubernetes vs. docker vs. heat vs. terra form vs...Orchestration tool roundup   kubernetes vs. docker vs. heat vs. terra form vs...
Orchestration tool roundup kubernetes vs. docker vs. heat vs. terra form vs...
 
OpenStack KOREA 정기 세미나_OpenStack meet iNaaS SDN Controller
OpenStack KOREA 정기 세미나_OpenStack meet iNaaS SDN ControllerOpenStack KOREA 정기 세미나_OpenStack meet iNaaS SDN Controller
OpenStack KOREA 정기 세미나_OpenStack meet iNaaS SDN Controller
 
Overview of OpenDaylight Container Orchestration Engine Integration
Overview of OpenDaylight Container Orchestration Engine IntegrationOverview of OpenDaylight Container Orchestration Engine Integration
Overview of OpenDaylight Container Orchestration Engine Integration
 
Status of Embedded Linux
Status of Embedded LinuxStatus of Embedded Linux
Status of Embedded Linux
 
Kubernetes Networking
Kubernetes NetworkingKubernetes Networking
Kubernetes Networking
 
4. CNCF kubernetes Comparison of-existing-cni-plugins-for-kubernetes
4. CNCF kubernetes Comparison of-existing-cni-plugins-for-kubernetes4. CNCF kubernetes Comparison of-existing-cni-plugins-for-kubernetes
4. CNCF kubernetes Comparison of-existing-cni-plugins-for-kubernetes
 
Obstacles & Solutions for Livepatch Support on ARM64 Architecture
Obstacles & Solutions for Livepatch Support on ARM64 ArchitectureObstacles & Solutions for Livepatch Support on ARM64 Architecture
Obstacles & Solutions for Livepatch Support on ARM64 Architecture
 

Viewers also liked

Viewers also liked (19)

Container Security
Container SecurityContainer Security
Container Security
 
OpenDaylight OpenStack Integration
OpenDaylight OpenStack IntegrationOpenDaylight OpenStack Integration
OpenDaylight OpenStack Integration
 
Linuxcon secureefficientcontainerimagemanagementharbor
Linuxcon secureefficientcontainerimagemanagementharborLinuxcon secureefficientcontainerimagemanagementharbor
Linuxcon secureefficientcontainerimagemanagementharbor
 
Simplify Networking for Containers
Simplify Networking for ContainersSimplify Networking for Containers
Simplify Networking for Containers
 
Zephyr: Creating a Best-of-Breed, Secure RTOS for IoT
Zephyr: Creating a Best-of-Breed, Secure RTOS for IoTZephyr: Creating a Best-of-Breed, Secure RTOS for IoT
Zephyr: Creating a Best-of-Breed, Secure RTOS for IoT
 
There is NO Open Source Business Model
There is NO Open Source Business ModelThere is NO Open Source Business Model
There is NO Open Source Business Model
 
Flowchain: A case study on building a Blockchain for the IoT
Flowchain: A case study on building a Blockchain for the IoTFlowchain: A case study on building a Blockchain for the IoT
Flowchain: A case study on building a Blockchain for the IoT
 
Rebuild - Simplifying Embedded and IoT Development Using Linux Containers
Rebuild - Simplifying Embedded and IoT Development Using Linux ContainersRebuild - Simplifying Embedded and IoT Development Using Linux Containers
Rebuild - Simplifying Embedded and IoT Development Using Linux Containers
 
Open Source Software Business Models Redux
Open Source Software Business Models ReduxOpen Source Software Business Models Redux
Open Source Software Business Models Redux
 
Secure Containers with EPT Isolation
Secure Containers with EPT IsolationSecure Containers with EPT Isolation
Secure Containers with EPT Isolation
 
UEFI HTTP/HTTPS Boot
UEFI HTTP/HTTPS BootUEFI HTTP/HTTPS Boot
UEFI HTTP/HTTPS Boot
 
From Resilient to Antifragile Chaos Engineering Primer
From Resilient to Antifragile Chaos Engineering PrimerFrom Resilient to Antifragile Chaos Engineering Primer
From Resilient to Antifragile Chaos Engineering Primer
 
Introduction to OCI Image Technologies Serving Container
Introduction to OCI Image Technologies Serving ContainerIntroduction to OCI Image Technologies Serving Container
Introduction to OCI Image Technologies Serving Container
 
Make Accelerator Pluggable for Container Engine
Make Accelerator Pluggable for Container EngineMake Accelerator Pluggable for Container Engine
Make Accelerator Pluggable for Container Engine
 
Build Robust Blockchain Services with Hyperledger and Containers
Build Robust Blockchain Services with Hyperledger and ContainersBuild Robust Blockchain Services with Hyperledger and Containers
Build Robust Blockchain Services with Hyperledger and Containers
 
How Open Source Communities do Standardization
How Open Source Communities do StandardizationHow Open Source Communities do Standardization
How Open Source Communities do Standardization
 
Quickly Debug VM Failures in OpenStack
Quickly Debug VM Failures in OpenStackQuickly Debug VM Failures in OpenStack
Quickly Debug VM Failures in OpenStack
 
Unikernelized Linux
Unikernelized LinuxUnikernelized Linux
Unikernelized Linux
 
Fully automated kubernetes deployment and management
Fully automated kubernetes deployment and managementFully automated kubernetes deployment and management
Fully automated kubernetes deployment and management
 

Similar to See what happened with real time kvm when building real time cloud pezhang@redhat.com.pdf

20151222_Interoperability with ML2: LinuxBridge, OVS and SDN
20151222_Interoperability with ML2: LinuxBridge, OVS and SDN20151222_Interoperability with ML2: LinuxBridge, OVS and SDN
20151222_Interoperability with ML2: LinuxBridge, OVS and SDN
Sungman Jang
 
[OpenStack 하반기 스터디] Interoperability with ML2: LinuxBridge, OVS and SDN
[OpenStack 하반기 스터디] Interoperability with ML2: LinuxBridge, OVS and SDN[OpenStack 하반기 스터디] Interoperability with ML2: LinuxBridge, OVS and SDN
[OpenStack 하반기 스터디] Interoperability with ML2: LinuxBridge, OVS and SDN
OpenStack Korea Community
 
DockerCon14 Performance Characteristics of Traditional VMs vs. Docker Containers
DockerCon14 Performance Characteristics of Traditional VMs vs. Docker ContainersDockerCon14 Performance Characteristics of Traditional VMs vs. Docker Containers
DockerCon14 Performance Characteristics of Traditional VMs vs. Docker Containers
Docker, Inc.
 

Similar to See what happened with real time kvm when building real time cloud pezhang@redhat.com.pdf (20)

LF_OVS_17_OVS-DPDK Installation and Gotchas
LF_OVS_17_OVS-DPDK Installation and GotchasLF_OVS_17_OVS-DPDK Installation and Gotchas
LF_OVS_17_OVS-DPDK Installation and Gotchas
 
Known basic of NFV Features
Known basic of NFV FeaturesKnown basic of NFV Features
Known basic of NFV Features
 
[OpenStack 하반기 스터디] HA using DVR
[OpenStack 하반기 스터디] HA using DVR[OpenStack 하반기 스터디] HA using DVR
[OpenStack 하반기 스터디] HA using DVR
 
Network Automation Tools
Network Automation ToolsNetwork Automation Tools
Network Automation Tools
 
SR-IOV, KVM and Emulex OneConnect 10Gbps cards on Debian/Stable
SR-IOV, KVM and Emulex OneConnect 10Gbps cards on Debian/StableSR-IOV, KVM and Emulex OneConnect 10Gbps cards on Debian/Stable
SR-IOV, KVM and Emulex OneConnect 10Gbps cards on Debian/Stable
 
Implementing an IPv6 Enabled Environment for a Public Cloud Tenant
Implementing an IPv6 Enabled Environment for a Public Cloud TenantImplementing an IPv6 Enabled Environment for a Public Cloud Tenant
Implementing an IPv6 Enabled Environment for a Public Cloud Tenant
 
20151222_Interoperability with ML2: LinuxBridge, OVS and SDN
20151222_Interoperability with ML2: LinuxBridge, OVS and SDN20151222_Interoperability with ML2: LinuxBridge, OVS and SDN
20151222_Interoperability with ML2: LinuxBridge, OVS and SDN
 
[OpenStack 하반기 스터디] Interoperability with ML2: LinuxBridge, OVS and SDN
[OpenStack 하반기 스터디] Interoperability with ML2: LinuxBridge, OVS and SDN[OpenStack 하반기 스터디] Interoperability with ML2: LinuxBridge, OVS and SDN
[OpenStack 하반기 스터디] Interoperability with ML2: LinuxBridge, OVS and SDN
 
Spectre meltdown performance_tests - v0.3
Spectre meltdown performance_tests - v0.3Spectre meltdown performance_tests - v0.3
Spectre meltdown performance_tests - v0.3
 
Intel DPDK Step by Step instructions
Intel DPDK Step by Step instructionsIntel DPDK Step by Step instructions
Intel DPDK Step by Step instructions
 
Achieving the Ultimate Performance with KVM
Achieving the Ultimate Performance with KVMAchieving the Ultimate Performance with KVM
Achieving the Ultimate Performance with KVM
 
Understanding kube proxy in ipvs mode
Understanding kube proxy in ipvs modeUnderstanding kube proxy in ipvs mode
Understanding kube proxy in ipvs mode
 
Libvirt/KVM Driver Update (Kilo)
Libvirt/KVM Driver Update (Kilo)Libvirt/KVM Driver Update (Kilo)
Libvirt/KVM Driver Update (Kilo)
 
Achieving the Ultimate Performance with KVM
Achieving the Ultimate Performance with KVMAchieving the Ultimate Performance with KVM
Achieving the Ultimate Performance with KVM
 
SR-IOV+KVM on Debian/Stable
SR-IOV+KVM on Debian/StableSR-IOV+KVM on Debian/Stable
SR-IOV+KVM on Debian/Stable
 
DockerCon14 Performance Characteristics of Traditional VMs vs. Docker Containers
DockerCon14 Performance Characteristics of Traditional VMs vs. Docker ContainersDockerCon14 Performance Characteristics of Traditional VMs vs. Docker Containers
DockerCon14 Performance Characteristics of Traditional VMs vs. Docker Containers
 
Openstack Networking Internals - first part
Openstack Networking Internals - first partOpenstack Networking Internals - first part
Openstack Networking Internals - first part
 
KVM Enhancements for OPNFV
KVM Enhancements for OPNFVKVM Enhancements for OPNFV
KVM Enhancements for OPNFV
 
NetBSD on Google Compute Engine (en)
NetBSD on Google Compute Engine (en)NetBSD on Google Compute Engine (en)
NetBSD on Google Compute Engine (en)
 
Intel's Out of the Box Network Developers Ireland Meetup on March 29 2017 - ...
Intel's Out of the Box Network Developers Ireland Meetup on March 29 2017  - ...Intel's Out of the Box Network Developers Ireland Meetup on March 29 2017  - ...
Intel's Out of the Box Network Developers Ireland Meetup on March 29 2017 - ...
 

More from LinuxCon ContainerCon CloudOpen China

More from LinuxCon ContainerCon CloudOpen China (6)

kdump: usage and_internals
kdump: usage and_internalskdump: usage and_internals
kdump: usage and_internals
 
Building a Better Thermostat
Building a Better ThermostatBuilding a Better Thermostat
Building a Better Thermostat
 
OCI Support in Mesos
OCI Support in MesosOCI Support in Mesos
OCI Support in Mesos
 
Libvirt API Certification
Libvirt API CertificationLibvirt API Certification
Libvirt API Certification
 
Rethinking the OS
Rethinking the OSRethinking the OS
Rethinking the OS
 
Releasing a Distribution in the Age of DevOps.
Releasing a Distribution in the Age of DevOps. Releasing a Distribution in the Age of DevOps.
Releasing a Distribution in the Age of DevOps.
 

Recently uploaded

Recently uploaded (20)

Buy Epson EcoTank L3210 Colour Printer Online.pptx
Buy Epson EcoTank L3210 Colour Printer Online.pptxBuy Epson EcoTank L3210 Colour Printer Online.pptx
Buy Epson EcoTank L3210 Colour Printer Online.pptx
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. Startups
 
What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024
 
UiPath Test Automation using UiPath Test Suite series, part 1
UiPath Test Automation using UiPath Test Suite series, part 1UiPath Test Automation using UiPath Test Suite series, part 1
UiPath Test Automation using UiPath Test Suite series, part 1
 
Connecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAKConnecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAK
 
Introduction to Open Source RAG and RAG Evaluation
Introduction to Open Source RAG and RAG EvaluationIntroduction to Open Source RAG and RAG Evaluation
Introduction to Open Source RAG and RAG Evaluation
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
 
Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutes
 
Top 10 Symfony Development Companies 2024
Top 10 Symfony Development Companies 2024Top 10 Symfony Development Companies 2024
Top 10 Symfony Development Companies 2024
 
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
 
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdfWhere to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
 
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
 
Optimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityOptimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through Observability
 
A Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System StrategyA Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System Strategy
 
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdfLinux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
 
Agentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdfAgentic RAG What it is its types applications and implementation.pdf
Agentic RAG What it is its types applications and implementation.pdf
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024
 
IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024
 

See what happened with real time kvm when building real time cloud pezhang@redhat.com.pdf

  • 1. See What Happened With Real-Time KVM When Building Real Time Cloud? Pei Zhang (张 培) pezhang@redhat.com June 20, 2017
  • 2. Agenda 1. Background 2. Build real time cloud 3. Configure Open vSwitch, DPDK and vhost-user in real time environment 4. Show performance testing results 2
  • 4. 1.1 Real-Time Real-Time is about determinism, constant and predictive behaviour. 4
  • 5. 1.2 Network Function Virtualization(NFV) 5 Network functions will be implemented in software that can run on a range of industry standard server hardware. Dedicated network appliances will be replaced by virtualization and software.
  • 6. 1.3 FCAPS in Telecommunications 6 ● Fault management ● Configuration management ● Accounting management ● Performance management ○ Throughput, network response times, packet loss rates, etc ● Security management
  • 7. 2. Build real time cloud 7
  • 8. 8 2.1 Real-Time in OpenStack
  • 9. 2.2 Strict pre-requisites for compute nodes 9 - Set up BIOS - Install real time kernel/kvm - Isolate cores - Set hugepages
  • 10. - Real time policy - Pin vCPUs to individual cores - Set housekeeping cores and real time cores - Reserve hugepages - Avoid devices which cause high latency 10 2.3 Configure Nova for launching real time VM
  • 11. 3. Configure Open vSwitch, DPDK and vhost-user in real time environment 11
  • 12. 12 ● Real-Time OS in the VNF ● Real-Time KVM in the NFVI 3.1 NFV Solution
  • 13. 3.2 VM with Open vSwitch, DPDK and vhost-user 13 - Poll mode driver - Hugepage - User space - Strict NUMA policy - Real time scheduling - Individual cores pinning
  • 16. 4. Show performance testing results 16
  • 17. 4.1 Real-Time KVM latency testing 17 Versions: Red Hat Enterprise Linux 7 Testing environment: CPU: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz vCPU 1 vCPU 2 vCPU 3 vCPU 4 vCPU 5 vCPU 6 vCPU 7 vCPU 8 Min(us) 00006 00006 00006 00006 00006 00005 00006 00006 Avg(us) 00006 00006 00006 00006 00006 00006 00006 00006 Max(us) 00012 00012 00012 00011 00016 00012 00012 00017 - Multiple vCPUs in VM
  • 18. 4.1 Real-Time KVM latency testing 18 Versions: Red Hat Enterprise Linux 7 Testing environment: CPU: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz VM 1 VM 2 VM 3 VM 4 Min(us) 00005 00005 00005 00005 Avg(us) 00006 00006 00006 00006 Max(us) 00012 00011 00013 00013 - Multiple VMs
  • 19. 4.2 Network layer 2 latency testing 19 1 hour Min(us) Avg(us) Max(us) v_95p v_99p v_9999p KVM / 2Q 15.987 17.077 777.990 17.555 18.547 23.692 RT-KVM /2Q 15.916 16.429 39.084 17.004 17.171 17.683 Testing environment: CPU: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz NIC: 10-Gigabit X540-AT2 Versions: Red Hat Enterprise Linux 7
  • 21. 21 # cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.10.0-661.rt56.579.el7.x86_64 ...default_hugepagesz=1G iommu=pt intel_iommu=on isolcpus=1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,30,28,26,24,22,20,18,16 nohz=on nohz_full=1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,30,28,26,24,22,20,18,16 rcu_nocbs=1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,30,28,26,24,22,20,18,16 intel_pstate=disable nosoftlockup # tuned-adm active Current active profile: realtime-virtual-host # cat /usr/lib/tuned/realtime-virtual-host/lapic_timer_adv_ns 3500 # cat /sys/module/kvm/parameters/lapic_timer_advance_ns 3500 # cat /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages 20 - Configuring compute node Appendix
  • 22. [root@compute1 ~]# cat /etc/nova/nova.conf vcpu_pin_set=2,4,6,8,10,12,14,16,18 … [root@compute1 ~]# source admin-openrc [root@compute1 ~]# nova flavor-key 1 set hw:cpu_policy=dedicated [root@compute1 ~]# nova flavor-key 1 set hw:cpu_realtime=yes [root@compute1 ~]# nova flavor-key 1 set hw:cpu_realtime_mask="^0-1" [root@compute1 ~]# nova flavor-key 1 set hw:mem_page_size=1GB 22 - configuring nova
  • 23. 23 vCPU pin Real time Hugepage <memoryBacking> <hugepages> <page size='1048576' unit='KiB' nodeset='0'/> </hugepages> <locked/> </memoryBacking> <vcpu placement='static'>2</vcpu> <cputune> <vcpupin vcpu='0' cpuset='19'/> <vcpupin vcpu='1' cpuset='18'/> <emulatorpin cpuset='1,3,5,7,9'/> <vcpusched vcpus='0-1' scheduler='fifo' priority='1'/> </cputune> <cpu mode='host-passthrough'> <feature policy='require' name='tsc-deadline'/> </cpu> Note: Remove devices which will cause high latency: 1. All USB support 2. All Spice support and QXL hardware 3. Virtio serial and guest agent sockets 4. Sound card - booting VM
  • 24. 24 # cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.10.0-661.rt56.579.el7.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet LANG=en_US.UTF-8 console=tty0 console=ttyS0,115200 isolcpus=1 nohz=on nohz_full=1 rcu_nocbs=1 intel_pstate=disable nosoftlockup # tuned-adm active Current active profile: realtime-virtual-guest - Configuring VM
  • 25. - Set up of Open vSwitch 1. Bind network devices to the driver dpdk supported, like vfio-pci 2. Open DPDK support option # ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true 3. Use hugepage # ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem="1024,1024" 4. Create DPDK ports and vhost-user ports #ovs-vsctl add-port ovsbr1 dpdk1 -- set Interface dpdk1 type=dpdk # ovs-vsctl add-port ovsbr1 vhost-user1 -- set Interface vhost-user1 type=dpdkvhostuserclient options:vhost-server-path=/tmp/vhostuser1.sock # ovs-vsctl add-port ovsbr0 vhost-user0 -- set Interface vhost-user0 type=dpdk vhostuser 5. Pin each PMD thread to individual isolate core Eg. 4 port, 4 pmd thread, 4 individual cores. 6. Follow strict NUMA placement policy Should use cores and hugepage of same NUMA with network devices locate 7. Change the real time scheduling of all PMD processes. # chrt -fp 95 $processID 25
  • 26. - Set up of VM 1. Follow strict NUMA placement policy Should use cores and hugepage of same NUMA with host network devices(or Open vSwitch) 2. Boot with enough vCPUs 3. Add network devices with vhost-user client/server mode If Open vSwitch vhost-user ports is client mode, then guest should be server mode. Vice visa. 26