Huawei Cloud applies xen platform to many customer scenarios. This talk will introduce our optimizations on the xen platform to solve problems occuring in these scenarios. E.g 1. Redesign the implementation of kernel locks to improve the scalability of the Xen platform in large-scale server scenarios. 2. Develop LazyFPU and L3 cache affinity features to improve virtual machine performance in SAP HANA database service scenarios. 3. Develop HostNUMA and GuestNUMA features to enhance virtual machine performance in specvirt test and desktop cloud scenarios. 4. Shorten the time cost of concurrent life-cycle operations for large scales of virtual machines, to achieve quick change of classes in the cloud classroom.

1. HUAWEI TECHNOLOGIES CO., LTD.
www.huawei.com
Huawei Confidential
Security Level:2018/7/2
Bo Zhang <oscar.zhangbo@Huawei.com>
Yifei Jiang <jiangyifei@Huawei.com>
Performance tuning on
Xen platform

2. Agenda
◼ Application Scenarios of Huawei Cloud
◼ Scenario 1: SAP HANA
◼ Scenario 2: Desktop Cloud
◼ Scenario 3: Cloud Classroom

3. HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Application Scenarios of Huawei Cloud
Page 3
• Huawei UVP supports 100 cloud services under 14 catagories
• Public cloud
• Private cloud
• NFV
• Usage scenarios that have performance demands for virtualization
platform
• SAP HANA
• Desktop cloud(VDI)
• Cloud classroom
• ……

4. HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Scenario 1: SAP HANA
Page 4
• scenario
• Large-sized servers & virtual machines
• Un-overcommitted CPU & memory
• Performance demands
• Meet the SAP HANA virtualization certification’s performance index.
• Performance Barriers：
 Single-vm：the performance of a vm should be lower no more than 10%
compared to the physical machine of the same size.
 Multi-vm: no more than 5% performance degradation when other guests
are under pressure, compared with that other guests are idle.

5. HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Scenario 1: SAP HANA
Page 5
• Approach
• Keep vm’s hardware attributes consitent with physical machines
• Cpu topology & L3 cache affinity
• NUMA
• Virtualization platform cost
• Save the cost of Vmexit/vmentry
• Kernel lock optimization
• Resource isolation of CPU, memory and cache
• Dom0 and DomU
• DomU and DomU

6. HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Cpu topology & L3 cache affinity
Page 6
• Problem
• the server & client thread inside the guest that communicate via a socket share the
same VCPU.
• Thus both the server thread & client thread could not obtain the computing power
of a single pcpu.
• Root cause
• The linux scheduler inside guest senses no MC scheduling domain.
• The cache-affinity threads are scheduled on the same VCPU.
• Technical solution
• Present CPU topology information to guests.
• Make L3 cache shared among vcpus that is under the same socket of the guest.

7. HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Save the cost of vmexit/vmentry
Page 7
• Problem
• Large amount of CLTS instructions cause massive vmexit cost
• PLE may casue vmexit when vcpu paused for too long
• Technical solution
• Reduce vmexit times when CLTS instruction is executed.
• Disable PLE.

8. HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Kernel lock optimization
Page 8
• Problem
• Xen uses ticket spinlock by default. Ticket spinlock is quite performace consuming
for many core.
 Spin on global shared variable, among large amounts of pcpus
 Expensive cache entries invalidation
• Technical solution
• MCS lock is scalable.
 Spin on local variable
 Generate a constant number of cache misses per acquisition, avoid the
performance collapse with many cores.

9. HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Resource isolation
Page 9
• Problem
• On multi-vm scenario, other domU and dom0 may have large pressure, and may
affect the tested domU.
• Technical solution
• Isolation on domU and domU
• Isolation on domU and dom0

10. HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Scenario 2: VDI cloud
Page 10
• Desktop cloud demands:
• Large amont of small-sized virtual machines
• Performance demands：VSI density test
• Approach
• PLE enabled
• Guests’ memory and cpu location optimization
• Qemu resources limited: rm unused devices&slowdown devices’timer’s
frequency
• Hyperv to guests
• ……

11. HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Scenario 3: Cloud Classroom
Page 11
• Scenario introduction
• Cloud classroom customers requires rapid completion of server power-up
to a large numbers guests get online.
• Fast virtual machines’shutdown-startup-image replacement operations
during class change.
• Approach
• Virtual machines’ concurrent lifecycle operation timecost optimization
• Optimize the poweron/poweroff timecost inside guests

12. HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
lifecycle operation timecost optimization
Page 12
• Problem
• Xenstore has poor concurrent write permance
• Approach
• Use serialized xenstore write to avoid atomic breakup during guests concurrent
startup.
• Weak write that do not disturb transactions

13. HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
Timecost optimization inside guests
Page 13
• Problem
• When a large scale of virtual machines start/stop concurrently, the power on/off
timecost inside guests takes long too.
• Root cause
• Lots of IO r/w operation inside virtual machine during power on/off
• Technical solution
• Use memory disks for virtual machines
• Expand the number of grant-table to improve disk IO r/w performance
• Destroy virtual machines, rather than shutdown.

14. Thank you
www.huawei.com