As eBay is moving to OpenStack, we need to find capacity conversion ratio between ESX and KVM. Moreover, we hope to tunning KVM performance that make KVM to be same as or better than ESX
Konrad Wilk is a Software Development Manager at Oracle. His group’s mission is to make Linux and Xen Project virtualization better and faster. As part of this work, Konrad has been the maintainer of the Xen Project subsystem in Linux, Xen Project maintainer and now also Release Manager for the 4.5 release of the Xen Project Hypervisor. Konrad has been active in the Linux and Xen Project communities for more than 6 years and was instrumental in adding Xen Project support to the Linux Kernel.
XPDS14 - Scaling Xen's Aggregate Storage Performance - Felipe Franciosi, CitrixThe Linux Foundation
Storage systems continue to deliver better performance year after year. High performance solutions are now available off-the-shelf, allowing users to boost their servers with drives capable of achieving several GB/s worth of throughput per host. To fully utilise such devices, workloads with large queue depths are often necessary. In virtual environments, this translates into aggregate workloads coming from multiple virtual machines.
Having previously addressed the impact of low latency devices in virtualised platforms, we are now aiming at optimising aggregate workloads. We will discuss the existing memory grant technologies available in Xen and compare trade-offs and performance implications of each: grant mapping, persistent grants and grant copy. For the first time, we will present grant copy as an alternative and show measurements over 7 GB/s, maxing out a set of local SSDs.
Presentation from OpenStack Summit Tokyo
Online video link is below.
https://www.openstack.org/summit/tokyo-2015/videos/presentation/approaching-open-source-hyper-converged-openstack-using-40gbit-ethernet-network
XPDS14 - Intel(r) Virtualization Technology for Directed I/O (VT-d) Posted In...The Linux Foundation
With the development of virtualization, there are more device assignment requirements. Based on VT-d interrupt remapping, Intel introduces VT-d interrupt posting as a more enhanced method to handle interrupts in the virtualization environment. The Posted Interrupts (PI) on CPU side has been already supported in Intel CPUs, with VT-d Posted Interrupt we can get some additional advantages, it can directly deliver external interrupts to running vCPUs without hypervisor involvement, decease the interrupt migration complexity, differentiate between urgent and non-urgent external interrupt, and avoid consuming host-vector for each interrupt to vCPU. In this presentation, Feng will talk about the mechanism of VT-d PI and its advantages, as well as some performance data of I/O intensive workload in Xen, which will show the performance gain after using VT-d PI.
Improving the Performance of the qcow2 Format (KVM Forum 2017)Igalia
By Alberto García.
qcow2 is QEMU's native file format for storing disk images. One of its features is that it grows dynamically, so disk space is only allocated when the virtual machine needs to store data. This makes the format efficient in terms of space requirements, but has an impact on its I/O performance. This presentation will describe some of those performance problems and will discuss possible ways to address them. Some of them can be solved by simply adjusting configuration parameters, others require improving the qcow2 driver in QEMU, and others need extending the file format itself.
(c) KVM Forum 2017
October 25 - 27, 2017
Hilton Prague, Prague, Czech Republic
http://events.linuxfoundation.org/events/archive/2017/kvm-forum
Konrad Wilk is a Software Development Manager at Oracle. His group’s mission is to make Linux and Xen Project virtualization better and faster. As part of this work, Konrad has been the maintainer of the Xen Project subsystem in Linux, Xen Project maintainer and now also Release Manager for the 4.5 release of the Xen Project Hypervisor. Konrad has been active in the Linux and Xen Project communities for more than 6 years and was instrumental in adding Xen Project support to the Linux Kernel.
XPDS14 - Scaling Xen's Aggregate Storage Performance - Felipe Franciosi, CitrixThe Linux Foundation
Storage systems continue to deliver better performance year after year. High performance solutions are now available off-the-shelf, allowing users to boost their servers with drives capable of achieving several GB/s worth of throughput per host. To fully utilise such devices, workloads with large queue depths are often necessary. In virtual environments, this translates into aggregate workloads coming from multiple virtual machines.
Having previously addressed the impact of low latency devices in virtualised platforms, we are now aiming at optimising aggregate workloads. We will discuss the existing memory grant technologies available in Xen and compare trade-offs and performance implications of each: grant mapping, persistent grants and grant copy. For the first time, we will present grant copy as an alternative and show measurements over 7 GB/s, maxing out a set of local SSDs.
Presentation from OpenStack Summit Tokyo
Online video link is below.
https://www.openstack.org/summit/tokyo-2015/videos/presentation/approaching-open-source-hyper-converged-openstack-using-40gbit-ethernet-network
XPDS14 - Intel(r) Virtualization Technology for Directed I/O (VT-d) Posted In...The Linux Foundation
With the development of virtualization, there are more device assignment requirements. Based on VT-d interrupt remapping, Intel introduces VT-d interrupt posting as a more enhanced method to handle interrupts in the virtualization environment. The Posted Interrupts (PI) on CPU side has been already supported in Intel CPUs, with VT-d Posted Interrupt we can get some additional advantages, it can directly deliver external interrupts to running vCPUs without hypervisor involvement, decease the interrupt migration complexity, differentiate between urgent and non-urgent external interrupt, and avoid consuming host-vector for each interrupt to vCPU. In this presentation, Feng will talk about the mechanism of VT-d PI and its advantages, as well as some performance data of I/O intensive workload in Xen, which will show the performance gain after using VT-d PI.
Improving the Performance of the qcow2 Format (KVM Forum 2017)Igalia
By Alberto García.
qcow2 is QEMU's native file format for storing disk images. One of its features is that it grows dynamically, so disk space is only allocated when the virtual machine needs to store data. This makes the format efficient in terms of space requirements, but has an impact on its I/O performance. This presentation will describe some of those performance problems and will discuss possible ways to address them. Some of them can be solved by simply adjusting configuration parameters, others require improving the qcow2 driver in QEMU, and others need extending the file format itself.
(c) KVM Forum 2017
October 25 - 27, 2017
Hilton Prague, Prague, Czech Republic
http://events.linuxfoundation.org/events/archive/2017/kvm-forum
Performance Tuning a Cloud Application: A Real World Case Studyshane_gibson
During the OpenStack Icehouse summit in Atlanta, Symantec presented on our vision for a Key Value as a Service storage technology utilizing MagnetoDB. Since then our Cloud Platform Team has rolled the service out in our production environments. Through that process we have learned about tuning requirements of the solution on bare metal versus hosted VMs within an OpenStack environment.
Our initial performance testing was done with MagnetoDB running on bare metal nodes. After migrating the service from bare metal to an OpenStack VM hosted environment, we observed a 50% reduction in performance.
This presentation will dig into the details of the performance baselines, the tuning of the Nova Compute servers, Virtual Machine settings, and the applications itself to increase our performance.
Why larger community will be interested in this topic
This presentation will dig in to the technical details of performance tuning an application running on an OpenStack Nova Compute cluster. We will examine the performance related configuration settings necessary to improve the hosted application from three different angles:
the underlying compute node Operating System configuration
the hypervisor virtualization layer
and the Guest VM and Application stack
This presentation will provide a real world analysis of the steps taken. In addition, it will provide an outline for other cloud operators to follow when they work towards performance tuning their own cloud stack.
VMworld 2013
Lenin Singaravelu, VMware
Haoqiang Zheng, VMware
Learn more about VMworld and register at http://www.vmworld.com/index.jspa?src=socmed-vmworld-slideshare
XPDS16: Xen Scalability Analysis - Weidong Han, Zhichao Huang & Wei Yang, HuaweiThe Linux Foundation
As CPU integrates more cores, server will have more and more cores. It requires hypervisor to have good scalability. This talk will introduce our analysis on many core scalability of Xen, and share some findings and lessons.
XPDDS18: Performance tuning on Xen platform - Bo Zhang & Yifei Jiang, HuaweiThe Linux Foundation
Huawei Cloud applies xen platform to many customer scenarios. This talk will introduce our optimizations on the xen platform to solve problems occuring in these scenarios.
E.g
1. Redesign the implementation of kernel locks to improve the scalability of the Xen platform in large-scale server scenarios.
2. Develop LazyFPU and L3 cache affinity features to improve virtual machine performance in SAP HANA database service scenarios.
3. Develop HostNUMA and GuestNUMA features to enhance virtual machine performance in specvirt test and desktop cloud scenarios.
4. Shorten the time cost of concurrent life-cycle operations for large scales of virtual machines, to achieve quick change of classes in the cloud classroom.
Power management has become increasingly important in large-scale datacenters to address costs and limitations in cooling or power delivery, and it is much critical in mobile client where battery lifecycle is considered as one of the critical characteristics of the platform of choice. Good power management helps to achieve great energy efficiency. Virtualization imposes additional challenge to power management. It involves multiple software layers: VMM, OS, APP. For example, a good OS software stack may result in bad power consumption, if the hypervisor is not the timer unalignment, etc.
In this session, we will introduce what we did to improve power efficiency to achieve better power efficiency in both server and client virtualization environment.
In server side, we will introduce additional optimization technologies (e.g., eliminate unnecessary activities, align periodic timers to create long-idle period), to improve package C6 residency to be within 5% overhead with native. In client side, we will share our client power optimization technologies (e.g. graphics, ATA and wireless), which successfully reduce XenClient idle power overhead to be within 5%.
LCNA14: Why Use Xen for Large Scale Enterprise Deployments? - Konrad Rzeszute...The Linux Foundation
For many years, the Xen community has been delivering a solid virtualization platform for the enterprise. In support of the Xen community innovation effort, Oracle has been translating our enterprise experience with mission-critical workloads and large-scale infrastructure deployments into upstream contributions for the Linux and Xen efforts. In this session, you'll hear from a key Oracle expert, and community member, about Oracle contributions that focus on large-scale Xen deployments, networking, PV drivers, new PVH architecture, performance enhancements, dynamic memory usage with ‘tmem', and much more. This is your chance to get an under the hood view and see why the Xen architecture is the ideal choice for the enterprise.
This talk explores what has gone in so far in the Linux kernel (version 3.0 and 3.1) and which Linux distributions are deliverinbg Xen again. The otalk explores outstanding challenges and the pieces that are missing and what we can do, and what we cannot do working with Linux.
Mastering kvm virtualization- A complete guide of KVM virtualizationHumble Chirammal
Mastering KVM virtualization is a complete guide to understand KVM virtualization. Mastering KVM Virtualization is a culmination of all the knowledge we gained by
troubleshooting, configuring and fixing bug on KVM virtualization. We
authored this book for system administrators, DevOps practitioners and developers who have
a good hands-on knowledge of Linux and would like to sharpen their skills of open
source virtualization. The chapters in this book are written with a focus on practical
examples that should help you deploy a robust virtualization environment, suiting
your organization's needs. Our expectation is that, once you have finished the book,
you should have a good understanding of KVM virtualization, its tools to build
and manage diverse virtualization environments.
Performance Tuning a Cloud Application: A Real World Case Studyshane_gibson
During the OpenStack Icehouse summit in Atlanta, Symantec presented on our vision for a Key Value as a Service storage technology utilizing MagnetoDB. Since then our Cloud Platform Team has rolled the service out in our production environments. Through that process we have learned about tuning requirements of the solution on bare metal versus hosted VMs within an OpenStack environment.
Our initial performance testing was done with MagnetoDB running on bare metal nodes. After migrating the service from bare metal to an OpenStack VM hosted environment, we observed a 50% reduction in performance.
This presentation will dig into the details of the performance baselines, the tuning of the Nova Compute servers, Virtual Machine settings, and the applications itself to increase our performance.
Why larger community will be interested in this topic
This presentation will dig in to the technical details of performance tuning an application running on an OpenStack Nova Compute cluster. We will examine the performance related configuration settings necessary to improve the hosted application from three different angles:
the underlying compute node Operating System configuration
the hypervisor virtualization layer
and the Guest VM and Application stack
This presentation will provide a real world analysis of the steps taken. In addition, it will provide an outline for other cloud operators to follow when they work towards performance tuning their own cloud stack.
VMworld 2013
Lenin Singaravelu, VMware
Haoqiang Zheng, VMware
Learn more about VMworld and register at http://www.vmworld.com/index.jspa?src=socmed-vmworld-slideshare
XPDS16: Xen Scalability Analysis - Weidong Han, Zhichao Huang & Wei Yang, HuaweiThe Linux Foundation
As CPU integrates more cores, server will have more and more cores. It requires hypervisor to have good scalability. This talk will introduce our analysis on many core scalability of Xen, and share some findings and lessons.
XPDDS18: Performance tuning on Xen platform - Bo Zhang & Yifei Jiang, HuaweiThe Linux Foundation
Huawei Cloud applies xen platform to many customer scenarios. This talk will introduce our optimizations on the xen platform to solve problems occuring in these scenarios.
E.g
1. Redesign the implementation of kernel locks to improve the scalability of the Xen platform in large-scale server scenarios.
2. Develop LazyFPU and L3 cache affinity features to improve virtual machine performance in SAP HANA database service scenarios.
3. Develop HostNUMA and GuestNUMA features to enhance virtual machine performance in specvirt test and desktop cloud scenarios.
4. Shorten the time cost of concurrent life-cycle operations for large scales of virtual machines, to achieve quick change of classes in the cloud classroom.
Power management has become increasingly important in large-scale datacenters to address costs and limitations in cooling or power delivery, and it is much critical in mobile client where battery lifecycle is considered as one of the critical characteristics of the platform of choice. Good power management helps to achieve great energy efficiency. Virtualization imposes additional challenge to power management. It involves multiple software layers: VMM, OS, APP. For example, a good OS software stack may result in bad power consumption, if the hypervisor is not the timer unalignment, etc.
In this session, we will introduce what we did to improve power efficiency to achieve better power efficiency in both server and client virtualization environment.
In server side, we will introduce additional optimization technologies (e.g., eliminate unnecessary activities, align periodic timers to create long-idle period), to improve package C6 residency to be within 5% overhead with native. In client side, we will share our client power optimization technologies (e.g. graphics, ATA and wireless), which successfully reduce XenClient idle power overhead to be within 5%.
LCNA14: Why Use Xen for Large Scale Enterprise Deployments? - Konrad Rzeszute...The Linux Foundation
For many years, the Xen community has been delivering a solid virtualization platform for the enterprise. In support of the Xen community innovation effort, Oracle has been translating our enterprise experience with mission-critical workloads and large-scale infrastructure deployments into upstream contributions for the Linux and Xen efforts. In this session, you'll hear from a key Oracle expert, and community member, about Oracle contributions that focus on large-scale Xen deployments, networking, PV drivers, new PVH architecture, performance enhancements, dynamic memory usage with ‘tmem', and much more. This is your chance to get an under the hood view and see why the Xen architecture is the ideal choice for the enterprise.
This talk explores what has gone in so far in the Linux kernel (version 3.0 and 3.1) and which Linux distributions are deliverinbg Xen again. The otalk explores outstanding challenges and the pieces that are missing and what we can do, and what we cannot do working with Linux.
Mastering kvm virtualization- A complete guide of KVM virtualizationHumble Chirammal
Mastering KVM virtualization is a complete guide to understand KVM virtualization. Mastering KVM Virtualization is a culmination of all the knowledge we gained by
troubleshooting, configuring and fixing bug on KVM virtualization. We
authored this book for system administrators, DevOps practitioners and developers who have
a good hands-on knowledge of Linux and would like to sharpen their skills of open
source virtualization. The chapters in this book are written with a focus on practical
examples that should help you deploy a robust virtualization environment, suiting
your organization's needs. Our expectation is that, once you have finished the book,
you should have a good understanding of KVM virtualization, its tools to build
and manage diverse virtualization environments.
Virtualization with KVM (Kernel-based Virtual Machine)Novell
As a technical preview, SUSE Linux Enterprise Server 11 contains KVM, which is the next-generation virtualization software delivered with the Linux kernel. In this technical session we will demonstrate how to set up SUSE Linux Enterprise Server 11 for KVM, install some virtual machines and deal with different storage and networking setups.
To demonstrate live migration we will also show a distributed replicated block device (DRBD) setup and a setup based on iSCSI and OCFS2, which are included in SUSE Linux Enterprise Server 11 and SUSE Linux Enterprise 11 High Availability Extension.
This presentation talks about how to use GlusterFS containers in Docker. If you need more details please refer http://www.humblec.com/building-glusterfs-in-a-docker-container/
Demo Video # https://youtu.be/4Xf8pmDEZYw
The subject of the thesis is to clarify theoretical basis of visual means used in digital
games, to define the trends across the history and interconnection of the design in games
with art. After a comprehensive theory we move on to the detailed analysis of selected digital game: Journey (Thatgamecompany, 2012).
During Kylin OLAP development, we setup many engineering principles in the team. These principles are very important to delivery Kylin with high quality and on schedule.
View On-Demand
http://ecast.opensystemsmedia.com/331
Learn how to avoid commonly made mistakes when designing your distributed system. Whether it's misunderstanding your data flow or underestimating scale – learn from and apply real-world technical experience to your project.
RTI is the world's leading vendor of real-time middleware. We have direct experience with hundreds of teams building thousands of complex distributed systems. These range from a few computers to thousands, and soon, millions. We have seen, across the dozens of industries we engage, patterns of use, and critical mistakes, that even the best designers can make.
This webinar targets chief engineers and software architects for high-performance distributed systems. It captures some of our experience with these types of systems.
Session ID: SFO17-307
Session Name: WALT vs PELT : Redux
- SFO17-307
Speaker: Pavan Kumar Kondeti
Track: LMG
★ Session Summary ★
New data on the comparison of the WALT and PELT load tracking schemes in the scheduler
---------------------------------------------------
★ Resources ★
Event Page: http://connect.linaro.org/resource/sfo17/sfo17-307/
Presentation:
Video: https://www.youtube.com/watch?v=r3QKEYpyetU
---------------------------------------------------
★ Event Details ★
Linaro Connect San Francisco 2017 (SFO17)
25-29 September 2017
Hyatt Regency San Francisco Airport
---------------------------------------------------
Keyword:
'http://www.linaro.org'
'http://connect.linaro.org'
---------------------------------------------------
Follow us on Social Media
https://www.facebook.com/LinaroOrg
https://twitter.com/linaroorg
https://www.youtube.com/user/linaroorg?sub_confirmation=1
https://www.linkedin.com/company/1026961
The Economics of Scaling Cassandra - By Alex Bordei, Techie Product Manager at Bigstep
This presentation was made during the "Cassandra Summit 2014" Event, in London.
We benchmarked Cassandra on a number of configurations and we show what's the scaling profile. We test Cassandra on Docker as well as Cassandra's In-memory feature.
Follow Alex on Twitter: @alexandrubordei
Bigstep on Twitter: @BigStepInc
If you have any questions, let us know at hello@bigstep.com and we'll do our best to answer.
Stay informed: http://blog.bigstep.com/
Dell EMC VMAX All Flash and VMAX3 – powered by the universally trusted Hypermax/Enginuity operating system - continues to revolutionize the ways organizations are deploying, provisioning, protecting, and managing enterprise storage. This interactive session allows attendees to discuss new Dell EMC VMAX features and functionality in an open forum with specialists and engineering leaders. Bring your questions and top of mind discussion topics for this always-lively session.
XPDDS18: Real Time in XEN on ARM - Andrii Anisov, EPAM Systems Inc.The Linux Foundation
Currently, several initiatives promote XEN hypervisor into the automotive area as a base of complex virtualized systems. To support those initiatives and plunge into the automotive world XEN should fit at least two requirements: it should be appropriately certified and to be able to host a security domain. Leaving behind certification topic, here we focus on security domain hosting capability of XEN. Particularly on keeping RT guarantees for the specific domain.
This talk is a presentation of the investigation on a XEN hypervisor applicability to building a multi-OS system with real-time guarantees being kept for one of the hosted OSes.
During this presentation following topics would be outlined:
- experimental setup
- experimental use-cases and their motivation
- received results and discovered issues
- solutions and mitigation measures for discovered issues
Large-Scale Optimization Strategies for Typical HPC Workloadsinside-BigData.com
In this deck from PASC 2019, Liu Yu from Inspur presents: Large-Scale Optimization Strategies for Typical HPC Workloads.
"Ensuring performance of applications running on large-scale clusters is one of the primary focuses in HPC research. In this talk, we will show our strategies on performance analysis and optimization for applications in different fields of research using large-scale HPC clusters. Our strategies are designed to comprehensively analyze runtime features of applications, parallel mode of the physical model, algorithm implementation and other technical details. This three levels of strategy covers platform optimization, technological innovation, and model innovation, and targeted optimization based on these features. State-of-the-art CPU instructions, network communication and other modules, and innovative parallel mode of some applications have been optimized. After optimization, it is expected that these applications will outperform their non-optimized counterparts with obvious increase in performance."
Watch the video: https://wp.me/p3RLHQ-kwB
Learn more: http://en.inspur.com/en/2403285/2403287/2403295/index.html
and
https://pasc19.pasc-conference.org/program/keynote-presentations/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Intel(R) Xeon(R) E7 v3-based X6 platforms + Lenovo Flex System Interconnect Fabric solutions deliver a highly-reliable, cost-efficient and scalable system for your data center.
Tizen Developer Conference 2017 San Francisco - Tizen Power Management Servic...Chanwoo Choi
Power Management in the key point of embedded devices because of the limited power capacity. On Tizen 4.0, PASS (Power Aware System Service) is the power management service which provides the common interface to allocate the limited resources efficiently in order to meet the performance demands and power-consumption requirements. Tizen 4.0 platform developers or vendors may tune the behaviors of PASS governor for their own devices by modifying the configuration file of PASS. It allocates the power-related resources according to the configurations, which may emphasize either power-consumption or performance (or even "balanced") and the current user context (e.g., launching a heavy app, touching or swiping the touchscreen and so on). PASS supports PM_QOS based the scenarios as well. The configuration file of PASS includes CPU, Memory Bus, GPU information and the scenario list.
In this deck from the NVIDIA GPU Technology Conference, Axel Koehler presents: Inside the Volta GPU Architecture and CUDA 9.
"The presentation will give an overview about the new NVIDIA Volta GPU architecture and the latest CUDA 9 release. The NVIDIA Volta architecture powers the worlds most advanced data center GPU for AI, HPC, and Graphics. Volta features a new Streaming Multiprocessor (SM) architecture and includes enhanced features like NVLINK2 and the Multi-Process Service (MPS) that delivers major improvements in performance, energy efficiency, and ease of programmability. New features like Independent Thread Scheduling and the Tensor Cores enable Volta to simultaneously deliver the fastest and most accessible performance. CUDA is NVIDIA''s parallel computing platform and programming model. You''ll learn about new programming model enhancements and performance improvements in the latest CUDA9 release."
Watch the video: https://wp.me/p3RLHQ-iB7
Learn more: https://www.nvidia.com/en-us/gtc/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
VMworld 2013
Peter Boone, VMware
Seongbeom Kim, VMware
Learn more about VMworld and register at http://www.vmworld.com/index.jspa?src=socmed-vmworld-slideshare
VMworld 2015: Extreme Performance Series - vSphere Compute & MemoryVMworld
In this session we'll dive deep into how the vSphere compute and memory schedulers work to provide the same level of performance as bare metal. Hosted by two outstanding performance engineers, they will review concepts like how and when vSphere schedules vCPUs, how virtual machines are idles, understand virtual machine memory overhead and how large memory pages help or hurt performance. If you want to understand what vSphere does at an atomic level you don't want to miss this advanced session.
Bridging the Digital Gap Brad Spiegel Macon, GA Initiative.pptxBrad Spiegel Macon GA
Brad Spiegel Macon GA’s journey exemplifies the profound impact that one individual can have on their community. Through his unwavering dedication to digital inclusion, he’s not only bridging the gap in Macon but also setting an example for others to follow.
Instagram has become one of the most popular social media platforms, allowing people to share photos, videos, and stories with their followers. Sometimes, though, you might want to view someone's story without them knowing.
Ready to Unlock the Power of Blockchain!Toptal Tech
Imagine a world where data flows freely, yet remains secure. A world where trust is built into the fabric of every transaction. This is the promise of blockchain, a revolutionary technology poised to reshape our digital landscape.
Toptal Tech is at the forefront of this innovation, connecting you with the brightest minds in blockchain development. Together, we can unlock the potential of this transformative technology, building a future of transparency, security, and endless possibilities.
Gen Z and the marketplaces - let's translate their needsLaura Szabó
The product workshop focused on exploring the requirements of Generation Z in relation to marketplace dynamics. We delved into their specific needs, examined the specifics in their shopping preferences, and analyzed their preferred methods for accessing information and making purchases within a marketplace. Through the study of real-life cases , we tried to gain valuable insights into enhancing the marketplace experience for Generation Z.
The workshop was held on the DMA Conference in Vienna June 2024.
Meet up Milano 14 _ Axpo Italia_ Migration from Mule3 (On-prem) to.pdfFlorence Consulting
Quattordicesimo Meetup di Milano, tenutosi a Milano il 23 Maggio 2024 dalle ore 17:00 alle ore 18:30 in presenza e da remoto.
Abbiamo parlato di come Axpo Italia S.p.A. ha ridotto il technical debt migrando le proprie APIs da Mule 3.9 a Mule 4.4 passando anche da on-premises a CloudHub 1.0.
3. Project Background
KVM Tuning @ eBay
3
•Production environment is moving to OpenStratus, KVM based.
•Find capacity conversion ratio between ESX and KVM.
•Tuning KVM performance, make it same as or better than ESX.
8. Profiling and Tracing Tools
•Profiling and Tracing Tools
–perf
–ftrace
–turbostat/powertop
–vmstat
–itop
–/proc and /sys
–systemtap
–JMeter perfmon plugin
KVM Tuning @ eBay
8
9. KVM Performance - Baseline
• Baseline, KVM vs ESX
– Without tuning,
KVM performance is poorer than ESX..
KVM Tuning @ eBay 9
ESX Base improvement ESX Base improvement
TPS 9.82 9.54 -2.85% 23.28 18.6 -20.10%
Response Time 967 985 -1.86% 1278 1529 -19.64%
VM Total CPU 34.66 47.99 -38.46% 89.7 93.04 -3.72%
Normal Load (8 clients) High Load (24 clients)
10. KVM Performance – Optimization List
Presentation Title Goes Here
10
OptimizationRationalvirtioPara-virtualizaiton driver, reduce vmexit eventsvirtio+vhost_netPutting the device backend into host kernelinstead of userspace. Reduce context switches. THP(Transparent Huge Pages) 2MB page size instead of default 4KB page size, reduce pagefaults and TLB misses. Dynamically allocation. HugeTLBfs2MB page size instead of default 4KB page size, reduce pagefaults and TLB misses. Memory statically pre-allocation. ACPI c1 andperformance governorACPI in c1 (idle) and p0 (active) state. Reduce latency when entering/exiting idle stateand CPU is in maximum frequence if it's active. CFS parameter(Completely Fair Scheduler) Increase the time slice of process scheduler. Reduce the context switch overhead. NUMA pinnedPin each VM's memory and cpu in same NUMA node. To avoid remote memory access overhead.
11. KVM Performance – Optimizations, Normal Load
KVM Tuning @ eBay
11
THP has 5.08% improvement as highest, other tunings almost don’t affect TPS in the normal load.
The cpu usage decreases in all the tunings except CFS case, THP improve the cpu usage most, it’s 14.55%.
12. KVM Performance – Optimizations, High Load
KVM Tuning @ eBay
12
Only CFS tuning doesn’t improve the TPS,
the other tunings improve the TPS ranging from 3% to 12%,
THP improves the most, and it’s 12.11%.
14. KVM Performance – After Tuning
KVM Tuning @ eBay 14
ESX virtio + vhost-net+THP improvement ESX virtio + vhost-net+THP improvement
TPS 9.98 9.58 -4.01% 23.28 22.44 -3.61%
Response Time 953 994 -4.30% 1279 1311 -2.50%
VM Total CPU 35.4 39.13 -10.54% 89.36 87.61 1.96%
Normal Load (8 clients) High Load (24 clients)
After tuning, KVM vs ESX
• virtio+vhost_net+THP
15. KVM Scalability – Normal Load
KVM Tuning @ eBay
15
•Total TPS scale well from 1 VM to 6 VMs
•Average TPS of 6 VMs is 95% of average TPS of 1 VM
16. KVM Scalability – High Load
KVM Tuning @ eBay
16
•Total TPS in KVM scales well from 1 VM to 5 VMs, but it stops increasing from 5 VMs to 6 VMs if no NUMA pinned.
•After NUMA pinned, KVM reaches 86.64 TPS on 6 VMs, while ESX is 84.4 on 6VMs
•Average TPS of 6 VMs is 65% of average TPS of 1 VM, both ESX and KVM are the same.
17. KVM Scalability – Optimizations Comparison
KVM Tuning @ eBay
17
•Default tunned parameters (virtio+vhost_net+THP), improves TPS 23.88% in high load compared to no tunned KVM
•ACPI-c1 improves the performance in high load, compared to default tunned parameters, it’s about 6.8%,
•NUMA pinned improvement is big in high load, compared to default tunned parameters, it’s 32.9%.
18. KVM Over-Subscription – Add comments
KVM Tuning @ eBay
18
In the normal load, the total TPS has no much decreasing from 6 VMs to 18 VMs
In the high load,
•NUMA pinned, decreases from 86.64 to 71.48 linear
•No NUMA pinned, no much decreasing from 6 VMs to 18 VMs.
19. Conclusion
• Four optimizations for KVM
–virtio + vhost_net (production ready)
–THP (production ready)
–ACPI c1 + performance governor (power consuming trade-off)
–NUMA pinned, both memory and cpu (works for automation)
•Interestingly, we find similar optimization in ESX
–Paravirtualization vmxnet3 driver in ESX guest.
–Use large page by default
–Always put cpu in C0/C1 and P0 state (i.e. esxtop can show it)
–Use NUMA based scheduler by default
KVM Tuning @ eBay
19
20. How to go production?
•virtio + vhost_net + THP
–These parameters can be easily deployed.
–24* 7 stability testing has been done, no issue.
•NUMA pinned
–Two solutions
•NUMA aware VM allocation.
–Changes in the openstack are required.
•Dynamically NUMA pinned.
–numad from fedora doesn’t work well, e.g. 6 VMs running on two NUMA nodes, numad finally pins 5 VMs on one node, and 1 VM on the other node, not balance!
•ACPI c1 + performance governor
–Power consumption evaluation is required.
–ESX enable this feature by default.
KVM Tuning @ eBay
20
21. Case Study – Scalability Bottleneck
• Why KVM doesn’t scale well from 5 VMs to 6 VMs ?
KVM Tuning @ eBay
21
0
10
20
30
40
50
60
70
80
90
1
3
5
6
Total TPS, ESX vs KVM
ESX
KVM
23. Root Cause?
From 5 VMs to 6VMs
•CPU usage increases from 82% to 97%, but TPS only increases from 70 to 72.
•What’s CPU usage? What’s the TPS?
–CPU = Cycles (600B -> 690B)
–CPU <> Instruction
– Instruction == TPS (261B -> 2760B)
•The IPC(Instructions Per Cycle) reduces from 0.44 to 0.39, so the instruction latency increases about 10%
Presentation Title Goes Here
23
25. Memory Bottleneck?
•Cycle Accounting
– pipeline execution (same for 5 VM and 6 VM)
– data access (L1/L2/LLC/Memory)
•Instruction branch misses ratio is almost same.
•Cache misses ratio and TLB misses ratio are almost same.
•The LLC (Last Level Cache) misses show memory accesses are huge.
Presentation Title Goes Here
25
26. NUMA
•NUMA ?
•Local Dram ~60 ns
•Remote Dram ~100 ns
KVM Tuning @ eBay
26
28. NUMA Pinned
•Solution_1: Memory and CPU pinned when application starts.
#pin to numa node 0
echo 0 > /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.mems
echo '0-5,12-17' > /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.cpus
•Solution_2: Do memory and CPU migration after application starts.
echo 1 > /sys/fs/cgroup/cpuset/libvirt/qemu/instance-00000001/cpuset.memory_migrate
echo 1 > /sys/fs/cgroup/cpuset/libvirt/qemu/instance-00000001/cpuset.mems
echo '6-11,18-23' > /sys/fs/cgroup/cpuset/libvirt/qemu/instance-00000001/cpuset.cpus
Presentation Title Goes Here
28
29. NUMA Pin Issue – Can’t pin memory on Node 1
–Static pin Node 1 fails when VM starts.
KVM Tuning @ eBay
29
30. After NUMA pinned
KVM Tuning @ eBay
30
0
10
20
30
40
50
60
70
80
90
100
1
3
5
6
Total TPS, ESX vs KVM, High Load
ESX
KVM-tunned no NUMA pinned
KVM-tunned NUMA pinned
31. Intel PMU Comparison before & after tuning
• MEM_UNCORE_RETIRED.REMOTE_DRAM (0x53100f) ratio is 1:56 between pinned and unpinned
KVM Tuning @ eBay
31