Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

XPDDS17: Xen Schedulers and Their Impact on Interrupt Latency - Stefano Stabellini, Aporeto & Dario Faggioli, Citrix

313 views

Published on

The Xen hypervisor is a great fit for embedded, automotive and avionics, thanks to its lean architecture, small code base (even smaller on ARM), flexibility and security. Low interrupt latency is an essential requirement for many embedded use cases. Unavoidably, virtualization increases interrupt latency, but different Xen configurations lead to very different results.

This talk will explain how to achieve minimal interrupt latency. It will cover how scheduling interacts with interrupt delivery in Xen. It will introduce the new Xen "null" scheduler, designed for maximum determinism and minimum overhead on embedded platforms. Findings will be corroborated by showing interrupt latency numbers taken on Xen on ARM in various configurations, such as vCPU pinning, Credit 1 and 2, and null schedulers.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

XPDDS17: Xen Schedulers and Their Impact on Interrupt Latency - Stefano Stabellini, Aporeto & Dario Faggioli, Citrix

  1. 1. Xen Schedulers and Interrupt Latency Dario Faggioli & Stefano Stabellini
  2. 2. The case for embedded virtualization
  3. 3. Galois SMACCMPPilot Demo Xen Summit 2014
  4. 4. Why Xen? Why an hypervisor? • Efficiency and Consolidation • Isolation and Partitioning • Componentization • Resilience • Scaling • Portability
  5. 5. Embedded != Cloud Different requirements: • short boot times • small footprint • small codebase (certifications) • non-PCI device assignment • driver domains • co-processor virtualization • low, deterministic irq latency • real time schedulers
  6. 6. Embedded != Cloud Different requirements: • short boot times • small footprint • small codebase (certifications) • non-PCI device assignment • driver domains • co-processor virtualization • low, deterministic irq latency • real time schedulers Xen supports/enables
  7. 7. Xen Schedulers CPU CPU CPU CPU CPU CPU CPU CPU
  8. 8. Xen Schedulers CPU CPU CPU CPU CPU CPU CPU CPU Real Time Scheduler ARINC 653 Regular VM Scheduler Credit Dedicated to 1 VCPU Dedicated to 1 VCPU
  9. 9. Automotive Hardware Xen Dom0 Linux Control Domain UI Domain Automotive Grade Android HW Drivers GPU Driver PV Block & Net frontends PV Block & Net Backends Audio Driver
  10. 10. GlobalLogic
  11. 11. EPAMEPAM
  12. 12. Xilinx Zynq MPSoC Xen Dom0 Linux Baremetal App Toolstack FPGA Driver Baremetal App FPGA Driver Baremetal App FPGA Driver Baremetal App FPGA Driver FPGA Dedicated CPU Dedicated CPU Dedicated CPU Dedicated CPU
  13. 13. Latency Impact of Schedulers
  14. 14. pCPUs and vCPUs... pCPU0 pCPU1 pCPU2 pCPU3 vcpu vcpu vcpu vcpu vcpu vcpu vcpu vcpu vcpu vcpu vcpu vcpu vcpu vcpu We want to run!! We are blocked... vcpu vcpu I’m running
  15. 15. Keeping vCPUs “organised”: runqueues pCPU0 pCPU1 pCPU2 pCPU3 vcpu vcpu vcpuvcpuvcpurunq vcpu vcpurunq runq vcpu vcpu vcpurunq
  16. 16. Runqueues in Credit1 pCPU0 pCPU1 pCPU2 pCPU3 vcpu vcpu vcpuvcpuvcpurunq0 vcpu vcpurunq1 runq2 vcpu vcpu vcpurunq3 1 runqueue x pCPU vcpu vcpu vcpu
  17. 17. Runqueues in Credit2 pCPU0 pCPU1 pCPU2 pCPU3 vcpu vcpu vcpuvcpurunq runq Runqueues are shared vcpu vcpu vcpu
  18. 18. A vCPU Wake-Up in Credit1 pCPU0 pCPU1 pCPU2 pCPU3 vcpu vcpu vcpuvcpuvcpurunq0 vcpu vcpurunq1 runq2 vcpu vcpu vcpurunq3 Case a: 1. vcpu goes in a runq1 2. where can vcpu run? Hey, pCPU2 is idle! 3. put vcpu in runq2 4. pCPU2 picks up vcpu from its runqueue Case b: 1. vcpu goes in a runq3 2. hey, vcpu can prempt what’s running on pCPU3! 3. context switch vcpu vcpu vcpu vcpu vcpu vcpu vcpu (1) (2) (3) (4) vcpu vcpu (1) (2) (3)
  19. 19. A vCPU Wake-Up in Credit2 pCPU0 pCPU1 pCPU2 pCPU3 vcpu vcpu vcpurunqA runqB vcpu vcpu vcpu vcpu vcpu vcpu vcpu vcpu vcpu Case a: 1. vcpu goes in a runqA 2. load-balancer moves vcpu to a less loaded runq 3. pCPU2 picks up vcpu from its runqueue Case b: 1. vcpu goes in runqB 2. pCPU3 picks up vcpu
  20. 20. A vCPU Wake-Up in Credit20.000971102 irq_enter 0.000971102 irq_direct, vec fa, handler = apic_timer_interrupt 0.000971649 raise_softirq TIMER_SOFTIRQ 0.000971962 irq_exit, in_irq = 0 0.000974070 softirq_handler TIMER_SOFTIRQ 0.000976010 tasklet_schedule fn=hvm_assert_evtchn_irq, sched_on=6 (softirq) 0.000976010 tasklet_enqueue fn=hvm_assert_evtchn_irq 0.000976510 raise_softirq TASKLET_SOFTIRQ on cpu 6 0.000978213 softirq_handler TASKLET_SOFTIRQ 0.000981562 tasklet_do_work fn=hvm_assert_evtchn_irq 0.000982017 vcpu_wake d1v1 0.000982437 runstate_change d1v1 blocked->runnable 0.000982987 csched2:update_load 0.000983230 csched2:update_rq_load rq# 0, load = 1, avgload = 34.830% 0.000983430 csched2:update_vcpu_load d1v1, vcpu_load = 11.824% 0.000983735 csched2:runq_insert d1v1, position 0 0.000984060 csched2:runq_tickle_new d1v1, processor = 6, credit = 5292567 0.000984490 csched2:runq_tickle cpu 6 0.000984842 raise_softirq SCHEDULE_SOFTIRQ on cpu 6 0.000985500 softirq_handler SCHEDULE_SOFTIRQ 0.000988941 csched2:schedule cpu 6, rq# 0, idle, SMT idle, tickled 0.000989344 csched2:runq_cand_check d1v1 0.000989611 csched2:runq_candidate d1v1, credit = 5292567 0.000990881 sched_switch prev idle, run for 344.6us 0.000991199 sched_switch next d1v1, was runnable for 5.862us, next slice 1000.0us 0.000991377 sched_switch prev idle next d1v1 0.000991697 runstate_change idle running->runnable 0.000991979 runstate_change d1v1 runnable->running vcpu wakes-up; goes in runq Scheduler triggered on CPU 6 CPU 6 schedules vcpu runs Interrupt arrives Scheduling Introduced latency: 9.9620 us BEST CASE
  21. 21. _This_ _is_ _all_ _good_ ... … Because, thanks to this, we can offer VMs/users: • Overcommitting (i.e., having more vCPUs than pCPUs) • Weighted fair share of pCPU time • Hard and soft affinity • Cache and NUMA awareness • Caps and reservations on pCPU time … but it _comes_ _at_ _a_ _price_
  22. 22. The ‘null’ Scheduler
  23. 23. A scheduler that does nothing If we want features & flexibility, we must pay the price :-( What if we don’t want (or need) them, e.g.: • Static environments (some embedded usecases) • Systems (or cpupools, where we know we’ll never have overcommit) • For testing/benchmarking (as reference) Then you can use the ‘null’ scheduler
  24. 24. Runqueues in ‘null’. No, wait... pCPU0 pCPU1 pCPU2 pCPU3 vcpu vcpu vcpu vcpu There are no runqs at all! • vCPUs are statically assigned to pCPUs • Only 1 vCPUs per pCPU • Overcommit is possible (i.e.: the system won’t explode), but use only if you really know what you’re doing (i.e.: the VMs will likely explode!)
  25. 25. A vCPU Wake-Up in ‘null’ pCPU0 pCPU1 pCPU2 pCPU3 1. vcpu wakes up and run :-)vcpu vcpuvcpu (1)
  26. 26. A vCPU Wake-Up in ‘null’ 0.636884641 irq_enter 0.636884641 irq_direct, vec fa, handler = 0xffff82d080267ec4 0.636885492 raise_softirq TIMER_SOFTIRQ 0.636885922 irq_exit, in_irq = 0 0.636889583 softirq_handler TIMER_SOFTIRQ 0.636892021 tasklet_schedule fn=hvm_assert_evtchn_irq, sched_on=5 (softirq) 0.636892021 tasklet_enqueue fn=hvm_assert_evtchn_irq 0.636892836 raise_softirq TASKLET_SOFTIRQ on cpu 5 0.636895074 softirq_handler TASKLET_SOFTIRQ 0.636895607 tasklet_do_work fn=hvm_assert_evtchn_irq 0.636896202 vcpu_wake d1v1 0.636896712 runstate_change d1v1 blocked->runnable 0.636897197 raise_softirq SCHEDULE_SOFTIRQ on cpu 5 0.636898470 softirq_handler SCHEDULE_SOFTIRQ 0.636899465 null:schedule cpu 5, vcpu d1v1 0.636899720 sched_switch prev idle, run for 999936.973us 0.636899970 sched_switch next d1v1, was runnable for 2.411us 0.636900155 sched_switch prev idle next d1v1 0.636900448 runstate_change idle running->runnable 0.636900738 runstate_change d1v1 runnable->running vcpu wakes-up; goes in runq Scheduler triggered on CPU 6 CPU 6 schedules vcpu runs Interrupt arrives Scheduling Introduced latency: 4.5360 us (less than half of Credit2) NORMAL CASE
  27. 27. Benchmarks
  28. 28. Hardware and Software configuration Hardware: Xilinx Zynq MPSoC: 4 ARM A53 Cores Physical Timer Software: Xen 4.9.0-rc7 (+ phys_timer forwarding patch) Dom0: Linux 4.9, dom0_mem=1G, max_dom0_vcpus=2 1 vcpu TBM ctest
  29. 29. Fin

×