Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Luca Abeni - Real-Time Virtual Machines with Linux and kvm

66 views

Published on

This talk describes how to use some available technologies (the SCHED_DEADLINE scheduling policy, the PREEMPT_RT patchset, etc…) to execute real-time applications in kvm-based virtual machines while still providing performance guarantees to the virtualized applications.

In recent years, there has been a growing interest in supporting virtualized services even in embedded and real-time systems. However, executing real-time applications (characterized by temporal constraints) in virtual machines is not straightforward and presents some non-trivial challenges. This talk will describe how to use some technologies already available in the Linux kernel (the SCHED_DEADLINE scheduling policy, the PREEMPT_RT patchset, etc…) to execute real-time applications in kvm-based virtual machines while still providing performance guarantees to the virtualized applications. After presenting the problem (and providing a quick summary about real-time scheduling), it will be shown how to configure the host and guest kernels and the virtual machine, and how to schedule the VM threads in order to achieve predictable response times and to provide real-time guarantees.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Luca Abeni - Real-Time Virtual Machines with Linux and kvm

  1. 1. Real-Time Virtual Machines with Linux and kvm Luca Abeni luca.abeni@santannapisa.it
  2. 2. Real-Time Application LinuxLab 2018 Luca Abeni – 2 / 36 ■ The time when a result is produced matters ◆ A correct result produced too late is equivalent to a wrong result (or to no result) ■ What does “too late” mean, here? ◆ Applications characterised by temporal constraints ■ Temporal constraints are modelled through deadlines ◆ Finish some activity before a time (deadline) ◆ Generate some data before a deadline ◆ Terminate some process/thread before a deadline ◆ ...
  3. 3. Real-Time Task LinuxLab 2018 Luca Abeni – 3 / 36 Task (process or thread): sequence of actions characterized by deadlines and maximum execution time. Example: periodic task with period 8 and maximum execution time 3. 0 2 4 6 8 10 12 14 16 18 20 22 24 τ1 Note: while the first and the third jobs execute for 3 time units the second one executes for only 2 time units.
  4. 4. Formal Task Model LinuxLab 2018 Luca Abeni – 4 / 36 ■ Real-Time task τi: stream of jobs (or instances) Ji,k ■ Each job Ji,k = (ri,k, ci,k, di,k): ◆ Arrives at time ri,k (activation time) ◆ Executes for a time ci,k ◆ Finishes at time fi,k ◆ Should finish within an absolute deadline di,k ri,k fi,k di,k ci,k
  5. 5. Periodic Tasks LinuxLab 2018 Luca Abeni – 5 / 36 Periodic task τi = (Ci, Di, Pi): stream of jobs Ji,k, with ri,k+1 = ri,k + Pi di,k = ri,k + Di Ci = max k {ci,k} ■ Pi is the task period; ■ Di is the task relative deadline; ■ Ci is the task worst-case execution time (WCET); ■ Ri is the worst-case response time: Ri = maxk{ρi,k} = maxk{fi,k − ri,k}; ◆ Constraints respected ⇔ Ri ≤ Di.
  6. 6. Periodic Task Model LinuxLab 2018 Luca Abeni – 6 / 36 ■ A periodic task has a regular structure (cycle): ◆ activate periodically (period Pi) ◆ execute a computation ◆ suspend waiting for the next period void ∗ PeriodicTask ( void ∗ arg ) { <i n i t i a l i z a t i o n >; <s t a r t p e r i o d i c timer , period = P>; while ( cond ) { <do something . . . > ; <wait next a c t i v a t i o n >; } }
  7. 7. Real-Time Scheduling LinuxLab 2018 Luca Abeni – 7 / 36 ■ A real-time task τi = (Ci, Di, Pi) (or τi = (Ci, Pi) if Di = Pi) is properly served if all jobs respect their deadline... ■ ...Appropriate scheduling is important! ◆ The scheduler must somehow know the temporal constaints of the tasks... ◆ ...In order to schedule them so that such temporal constraints are respected ■ How to schedule real-time tasks? (scheduling algorithm) ■ Is it possible to respect all the deadlines? ■ Does Linux provide an appropriate scheduling algorithm?
  8. 8. Fixed Priorities LinuxLab 2018 Luca Abeni – 8 / 36 ■ The Linux fixed-priority scheduler is often used for real-time tasks... ■ ...Is this ok in general, or only in some cases? ◆ Given a set of real-time tasks Γ = {τi}, can a fixed priority scheduler allow to respect all the deadlines? ◆ Is it possible to know in advance if some deadline will be missed? ◆ How to assign the priorities? ■ If fixed priorities are enough, the SCHED FIFO and SCHED RR policies can be used!
  9. 9. Schedulability Analysis LinuxLab 2018 Luca Abeni – 9 / 36 ■ Given a set of tasks, is it possible to know in advance if deadlines will be missed? ◆ A task is schedulable if it will not miss any deadline ■ Depends on the task ■ Depends on the scheduler ■ Depends on the other tasks, ... ■ Schedulability test: mathematical test to check if τ = (C, D, P) will miss any deadline ◆ Possible idea: compute the amount of time needed by all the tasks to respect all their deadlines in an interval (t0, t1) and compare with t1 − t0
  10. 10. Partitioned Scheduling LinuxLab 2018 Luca Abeni – 10 / 36 ■ How to schedule tasks on multiple CPUs / cores? ◆ First idea: partitioned scheduling ■ Statically assign tasks to CPUs ■ Reduce the problem of scheduling on M CPUs to M instances of uniprocessor scheduling CPU CPU CPU CPU M
  11. 11. Global Scheduling LinuxLab 2018 Luca Abeni – 11 / 36 ■ One single task queue, shared by M CPUs ◆ The first M ready tasks from the queue are selected ◆ What happens using fixed priorities (or EDF)? ◆ Tasks are not bound to specific CPUs ◆ Tasks can often migrate between different CPUs ■ Problem: schedulers designed for UP do not work well M CPU CPU CPU CPU {M
  12. 12. Fixed Priorities in Linux LinuxLab 2018 Luca Abeni – 12 / 36 ■ SCHED FIFO and SCHED RR use fixed priorities ◆ They can be used for real-time tasks, to implement RM and DM ◆ Real-time tasks have priority over non real-time (SCHED OTHER) tasks ■ Difference between the two policies: visible when more tasks have the same priority ■ The administrator can use sched setscheduler() (or similar) to schedule real-time tasks ◆ Might be a little bit impractical, but can be used ■ However...
  13. 13. Periodic Task Example LinuxLab 2018 Luca Abeni – 13 / 36 ■ Consider a periodic task /∗ . . . ∗/ while (1) { /∗ Job body ∗/ c l o c k n a n o s l e e p (CLOCK REALTIME , TIMER ABSTIME , &r , NULL ) ; timespec add us (&r , period ) ; } ■ The task expects to be executed at time r (= r0 + jT)... ■ ...But is sometimes delayed to r0 + jT + δ
  14. 14. Theoretical Schedule LinuxLab 2018 Luca Abeni – 14 / 36 0 2 4 6 8 10 12 14 16 18 20 22 τ1 τ2
  15. 15. Actual Schedule LinuxLab 2018 Luca Abeni – 15 / 36 0 2 4 6 8 10 12 14 16 18 20 22 τ1 τ2 ■ What happens if the 2nd job of τ1 arrives a little bit later??? ◆ The 2nd job of τ2 misses a deadline!!!
  16. 16. Theory vs Real Schedule LinuxLab 2018 Luca Abeni – 16 / 36 ■ The delay δ in scheduling a task is due to kernel latency ■ Kernel latency can be modelled as a blocking time ■ In real world, high priority tasks often suffer from blocking times coming from the OS (more precisely, from the kernel) ◆ Why? ◆ How? ◆ What can we do? ■ To answer the previous questions, we need to recall how the hardware and the OS work...
  17. 17. Latency LinuxLab 2018 Luca Abeni – 17 / 36 ■ Latency: measure of the difference between the theoretical and actual schedule ◆ Task τ expects to be scheduled at time t . . . ◆ . . . but is actually scheduled at time t′ ◆ ⇒ Latency L = t′ − t ■ The latency L can be modelled as a blocking time ⇒ affects the guarantee test ◆ Similar to what done for shared resources ◆ Blocking time due to latency, not to priority inversion
  18. 18. Effects of the Latency LinuxLab 2018 Luca Abeni – 18 / 36 ■ Upper bound for L? If not known, no schedulability tests!!! ◆ The latency must be bounded: ∃Lmax : L < Lmax ■ If Lmax is too high, only few task sets result to be schedulable ◆ Large blocking time experienced by all tasks! ◆ The worst-case latency Lmax cannot be too high
  19. 19. Latency in Linux LinuxLab 2018 Luca Abeni – 19 / 36 ■ Tool (cyclictest) to measure the latency ■ Vanilla kernel: depends on the configuration ◆ Can be tens of milliseconds ■ PREEMPT RT patchset (https://wiki.linuxfoundation.org/realtime): reduce latency to less than 100 microseconds ◆ Tens of microseconds on well-tuned systems! ■ So, scheduling real-time tasks in Linux is not an issue anymore...
  20. 20. Real-Time in VMs??? LinuxLab 2018 Luca Abeni – 20 / 36 ■ So, serving real-time applications in Linux-based systems is not a problem today... ■ ...Can real-time applications run in virtual machines? ◆ Real-Time in Virtual Machines??? But... Why? ■ Component-Based Development ◆ Complex applications: sets of smaller components ◆ Both functional and temporal interfaces ■ Security ◆ Isolate real-time applications in a VM ■ Easy deployment ■ ...
  21. 21. Real-Time Components LinuxLab 2018 Luca Abeni – 21 / 36 ■ Assume to have N components... ◆ Each component is described by its interface... ◆ ...But a description of its temporal behaviour is also somehow provided... ■ Component are assumed to respect some temporal constraints ◆ Some kind of “contract” provided by real-time guarantees ■ Is it possible to combine the components so that the timing of the system is predictable? ◆ Can real-time guarantees be combined? How?
  22. 22. Combining Real-Time Apps LinuxLab 2018 Luca Abeni – 22 / 36 ? ■ Real-time analysis for each application / in each VM... ■ What about the resulting system?
  23. 23. 1 Real-Time Task per VM LinuxLab 2018 Luca Abeni – 23 / 36 ■ Single real-time thread/process in a VM → combining VMs is easy... ◆ VMs can be modelled as real-time tasks ◆ Simple (C, D, T) model, or something more complex... ■ What about VMs containing multiple real-time tasks? ◆ Model for a VM? ◆ How to summarize the VM’s temporal requirements? ◆ Scheduler in the VM? How to model it?
  24. 24. More Complex VMs LinuxLab 2018 Luca Abeni – 24 / 36 ■ Example: component Ci composed by ni tasks ■ More complex model... How to handle it? ◆ We only know how to schedule single tasks... ◆ And we need to somehow “summarise” the requirements of a component! Ci = {(Ci 0, Di 0, Ti 0), (Ci 1, Di 1, Ti 1), . . . , (Ci ni, Di ni, Ti ni)} ■ So, 2 main issues: 1. Describe the temporal requirements of a VM in a simple way 2. Schedule the VMs, and somehow “combine” their temporal guarantees
  25. 25. Practical Issues LinuxLab 2018 Luca Abeni – 25 / 36 ■ VMs can contain sets of real-time tasks ◆ The scheduler only sees a VM per taskset, but cannot see the tasks inside it ◆ Para-virtualization (of the OS scheduler) could be used to address this issue, but it is not so simple... ■ So, how to schedule VMs? ■ Two-level hierarchical scheduling system ◆ Host (global / root) scheduler, scheduling VMs ◆ Each VM contains its (local / 2nd level) scheduler
  26. 26. 2-Levels Hierarchy LinuxLab 2018 Luca Abeni – 26 / 36 1τ 1τ 1ττn τn Local Scheduler Local Scheduler Local Scheduler 2 τn τ Global Scheduler ■ Global Scheduler assigns CPU to virtual machines ■ Local Schedulers assign CPU to single tasks
  27. 27. Hierarchical Scheduling LinuxLab 2018 Luca Abeni – 27 / 36 ■ The global scheduler does not “see” all the tasks ■ Each VM has its own scheduler (fixed priorities for real-time tasks) ◆ Global scheduler: host / hypervisor scheduler ◆ Local scheduler: guest scheduler ◆ No problems in assigning fixed priorities to tasks! ■ Problem: what to use as a global scheduler? ◆ We must have a model for it ◆ Must allow to compose the “local guarantees” ◆ This time, fixed priorities are not enough :(
  28. 28. Schedulability Analysis LinuxLab 2018 Luca Abeni – 28 / 36 ■ (Over?)Simplifying things a little bit... ■ ...Suppose to know the amount of time needed by a VM to respect its temporal constraints and the amount of time provided by the global scheduler ■ A VM is “schedulable” if demanded time ≤ supplied time ◆ “demanded time”: amount of time (in a time interval) needed by a VM ◆ “supplied time”: amount of time (in a time interval) given by the global scheduler to a VM ■ Of course the devil is in the details
  29. 29. Supplied Time LinuxLab 2018 Luca Abeni – 29 / 36 ■ Description of the root scheduler temporal behaviour ■ More formally: ◆ Depends on the time interval t we are considering ◆ Depends on the root scheduler S ■ Minimum amount of time given by S to a VM in a time interval of size s ◆ Given all the time interval (t0, t1) : t1 − t0 = s... ◆ ...Compute the size of the sub-interval in which the VM is scheduled... ◆ ...And then find the minimum! ■ Fixed priorities do not guarantee a minimum time to the VM!!!
  30. 30. Host Scheduling LinuxLab 2018 Luca Abeni – 30 / 36 ■ Issue: how to guarantee a minimum amount of CPU time to each VM? ◆ As said, SCHED FIFO / SCHED RR are not ok... ■ SCHED DEADLINE policy: schedule a thread / process for an amount of time Q every period P ◆ Q: runtime ◆ P: reservation period ■ This is what we need!!! Allows to compute a supply function!
  31. 31. In Practice... LinuxLab 2018 Luca Abeni – 31 / 36 ■ Real-Time applications running in a VM? ◆ As for OSs, two different aspects ■ Resource allocation/management (scheduling) ■ Latency (host and guest) ◆ CPU allocation: interesting issues with multiple vCPUs ■ Virtualisation: full hw virtualisation or OS-level virtualisation ◆ Full hw virtualisation: focus on kvm ◆ OS-level virtualisation: containers (lxc, Docker, ...)
  32. 32. kvm-Based VMs LinuxLab 2018 Luca Abeni – 32 / 36 ■ kvm: Linux driver to use the CPU virtualisation extensions (Intel VT-X and similar technologies) ◆ User-space VMM (qemu, kvmtool, TinyEMU, Amazon’s firecracker...) for virtual devices, etc... ◆ A thread per virtual CPU (vCPU thread) ■ Use SCHED DEADLINE for the vCPU threads ◆ Real-Time theory provides some (boring) algorithms to dimension Q and P ■ Global guest scheduler ← para-virtualised scheduling!!! ◆ To always schedule on physical CPUs the highest priority guest tasks
  33. 33. Implementation LinuxLab 2018 Luca Abeni – 33 / 36 ■ Use PREEMPT RT for the host kernel ■ Use PREEMPT RT for the guest kernel ■ Use kvm-based VMs ◆ qemu as a user-space VMM (you can use kmvtool or similar tools, too) ■ After starting the VM, use SCHED DEADLINE for the vCPU threads ◆ Runtime and period computed according to the real-time load of the VM ◆ There are existing tools to compute the values ■ Use partitioned fixed priority scheduling in the guest
  34. 34. Scheduler Setup LinuxLab 2018 Luca Abeni – 34 / 36 Host Scheduling ■ CARTS: https://rtg.cis.upenn.edu/carts ■ Retina Tools: retinaproject.eu/publications-deliverables#products Guest Scheduling ■ Use fixed priorities (SCHED FIFO / SCHED RR) ■ Avoid global scheduling (set affinities to single CPUs) ◆ The Retina Tools provide support for partitioning real-time tasks Starting the VM ■ Retina Tools → example scripts using qemu/kvm
  35. 35. Conclusions LinuxLab 2018 Luca Abeni – 35 / 36 ■ Deterministic scheduling of real-time tasks in VMs is possible ◆ But can be tricky; you need to know what to do ■ Use real-time support provided by Linux community → PREEMPT RT ■ Use theory developed in real-time research → SCHED DEADLINE ■ Use the virtualization support provided by Linux → kvm, virtio, ... ◆ Some tools can help you in putting everything together
  36. 36. Demo? LinuxLab 2018 Luca Abeni – 36 / 36 ■ “I have a truly marvelous demo of this, which the margin of these slides is too narrow to contain...” ■ So, let’s try a less marvelos demo...

×