Download as PDF, PPTX
More Related Content
Similar to Luca Abeni - Real-Time Virtual Machines with Linux and kvm
Luca Abeni - Real-Time Virtual Machines with Linux and kvm
- 1. Real-Time Virtual Machines withLinux and kvm Luca Abeni luca.abeni@santannapisa.it
- 2. Real-Time Application LinuxLab 2018Luca Abeni – 2 / 36 ■ The time when a result is produced matters ◆ A correct result produced too late is equivalent to a wrong result (or to no result) ■ What does “too late” mean, here? ◆ Applications characterised by temporal constraints ■ Temporal constraints are modelled through deadlines ◆ Finish some activity before a time (deadline) ◆ Generate some data before a deadline ◆ Terminate some process/thread before a deadline ◆ ...
- 3. Real-Time Task LinuxLab 2018Luca Abeni – 3 / 36 Task (process or thread): sequence of actions characterized by deadlines and maximum execution time. Example: periodic task with period 8 and maximum execution time 3. 0 2 4 6 8 10 12 14 16 18 20 22 24 τ1 Note: while the first and the third jobs execute for 3 time units the second one executes for only 2 time units.
- 4. Formal Task Model LinuxLab2018 Luca Abeni – 4 / 36 ■ Real-Time task τi: stream of jobs (or instances) Ji,k ■ Each job Ji,k = (ri,k, ci,k, di,k): ◆ Arrives at time ri,k (activation time) ◆ Executes for a time ci,k ◆ Finishes at time fi,k ◆ Should finish within an absolute deadline di,k ri,k fi,k di,k ci,k
- 5. Periodic Tasks LinuxLab 2018Luca Abeni – 5 / 36 Periodic task τi = (Ci, Di, Pi): stream of jobs Ji,k, with ri,k+1 = ri,k + Pi di,k = ri,k + Di Ci = max k {ci,k} ■ Pi is the task period; ■ Di is the task relative deadline; ■ Ci is the task worst-case execution time (WCET); ■ Ri is the worst-case response time: Ri = maxk{ρi,k} = maxk{fi,k − ri,k}; ◆ Constraints respected ⇔ Ri ≤ Di.
- 6. Periodic Task Model LinuxLab2018 Luca Abeni – 6 / 36 ■ A periodic task has a regular structure (cycle): ◆ activate periodically (period Pi) ◆ execute a computation ◆ suspend waiting for the next period void ∗ PeriodicTask ( void ∗ arg ) { <i n i t i a l i z a t i o n >; <s t a r t p e r i o d i c timer , period = P>; while ( cond ) { <do something . . . > ; <wait next a c t i v a t i o n >; } }
- 7. Real-Time Scheduling LinuxLab 2018Luca Abeni – 7 / 36 ■ A real-time task τi = (Ci, Di, Pi) (or τi = (Ci, Pi) if Di = Pi) is properly served if all jobs respect their deadline... ■ ...Appropriate scheduling is important! ◆ The scheduler must somehow know the temporal constaints of the tasks... ◆ ...In order to schedule them so that such temporal constraints are respected ■ How to schedule real-time tasks? (scheduling algorithm) ■ Is it possible to respect all the deadlines? ■ Does Linux provide an appropriate scheduling algorithm?
- 8. Fixed Priorities LinuxLab 2018Luca Abeni – 8 / 36 ■ The Linux fixed-priority scheduler is often used for real-time tasks... ■ ...Is this ok in general, or only in some cases? ◆ Given a set of real-time tasks Γ = {τi}, can a fixed priority scheduler allow to respect all the deadlines? ◆ Is it possible to know in advance if some deadline will be missed? ◆ How to assign the priorities? ■ If fixed priorities are enough, the SCHED FIFO and SCHED RR policies can be used!
- 9. Schedulability Analysis LinuxLab 2018Luca Abeni – 9 / 36 ■ Given a set of tasks, is it possible to know in advance if deadlines will be missed? ◆ A task is schedulable if it will not miss any deadline ■ Depends on the task ■ Depends on the scheduler ■ Depends on the other tasks, ... ■ Schedulability test: mathematical test to check if τ = (C, D, P) will miss any deadline ◆ Possible idea: compute the amount of time needed by all the tasks to respect all their deadlines in an interval (t0, t1) and compare with t1 − t0
- 10. Partitioned Scheduling LinuxLab 2018Luca Abeni – 10 / 36 ■ How to schedule tasks on multiple CPUs / cores? ◆ First idea: partitioned scheduling ■ Statically assign tasks to CPUs ■ Reduce the problem of scheduling on M CPUs to M instances of uniprocessor scheduling CPU CPU CPU CPU M
- 11. Global Scheduling LinuxLab 2018Luca Abeni – 11 / 36 ■ One single task queue, shared by M CPUs ◆ The first M ready tasks from the queue are selected ◆ What happens using fixed priorities (or EDF)? ◆ Tasks are not bound to specific CPUs ◆ Tasks can often migrate between different CPUs ■ Problem: schedulers designed for UP do not work well M CPU CPU CPU CPU {M
- 12. Fixed Priorities inLinux LinuxLab 2018 Luca Abeni – 12 / 36 ■ SCHED FIFO and SCHED RR use fixed priorities ◆ They can be used for real-time tasks, to implement RM and DM ◆ Real-time tasks have priority over non real-time (SCHED OTHER) tasks ■ Difference between the two policies: visible when more tasks have the same priority ■ The administrator can use sched setscheduler() (or similar) to schedule real-time tasks ◆ Might be a little bit impractical, but can be used ■ However...
- 13. Periodic Task Example LinuxLab2018 Luca Abeni – 13 / 36 ■ Consider a periodic task /∗ . . . ∗/ while (1) { /∗ Job body ∗/ c l o c k n a n o s l e e p (CLOCK REALTIME , TIMER ABSTIME , &r , NULL ) ; timespec add us (&r , period ) ; } ■ The task expects to be executed at time r (= r0 + jT)... ■ ...But is sometimes delayed to r0 + jT + δ
- 14. Theoretical Schedule LinuxLab 2018Luca Abeni – 14 / 36 0 2 4 6 8 10 12 14 16 18 20 22 τ1 τ2
- 15. Actual Schedule LinuxLab 2018Luca Abeni – 15 / 36 0 2 4 6 8 10 12 14 16 18 20 22 τ1 τ2 ■ What happens if the 2nd job of τ1 arrives a little bit later??? ◆ The 2nd job of τ2 misses a deadline!!!
- 16. Theory vs RealSchedule LinuxLab 2018 Luca Abeni – 16 / 36 ■ The delay δ in scheduling a task is due to kernel latency ■ Kernel latency can be modelled as a blocking time ■ In real world, high priority tasks often suffer from blocking times coming from the OS (more precisely, from the kernel) ◆ Why? ◆ How? ◆ What can we do? ■ To answer the previous questions, we need to recall how the hardware and the OS work...
- 17. Latency LinuxLab 2018 LucaAbeni – 17 / 36 ■ Latency: measure of the difference between the theoretical and actual schedule ◆ Task τ expects to be scheduled at time t . . . ◆ . . . but is actually scheduled at time t′ ◆ ⇒ Latency L = t′ − t ■ The latency L can be modelled as a blocking time ⇒ affects the guarantee test ◆ Similar to what done for shared resources ◆ Blocking time due to latency, not to priority inversion
- 18. Effects of theLatency LinuxLab 2018 Luca Abeni – 18 / 36 ■ Upper bound for L? If not known, no schedulability tests!!! ◆ The latency must be bounded: ∃Lmax : L < Lmax ■ If Lmax is too high, only few task sets result to be schedulable ◆ Large blocking time experienced by all tasks! ◆ The worst-case latency Lmax cannot be too high
- 19. Latency in Linux LinuxLab2018 Luca Abeni – 19 / 36 ■ Tool (cyclictest) to measure the latency ■ Vanilla kernel: depends on the configuration ◆ Can be tens of milliseconds ■ PREEMPT RT patchset (https://wiki.linuxfoundation.org/realtime): reduce latency to less than 100 microseconds ◆ Tens of microseconds on well-tuned systems! ■ So, scheduling real-time tasks in Linux is not an issue anymore...
- 20. Real-Time in VMs??? LinuxLab2018 Luca Abeni – 20 / 36 ■ So, serving real-time applications in Linux-based systems is not a problem today... ■ ...Can real-time applications run in virtual machines? ◆ Real-Time in Virtual Machines??? But... Why? ■ Component-Based Development ◆ Complex applications: sets of smaller components ◆ Both functional and temporal interfaces ■ Security ◆ Isolate real-time applications in a VM ■ Easy deployment ■ ...
- 21. Real-Time Components LinuxLab 2018Luca Abeni – 21 / 36 ■ Assume to have N components... ◆ Each component is described by its interface... ◆ ...But a description of its temporal behaviour is also somehow provided... ■ Component are assumed to respect some temporal constraints ◆ Some kind of “contract” provided by real-time guarantees ■ Is it possible to combine the components so that the timing of the system is predictable? ◆ Can real-time guarantees be combined? How?
- 22. Combining Real-Time Apps LinuxLab2018 Luca Abeni – 22 / 36 ? ■ Real-time analysis for each application / in each VM... ■ What about the resulting system?
- 23. 1 Real-Time Taskper VM LinuxLab 2018 Luca Abeni – 23 / 36 ■ Single real-time thread/process in a VM → combining VMs is easy... ◆ VMs can be modelled as real-time tasks ◆ Simple (C, D, T) model, or something more complex... ■ What about VMs containing multiple real-time tasks? ◆ Model for a VM? ◆ How to summarize the VM’s temporal requirements? ◆ Scheduler in the VM? How to model it?
- 24. More Complex VMs LinuxLab2018 Luca Abeni – 24 / 36 ■ Example: component Ci composed by ni tasks ■ More complex model... How to handle it? ◆ We only know how to schedule single tasks... ◆ And we need to somehow “summarise” the requirements of a component! Ci = {(Ci 0, Di 0, Ti 0), (Ci 1, Di 1, Ti 1), . . . , (Ci ni, Di ni, Ti ni)} ■ So, 2 main issues: 1. Describe the temporal requirements of a VM in a simple way 2. Schedule the VMs, and somehow “combine” their temporal guarantees
- 25. Practical Issues LinuxLab 2018Luca Abeni – 25 / 36 ■ VMs can contain sets of real-time tasks ◆ The scheduler only sees a VM per taskset, but cannot see the tasks inside it ◆ Para-virtualization (of the OS scheduler) could be used to address this issue, but it is not so simple... ■ So, how to schedule VMs? ■ Two-level hierarchical scheduling system ◆ Host (global / root) scheduler, scheduling VMs ◆ Each VM contains its (local / 2nd level) scheduler
- 26. 2-Levels Hierarchy LinuxLab 2018Luca Abeni – 26 / 36 1τ 1τ 1ττn τn Local Scheduler Local Scheduler Local Scheduler 2 τn τ Global Scheduler ■ Global Scheduler assigns CPU to virtual machines ■ Local Schedulers assign CPU to single tasks
- 27. Hierarchical Scheduling LinuxLab 2018Luca Abeni – 27 / 36 ■ The global scheduler does not “see” all the tasks ■ Each VM has its own scheduler (fixed priorities for real-time tasks) ◆ Global scheduler: host / hypervisor scheduler ◆ Local scheduler: guest scheduler ◆ No problems in assigning fixed priorities to tasks! ■ Problem: what to use as a global scheduler? ◆ We must have a model for it ◆ Must allow to compose the “local guarantees” ◆ This time, fixed priorities are not enough :(
- 28. Schedulability Analysis LinuxLab 2018Luca Abeni – 28 / 36 ■ (Over?)Simplifying things a little bit... ■ ...Suppose to know the amount of time needed by a VM to respect its temporal constraints and the amount of time provided by the global scheduler ■ A VM is “schedulable” if demanded time ≤ supplied time ◆ “demanded time”: amount of time (in a time interval) needed by a VM ◆ “supplied time”: amount of time (in a time interval) given by the global scheduler to a VM ■ Of course the devil is in the details
- 29. Supplied Time LinuxLab 2018Luca Abeni – 29 / 36 ■ Description of the root scheduler temporal behaviour ■ More formally: ◆ Depends on the time interval t we are considering ◆ Depends on the root scheduler S ■ Minimum amount of time given by S to a VM in a time interval of size s ◆ Given all the time interval (t0, t1) : t1 − t0 = s... ◆ ...Compute the size of the sub-interval in which the VM is scheduled... ◆ ...And then find the minimum! ■ Fixed priorities do not guarantee a minimum time to the VM!!!
- 30. Host Scheduling LinuxLab 2018Luca Abeni – 30 / 36 ■ Issue: how to guarantee a minimum amount of CPU time to each VM? ◆ As said, SCHED FIFO / SCHED RR are not ok... ■ SCHED DEADLINE policy: schedule a thread / process for an amount of time Q every period P ◆ Q: runtime ◆ P: reservation period ■ This is what we need!!! Allows to compute a supply function!
- 31. In Practice... LinuxLab 2018Luca Abeni – 31 / 36 ■ Real-Time applications running in a VM? ◆ As for OSs, two different aspects ■ Resource allocation/management (scheduling) ■ Latency (host and guest) ◆ CPU allocation: interesting issues with multiple vCPUs ■ Virtualisation: full hw virtualisation or OS-level virtualisation ◆ Full hw virtualisation: focus on kvm ◆ OS-level virtualisation: containers (lxc, Docker, ...)
- 32. kvm-Based VMs LinuxLab 2018Luca Abeni – 32 / 36 ■ kvm: Linux driver to use the CPU virtualisation extensions (Intel VT-X and similar technologies) ◆ User-space VMM (qemu, kvmtool, TinyEMU, Amazon’s firecracker...) for virtual devices, etc... ◆ A thread per virtual CPU (vCPU thread) ■ Use SCHED DEADLINE for the vCPU threads ◆ Real-Time theory provides some (boring) algorithms to dimension Q and P ■ Global guest scheduler ← para-virtualised scheduling!!! ◆ To always schedule on physical CPUs the highest priority guest tasks
- 33. Implementation LinuxLab 2018 LucaAbeni – 33 / 36 ■ Use PREEMPT RT for the host kernel ■ Use PREEMPT RT for the guest kernel ■ Use kvm-based VMs ◆ qemu as a user-space VMM (you can use kmvtool or similar tools, too) ■ After starting the VM, use SCHED DEADLINE for the vCPU threads ◆ Runtime and period computed according to the real-time load of the VM ◆ There are existing tools to compute the values ■ Use partitioned fixed priority scheduling in the guest
- 34. Scheduler Setup LinuxLab 2018Luca Abeni – 34 / 36 Host Scheduling ■ CARTS: https://rtg.cis.upenn.edu/carts ■ Retina Tools: retinaproject.eu/publications-deliverables#products Guest Scheduling ■ Use fixed priorities (SCHED FIFO / SCHED RR) ■ Avoid global scheduling (set affinities to single CPUs) ◆ The Retina Tools provide support for partitioning real-time tasks Starting the VM ■ Retina Tools → example scripts using qemu/kvm
- 35. Conclusions LinuxLab 2018 LucaAbeni – 35 / 36 ■ Deterministic scheduling of real-time tasks in VMs is possible ◆ But can be tricky; you need to know what to do ■ Use real-time support provided by Linux community → PREEMPT RT ■ Use theory developed in real-time research → SCHED DEADLINE ■ Use the virtualization support provided by Linux → kvm, virtio, ... ◆ Some tools can help you in putting everything together
- 36. Demo? LinuxLab 2018 LucaAbeni – 36 / 36 ■ “I have a truly marvelous demo of this, which the margin of these slides is too narrow to contain...” ■ So, let’s try a less marvelos demo...