Dario Faggioli, October 2 2007                  Università di Pisa                                                   Facol...
Implementation and study of the                             BandWidth Inheritance protocol in the Linux kernel      Real-t...
Implementation and study of the                             BandWidth Inheritance protocol in the Linux kernel      Real-t...
Real-time systems                                                        Definitions      Some definitions of a real-time ...
Real-time systems                                                     Definitions      Hard real-time systems           A...
Real-time systems                                                      Definitions      Soft real-time systems           ...
Real-time systems                                                                 Definitions      Real-time theory       ...
Real-time systems                                  Real-time scheduling and resource sharing      Real-time scheduling    ...
Real-time systems                                     Real-time scheduling and resource sharing      Sharing resources and...
Real-time systems                                                Real-time scheduling and resource sharing      PIP rules ...
Real-time systems                                     Server based scheduling and resource sharing      Server based sched...
Real-time systems                               Server based scheduling and resource sharing      Server based scheduling ...
Real-time systems                                      Server based scheduling and resource sharing      Constant Bandwidt...
Real-time systems                                     Server based scheduling and resource sharing      Some CBS propertie...
Real-time systems                                Server based scheduling and resource sharing      Sharing Resources with ...
Real-time systems                               Server based scheduling and resource sharing      BandWidth Inheritance pr...
Real-time systems                                Server based scheduling and resource sharing      BandWidth Inheritance p...
Real-time systems                                         Server based scheduling and resource sharing      BandWidth Inhe...
Real-time systems                             Soft real-time systems, QoS and Resource Reservation      Building a soft re...
Real-time systems                             Soft real-time systems, QoS and Resource Reservation      Soft real-time sys...
Real-time systems                              Soft real-time systems, QoS and Resource Reservation      Resource Reservat...
Real-time systems                             Soft real-time systems, QoS and Resource Reservation      Resource Reservati...
Implementation and study of the                             BandWidth Inheritance protocol in the Linux kernel      Real-t...
The AQuoSA Framework                                           Objectives and design goals      AquoSA Framework         ...
The AQuoSA Framework                                                     Objectives and design goals      AquoSA Framework...
The AQuoSA Framework                                            Objectives and design goals      Some AQuoSA design goals ...
The AQuoSA Framework                                                          Architecture and usage      AquoSA architect...
The AQuoSA Framework                                                        Architecture and usage      AQuoSA architectur...
The AQuoSA Framework                                                     Architecture and usage      AQuoSA architecture a...
The AQuoSA Framework                                                Architecture and usage      AquoSA usage           Qo...
Implementation and study of the                             BandWidth Inheritance protocol in the Linux kernel      Real-t...
BWI implementation inside AquoSA                                               Design and goals      BWI implementation   ...
BWI implementation inside AquoSA                                             Specific contributions      Some contribution...
BWI implementation inside AquoSA                                             Specific contributions      Some contribution...
BWI implementation inside AquoSA                                             Specific contributions      Some contribution...
BWI implementation inside AquoSA                                             Specific contributions      Some contribution...
BWI implementations inside AquoSA                                               Implementation characteristics      Descri...
BWI implementations inside AquoSA                                                             Implementation characteristi...
BWI implementations inside AquoSA                                               Implementation characteristics           ...
Implementation and study of the                             BandWidth Inheritance protocol in the Linux kernel      Real-t...
BWI implementation verification      Verification           Static analysis               −    during the whole process o...
BWI implementation verification           Verification           Static analysis               −    during the whole pro...
Implementation and study of the                             BandWidth Inheritance protocol in the Linux kernel           ...
BWI experimental results      Experimental simulations           We build and ran some simulations in order to           ...
BWI experimental results      Experimental results           bwi_test_bwi_1                                              ...
BWI experimental results      Experimental results           bwi_test_inversion               −    BWI not used          ...
BWI experimental results      Experimental results            bwi_test_isolationTh. A=           W0                      ...
BWI experimental results      Experimental results            bwi_test_isolation Th. A=          W0                      ...
BWI experimental results      Experimental results           bwi_test_bandwidth_1               (2 thread, 1 mutex)      ...
BWI experimental results      Experimental results           bwi_test_bandwidth_1               (3 thread, 2 mutex)      ...
BWI experimental results      Experimental results           bwi_test_bandwidth_1               (8 thread, 5 mutex)      ...
Implementation and study of the                             BandWidth Inheritance protocol in the Linux kernel      Real-t...
BWI implementation overhead evaluation      Overhead evaluation           Space/Memory               look at the size of ...
BWI implementation overhead evaluation      Overhead evaluation           Time/Execution               −    profile the f...
BWI implementation overhead evaluation      Overhead evaluation           Context Switches               −    investigate...
Upcoming SlideShare
Loading in …5
×

Implementation and study of the BandWidth Inheritance protocol in the Linux kernel

428 views

Published on

Just my master thesis presentation. It is about the implementation of the BandWidth Inheritance real-time synchronization protocol within a QoS framework for Linux called AQuoSA (http://www.aquosa.sf.net). Nothing particularly refined (just master thesis quality :-P), but it was fun to do it!

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

Implementation and study of the BandWidth Inheritance protocol in the Linux kernel

  1. 1. Dario Faggioli, October 2 2007 Università di Pisa Facoltà di Ingegneria Corso di Laurea Specialistica in Ingegneria Informatica Tesi di Laurea Specialistica Implementation and study of the BandWidth Inheritance protocol in the Linux kernel Candidato: Relatori: Dario Faggioli Prof. Paolo Ancilotti Prof. Giuseppe Lipari02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  2. 2. Implementation and study of the BandWidth Inheritance protocol in the Linux kernel Real-time systems − Definitions − Real-time scheduling and resource sharing − Server based scheduling and resource sharing − Soft real-time systems, QoS and Resource Reservation The AquoSA Framework − Framework objectives and design goals − AquoSA Framework architecture and usage BWI implementation inside AquoSA − Design and goals − Specific contribution − Implementation characteristics BWI implementation verification BWI experimental simulations results BWI implementation overhead evaluation02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  3. 3. Implementation and study of the BandWidth Inheritance protocol in the Linux kernel Real-time systems − Definitions − Real-time scheduling and resource sharing − Server based scheduling and resource sharing − Soft real-time systems, QoS and Resource Reservation The AquoSA Framework − Framework objectives and design goals − AquoSA Framework architecture and usage BWI implementation inside AquoSA − Design and goals − Specific contribution − Implementation characteristics BWI implementation verification BWI experimental simulations results BWI implementation overhead evaluation02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  4. 4. Real-time systems Definitions Some definitions of a real-time system − A set of activities with timing requirements − A system interacting with the external environment which directly imposes its rigid requirements on timing behaviour, usually as constraints on response and/or worst case computation time − A system the correctness of which depends both on the accuracy of the computation results as well as on the time at which those results are produced and presented as output Differences with “traditional” systems − “traditional” systems aims at achieving maximum possible throughput, fairness and complete exploitation of available resources − real-time systems aims at achieving predictability and determinism02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  5. 5. Real-time systems Definitions Hard real-time systems  An hard real-time system completely fail if its timing constraints are not honored, even only one single time  An hard real-time system typically handles critical activities − flight control systems − defense systems − nuclear power plants control systems − ...  An hard real-time system causes, if it fails, catastrophic consequences02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  6. 6. Real-time systems Definitions Soft real-time systems  A soft real-time system can tolerate some miss to happen considering such an event only a temporary failure  A soft real-time systems typically are are non critical systems − audio/video players − multimedia applications − (multimedia) streaming applications − VR or video games and entertainment systems − ...  A soft real-time system produces, if occasional constraints violations occur, no cataclysmic effects  A soft real-time system produces, if occasional constraints violations occur, a degradation of the Quality of Service perceived by the user02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  7. 7. Real-time systems Definitions Real-time theory the scientific discipline that provides solutions to the problem of being able to design systems with full predictable timing behavior, also showing (mathematically) evidences that they meets all their timing requirements Real-time tasks An “activity” to be performed inside a real-time system is usually called a task and is characterized by some parameters periodic task ti, job j and j+1 non periodic task ti, job j and j+1 ai,j ci,j fi, j di,j ai,j ci,j fi, j di,j ai,j+1 c fi, j+1 i,j+1 ai,j+1 ci,j+1 fi, j+1 di,j+1 Ti,j,j+1 Di,j=Ti Di di,j+1 ci max j {c i, j } Utilisation factor (for each task) U i= = T i min j { T i, j, j+1 }02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  8. 8. Real-time systems Real-time scheduling and resource sharing Real-time scheduling Trying to find an order of execution of all the jobs of all the tasks in the system so that any of them meets their deadlines Real-time scheduling algorithms − Rate Monotonic (RM) tasks are executed in priority order, with the highest priority task being the one with the shorter period 1 c U =  U i =  < n2 n −1 i T i − Earliest Deadline First (EDF) tasks are executed in priority order, with the highest priority task being the one with the earliest (absolute) deadline c U =  U i =  <1 i T i02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  9. 9. Real-time systems Real-time scheduling and resource sharing Sharing resources and synchronization between tasks In complex systems tasks may need to compete or cooperate by means of shared resources collocated in shared memory areas and protected by mutex semaphores providing mutual exclusive access Priority inversion The use of classical semaphores is prone to the well known priority inversion problem an high priority task is blocked by a low priority one for an unbounded amount of time due to the interference of some other task, with medium priority between the two Caused problem also during Mars Pathfinder mission !! Priority Inversion avoidance: − Non Preemptive Protocol (NPP) T1 WR SR R − Highest Lock Priority Protocol (HLPP) T2 − Priority Inheritance Protocol (PIP) − Priority Ceiling Protocol (PCP) T3 WR SR R R R − Stack Resource Policy (SRP) Priority inversion !02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  10. 10. Real-time systems Real-time scheduling and resource sharing PIP rules The PIP achieve its objectives through two simple rules − when a task TH is blocked accessing a critical section whose mutex is already locked by TL (and if TH has higher priority than TL) then TL inherits the priority of TH − when TL releases the mutex it is returned the priority it had before acquiring it PIP ExampleT1 WR SR RT2 T1 WR SR RT3 WR SR T2 R R R Priority inversion ! T3 WR SR R R Priority inheritance02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  11. 11. Real-time systems Server based scheduling and resource sharing Server based scheduling  A server Si is an entity characterized three parameters − Qi the budget or capacity − Pi the period Qi Ui= − the server bandwidth Pi  Servers are the actual scheduling entities  A server serves one or more tasks T1 − hard or soft Hard T2 tasks EDF − periodic, sporadic or aperiodic T3 ready queue CPU − typically T4 S1 Soft T5  sporadic soft real-time tasks tasks S2 T6 S3  aperiodic non real-time tasks  When a server is scheduled its tasks are given a chance to run02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  12. 12. Real-time systems Server based scheduling and resource sharing Server based scheduling  While a server executes its actual budget is decreased  When a servers budget reaches zero it is periodically recharged  It is usually possible to ensure a task it will run at least for Qi units of time during each Pi period (depending on the server algorithm)  Some server algorithms − Polling Server (PS), − Deferrable Server (DS) − Sporadic Server (SS) − Total Bandwidth Server (TBS) − Constant Bandwidth Server (CBS)02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  13. 13. Real-time systems Server based scheduling and resource sharing Constant Bandwidth Server (CBS) based scheduling  Each CBS server is also characterized by − qi (qi(t)) the current server budget at time instant t − di (di(t)) the current server deadline at time instant t  The CBS algorithm works as follows: − CBS initialization: for each server Si, qi=Qi , di=0 − CBS rule A: when a new job of Ti, within Si, arrives the server checks qi ( di−ai )U i  if true nothing has to be done  if false qi=Qi, di=di+Pi − CBS rule B: when a Si executes for Dt time units qi=qi-Dt − CBS rule C: when qi=0 qi=Qi , di=di+Pi CBS fundamental property The contribution of the tasks served by Si to U is guaranteed to be not greater than Ui02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  14. 14. Real-time systems Server based scheduling and resource sharing Some CBS properties  The basic ideas behind the algorithm are − when a request for a task execution arrives it is assigned a deadline calculated taking into account its servers bandwidth − when a task tries to execute more than what it is expected its deadline is postponed, so that it is slowed down  The algorithm works − without requiring the WCET to be a-priori known − without requiring the period or MIT to be a-priori known  The algorithm provides the Bandwidth Isolation Property (BIP) − the fraction of the CPU bandwidth of a CBS server is U=Q/P  independently from the computation time of the served tasks  independently from the arrival time pattern of the served tasks − the fraction of the CPU bandwidth the server can exploit is U=Q/P  independently from the computation time of other tasks in the system  independently from the arrival time pattern of other tasks in the system02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  15. 15. Real-time systems Server based scheduling and resource sharing Sharing Resources with CBS  Within CBS algorithm no resource sharing is allowed if a task blocks itself we are no longer sure the executing server is the one with the earliest deadline  PIP can not be “simply” applied here − what happens if SL exhausts its budget after it inherited SHs priority (deadline)? S1=(4, 16) WR SR − which deadline has to be postponed R and whose budget recharged ?  This leads to S2=(4, 24) − priority inversions ! WR SR S3=(9, 36) R R R − BIP breakout ! deadline miss ! S4=(1, 4) 0 2 4 6 8 10 12 14 16 18 Priority inversion !02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  16. 16. Real-time systems Server based scheduling and resource sharing BandWidth Inheritance protocol  Only adds two rules to the CBS algorithm − CBS rule D: when Ti blocks while accessing R owned by Tj, then Tj is added to the list of tasks served by the server Si. If even Tj is blocked the chain of blocked tasks is followed and all the encountered tasks are added until one that is not blocked is found − CBS rule E: when Tj releases R and Ti is resumed the server Si discards Tj from its list and all the servers that has previously added Tj replace it with Ti deadline postponements S1=(2, 6) WR SR SR R R R S2=(2, 6) S3=(6, 18) WR R 0 2 4 6 8 10 12 14 16 18 2002/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  17. 17. Real-time systems Server based scheduling and resource sharing BandWidth Inheritance protocol  Natural extension of Priority Inheritance Protocol toward the CBS − bandwidth isolation property − bounded blocking delay and no priority inversions  Real inheritance protocol when a task blocks himself on a mutex the lock-owner inherits its whole server  Addresses the open issues of PIP and CBS − what happens if SL exhausts its budget deadline postponements after it inherited SHs priority (deadline)? A task is able of being scheduled with S1=(2, 6) WR SR SR R R R the earliest deadline between all the eligible servers it has been added to S2=(2, 6) − which deadline has to be postponed and whose budget recharged ? They are the ones of the server S3=(6, 18) WR in which the task is executing R 0 2 4 6 8 10 12 14 16 18 2002/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  18. 18. Real-time systems Server based scheduling and resource sharing BandWidth Inheritance protocol and isolation  Interacting tasks − Ti and Tj have to be considered interacting tasks if  Ti and Tj share a resource R  a chain of tasks T1, T2, T3, ... Tn exists such that Ti=T1 and Tj=Tn and, for each k=1, 2, 3, ..., n, Tk and Tk-1 share a resource Rk  BWI is able to provide isolation among non-interacting task  It is impossible to provide isolation among interacting tasks !! S1=(4, 16) WR SR S1=(4, 16) WR SR R R R S2=(4, 24) S2=(4, 24) WR SR W SR S3=(9, 36) S3=(9, 36) RR R R R R deadline miss ! S4=(1, 4) S4=(1, 4) 0 2 4 6 8 10 12 14 16 18 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 3402/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  19. 19. Real-time systems Soft real-time systems, QoS and Resource Reservation Building a soft real-time system  Soft real-time applications are characterized by more relaxed constraints − some deadline misses can be tolerated up to a certain degree − control on the system performance and resource usage is still required  The main goal dealing with soft real-time applications is not to prevent all the deadline to be missed, beside to keep this phenomenon under control  What it is desirable in a soft real-time system is − maximize the utilization of the system resources (not too pessimistic assumptions) − remove any need of precise a-priori knowledge about the applications − achieve graceful degradation of the performances − provide isolation to each application, so that an overrun in any of them do not influence the normal execution of the others  An hard real-time approach would mean an unacceptable waste of resource and it is definitely not appropriate02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  20. 20. Real-time systems Soft real-time systems, QoS and Resource Reservation Soft real-time systems and Quality of Service (QoS)  The users satisfaction is a decreasing function of − the number of the timing constraints violations − the severity (e.g. mean tardiness) of the constraints violations An optimal measure of the provided QoS could right be the ratio of deadline miss over a given time interval the lower the deadline miss ratio, the better the QoS The main challenges with soft real-time systems are − increasing efficiency in resource usage − keep the overall cost of the system low − removing any assumptions on the exact knowledge of execution times − ensure some guarantees on the performances of the system not an absolute “yes-or-no” as by the notion of QoS02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  21. 21. Real-time systems Soft real-time systems, QoS and Resource Reservation Resource Reservation  A very effective approach to soft real-time systems − temporal protected (isolated) access to shared systems resources (e.g. CPU)  each application is reserved a fraction of a systems resource utilisation time  its ability to meet the timing constraints is not influenced by any other application  Each soft real-time task is assigned a fraction of the CPU time (bandwidth) and is scheduled in such a way it will never demand more that its bandwidth  If the task requires to execute more than its bandwidth it is slowed down − overruns occurring on Ti will only affect and delay the execution of Ti itself − no one is able to steal any crumb of bandwidth reserved to the other tasks  No reciprocal tasks interference allowed − a task executes on a dedicated virtual processor running at a fraction of the speed of the real one equal to its reserved bandwidth − the problem of schedulability analysis reduces to the estimation of the computation time of a task, without considering the rest of the system, in order to assign it the proper bandwidth02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  22. 22. Real-time systems Soft real-time systems, QoS and Resource Reservation Resource Reservation and CBS  We can effectively realize the RR utilising the CBS algorithm The BIP provided by the CBS is the perfect vehicle for providing temporal isolation Resource Reservation and BWI  We can also utilise the BWI-modified CBS algorithm − the arrival times of the tasks need no to be known − the execution times of the tasks need not to be know − the mutual exclusive resources a task will access is not to be known − the duration of each critical section is not to be known02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  23. 23. Implementation and study of the BandWidth Inheritance protocol in the Linux kernel Real-time systems − Definition − Real-time scheduling and resource sharing − Server based scheduling and resource sharing − Soft real-time systems, QoS and Resource Reservation The AquoSA Framework − Objectives and design goals − Architecture and usage BWI implementation inside AquoSA − Design and goals − Specific contribution − Implementation characteristics BWI implementation verification BWI experimental simulations results BWI implementation overhead evaluation02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  24. 24. The AQuoSA Framework Objectives and design goals AquoSA Framework  Adaptive Quality of Service Architecture (AquoSA) − Free Software project developed at the Real-Time Systems Laboratory (RETIS Lab, Scuola Superiore SantAnna, Pisa) − aims at providing Quality of Service management on a GNU/Linux system02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  25. 25. The AQuoSA Framework Objectives and design goals AquoSA Framework  What GNU/Linux offers to soft real-time applications − a low-latency and fully preemptable kernel − high-resolution timing support − (almost) all the real-time capabilities defined in the POSIX real-time extensions  What GNU/Linux lacks − a predictable scheduler with timing and/or QoS guarantees − a temporal protection and isolation mechanism so that  an application can access the soft real-time scheduler but not exploit it to starve all the other applications in the system  an application can have the guarantees it has been provided respected in all systems load conditions and independently on misbehaviour of any other one  AQuoSA wants to fill this gap provides a standard GNU/Linux system with all its needed to offer time-sensitive applications a guaranteed behaviour and the temporal protection property02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  26. 26. The AQuoSA Framework Objectives and design goals Some AQuoSA design goals  AQuoSA is developed trying to achieve the following key goals − Portability: layered structure where kernel dependent code is confined in the lowermost level − Backward compatibility: no change to the API and ABI of a standard Linux kernel − Flexibility: easy to introduce new scheduling algorithms and policies − Efficiency: overhead introduced reduced to minimal measure − Security: control on the assigned bandwidth by the system administrator in order to prevent erroneous or voluntary denial of services to happen02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  27. 27. The AQuoSA Framework Architecture and usage AquoSA architecture  Resource Reservation (RR) layer − provides the system with  the EDF scheduler  the RR mechanism (IRIS)  the reservation supervisor QoS Managed QoS Reserved GNU/Linux Application Application Application − is responsible for all that regards  the reservations creation  the reservations deletion QoS Manager layer  the enforcement of the timing behaviour QoS Resource Reservation of the reserved tasks layer  QoS Manager layer GNU/Linux system a set of predictor and controllers for feedback based scheduling02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  28. 28. The AQuoSA Framework Architecture and usage AQuoSA architecture and components:  The architecture is realized by the following main components: − Generic Scheduler Patch (GSP): a small patch to the Linux kernel extending the Linux scheduler functionalities − Kernel Abstraction Layer (KAL): a set of functions and macros abstracting all we need from the kernel − QoS Reservation component:  resource reservation kernel module (rresmod) which implements EDF and the RR scheduling algorithms  resource reservation supervisor kernel module (qresmod) which grants no system overload to occur  resource reservation and supervisor libraries (qreslib and qsuplib) which provides the application level APIs − QoS Manager component:  QoS manager loadable kernel module (qmgrmod)  QoS manager library (qmgrlib)02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  29. 29. The AQuoSA Framework Architecture and usage AQuoSA architecture and components: ● The architecture is realized by the following main components: ● Generic Scheduler Patch (GSP): a small patch to the Linux kernel extending the Linux scheduler functionalities ● Kernel Abstraction Layer (KAL): a set of functions and macros abstractingApplication need from the kernel QoS Managed Application all we QoS Reserved GNU/Linux Application ● QoS Reservation component: user-space ● resource reservation kernel module (rresmod) which implements EDF and the RR scheduling algorithms qmgrlib qsuplib qreslib glibc ● resource reservation supervisor kernel module (qresmod) which grants no system overload to occur QoS Manager qmgrmod component qresmod ● resource reservation and supervisor libraries (qreslib and qsuplib) which provides the application level APIs EDF Scheduler rresmod ● QoS Manager component: kernel-space Kernel Abstraction Layer ● QoS manager loadable kernel module (qmgrmod) QoS Reservation component ● QoS manager library (qmgrlib) GSP Linux kernel02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  30. 30. The AQuoSA Framework Architecture and usage AquoSA usage  QoS resource reservation library − from within application code  From the Command Line Interface qres and qres-wrap utilities with all the same API functionalities  From the Graphical User Interface AQuoSA Monitor much more user friendly interface02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  31. 31. Implementation and study of the BandWidth Inheritance protocol in the Linux kernel Real-time systems − Definition − Real-time scheduling and resource sharing − Server based scheduling and resource sharing − Soft real-time systems, QoS and Resource Reservation The AquoSA Framework − Framework objectives and design goals − AquoSA Framework architecture and usage BWI implementation inside AquoSA − Design and goals − Specific contributions − Implementation characteristics BWI implementation verification BWI experimental simulations results BWI implementation overhead evaluation02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  32. 32. BWI implementation inside AquoSA Design and goals BWI implementation  Before this work AQuoSA offered no support for critical sections access when a task blocks it is assumed an instance is finished !!  We started this work with the following specifications − to realize a full and functional implementation of the BWI protocol inside AQuoSA − to make it possible to enable or disable the protocol at compile time − to make it be possible to use or not use the protocol at run time (for each specific mutex and critical section) − to alter as little as possible impact as possible the AQuoSA code and architecture − to add as little as possible overhead both in space (memory occupation) and in execution time02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  33. 33. BWI implementation inside AquoSA Specific contributions Some contributions  Lightweight deadlock check Since each time a task Ti blocks we walk through the whole chain of blocked tasks starting from it, it is sufficient we to encounter Ti again to identify a deadlock situation with zero overhead  Adding only one task − As a performance optimization each time Ti blocks and we follow the chain of blocked tasks till a running one, we can add only such an able to run task − This means, since each task Ti can, at each time instant t, be blocked at most by only one other task Tj, the following property always holds: BWI one blocked task property: at each time instant t each task Ti is being blocked by at most one task Tj, and so only Tj is executing inside its server Si due to Ti itself02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  34. 34. BWI implementation inside AquoSA Specific contributions Some contributions  Nested critical section handling enhancement A slight but effective modification of the algorithm − the CBS rule D does not account for the case a task blocks when it already has been added to some servers (i.e. while holding at least a mutex with at least one another blocked task) − we have modified it into the CBS rule D (and implemented the latter) CBS rule D: when Ti blocks while accessing R owned by Tj, then Tj is added to all the list of tasks of all the servers which also serves Ti. If even Tj is blocked the chain of blocked task is followed and all the encountered tasks are added to all these servers list, until one that is not blocked is found02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  35. 35. BWI implementation inside AquoSA Specific contributions Some contributions  Nested critical section handling enhancement A slight but effective modification of the algorithm − the CBS rule D does not account for the case a task blocks when it already has been added to some servers (i.e. while holding at least a mutex with at least one another blocked task) − we have modified it into the CBS rule D (and implemented the latter) CBS rule D: when Ti blocks while accessing R owned by Tj, then Tj is added to all the list of tasks of all the servers which also serves Ti. If even Tj is blocked the chain of blocked task is followed and all the encountered tasks are added to all these servers list, until one that is not blocked is found CBS rule D: when Ti blocks while accessing R owned by Tj, then Tj is added to the list of tasks served by the server Si. If even Tj is blocked the chain of blocked tasks is followed and all the encountered tasks are added until one that is not blocked is found02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  36. 36. BWI implementation inside AquoSA Specific contributions Some contributions  Nested critical section handling enhancement A slight but effective modification of the algorithm − the CBS rule D does not account for the case a task blocks when it already has been added to some servers (i.e. while holding at least a mutex with at least one another blocked task) − we have modified it into the CBS rule D (and implemented the latter) CBS rule D: when Ti blocks while accessing R owned by Tj, then Tj is added to all the list of tasks of all the servers which also serves Ti. If even Tj is blocked the chain of blocked task is followed and all the encountered tasks are added to all these servers list, until one that is not blocked is found CBS rule D: when Ti blocks while accessing R owned by Tj, then Tj is added to the list of tasks served by the server Si. If even Tj is blocked the chain of blocked tasks is followed and all the encountered tasks are added until one that is not blocked is found02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  37. 37. BWI implementations inside AquoSA Implementation characteristics Description of the implementation  Non intrusive approach Do not modify the core features of AquoSA, that is  the scheduler and scheduler related functions  the RR enforcing mechanisms  the servers handling (list of tasks, dispatching, etc.) mechanisms  BWI added functions only called in very few situations − when a task is attached or detached to/from a server − when a task blocks or unblocks on/from a mutex  Quantitative data − only 4 core functions, 3 utility ones and some KAL macro added − only 2 files have been newly created − only 6 files lightly modified modified − totally 260 lines of code added − totally 6 lines of code modified02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  38. 38. BWI implementations inside AquoSA Implementation characteristics  Description of the implementation QoS Managed QoS Reserved GNU/Linux Application Application Application  Non intrusive approach user-space − Do not modify the core features of AquoSA, that is qmgrlib qsuplib qreslib glibc  the scheduler and scheduler related functions QoS  the RR enforcing mechanisms Manager qmgrmod component  the servers handling (list of tasks, dispatching, etc.) mechanisms qresmod EDF  BWI added functionsrresmod called in very few situations only Scheduler rres_bwi − when a task is attached or detached to/from a server Kernel Abstraction Layer kal_bwi kernel-space QoS Reservation − component when a task blocks or unblocks on/from a mutex GSP  Quantitative data Linux kernel − only 4 core functions, 3 utility ones and some KAL macro added − only 2 files have been newly created − only 6 files lightly modified modified − totally 260 lines of code added − totally 6 lines of code modified02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  39. 39. BWI implementations inside AquoSA Implementation characteristics  Description of the implementation  Non intrusive approach − Do not modify the core features of AquoSA, that is  the scheduler and scheduler related functions  the RR enforcing mechanisms  the servers handling (list of tasks, dispatching, etc.) mechanisms task_struct task_list server_t  BWI added functions only called in very few situations T1 ... private_data ... task others ... tasks srv ... bwi_blocked_on − when a task is attached or detached to/from a server bwi_wait_task bwi_orig_srv blocked on bwi_entries S1 − when a task blocks or unblocks on/from a mutex T2  Quantitative data task_struct ... task_list task server_t ... private_data others tasks ... srv ... − only 4 core functions, 3 utility ones and some KAL macro added bwi_blocked_on bwi_wait_task bwi_orig_srv S1 − only 2 files have been newly created bwi_entries task_list − only 6 files lightly modified modified task others srv − totally 260 lines of code added bwi_blocked_on bwi_wait_task bwi_orig_srv − totally 6 lines of code modified bwi_entries02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  40. 40. Implementation and study of the BandWidth Inheritance protocol in the Linux kernel Real-time systems − Definition − Real-time scheduling and resource sharing − Server based scheduling and resource sharing − Soft real-time systems, QoS and Resource Reservation The AquoSA Framework − Framework objectives and design goals − AquoSA Framework architecture and usage BWI implementation inside AquoSA − Design and goals − Specific contribution − Implementation characteristics BWI implementation verification BWI experimental simulations results BWI implementation overhead evaluation02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  41. 41. BWI implementation verification Verification  Static analysis − during the whole process of code development − assertions placed throughout the code to ensure consistency in some critical points  Dynamic analysis (testing) − a lot of test programs have been run checking  the output they produces is exactly what we could have expected  none of the assertions added during static analysis is violated  Stress tests − a lot of test programs have been run caring of  all the code paths are covered  all the identified abnormal situations are correctly handled − all the test programs have been run in different system load conditions  (Inverse) integration tests − all test programs have been run whit BWI not being used by them − all test programs have been run with our code compiled out02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  42. 42. BWI implementation verification  Verification  Static analysis − during the whole process of code development − assertions placed throughout the code to ensure consistency in some critical points  Dynamic analysis (testing) − a lot of test programs have been run checking  the output they produces is exactly what we could have expected E. W. Dijkstra:assertions added during static analysis is violated none of the   Stress tests can only reveal the existence of programming errors, <<testing − a not prove their absence>> caring of lot of test programs have been run  all the code paths are covered  all the identified abnormal situations are correctly handled − all the test programs have been run in different system load conditions  (Inverse) integration tests − all test programs have been run whit BWI not being used by them − all test programs have been run with our code compiled out02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  43. 43. Implementation and study of the BandWidth Inheritance protocol in the Linux kernel  Real-time systems − Definition − Real-time scheduling and resource sharing − Server based scheduling and resource sharing − Soft real-time systems, QoS and Resource Reservation  The AquoSA Framework − Framework objectives and design goals − AquoSA Framework architecture and usage  BWI implementation inside AquoSA − Design and goals − Specific contribution − Implementation characteristics  BWI implementation verification BWI experimental simulations results BWI implementation overhead evaluation02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  44. 44. BWI experimental results Experimental simulations  We build and ran some simulations in order to − prove BWI is an effective solution for handling critical sections and resource sharing in a real and running RR based soft real-time system − point out some if its fundamental properties and benefits it introduces  List of the simulations we ran − BWI enforcement  bwi_test_bwi_1  bwi_test_bwi_2 − Priority inversion control  bwi_test_inversion − Compliance with the BIP  bwi_test_isolation − Full bandwidth exploitation  bwi_test_bandwidth_1  bwi_test_bandwidth_2  bwi_test_bandwidth_302/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  45. 45. BWI experimental results Experimental results  bwi_test_bwi_1 deadline misses ! − BWI not used S0 S0 Th. A=(10, 10, 100) W0 W0 0 0 W0 S0 W0 S0 Th. B=(30, 10, 100) 0 0 0 deadline hit 0 0 0 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 320 340 360 380 400 420 440 460 480 500 520 deadline misses ! − BWI used W0 S0 W0 S0 W0 S0 W0 S0 Th. A=(10, 10, 100) 0 0 0 0 0 0 W0 S0 W0 S0 Th. B=(10, 10, 100) 0 0 0 0 deadline hit Thread As tardiness is significantly decreased 407.80196 msec instead of 1905.8146 msec after 10 runs02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  46. 46. BWI experimental results Experimental results  bwi_test_inversion − BWI not used W0 S0 W0 S0 Th. A=(30, 100) Th. A=(30, 100) 0 0 Th. B=(40, 320) Th. B=(60, 320) W0 S0 W0 S0 Th. C=(40, 480) Th. C=(40, 480) 0 0 0 0 0 0 0 20 40 60 80 100 120 0 20 40 60 80 100 120 140 − BWI used W0 S0 S0 W0 S0 S0 Th. A=(30, 100) Th. A=(30, 100) 0 0 0 0 Th. B=(40, 320) Th. B=(60, 320) W0 W0 Deterministic Th. C=(40, 480) 0 Th. C=(40, 480) 0 behaviour achieved 0 20 40 60 80 100 120 140 0 20 40 60 80 100 120 14002/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  47. 47. BWI experimental results Experimental results  bwi_test_isolationTh. A= W0 S0(40, 140) 0Th. B=(100, 380) Th. A= W0S0 S0 (40, 140)Th. C= W0 S0 0 0 0(20, 600) 0 0 Th. B=Th. D= W1 S1 (100, 380)(40, 150) 1 Th. C= W0 W1 S1 (20, 600) 0Th. E=(40, 500) 111 1 Th. D= W1 S1 S1 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 320 340 (40, 150) 1 1 1 Th. E= W1 (40, 500) 1 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 320 34002/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  48. 48. BWI experimental results Experimental results  bwi_test_isolation Th. A= W0 S0 (40, 140) 0 Th. B= (20, 300) Th. A= W0S0 S0 (40, 140) Th. C= W0 S0 0 0 0 (20, 600) 0 0 Th. B= Th. D= (20, 300) W1 S1 (40, 150) 1 Th. C= W0 Th. E= (20, 600) W1 S1 0 (80, 500) 111 1 Th. D= W1 S1 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 320 340 (40, 150) 1 1 Th. E= W1 S1 (80, 500) 1 1 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 320 34002/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  49. 49. BWI experimental results Experimental results  bwi_test_bandwidth_1 (2 thread, 1 mutex) Full Bandwidth exploitation Boosted finishing time02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  50. 50. BWI experimental results Experimental results  bwi_test_bandwidth_1 (3 thread, 2 mutex) Full Bandwidth exploitation Boosted finishing time02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  51. 51. BWI experimental results Experimental results  bwi_test_bandwidth_1 (8 thread, 5 mutex) Full Bandwidth exploitation Boosted finishing time02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  52. 52. Implementation and study of the BandWidth Inheritance protocol in the Linux kernel Real-time systems − Definition − Real-time scheduling and resource sharing − Server based scheduling and resource sharing − Soft real-time systems, QoS and Resource Reservation The AquoSA Framework − Framework objectives and design goals − AquoSA Framework architecture and usage BWI implementation inside AquoSA − Design and goals − Specific contribution − Implementation characteristics BWI implementation verification BWI experimental simulations results BWI implementation overhead evaluation02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  53. 53. BWI implementation overhead evaluation Overhead evaluation  Space/Memory look at the size of difference between the two cases BWI is compiled or not of  the rresmod.ko file, i.e. the loadable kernel module  its .text ELF section, i.e. the code actually loaded and executed BWI not enabled BWI enabled Difference Difference % module size (byte) 563055 609796 46741 8.3 .text length (byte) 31852 42196 10344 32.4802/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  54. 54. BWI implementation overhead evaluation Overhead evaluation  Time/Execution − profile the functions in which the call to our 4 core routines have been inserted  when BWI is compiled but not used  when BWI is compiled and used − look at the differences between mean and maximum execution time obtained values and the case BWI is not compiled in the framework Max. Execution Time (usec) BWI compiled out BWI not used Difference BWI used Differences rres_attach_task_nosched 7 8 1 7 0 rres_block_hook 6 6 0 7 1 rres_unblock_hook 9 9 0 12 3 rres_detach_task_nosched 6 6 0 6 0 Mean Execution Time (usec) BWI compiled out BWI not used Difference BWI used Differences rres_attach_task_nosched 4.840 5.080 0.240 5.000 0.160 rres_block_hook 3.467 3.477 0.010 3.636 0.169 rres_unblock_hook 5.721 5.773 0.052 5.837 0.116 rres_detach_task_nosched 4.720 4.920 0.200 4.960 0.24002/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
  55. 55. BWI implementation overhead evaluation Overhead evaluation  Context Switches − investigate how BWI behaves with respect to context switches  does it introduce any additional context switch ?  does it reduce the number of context switch ? − count the context switches and evaluate the differences between the cases  BWI is not used  BWI is used bwi_test_bwi_1 (periodic tasks) BWI not used BWI used Difference Thread A 26 24 -2 Thread B 34 34 0 bwi_test_bandwidth_2 (greedy tasks) BWI not used BWI used Difference Thread A 607 343 -264 Thread B 414 316 -98 Thread C 405 405 002/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel

×