Priority Inversion is an

1. RAVINDRA V. JOSHI

2. MARS – PATH FINDER

3. Description of the Problem
occurred
 Path Finder was getting reset every now and then
 System appeared to be unstable
 Crucial data was getting lost as system was getting
reset
 No clue what was the problem, as there was no test
report suggested this behavior!!!
 How to analyze? How to debug?? How to correct???

4. Analysis Of the Problem
 Information bus – High Priority Task
 Data Gathering Task – Low Priority Task
 Information Bus Task was getting reset continuously
 Finally got zeroed-on to Priority Inversion Problem

5. PROCESSES, TASKS, AND THREADS
 An instance of Program in Execution
 For simplicity’s sake let us use the term, tasks
PROCESSES
TASKS

6. TASKS AND RESOURCES
 A typical system contains a set of tasks and set of
resources

7. SCHEDULING – ART OF ASSIGNING
RESOURCES TO TASKS
 FIFO
 Shortest Job First
 Priority based scheduling
 Round Robin
 Pre-emptive and Non-PreEmptive ( Co-operative )
 Longest Waiting First

8. Priority Based
Pre-Emptive Scheduling

9. Priority Based
Pre-Emptive Scheduling

10. Priority Based
Pre-Emptive Scheduling

11. Priority Based
Pre-Emptive Scheduling

12. Locking a Resource

13. DESCRIPTION OF
PRIORITY INVERSION
 Consider 3 Tasks with Priority J<Q<K and a
resource JOKER which is required by J and K but
not by Q

14.  Currently J is scheduled and it locks JOKER
DESCRIPTION OF
PRIORITY INVERSION

15.  Now Q becomes Ready and Pre Empts J as it does not
require JOKER
DESCRIPTION OF
PRIORITY INVERSION

16.  Q got scheduled though K with higher priority is
waiting. This is PRIORITY INVERSION.
DESCRIPTION OF
PRIORITY INVERSION

17. Solution
 Priority Inheritance = Current Highest Priority
 Priority Ceiling = Statically scheduled

18. Debugging and Correcting the
problem
 Problem was tried for reproduction on ground by
collected Traces and Logs
 At last, the problem got reproduced when everybody
gave up and went home except one engineer who
finally saw the reproduction
 It turned out that VxWorks( the operating system
inside Path Finder ) had implemented a solution (
with Priority Inheritance ) but that function was called
with parameter false
 A short patch overwriting the parameter was beamed
to Mars and Program was “Corrected” in Mars

19. LESSONS LEARNT
 Never leave distant machine in Un-debuggable
method. Some format of Remote debugging MUST be
present
 Exercise the options conservatively. When cost of
using or not using “Priority Inheritance” is not
extraordinarily large, use Priority Inheritance
 Do not give up hope 

20. Unsung Heroes
 But who had found that such problem can occur? It
was quiet three Academicians from Carnegie Mellon
University
 Lui Sha
 John Lehoczky and
 Raj Rajkumar
 Standing Applause

21. Questions??
THANK YOU

22. WATCHDOG – The Ultimate Savior
 Basic Principle is – Never to hang
 If required, Automatically Resetting is preferred
 If it finds, that the system is “HUNG”, it will reset the
system