This document discusses protection and security in operating systems. It begins by defining protection as mechanisms within the OS to control access to programs and resources, while security refers to issues external to the OS like authentication. The document then outlines topics on protection mechanisms and security issues. It discusses the first computer virus as an example of how a program can reproduce itself and propagate in a way that is hard to detect. Finally, it covers goals of protection like least privilege and separating policy from mechanism in OS design.

1. Protection and Security
CS-502 Fall 2007 1
Protection and Security
CS-502 Operating Systems
Fall 2007
(Slides include materials from Operating System Concepts, 7th ed., by Silbershatz, Galvin, & Gagne and
from Modern Operating Systems, 2nd ed., by Tanenbaum)

2. Protection and Security
CS-502 Fall 2007 2
Concepts
• Protection:
• Mechanisms and policy to keep programs and users
from accessing or changing stuff they should not do
• Internal to OS
• Chapter 14 in Silbershatz
• Security:
• Issues external to OS
• Authentication of user, validation of messages,
malicious or accidental introduction of flaws, etc.
• Chapter 15 of Silbershatz

3. Protection and Security
CS-502 Fall 2007 3
Outline
• Part 1
• The first computer virus
• Protection mechanisms
• Part 2
• Security issues
• Some cryptographic themes

4. Protection and Security
CS-502 Fall 2007 4
The First Computer Virus
• Reading assignment:–
Ken Thompson, “Reflections on Trusting Trust,”
Communications of ACM, vol.27, #8, August
1984, pp. 761-763 (pdf)
• Three steps
1. Program that prints a copy of itself
2. Training a compiler to understand a constant
3. Embedding a Trojan Horse without a trace

5. Protection and Security
CS-502 Fall 2007 5
Step 1 – Program to print copy of itself
• How do we do this?
• First, store character array representing text of
program
• Body of program
• Print declaration of character array
• Loop through array, printing each character
• Print entry array as a string
• Result: general method for program to reproduce
itself to any destination!

6. Protection and Security
CS-502 Fall 2007 6
Step 2 – Teaching constant values to compiler
/* reading string constants */
if (s[i++] == '')
if (s[i] == 'n') insert ('n');
elseif (s[i] == 'v') insert ('v');
elseif …
• Question: How does compiler know what integer
values to insert for 'n‘, 'v‘, etc.?

7. Protection and Security
CS-502 Fall 2007 7
Step 2 (continued)
• Answer: In the first compiler for this machine
type, insert the actual character code
• i.e., 11 (decimal) for ‘v’, etc.
/* reading string constants */
if (s[i++] == '')
if (s[i] == 'n') insert ('n');
elseif (s[i] == 'v') insert (11);
elseif …
• Next: Use the first compiler to compile itself!

8. Protection and Security
CS-502 Fall 2007 8
Step 2 (continued)
• Result: a compiler that “knows” how to interpret
the sequence “v”
• And all compilers derived from this one, forever after!
• Finally: replace the value “11” in the source code
of the compiler with ‘v’ and compile itself again
• Note: no trace of values of special characters in …
– The C Programming Language book
– source code of C compiler
• I.e., special character values are self-reproducing

9. Protection and Security
CS-502 Fall 2007 9
Step 3 – Inserting a Trojan Horse
• In compiler source, add the text
if (match(sourceString, pattern)
insert the Trojan Horse code
where “pattern” is the login code (for example)
• In compiler source, add additional text
if (match(sourceString, pattern2)
insert the self-reproducing code
where “pattern2” is a part of the compiler itself
• Use this compiler to recompile itself, then
remove source

10. Protection and Security
CS-502 Fall 2007 10
Step 3 – Concluded
• Result: an infected compiler that will
a. Insert a Trojan Horse in the login code of any Unix
system
b. Propagate itself to all future compilers
c. Leave no trace of Trojan Horse in its source code
• Like a biological virus:
– A small bundle of code that uses the compiler’s own
reproductive mechanism to propagate itself

11. Protection and Security
CS-502 Fall 2007 11
Questions?

12. Protection and Security
CS-502 Fall 2007 12
Goals of Protection
• Operating system consists of a collection of
objects (hardware or software)
• Each object has a unique name and can be
accessed through a well-defined set of operations.
• Protection problem – to ensure that each object is
accessed correctly and only by those processes
that are allowed to do so.

13. Protection and Security
CS-502 Fall 2007 13
Guiding Principles of Protection
• Principle of least privilege
– Programs, users and systems should be given
just enough privileges to perform their tasks
• Separate policy from mechanism
– Mechanism: the stuff built into the OS to make
protection work
– Policy: the data that says who can do what to
whom

14. Protection and Security
CS-502 Fall 2007 14
Domain Structure
• Access-right = <object-name, rights-set>
where rights-set is a subset of all valid operations
that can be performed on the object.
• Domain = set of access-rights

15. Protection and Security
CS-502 Fall 2007 15
Conceptual Representation – Access Matrix
• View protection as a matrix (access matrix)
• Rows represent domains
• Columns represent objects
• Access(i, j) is set of operations that process
executing in Domaini can invoke on Objectj

16. Protection and Security
CS-502 Fall 2007 16
Textbook Access Matrix
• Columns are access control lists (ACLs)
• Associated with each object
• Rows are capabilities
• Associated with each user, group, or domain

17. Protection and Security
CS-502 Fall 2007 17
Unix & Linux
• System comprises many domains:–
– Each user
– Each group
– Kernel/System
• (Windows has even more domains than
this!)

18. Protection and Security
CS-502 Fall 2007 18
Unix/Linux Matrix
file1 file 2 file 3 device domain
User/Domain 1 r rx rwx – enter
User/Domain 2 r x rx rwx –
User/Domain 3 rw – – – –
…
• Columns are access control lists (ACLs)
• Associated with each object
• Rows are capabilities
• Associated with each user or each domain

19. Protection and Security
CS-502 Fall 2007 19
Changing Domains (Unix)
• Domain = uid or gid
• Domain switch via file access controls
– Each file has associated with it a domain bit (setuid bit).
• rwS instead of rwx
– When executed with setuid = on, then uid or gid is
temporarily set to owner or group of file.
– When execution completes uid or gid is reset.
• Separate mechanism for entering kernel domain
– System call interface

20. Protection and Security
CS-502 Fall 2007 20
General (textbook) representation
• Domains as objects added to Access Matrix

21. Protection and Security
CS-502 Fall 2007 21
Practicalities
• At run-time…
– What does the OS know about the user?
– What does the OS know about the resources?
• What is the cost of checking and enforcing?
– Access to the data
– Cost of searching for a match
• Impractical to implement full Access Matrix
– Size
– Access controls disjoint from both objects and domains

22. Protection and Security
CS-502 Fall 2007 22
ACLs vs. Capabilities
• Access Control List: Focus on resources
– Good if resources greatly outnumber users
– Can be implemented with minimal caching
– Can be attached to objects (e.g., file metadata)
– Good when the user who creates a resource has
authority over it
• Capability System: Focus on users
– Good if users greatly outnumber resources
– Lots of information caching is needed
– Good when a system manager has control over all
resources

23. Protection and Security
CS-502 Fall 2007 23
Both are needed
• ACLs for files and other proliferating resources
• Capabilities for major system functions
• The common OSs offer BOTH
– Linux emphasizes an ACL model
• provides good control over files and resources that are file-like
– Windows 2000/XP emphasize Capabilities
• provides good control over access to system functions (e.g.
creating a new user, or doing a system backup…)
• Access control lists for files

24. Protection and Security
CS-502 Fall 2007 24
…and good management, too!
• What do we need to know to set up a new
user or to change their rights?
• …to set up a new resource or to change the
rights of its users?
• …Who has the right to set/change access
rights?
• No OS allows you to implement all the
possible policies easily.

25. Protection and Security
CS-502 Fall 2007 25
Enforcing Access Control
• User level privileges must always be less than OS
privileges!
– For example, a user should not be allowed to grab
exclusive control of a critical device
– or write to OS memory space
• …and the user cannot be allowed to raise his
privilege level!
• The OS must enforce it…and the user must not be
able to bypass the controls
• In most modern operating systems, the code which
manages the resource enforces the policy

26. Protection and Security
CS-502 Fall 2007 26
(Traditional) Requirements–System Call Code
• No user can interrupt it while it is running
• No user can feed it data to make it
– violate access control policies
– stop serving other users
• No user can replace or alter any system call
code
• No user can add functionality to the OS!
• Data must NEVER be treated as code!

27. Protection and Security
CS-502 Fall 2007 27
“Yeah, but …”
• No user can interrupt it while it is running
• Windows, Linux routinely interrupt system calls
• No user can feed it data to make it
• violate access control policies
• stop serving other users
• No user can replace or alter any system call code
• Except your average virus
• No user can add functionality to the OS!
• Except dynamically loaded device drivers
• Data must NEVER be treated as code!
• “One man’s code is another man’s data” A. Perlis

28. Protection and Security
CS-502 Fall 2007 28
Saltzer-Schroeder Guidelines
• System design should be public
• Default should be no access
• Check current authority – no caching!
• Protection mechanism should be
– Simple, uniform, built into lowest layers of system
• Least privilege possible for processes
• Psychologically acceptable
• KISS!

29. Protection and Security
CS-502 Fall 2007 29
Reading Assignment
Silbershatz, Chapter 14

30. Protection and Security
CS-502 Fall 2007 30
Questions?