We propose an approach to the control of system calls which requires minimal additions to the kernel code and neither changes to the syntax and semantics of the system calls nor modifications of existing kernel data structures and algorithms. All kernel enhancements are transparent to existing or new programs. No change in the source code or special compiling procedure is required.
Basically when controlled system calls are invoked, our mechanism
checks whether the invoking process and the value of the arguments
comply with the rules kept in an Access Control Database (ACD) placed
within the kernel.
To reduce the cost of the checks, a detailed analysis of privileged
applications and system calls is carried out. This allows to identify
the set of primitives which may be dangerous for the system security.
As an example of this methodology, we have designed and implemented a prototype which prevents privileged processes from compromising the security and integrity of the Linux operating system when they are subverted by means of techniques like buffer overflow.
Buffer overflow is a widely known technique [AlephOne] which allows to force a process generated by a buggy program in executing ``fake'' instructions injected by the attacker. If the technique is successfully applied to a privileged process and the fake code is used, for instance, to start the execution of an interactive shell, the attacker gains the access to a privileged shell.
During the design phase, the complete set of Linux system calls has been analyzed. The result of the analysis shows that by adding access control tests to a small number of system calls, the protection against buffer overflow is complete and can not be bypassed by executing unprotected system calls. This reduces the cost of system call interception since the invocation of most system calls is not checked.
Any process running with root privileges is a potential target of a buffer overflow attack. Our control mechanism prevents these processes from executing unexpected system calls if they undergo an attack.