System calls analysis

A drastic technique to prevent the damages produced by buffer overflow attacks would be to abort the execution of all system calls invoked by non-interactive root processes. However, the abort of all system calls is not possible (root processes can legally invoke some system calls), and is not needed (many system calls are not dangerous even when invoked via a buffer overflow attack).

The system calls available in Linux 2.2 have been grouped in categories according to their functionality as reported in table 1. In addition, each primitive has been classified according to the level of threat that it may introduce following the definition reported in table 2.

 

Table 1: System calls categories

I	File system and devices	V	Communication
II	Process management	VI	Time and timers
III	Module management	VII	System information
IV	Memory management	VIII	Reserved
		IX	Not implemented

 

Table 2: Threat level classification

threat level	threat description
1	May be used to get full control of the system
2	May be used for a denial of service attack
3	May be used for subverting the behavior of the invoking process
4	It is harmless

 

Table 3: Threat level 1 system calls

system calls	dangerous parameter
chmod, fchmod	a system file or a directory
chown, fchown, lchown	a system file or a directory
execve	an executable file
mount	on a system directory
rename, open	a system file
link, symlink, unlink	a system file
setuid, setresuid, setfsuid, setreuid	UID set to zero
setgroups, setgid, setfsgid, setresgid, setregid	GID set to zero
create_module	a malicious module

Currently we address the issues raised by the system calls classified as threat level 1 which are reported in table 3.

For different reasons no system call in the groups IV-IX can be used to gain the control of the system. For instance system calls in group IX return immediately the error ENOSYS, whereas primitives in group VIII can not be invoked by a generic user process (even if it is a privileged process).

As to the system calls in group III, a subverted process may use them for loading a malicious module. Our investigation shows that create_module is the only primitive which reaches the threat level 1 since no module can be activated without invoking create_module. The term malicious in table 3 has a very broad meaning. Basically we block any module which is not listed in the ACD.

Among the primitives related to process management, ten reach the highest level of danger for the system security. The execve can be used to start a root shell. The other nine primitives set user and group identifiers. It is worth noting that capset allows a process only to restrict its capabilities, so it belongs to class 3.

The ten system calls related to the file system classified as threat level 1 require special attention. It is apparent that the execution of

chmod(``/etc/passwd'',0666)

compromises the OS authentication mechanisms. However, it is necessary to consider chains of system calls as well. For instance, chown and chmod primitives can be used in a two-steps procedure to create a setuid shell, whereas the following sequence:

unlink(``/etc/passwd'')
link(``/tmp/passwd'',``/etc/passwd'')

produces the same result of the rename primitive.

Moreover, by means of a buffer overflow, it is possible to execute code that adds to the /etc/passwd (and/or /etc/shadow) file a new user with UID=0 or adds a fake .rhosts file to the root home directory. Although less common, these exploits are, by no means, less dangerous than those based on the classic shellcode. Most attacks that involve a file require at least two system calls. A first one to open the file and a second one to modify it. In this case, in accordance with [Goldberg] we assume that it is necessary to monitor just the open primitive. That is the reason why the write system call is not considered threat level 1.

In building an ACD for the open primitive a number of different situations must be considered. Several setuid programs or root daemons may open, for good reasons, critical files (e.g., /etc/hosts.allow). For the time being, we assume that opening such files in read-only mode is harmless. We understand that it may be possible to steal precious information like the encrypted passwords or the names of ``trusted'' hosts but we are going to address such issues in a second time. We recall that the checks are enforced on setuid or daemon programs just in case they invoke critical system calls with root privileges (that is with EUID=0). Since most of the times these programs give up to their privileges (the EUID is set equal to the real UID) before opening files in write mode, the access to user files does not need to be authorized. A detailed study of setuid and daemon programs has been carried out to define which directories and files must be included in the ACD. This has been realized by both source code inspection and analysis of the results produced by tools like the strace command which intercepts and records the system calls invoked by a process.