Can pH detect and stop attacks in time to prevent system compromise?

Figure 2: A graph showing the pH-induced system-call delay during the sshd backdoor intrusion. Note the exponential increase (from 0 to 8, 16, then 32) and decrease, with a constant delay for most calls within the locality frame. The process shown is the child process, and it terminates with a shell being exec'd. The pair window size is 6, the locality frame size is 128, and the $delay\_factor$ is set to 4. Time is measured in jiffies, which are 1/100 of a second on Linux running on i386-compatible machines.

To test how pH could respond to security violations, we tested its behavior by seeing how it could detect and respond to a Secure Shell (SSH) daemon [29] backdoor, an SSH daemon buffer overflow, and a sendmail [13] attack that exploits a bug in the Linux kernel's capabilities code. These three violations all allow an attacker to obtain root privileges, using different techniques to gain access. Delays alone are significant inhibitors of these attacks; with execve aborts, pH can effectively stop all of them.

To test the SSH attacks, the sshd program in Debian 2.2's packaged version of Secure Shell (ssh-nonfree), version 1.2.27-6 was modified in two basic ways. First, it was made to link against the RSAREF2 library, to make it vulnerable to a buffer overflow attack script published on the BUGTRAQ mailing list [2]. Second, the source was modified using the rkssh5 trojan patch [37], and was built using the ``global password'' flag. This option allows an attacker to access to any account on the system using a compiled in, MD5-encoded password. In addition, use of this password disables most logging, minimizing the evidence of the intrusion.

A normal profile for this modified sshd binary was created by exercising the program on a personal workstation. Normal logins via root and a regular user were tested, using the password, RSA-secured rhosts, and pure RSA methods of authentication. Also, failed logins were tested, using nonexistent users and incorrect passwords. Together these produced 687 sequences, and a profile with 1725 pairs, over 47756 system calls.

Relative to this synthetic normal profile, we first tested whether pH could detect the use of the global password to gain access to the root account. With all responses disabled, the backdoor produced 5 anomalies, 3 in the child (which exec's the remote user's shell), and 2 in the parent (which maintains the network connection). Setting $delay\_factor$ to 4 produced the same anomaly profile, but did not prevent the remote user from logging in; however, the resulting connection was slowed down significantly, as shown in Figure 2. With $abort\_execve$ set to 1, the backdoor was closed, whether delays were enabled or not.

With all responses disabled, the buffer overflow attacked produced 4 clustered anomalies in the parent SSH process. Setting $delay\_factor$ to 4 produced the same anomalies, and allowed the attacker to obtain a root shell; however, this shell was less useful than might be supposed. Recall that pH delays every process with a non-zero LFC, and the LFC is only updated if the program has a valid normal (test) profile. As it turns out, bash, the standard shell on most Linux systems, is a large, complicated program that tends not to reach a stable profile. Thus, the 64 jiffy (0.64s) delay incurred by the overflowed sshd is passed on to the exec'd bash, and bash keeps this delay forever! Even if this weren't the case, because of the 128 entry locality frame, we'd see the delay for 125 system calls, giving us at least an 80s delay. Not a huge amount of time, but possibly enough to make a cracker think the attack isn't working.

With execve aborts enabled, the overflow attack was stopped, whether delays were enabled or not. The attack script does not simply fail, though; since the overflow code keeps retrying the execve call if it fails, the aborts cause an infinite loop. Each pass through the loop generates 3 anomalies, due to the failed execve; a few times through this tight loop thus causes the $tolerization\_limit$ to be exceeded, causing sshd's training profile to be reset.

The Linux capability vulnerability allows a non-privileged program to prevent a privileged one from dropping its capabilities on systems running recent 2.2 kernels (2.2.14 and 2.2.15 are both vulnerable). An exploit was published on BUGTRAQ [28] which uses sendmail to take advantage of this hole. Because this is a flaw in the kernel, it can succeed even though sendmail does the right thing and tries to drop its privileges.

A normal profile for sendmail (Debian version 8.9.3-22) was first generated, based on normal usage on a personal workstation. This normal had 3443734 system calls with 1061 unique sequences, and produced a profile with 2412 system call pairs. Relative to this normal, the exploit was extremely noticeable. The exploit generates several different sendmail processes, and just one of them had 47 anomalies! Indeed, the numerous anomalies caused the $tolerization\_limit$ to be reached numerous times. Enabling execve aborts did nothing to inhibit the attack; this makes sense, since the exploit doesn't have sendmail directly run a privileged shell; instead, it creates a setuid-root shell in /tmp. However, a $delay\_factor$ of 4 effectively stopped the attack -- delays were produced which lasted for at least two hours. Time delays of this magnitude would almost certainly frustrate a normal cracker; a patient one could be addressed by automatically killing any process that had been delayed for a long time period, say 30 minutes or more.