Reliable RSTs

With TCP, the control signals for connection establishment and completion (SYN and FIN, respectively) are delivered reliably, but the ``abrupt termination'' (RST) signal is not. This leads to a significant problem: in general, both a normalizer and a NIDS needs to tear down state for an existing connection once that connection completes, in order to recover the associated memory. But it is not safe to do so upon seeing a RST, because the RST packet might be lost prior to arriving at the receiver, or might be rejected by the receiver.

Thus, a monitor cannot tell whether a given RST does in fact terminate its corresponding connection. If the monitor errs and assumes it does when in fact it did not, then an attacker can later continue sending traffic on the connection, and the monitor will lack the necessary state (namely, that the connection is still established, and with what sequence numbers, windows, etc.) to correctly interpret that traffic. On the other hand, if the monitor assumes the RST does not terminate the connection, then it is left holding the corresponding state potentially indefinitely. (Unfortunately, RST-termination is not uncommon in practice, so even for benign traffic, this state will grow significantly over time.)

The RST might fail to arrive at the receiver because of normal loss processes such as buffer overflows at congested routers, or because of manipulation by an attacker, such as the TTL games discussed in the context of Figure 1. In addition, the rules applied by receivers to determine whether a particular RST is valid vary across different operating systems, which the NIDS likely cannot track.

A general solution to this problem would be to ensure that RSTs are indeed delivered and accepted, i.e., we want ``reliable RSTs.'' We can do so, as follows. Whenever the normalizer sees a RST packet sent from A to B, after normalizing it and sending it on, it synthesizes a second packet and sends that to B, too. This additional packet takes the form of a TCP ``keep-alive,'' which is a dataless⁴ ACK packet with a sequence number just below the point cumulatively acknowledged by B. The TCP specification requires that B must in turn reply to the keep-alive with an ACK packet of its own, one with the correct sequence number to be accepted by A, to ensure that the two TCP peers are synchronized. However, B only does this if the connection is still open; if it is closed, it sends a RST in response to the keep-alive.

Thus, using this approach, there are four possible outcomes whenever the normalizer forwards a RST packet (and the accompanying keep-alive):

(i)

The RST was accepted by B, and so B will generate another RST back to A upon receipt of the keep-alive;

(ii)

the RST either did not make it to B, or B ignored it, in which case B will generate an ACK in response to the keep-alive;

(iii)

the keep-alive did not make it to B, or B ignored it (though this latter shouldn't happen);

(iv)

or, the response B sent in reply to the keep-alive was lost before the normalizer could see it.

The normalizer then uses the following rule for managing its state upon seeing a RST: upon seeing a RST from A to B, retain the connection state; but subsequently, upon seeing a RST from B to A, tear down the state.³ Thus, the normalizer only discards the connection state upon seeing proof that B has indeed terminated the connection. Note that if either A or B misbehaves, the scheme still works, because one of the RSTs will still have been legitimate; only if A and B collude will the scheme fail, and, as noted earlier, in that case there is little a normalizer or a NIDS can do to thwart evasion.

The rule above addresses case (i). For case (ii), the normalizer needn't do anything special (it still retains the connection state, in accordance with the rule). For cases (iii) and (iv), it will likewise retain the state, perhaps needlessly; but these cases should be rare, and are not subject to manipulation by A. They could be created by B if B is malicious; but not to much effect, as in that case the connection is already terminated as far as A is concerned.