Figure 1: GNIC-II, Alteon, & SysKonnect: TCP Throughput and Latency
Table 1: TCP/IP Performance with Large Socket Buffer and MTU
Figure 2: GNIC-II: M-VIA Throughput and Latency
Since TCP was originally engineered to provide a general transport protocol, it is not by default optimized for streams of data coming in and out of the system at high transmission rates (e.g 1Gbps). In [Farrell], it is shown that communication performance is affected by a number of factors and indicated that one can tune certain network parameters to achieve high TCP/IP performance especially for a high speed network such as a Gigabit Ethernet network. We have taken care to tune the TCP parameters according to RFC 1323 TCP/IP Extension for High Performance [RFC1323] in order to achieve high speed TCP/IP communication. We have also set the window size to 128KB rather than the default 64KB in the Linux 2.2.12 kernel.
Figure 1 shows the TCP/IP throughput and latency for various Gigabit Ethernet NICs. Since ACEnic and SK-NET support frame sizes larger than the default of 1500 bytes, we tested them with different MTU sizes. In the figure, we present the TCP/IP performance with MTU of 1500 bytes for all Gigabit Ethernet NICs, and also with MTU equals to 9000 for ACEnic and SK-NET which achieves the highest peak throughput.
One obvious observation from the figures is there are many severe dropouts in ACEnic TCP/IP performance. The reason for these dropouts is supposedly due to the ACEnic device driver. For instance, using ACEnic device driver v0.32, we obtained maximum TCP/IP throughput of 356Mbps using MTU equals to 1500 and 468 Mbps using an MTU of 6500 bytes as opposed to an MTU of 9000 bytes. Furthermore, the latency of ACEnic driver v0.32 is approximately 40% less than the latency of ACEnic device driver v0.45. In addition, with MTU of 1500 bytes and the ACEnic device driver v0.32, the TCP throughput performance is better than that presented here. However, the TCP/IP performance of the ACEnic using device driver v0.45 with large MTU has improved substantially. In general, the overall TCP behavior for both ACEnic device drivers v0.32 and v0.45 have not been improved since v0.28, i.e., the performance graphs have many severe dropouts. In [Farrell], the ACEnic device driver v0.28 running on the Linux 2.2.1 kernel has a smoother performance curve and achieved its maximum throughput of 470 Mbps, using an MTU of 9000.
For MTU of 1500, the maximum attainable throughput is approximately 371
Mbps, 301 Mbps, and 331 Mbps for GNIC-II, ACEnic, and SK-NET
respectively. And, the latency is approximately 137 
secs, 182
secs, and 45 secs for GNIC-II, ACEnic, and SK-NET
respectively. With the lowest latency, SK-NET is able to perform much
better than the ACEnic and than the GNIC-II for message sizes up to
49KB. For example, for message size of 16KB, SK-NET throughput is
approximately 32% more than the GNIC-II and 82% more than the ACEnic.
However, for message sizes greater than 49KB, SK-NET reaches its maximum
of 331 Mbps.
Tests on networks based on FDDI, ATM [Farrell2] and Fibre Channel have
shown that high speed networks perform better when the MTU is larger
than 1500 bytes. Similarly, we expect Gigabit Ethernet would also
perform better with an MTU greater than 1500 bytes. From
Figure 1, we see that ACEnic maximum attainable throughput
increases approximately 70% reaching 513 Mbps when the MTU is set to
9000; And, for SK-NET, the maximum attainable throughput has also
increased to approximately 613 Mbps. The latency of ACEnic has decreased
to 121 secs; and, the SK-NET has increased slightly to 46
secs. In order to benefit from the larger MTU, one must also use a
larger socket buffer size rather than the default socket buffer size of
64KB. Table 1 shows this effect for various sizes of MTU and
socket buffer sizes of 64KB and 128KB.
Figure 2 shows the throughput and latency of M-VIA on the
GNIC-II compared
with the best attainable performance for each card using TCP.
The maximum attainable throughput
for M-VIA remains yet to be determined. This is due to the fact that
vnettest.c stops when message size reaches 32KB which is the
maximum data buffer size supported by the M-VIA implementation. For
message sizes around 30KB, the throughput reaches approximately
448 Mbps with latency of only 16 secs. Thus, the throughput is
approximately 53%, 42% and 4% more than GNIC-II, ACEnic, and SK-NET
respectively.
The VIA specification only requires VIA developers to support the minimum data buffer of 32KB. However, developers may choose to support data buffer sizes greater than 32KB. In this case, developers must provide a mechanism for the VI consumer to determine the data buffer size. Thus, we expect a larger data buffer will give higher throughput as message sizes continue to increase. On the other hand allocating larger data buffers may result in memory wastage.
