We experimentally compare NFS versions 2, 3 and 4 with iSCSI using a combination of micro- and macro-benchmarks. The objective of our micro-benchmarking experiments is to measure the network message overhead of various file and directory operations in the two protocols, while our macro-benchmarks experimentally measure overall application performance.
Our micro-benchmarks measure the network message overhead (number of network messages) for a variety of system calls that perform file and directory operations. We first measure the network message overhead assuming a cold cache at the client and the server and then repeat the experiment for a warm cache. By using a cold and warm cache, our experiments capture the worst and the average case, respectively, for the network message overhead. Since the network message overhead depends on the directory depth (path length), we also measure these overheads for varying directory depths. In case of file reads and writes, the network message overhead is dependent on (i) the I/O size, and (ii) the nature of the workload (i.e., random or sequential). Consequently, we measure the network message overhead for varying I/O sizes as well as sequential and random requests. We also study the impact of the network latency between the client and the server on the two systems.
We also measure application performance using several popular benchmarks: PostMark, TPC-C and TPC-H. PostMark is a file system benchmark that is meta-data intensive due its operation on a large number of small files. The TPC-C and TPC-H database benchmarks are data-intensive and represent online transaction processing and decision support application profiles.
We use a variety of tools to understand system behavior for our experiments. We use Ethereal to monitor network packets, the Linux Trace toolkit and vmstat to measure protocol processing times, and nfsstat to obtain nfs message statistics. We also instrument the Linux kernel to measure iSCSI network message overheads. Finally, we use logging in the VFS layer to trace the generation of network traffic for NFS. While we use these tools to obtain a detailed understanding of system behavior, reported performance results (for instance, for the various benchmarks) are without the various monitoring tools (to prevent the overhead of these tools from influencing performance results).
The next two sections provide a summary of our key experimental results. A more detailed presentation of the results can be found in [9].