Check out the new USENIX Web site. next up previous
Next: Performance of personal workload Up: System evaluation Previous: Experimental settings


Baseline performance in a LAN

This section evaluates Pangaea's performance in a LAN using a sequential workload without data sharing. While such an environment is not Pangaea's main target, we conducted this study to test Pangaea's ability to serve people's daily storage needs and to understand the system's behavior in an idealized situation.

We created a variation of the Andrew benchmark6 that simulates a single-person, engineering-oriented workload. It has the same mix of operations as the original Andrew benchmark [13], but the volume of the data is expanded twenty-fold to allow for accurate measurements on modern hardware. This benchmark, denoted Andrew-Tcl hereafter, consists of five stages: (1) mkdir: creating 200 directories, (2) copy: copying the Tcl-8.4 source files from one directory to another, (3) stat: doing ``ls -l'' on the source files, (4) grep: doing ``du'' and ``grep'' on the source files, and (5) compile: compiling the source code. We averaged results from four runs per system, with 95% confidence interval below 3% for all the numbers presented.


Table 2: Andrew-Tcl benchmark results in a LAN environment. Numbers are in seconds. Label pang-N shows Pangaea's performance when it creates N replicas for each new file. Ext3 is Linux's native (local) file system.
pang-1 pang-2 pang-3 pang-4 NFS Coda-s Coda-w ext3
mkdir 2.04 2.04 2.18 2.28 0.316 2.25 0.047 0.021
copy 3.40 3.79 3.85 3.90 3.50 201.0 0.85 0.264
stat 0.91 0.90 0.90 0.91 0.87 0.86 0.86 0.162
grep 2.09 2.11 2.13 2.13 2.20 1.22 1.20 0.925
compile 74.4 75.3 75.8 75.9 77.2 90.2 62.1 61.5
Total 82.84 84.14 84.86 85.12 84.08 295.5 65.05 62.87


Table 2 shows the time to complete the benchmark. Throughout the evaluation, label pang-N stands for a Pangaea system with N (gold) replicas per file. Pangaea's performance is comparable to NFS. This is as expected, because both systems perform about the same amount of buffer flushing, which is the main source of overhead. Pangaea is substantially slower only in mkdir. This is because Pangaea must create a Berkeley DB file for each new directory, which is a relatively expensive operation. Pangaea's performance is mostly independent of a file's replication factor, thanks to optimistic replication, where most of the replication processing happens in the background.

Coda's weakly connected mode (coda-w) is very fast. This is due to implementation differences: whereas Pangaea and NFS flush buffers to disk after every update operation, Coda avoids that by intercepting low-level file-access (VFS) requests using a small in-kernel module.

Figure 7: Network bandwidth consumed during the Andrew benchmark. The ``overhead'' bars show bytes consumed by harbingers and duplicate updates. The numbers above the bars show the percentage of overhead.
\includegraphics[width=5in]{graphs/andrew_net.eps}

Figure 7 shows the network bandwidth used during the benchmark. ``Overhead'' is defined to be harbingers and update messages that turn out to be duplicates. Pang-1 does not involve any network activity since it stores files only on the local server. Numbers for pang-3 and -4 show the effect of Pangaea's harbinger algorithm in conserving network-bandwidth usage. In this benchmark, because all replicas are gold and they form a clique, Pangaea would have consumed 4 to 9 times the bandwidth of pang-2 were it not for harbingers. Instead, its network usage is near-optimal, with less than 2% of the bandwidth wasted.

Table 3 shows network bandwidth consumption for common file-system update operations. Operations such as creating a file or writing one byte show a high percentage of overhead, since they are sent directly without harbingers, but they have only a minor impact on the overall wasted bandwidth since their size is small. On the other hand, bulk writes, which make up the majority of the overall traffic, incur almost no overhead.


Table 3: Network bandwidth consumption for common file-system operations. Shows the total number of bytes transmitted between all the nodes for each operation. ``Overhead'' shows the percentage of the bandwidth used by harbingers and duplicate updates.
pang-1 pang-2 pang-3 pang-4 NFS coda-w coda-s
Bytes Bytes Overhead Bytes Overhead Bytes Overhead Bytes Bytes Bytes
create 0 248 0% 1.29K 60% 2.61K 68% 503 1.46K 1.96K
write 1B 0 323 0% 854 61% 2.01K 68% 667 944 935
write 50KB 0 52.04K 0% 104.98K 1.49% 157.44K 1.52% 53.21K 55.56K 82.13K
write 25MB 0 26.22M 0% 52.44M 0.01% 78.67M 0.02% 26.76M 1.56M 38.75M


Figure 8: Andrew-Tcl benchmark results on a node with a slow network link. The labels next to the bars indicate the link speeds. For Pangaea, these are the links between any two servers; for NFS and Coda, they are the links between clients and server. NFS took 1939 seconds in a 5Mb/s network, and it did not finish after two hours in a 1Mb/s network.
\includegraphics[width=5in]{graphs/andrew_latency2.eps}


next up previous
Next: Performance of personal workload Up: System evaluation Previous: Experimental settings
Yasushi Saito 2002-10-08