Most performance measurements and analysis of persistent object systems (and object-oriented database systems) have been done using synthetic benchmarks instead of using real applications. There are two reasons for this: first, there are few large, realistic applications that exercise all persistence mechanisms of the underlying system and of those that exist, few are available for general use; and second, it is typically extremely hard to adapt a large piece of code to any given persistence mechanism without having a detailed understanding of the application.
The OO1 and OO7 [3] benchmarks have become quite popular among various benchmarks, and have been used widely for measuring the performance of persistent systems. However, we posit that these benchmarks are not representative of typical real-world applications, because they have not been validated against applications in the domain they represent; other researchers [19] have also reached similar conclusions. As such, the experimental results from these benchmarks should be interpreted with caution. The apparently ``empirical'' nature of these ``experimental'' results is likely to lull people into relying on the results more than appropriate. It is important to always remember that while the results are obtained empirically, they are ultimately derived from a synthetic benchmark and are only as good as the mapping of benchmark behavior onto real applications.
Although OO1 is a crude benchmark and does not strongly correspond to a real application, we use it for several reasons. First, OO1 is simple for measuring raw performance of pointer traversals (which is what we are interested in) and is fairly amenable to modifications for different address translation granularities. Use of a synthetic benchmark is also appropriate in this situation because our performance is very good in some cases (i.e., zero overhead when there is no faulting) and dependent on the rate of faulting (usually minimal overhead compared to I/O costs) for other cases. As such, crude benchmarking is the most practical way to measure performance of different components of our system because it is easy to separate our costs from those of the underlying benchmark; this is usually more difficult with a real application. Further discussion on sythetic benchmarks and their applicability is available in [8].