A variety of tracing tools are available for characterizing workloads and evaluating storage systems [4,7,16,24,50]. However, these solutions assume no data dependencies, making accurate parallel trace replay difficult.
There are also a number of tools for tracing, replaying and debugging parallel applications [5,15,18,26,30,36]. Because these tools are used to reduce the inherent non-determinism in message passing programs in order to make debugging easier (e.g., to catch race conditions or deadlock), they deterministically replay non-deterministic applications in order to produce the same set of events, and hence synchronization times, that occurred during the traced run. In contrast, the goal of //TRACE is to replay I/O traces so as to reproduce (realistically) any non-determinism in the the global ordering of I/O being issued by the compute nodes.
Throttling has been used successfully elsewhere to correlate events [9,21]. By imposing variable delays in system components, one can confirm causal relationships and learn much about the internals of a complex distributed system. //TRACE follows this same philosophy, by delaying I/O at the system call level in order to expose the causal relationships among nodes in a parallel application; this information is then used to approximate the causal relationships during trace replay.
There are also black-box techniques for intelligently ``guessing'' causality, and these do not require throttling or perturbing the system. In particular, message-level traces can be correlated using signal processing techniques [1] and statistics [12]. The challenge is distinguishing causal relationships from coincidental ones.
Operating system events can be used to track the resource consumption of an application [6,45] and also determine the dominant causal paths in a distributed system. Such ``whitebox'' techniques would complement //TRACE, especially when debugging the performance of a system, by providing detail as to the source of a data dependency. In addition, system call tracing has been successfully used to discover dependencies among processes and files for intrusion detection [27] and result caching [48].